Microsoft’s Chinese-to-English translation AI matches human performance
By Digital News Asia March 20, 2018
- Reaching human parity is great but there are more challenges
- Technical breakthroughs could possibly apply to commercial products
A team of Microsoft researchers said March 14 that they believe they have created the first machine translation system that can translate sentences of news articles from Chinese to English with the same quality and accuracy as a person.
Researchers in the company’s Asia and US labs said that their system achieved human parity on a commonly used test set of news stories, called newstest2017, which was developed by a group of industry and academic partners and released at a research conference called WMT17 last year.
To ensure the results were both accurate and on par with what people would have done, the team hired external bilingual human evaluators, who compared Microsoft’s results to two independently produced human reference translations.
Xuedong Huang (pix, above), a technical fellow in charge of Microsoft’s speech, natural language and machine translation efforts, called it a major milestone in one of the most challenging natural language processing tasks.
“Hitting human parity in a machine translation task is a dream that all of us have had,” Huang said. “We just didn’t realise we’d be able to hit it so soon.”
Huang, who also led the group that recently achieved human parity in a conversational speech recognition task, said the translation milestone was especially gratifying because of the possibilities it has for helping people understand each other better.
“The pursuit of removing language barriers to help people communicate better is fantastic,” he said. “It’s very, very rewarding.”
Machine translation is a problem researchers have worked on for decades – and, experts say, for much of that time many believed human parity could never be achieved. Still, the researchers cautioned that the milestone does not mean that machine translation is a solved problem.
Ming Zhou, assistant managing director of Microsoft Research Asia and head of a natural language processing group that worked on the project, said that the team was thrilled to achieve the human parity milestone on the dataset. But he cautioned that there are still many challenges ahead, such as testing the system on real-time news stories.
Arul Menezes, partner research manager of Microsoft’s machine translation team, said the team set out to prove that its systems could perform about as well as a person when it used a language pair – Chinese and English – for which there is a lot of data, on a test set that includes the more commonplace vocabulary of general interest news stories.
“Given the best-case situation as far as data and availability of resources goes, we wanted to find out if we could actually match the performance of a professional human translator,” said Menezes, who helped lead the project.
Menezes said the research team can apply the technical breakthroughs they made for this achievement to Microsoft’s commercially available translation products in multiple languages. That will pave the way for more accurate and natural-sounding translations across other languages and for texts with more complex or niche vocabulary.
Inspired by human methods
Although academic and industry researchers have worked on translation for years, they have recently achieved substantial breakthroughs by using a method of training AI systems called deep neural networks. That has allowed them to create more fluent, natural-sounding translations that take into account an even broader context than the previous approach, known as statistical machine translation.
To reach the human parity milestone on this dataset, three research teams in Microsoft’s Beijing and Redmond, Washington, research labs worked together to add a number of other training methods that would make the system more fluent and accurate.
“Much of our research is really inspired by how we humans do things,” said Tie-Yan Liu (pix, above), a principal research manager with Microsoft Research Asia in Beijing, who leads a machine learning team that worked on this project.
The researchers also developed two new techniques to improve the accuracy of their translations – joint training and agreement regularisation. Zhou said he expects these methods and techniques to be useful for improving machine translation in other languages and situations as well. He said they also could be used to make other AI breakthroughs beyond translation.
“This is an area where machine translation research can apply to the whole field of AI research,” he said.
No ‘right’ answer
The test set the team used to reach the human parity milestone includes about 2,000 sentences from a sample of online newspapers that have been professionally translated.
Microsoft ran multiple evaluation rounds on the test set, randomly selecting hundreds of translations for evaluation each time. To verify that Microsoft’s machine translation was as good as a person’s translation, the company went beyond the specifications of the test set and hired a group of outside bilingual language consultants to compare Microsoft’s results against manually produced human translations.
The method of verifying the results highlights the complexity of teaching systems to translate accurately. With other tasks, such as speech recognition, it is quite straightforward to tell if a system is performing as well as a person, because the ideal result will be the exact same for a person and a machine. Researchers call that a pattern recognition task.
“Machine translation is much more complex than a pure pattern recognition task,” Zhou said. “People can use different words to express the exact same thing, but you cannot necessarily say which one is better.”
The researchers say that complexity is what makes machine translation such a challenging problem, but also such a rewarding one.
Liu said no one knows whether machine translation systems will ever get good enough to translate any text in any language pair with the accuracy and lyricism of a human translator. But, he said, these recent breakthroughs allow the teams to move on to the next big steps toward that goal and other big AI achievements, such as reaching human parity in speech-to-speech translation.
“What we can predict is that definitely we will do better and better,” Liu said.
Related stories:
Malaysia’s AI ambitions get City Brain boost from Alibaba Cloud
