Most people on Earth can speak two or more languages, but voice-operated virtual assistants have always forced them to pick and use just one—at least until today.
Google Assistant is now the first multilingual virtual assistant. Users can specify that they want listening done in two languages in the app’s settings on their phone or Google Home smart speaker. Then, a person can call out requests or commands in either language. Yell “Hey Google, turn off the hallway light!” as you walk out the house, and darkness will fall. “¡Hey Google, apaga la luz del pasillo!” will work too.
Johan Schalkwyk, a vice president on Google’s speech engineering team, says that new flexibility should help people feel more natural when talking to the company’s smart speaker, Google Home—particularly in households where not everyone speaks the same language. “I expect we may see some different behaviors, and an uptick in usage,” he adds.
Dylan Zwick, cofounder of Pulse Labs, which helps companies test applications for voice platforms such as Amazon’s Alexa, agrees. In homes with grandparents or other family members that don’t speak the family’s primary language fluently, knowing Google Home is multilingual could make the device more useful, he says.
For now, Google Assistant can be set to work in any two of six languages: English, Spanish, French, German, Italian, and Japanese. The company says it will add the five others supported by the assistant—Hindi, Indonesian, Korean, Brazilian Portuguese, and Thai—but declines to specify when.
As the list expands, the new feature might help Google’s ambitions to sign up new users in emerging markets such as India and Indonesia, where online ad markets are growing faster than in the West.
To make the multilingual feature work, Google had to train its algorithms to reliably determine which language a person was speaking. As a first pass, the assistant tries to identify the language just by the sound of the person’s voice—similarly to how a person unable to speak German or French can still know when they hear those languages spoken. To double check, the system also runs the audio through the recognizers for both languages a user has activated. Examining the output to see which makes most sense helps the assistant decide in which language to respond.
Google has made multilingual speech technology before. Its voice search and mobile voice typing services allow a user to use up to six languages. Schalkwyk says it took longer to bring that feature to the assistant because it’s more challenging to correctly identify a language in a conversational setting—and the stakes are higher when software will take action based on what it thinks you want.
In testing over the past nine months, Google Assistant’s ability to detect the language a person is using has improved to roughly 99 percent from around 90 percent. That’s the point at which a voice technology feels useful and can be let loose in the wild, according to Schalkwyk. “If it fails one in 10 it sucks,” he says. “One in 100? Wow, awesome!”
Part of that improvement came from training algorithms to be better at spotting when a person with an accent associated with one language, such as English, is actually speaking another, like French.
Emre Akkas, cofounder and CEO of Globalme, which gathers voice data used by companies developing voice interfaces, says Google’s move to make the assistant multilingual shows the technology is maturing: “It’s a complicated and ambitious thing to do.”
Akkas is curious to see how Google’s new system holds up in the variety of voices and situations in users’ homes. When Globalme helped The Washington Post test Google Assistant and Amazon’s Alexa on different accents, they found that performance varied. Google Assistant was better at understanding people hailing from its Silicon Valley home base of California, than those who speak with a Southern US accent, for example. Spanish and Chinese accents—hardly unusual in America—were misunderstood even more frequently. “Practical studies show this technology still has a long way to go,” Akkas says.
Schalkwyk of Google acknowledges that the technology isn’t perfect, but says more multilingual usage will provide example data that can be used to train algorithms to cope better with different accents. He’s also thinking about how to make it more truly multilingual.
One challenge: colloquial combinations of two languages, such as Spanglish, or the mixture of Hindi and English spoken in some communities in India. Schalkwyk dreams that one day Google Assistant will be smart enough to join in those conversations. “A future challenge is to build a single recognizer that can just recognize all the languages that you may speak,” he says.
More Great WIRED Stories
- It’s time to stop sending money on Venmo
- How to share an Instagram account with your partner
- The JavaScript developer taking on Google and Facebook
- Say hello to the most audacious flying machine ever
- Why universities need ‘public interest technology’ courses
- Looking for more? Sign up for our daily newsletter and never miss our latest and greatest stories
Source:WIRED
