Join Gaming Leaders, Alongside GamesBeat and Facebook Gaming, For Their 2nd Annual GamesBeat & Facebook Gaming Summit | GamesBeat: Into the Metaverse 2 January 25-27, 2022. Learn more about the event.
Following last year’s impressive personal AI solutions comparison, Loup Ventures today released the results of its Digital Assistant IQ Test 2019, and there’s good news if you enjoy giving voice commands to your phone, tablet, or speaker – all major digital assistants are getting better at their jobs.
Using a test consisting of the same 800 questions on each AI system, Google Assistant again led the pack, understanding 100% of the questions asked, just like last year, and answering 92.9 correctly. % between them. This is up from 85.5% last year and is quickly approaching a level of precision where errors will not be common.
In contrast, Apple’s Siri jumped in both categories, from a comprehension level of 99% last year to 99.8% this year, and from a correct answer level of 78.5% in 2018 to a correct level of 83.1% for 2019. Another way to look at this – although it may conflict with the real-world experiences of Siri users – is that Siri is almost as likely to answer this correctly. year that Google Assistant was last year.
Amazon’s Alexa took third place again, but made major progress this year, understanding 99.9% of questions and answering them correctly 79.8% of the time, better than Siri’s performance last year. . This is a big increase in the number of correct answers for Alexa, which jumped from a surprising low of 61.4% last year, and Wolf notes that this is the biggest jump it has ever seen. he’s known between the years since he started recording results.
Notably, Wolf left out Microsoft’s Cortana this year, which isn’t too surprisingly as the fourth place in AI has vanished from Microsoft’s third-party products and accessories. Cortana only answered 52.4% of last year’s questions correctly, meaning you might as well flip a coin or guess if your question could be answered binary .
One of the interesting things about Loup’s tests is that they cover five different categories: “local”, “commerce”, “navigation”, “information” and “order”, each designed to test a different area of support. potential of AI. So the best scores go to assistants who are well-rounded rather than just proficient in one area, so when Alexa focused heavily on Amazon commerce but not local news or navigation, that would suffer.
Google Assistant topped four of those five categories, opening a particularly large gap in commerce, where its 92% accuracy surpassed Alexa (71%) and Siri (68%). He actually had the highest scores in everything except “command,” where Siri beat him by a 93% to 86% margin – the only time Assistant fell below 92% in the correct answers.
Alexa ranks behind the two rivals in the “local”, “navigation” and “order” departments, while slightly ahead of Siri in “commerce”. Siri has also finished twice in second place and twice in third place, with its second largest gap in “information”, where it was significantly worse than other AI: 76% correct answers against 93% for Alexa. and 96% for Google.
As Wolf already mentioned, the continued march towards 100% is impressive, but should not be taken to mean that the assistants are in fact “intelligent”. While they can understand “within reason everything you tell them,” they only get good answers in their primary use cases and don’t exhibit higher-level reasoning skills. The next steps for digital assistants, says Wolf, are to add additional use cases that “voice is uniquely suited to solve” and provide simple user experiences to solve them.
VentureBeat’s mission is to be a digital public place for technical decision-makers to learn about transformative technology and conduct transactions. Our site provides essential information on data technologies and strategies to guide you in managing your organizations. We invite you to become a member of our community, to access:
- up-to-date information on the topics that interest you
- our newsletters
- Closed thought leader content and discounted access to our popular events, such as Transform 2021: Learn more
- networking features, and more
Become a member