IQ test for artificial intelligence systems – WSU Insider


Washington State University researchers create first-ever ‘IQ test’ for artificial intelligence (AI) systems that would rate systems on their ability to learn and adapt to new unfamiliar environments.

Diane Cook, Regents Professor and Huie-Rogers Chair Professor, and Larry Holder, Professor in the School of Electrical Engineering and Computer Science, received a grant of just over $1 million from Defense Advanced Research Projects Agency (DARPA) to create a framework for testing the “intelligence” of AI systems.

“Previously, research on measuring intelligence in AI systems was mostly theoretical,” Holder said. “They didn’t measure actual performance in new and never-before-seen environments and didn’t take into account task complexity.”

Holder and Cook will design a test that will rate AI systems based on the difficulty of the problems they can solve. Creating methods to rank problems according to their difficulty will be one of the main parts of the research.

Larry Holder

“The score will also take into account the precision, accuracy, time taken and amount of data they need to perform well,” Holder said.

One of the main challenges for AI researchers is to create machines that can learn in an unfamiliar environment like a human. Many recent advances in machine learning and artificial intelligence have focused on narrow, specialized tasks such as face detection and baseball game outcome prediction.

Creating machines that can learn and act intelligently in new environments remains the holy grail of AI research.

“We’re focused on testing and improving systems that can be more versatile, like a robot assistant that can help you with many of your daily tasks,” he said.

One of the behaviors Holder and Cook will examine is the ability of the machines they are testing to transfer their learning from one task and apply it to a new, unseen task.

“For example, you might want to learn checkers before chess because you can easily transfer your knowledge from one to the other,” Holder said.

Close up of Diane Cook
Diane Cook

An undergraduate student who did summer research with Holder helped him create an evaluation environment that would be used to test AI systems on tasks like playing video games, solving problems SAT test and solve the Rubik’s cube.

Cook is also the director of WSU’s Center for Advanced Adaptive Systems Studies (CASAS), which focuses on creating smart homes with robot assistants that care for the elderly, among other projects. These robots can be used to monitor the safety, health, mobility and social life of older people.

Holder currently runs a website that hosts the AIQ challenge and leaderboard. As part of the grant, Holder hopes to expand the publicly available software interface, which can be used by anyone to test their AI system while providing more data to researchers in the process.

This grant is part of DARPA’s Science of Artificial Intelligence and Learning for Open-World Novelty (SAIL-ON).

Media contacts:

  • Diane Cook, Regents Professor, School of Electrical Engineering and Computer Science, 509-335-4985, [email protected]
  • Larry Holder, Professor, School of Electrical Engineering and Computer Science, [email protected]
  • Tina Hilding, director of communications, Voiland College of Engineering and Architecture, 509-335-5095, [email protected]

Comments are closed.