Google Exec Proposes Novel AGI Test: Beyond Benchmarks

Google Exec Proposes Novel AGI Test: Beyond Benchmarks

The quest for Artificial General Intelligence (AGI) – AI with human-like cognitive abilities – is a central pursuit in the tech world. However, defining and testing for AGI has remained a significant challenge, with current benchmarks often criticized for being susceptible to manipulation or not reflecting true understanding. Demis Hassabis, CEO of Google DeepMind, recently outlined a compelling new benchmark for AGI that moves beyond traditional metrics, focusing on genuine scientific reasoning and creativity.

A Groundbreaking Test for True Intelligence

In a recent interview, Hassabis proposed a novel test: train an AI system with a knowledge cutoff, say, from 1911, and then assess if it can independently develop theories like Einstein’s General Relativity, which was published in 1915. This hypothetical test aims to distinguish between sophisticated pattern matching and genuine scientific discovery derived from first principles.

Hassabis articulated his long-standing definition of AGI: a system capable of exhibiting all the cognitive capabilities that humans possess. He noted that the human brain serves as the only known existence proof of general intelligence, a fact that historically drew him to study neuroscience. He emphasized that current AI systems, while impressive, fall short in critical areas such as true creativity, continual learning, and long-term planning.

“Today’s systems, although they’re very impressive and they’re improving, they they don’t do a lot of those things. So true creativity, continual learning, long-term planning, they’re not good at those things,” Hassabis stated. He further elaborated on the inconsistency of current AI, pointing out that while models can excel in complex tasks like international math olympiads, they can still falter on simpler problems if posed in a certain way. A true general intelligence, he argued, should not exhibit such “jagged” capabilities.

The Limitations of Current Benchmarks

The proposed test by Hassabis directly addresses a major criticism of current AI development: the concern that models are merely sophisticated data retrievers. If an AI could independently derive a novel scientific theory without explicit training data, it would represent a qualitative leap in intelligence. However, Hassabis acknowledged the difficulty of his proposed test, noting that Einstein’s discovery involved not just knowledge but also years of focused effort, physical intuition, experimental insights, and access to existing scientific frameworks like Maxwell’s equations. The AI would need to replicate this comprehensive context and creative leap.

This idea of moving goalposts for AGI has been a subject of discussion. Ray Dalio, in a separate interview, commented on how definitions of AGI seem to shift as AI capabilities advance. He posited his own definition, which he considers quite strict, requiring an AI to be an expert in thousands of different areas.

The Path Forward: Breakthroughs Needed

Hassabis believes that two or three more significant breakthroughs are necessary to achieve AGI. He highlighted key areas of development that Google is actively working on, including continual learning, enhanced memory capabilities (potentially through more efficient context window management akin to how the brain stores important information), and improved long-term reasoning and planning.

He also weighed in on the debate surrounding Large Language Models (LLMs), referencing the views of Yann LeCun, who has expressed skepticism about LLMs being the sole path to AGI. Hassabis, while not dismissing LLMs, sees them as a crucial component rather than the complete solution. “I’m not subscriber to someone like Yan Lun who thinks you know that they’re just sort of some kind of dead end. I think the only debate in my mind is are they a a key component or the only component?” he said.

The effectiveness of current benchmarks, such as the ARC (Abstraction and Reasoning Corpus) AGI leaderboard, is also being questioned. While models like Google’s Gemini 3 Deep Think have shown remarkable progress, reaching scores around 80%, research suggests these high scores may not always indicate genuine understanding. A study by Melanie Mitchell and her team found that LLMs can sometimes solve ARC tasks using shortcuts or spurious correlations in the data, such as exploiting numerical representations of colors. When the encoding was changed, accuracy dropped significantly. This highlights that high benchmark performance doesn’t necessarily equate to true comprehension, similar to the historical case of Clever Hans, the horse believed to be intelligent but was actually responding to subtle cues from its handler.

Mitchell’s research indicated that AI models can explain their reasoning correctly only about 70% of the time on ARC tasks, compared to humans at 90%. This suggests that a substantial portion of AI’s correct answers might be coincidental, not based on genuine understanding.

The Multimodal Future of AGI

Beyond textual understanding, the future of AGI is increasingly seen as multimodal. True general intelligence requires more than just language processing; it necessitates integrating vision, audio, touch, and the ability to reason within the physical world. Companies like Figure AI, with their humanoid robots and multimodal vision-language-action models, are exploring this direction.

Chris, in a tweet, speculated that robotics companies like Figure might inadvertently achieve AGI before pure LLM labs, due to their need to build accurate world models for physical navigation. The potential for a model that can perfectly predict the physics of the real world to form the foundation of AGI is a fascinating prospect. Figure’s CEO, Brett Adcock, has stated that current AI chatbots feel like research tools for advanced internet searches, and that true AGI will be multimodal, capable of listening, talking, seeing, remembering, and interacting with the world.

AGI as a Spectrum, Not a Moment

Yann LeCun and Yoshua Bengio, pioneers in AI, view AGI not as a singular event but as a spectrum of improving capabilities. Bengio explained that intelligence is not a single metric; humans excel in some areas while being less proficient in others, and the same applies to AI. He believes progress will occur across multiple fronts, but it’s unlikely AI will achieve human-level capabilities across the board simultaneously.

“It’s not a moment. The reason is simple. Intelligence isn’t just like one number. We have people who are very smart on some things and stupid on other things. And it’s the same with AI,” Bengio stated. He advocates for tracking specific skills as AI improves, assessing their utility and potential risks.

The anticipation for AGI’s arrival varies, with some expecting a gradual evolution akin to the past few decades, while others foresee a more transformative impact. The potential for AI to automate intellectual activities could lead to a world vastly different from today’s, potentially resembling 10,000 years of progress in a mere 25-50 year span.

Source: Google’s AGI Plan Just Got Clearer (Demis Hassabis Explains) (YouTube)