Cognitive Limitations of AI

Forecasters disagree strongly on whether incremental improvements through scaling and post-training enhancements will suffice to overcome AI's cognitive limitations. Low-forecast respondents tend to believe that “reasoning, memory, and embodiment are all something that can be improved with more computing power, more parameters, and more data.” High-forecast respondents often emphasize that limitations are “intrinsically interlinked,” and that to truly solve them, “we need entirely new architectures.”

Many low-forecast respondents emphasize that the frequency of hallucinations has already been substantially reduced with some predicting that “productized retrieval, tool use, and verifiers [will] cut hallucinations enough that only a minority will still call them serious.” Some also predict that users will become increasingly adept at prompting to avoid hallucinations. Several high-forecast respondents maintain that “even with retrieval-augmented generation and tool use, hallucinations will remain a widely recognized constraint especially in high-stakes settings like medicine and law.”

Low-forecast respondents often highlight recent progress, noting, “the advancements that have been made in just the last year to agentic systems and reasoning engines have [been] shocking levels of improvements.” They tend to think that test-time compute, tool-augmented reasoning, and other inference enhancements will lead to additional improvements. High-forecast respondents, however, tend to think that, “while AI can be expected to achieve superhuman performance in formal closed-system domains like math, code, and certain scientific domains, the ability to perform robust informal reasoning about the messy open world remains a frontier problem that will likely require new architectural breakthroughs.”

There is relative consensus that the memory issue is solvable, with disagreements mainly about timeline and completeness. Low-forecast respondents point to technical solutions: “Memory systems (vector databases, stateful agents, persistent contexts) are improving rapidly.” High-forecast respondents frequently acknowledge progress but emphasize limitations: “How to enable AI to work continuously over years like a human expert remains a significant and currently challenging problem.” They focus more on technical challenges, including “context rot which limits the use of explicit in-context prompting as a tool for memory,” and “the general limitations of RAG [retrieval augmented generation].”

Low-forecast respondents often emphasize that transfer learning will help, along with "the explosion, and use, of synthetic data, greater progress in test-time learning, and programmatic reasoning.” Some high-forecast respondents acknowledge that “few-shot learning and transfer learning continue to improve,” but tend to think that an inability to generalize is inherent to the underlying architecture, and that “failures on out-of-distribution or truly novel tasks will still be seen as a serious barrier especially by economists and policymakers.”

Many forecasters identify this as the core unsolved limitation. Low-forecast respondents are scarce and many high-forecast respondents are emphatic: “I deeply believe metacognition is the number one limitation today and that it will remain the case by 2030.” They emphasize that models “don't know what they don't know” in that they are unable to reliably self-correct, defer when appropriate, or guide themselves through the acquisition of new skills and knowledge—and there is no clear solution to address these issues on the horizon. One forecaster bluntly states: “I don't know how to solve this with RL [reinforcement learning]. Serious fundamental advances are needed.”

Disagreements center on timelines. Low-forecast respondents commonly point to recent progress: “If you view a Waymo vehicle as basically a robot on wheels, it is apparent that the problem of navigating in physical space in the real world is manageable.” High-forecast respondents tend to emphasize fundamental constraints: “To generate training data we need to deploy robots in real-world settings to collect data. However, doing this on a large scale presents practical challenges.” A roboticist forecaster agrees: “Embodied AI is significantly more challenging than many in the AI community believe. It is severely data restricted.” Many high-forecast respondents think use cases in unstructured environments (agriculture, construction) are likely to be particularly constrained.

Many low-forecast respondents emphasize that this “appears to be in large part a systems integration and standardization problem that will see major progress in light of strong commercial incentives.” They note rapid progress has already been made via the deployment of tools that “permit a supervised semi-autonomous optimization framework wherein instruction sets, for submission to subordinate models, are able to be optimized, evaluated, and aligned with human experts.” Some low-forecast respondents, however, point out that “AI systems represent various individuals, companies, and organizations, each with their own interests,” leading to a “lack of interoperable standards.” One notes: “While short-term collaboration between AI systems is feasible and not a major obstacle, sustained, coordinated collaboration over extended periods remains challenging.”