Accuracy on FrontierMath Benchmark

High-forecast respondents often cite impressive recent trends: “Evaluated scores across benchmark types have tended to jump over 3 year periods. For example: MATH from 7% in 2021 to 72% in 2023 and 90% in 2024.” Another observes, “The historical data show a very rapid increase in the accuracy [on FrontierMath] over a time period just short of a year. In that time the accuracy increased from 0% to 19%.”

High-forecast respondents frequently emphasize likely continued progress: “We've seen jumps of around 5 points on this benchmark every couple of months so far and these jumps will only accelerate as scores approach 50% (benchmark scores tend to be roughly sigmoid-shaped over time).” Among low-forecast respondents, a common sentiment is that “the fastest…progress is behind us and we are now approaching the flat/end point portion of the S-Curve of advancement.”

High-forecast respondents often highlight potential gains from scaling inference compute: “The current top scorer is a reasoning model, and the reasoning model paradigm is relatively new; this suggests that rapid improvements are likely as the paradigm evolves.” Another noted, “with very large amounts of inference compute, it's possible that o3 or o4-mini could already get well over 30% today.” Many low-forecast respondents, however, are skeptical that inference scaling will be sufficient to overcome fundamental architectural limitations.

High-forecast respondents tend to believe that labs will have the training data they need to drive improvements. One notes that an “increased use of synthetic data…will likely lead to broader problem-solving capabilities,” and another that, “by 2031, the frontier labs will have enormous amounts of math data if they believe it's valuable and the RL process will solve the problem.” Low-forecast respondents are more skeptical that training data will be suitable for the task: “One of the key bottlenecks is the scarcity of high-quality, human-generated mathematical data.”

Many high-forecast respondents highlight math's suitability for current architectures. One states, “Existing LLMs are well-suited to make progress in math because math is about logical pattern matching, can be well-represented by text, and the answers are typically verifiable allowing for easy rewards.” Low-forecast respondents express considerable doubt that the current architecture will be sufficient. One mathematician states: “I am deeply skeptical of LLMs ever being able to excel at complex math. LLMs are linguistic pattern-replication machines, and advanced math requires a high degree of conceptual creativity and flexibility that does not derive from linguistic patterns.”

Some high-forecast respondents expect sustained investment due to prestige and R&D value. One notes, “Math is highly relevant to many R&D domains, so progress in math has been, and is highly likely to continue to be, a focus for leading AI companies.” But many low-forecast respondents question that assumption: “Performance from 2028 to 2031 entirely depends on whether or not the frontier AI labs believe it's economically useful to continue improving performance on this benchmark,” writes one, with another bluntly concluding, “I don’t think current companies are focused on solving math.”