Year AI Beats Superforecasters on ForecastBench

Early-resolution respondents heavily rely on the linear projection of recent improvements in Brier scores. Many point to the language in the background information which reads, “A simple extrapolation of previous ForecastBench data suggests that LLMs may reach difficulty-adjusted Brier-score parity with superforecasters by October 2026.” Late-resolution respondents, however, challenge the validity of linear extrapolation in this domain, with several arguing that improvements will likely plateau as models bump up against irreducible uncertainty: “There probably is an absolute limit to how well one can forecast,” writes one, and another that “the inherent lower bound from aleatoric uncertainty might well be above 0.072.” One respondent argues that the trajectory “looks much more like a decaying exponential than a linear function... because ultimately the Brier score needs to plateau as it can’t go negative.”

Many early-resolution respondents argue that forecasting plays to AI strengths: “This type of task...is one that is well-suited to LLMs given that they were built to spot patterns and predict outcomes,” is a common sentiment. In addition, many note that ForecastBench questions tend to center on data-rich domains “where AI has a distinct advantage (like weather at specific times and places).” Some late-resolution respondents emphasize that while AI typically handles data-rich questions well, “ForecastBench covers a mix of domains, including geopolitics...where structural uncertainty, regime shifts, and sparse data limit purely algorithmic forecasting advantages.” Another echoes that sentiment, writing, “For geopolitical questions—where data is sparse, hard to structure, and where intuition, contextual interpretation, and complex human interactions on a highly aggregated level matter—I think models currently perform less well.”

Many high-forecast respondents—and in particular, superforecasters—emphasize that the way ForecastBench is structured advantages LLMs in a way that makes it more likely that one will hit the target score. The following arguments were common: “Improvements in Superforecaster accuracy over time are not taken into account, only our accuracy as of 2024. This freezes the target. AI models will improve, but for purposes of this question Superforecasters will not.”; “AI systems are improving rapidly and will keep improving, whereas they are being compared against a 2024 performance by a group of 39 superforecasters (who in the real world would have been constantly updating their predictions for unresolved questions…).” Some superforecasters also pointed to the limited number of human forecasters assigned to each question, as well as the non-complex, data-dependent nature of many of the questions, as additional factors that could contribute to an early resolution.

Many late-resolution respondents emphasize that the resolution criteria is exceptionally demanding. One states: “This a high bar—not just parity with 2024 superforecasters, but having a score outside the 95% confidence interval when, at present, all AIs have a score outside the 95% confidence interval in the other direction.” Another writes, “Beating the superforecaster median on ForecastBench after difficulty adjustment is a much higher bar than ‘be competitive’. The last bit of improvement is going to be brutally hard for AI: excellent calibration, restraint (knowing when not to be confident), and robustness across lots of weird question types.” Early-resolution respondents tend to minimize the gap between current performance and the specific target, with several suggesting that luck, paired with a high volume of attempts, will likely bridge it: “Given enough evaluations, eventually one will fall under this mark by chance…[Indeed, if] multiple models are tested every 2 weeks, the chance of this happening becomes quite high, so this question is really testing when models reach near parity, rather than any improvements over superforecasters."

Many early-resolution respondents suspect current architectures, augmented by “better scaffolding, tool use, [and] agentic search capabilities” will be sufficient to hit the target score: “Even if algo progress stopped, you can probably finish it off with better scaffolding,” writes one. Late-resolution respondents, however, often express doubt that current paradigms will be sufficient, with one noting that LLMs are “generally tuned to people-please,” and that this characteristic may prevent them from “making as extreme forecasts as they should.” In a similar vein, another argues, "Superforecasters are outliers. AI still averages. I think whatever it [is] that creates those outliers is difficult and it isn't clear it can be learned by a machine easily.”