Benchmark Performance of Open and Closed Models

Most high-polarity respondents emphasize that jumps in model capabilities increasingly require massive amounts of capital to pay for the high cost of compute, proprietary data, the best tool and retrieval stacks, and top-tier research teams. As a result, one forecaster concludes, “We should expect closed-weight models with heavy capital expenditure to perform particularly well at leading-edge tasks.” Low-polarity respondents often argue that capital-intensive techniques developed by closed labs will be adopted by open-source models. “We've seen this in the past,” notes one forecaster, adding “everything from telegraphy to computers (ENIAC onwards) to the internet, in which what starts as a more elite and constrained technology migrates widely into open knowledge.” Other low-polarity respondents believe that in lieu of massive capex, “aggressive community efforts” that result in rapid iteration, distributed innovation, and collaborative scaling will help open-source models close the gap, and that “open-weight progress will be [also] accelerated by expanded access to high-quality synthetic data.”

High-polarity respondents often point to historical trends that show a persistent gap between closed and open model performance: “Open-model performance has tended to lag closed-model performance by anywhere between about 6 and 22 months,” wrote one. Others estimated similar lags. Low-polarity respondents challenge this pattern, pointing to recent convergence: “According to Stanford's 2025 AI Index Report, the gap between closed and open models narrowed from 8.04% to 1.70% in just one year [on the Chatbot Arena Leaderboard]” (Stanford University Human Centered Artificial Intelligence 2025).

Many low-polarity respondents believe convergence could result from the four specific benchmarks becoming saturated prior to the end of 2030. SWE-bench was thought to be particularly susceptible: “I find it likely that the current version of SWE-Bench is fully saturated by 2027 by both closed and open models.” Many also thought the other benchmarks were also likely to saturate: “Eventually, benchmarks saturate (e.g., see what happened to GPQA),” wrote one forecaster, with another arguing “there is very little chance a benchmark with >1% score today is not completely saturated in 2030.” High-polarity respondents typically either do not view convergence due to benchmark saturation as likely, or focus less on the specific benchmarks than on polarity in general.

Most high-polarity respondents believe proprietary labs have advantages in breakthrough development. One notes that “a major architectural breakthrough (e.g., a successor to the Transformer), kept proprietary, could dramatically widen the gap.” As with the capex gap, low-polarity respondents tend to view rapid iteration, distributed innovation, and collaborative scaling as ways for open-source models to remain competitive. One points to the “proprietary and non-proprietary development of the human genome project” as an apt historical parallel.

Some high-polarity respondents believe that FrontierMath and ARC-AGI II, lacking clear practical utility, will be more of a focus of closed model developers: “Math to impress the general public, ARC-AGI for the specialists and the VCs.” Several low-polarity respondents, however, come to the opposite conclusion: “FrontierMath and ARC-AGI II don't have obvious economic utility, so they may become more of a focus of academics, etc. using open source models.”