Test-of-Time Winning AI-Written Paper

Early-resolution respondents usually take a non-restrictive interpretation of this language and believe that “‘written in part by AI’ is a very low bar” and perhaps already being cleared. One writes, “I think a lot of the most important AI-related papers in recent years have been ‘written at least in part by AI’ (e.g. Anthropic often notes that Claude was used to help write some of their important papers).” Another argues that “currently a significant fraction of the manuscripts being published (maybe 5 to 10% at least, probably more) could perfectly list AI as a contributing author (or as an important tool used during research).” Late-resolution respondents typically filter this language through a far more restrictive prism, assuming that any AI contribution must be significantly greater than what is occurring today to count: “Authors don't list other tools as their co-authors.”

Early-resolution respondents point to rapid advancements in AI capabilities, both “in other fields (protein design, materials, theorem proving),” and in writing papers: “The Agents4Science conference showed that fully AI-written papers are possible,” writes one, and another that “METR [Model Evaluation and Threat Research]'s research has shown AI task horizons doubling every ~7 months, suggesting month-long autonomous research projects may be achievable by 2028-2029.” Most late-resolution respondents express skepticism about AI's creative and conceptual abilities, and they deem these to be essential for the question to resolve: “I do not believe AI could create, on its own, any scientific paper beyond a literature review with significant insights because AI is primarily a predictive engine…”; “GenAI, as it is today, simply doesn't have the context or world view to consider nuanced and novel approaches to problems.”

Late-resolution respondents often emphasize what they perceive to be deep-seated resistance in the academic community to attributing authorship to AI. One writes that “major publication venues have rules against listing AI as an author, which I do not expect to change, as academics will likely keep seeing AI as a tool rather than a peer that deserves being listed as an author.” Another wrote that crediting AI is “fundamentally at odds with the purpose of a Test-of-Time award,” the point of which is to recognize “lasting intellectual human contribution and human insight.” Many early-resolution respondents, however, point out that the resolution panel’s independence from traditional conference rules allows for flexibility, and that while major conferences might ban AI authors, the panel can recognize work from open platforms like arXiv which “means a high-impact AI paper can be recognized by the expert panel even if it is structurally excluded from the prestige economy of major conferences.”

Most early-resolution respondents believe improvements to current AI architectures, even if they fall short of AGI, will suffice. One writes, “The genuinely hard part of research lies in identifying unsolved problems, combining existing ideas in novel ways, and iterating through experiments, and recent advances in AI coding and reasoning models are increasingly supporting even these processes.” But some late-resolution respondents argue that “new architectural paradigms in AI” would need “to be developed that could broaden a model / system's ability to consider the framing of problems in similar ways that a human expert would.”; “Barring/until about AGI,” writes another, “this feels very unlikely to me.”

Many late-resolution respondents point to the extraordinary difficulty of writing a test-of-time-worthy paper. One notes that such awards recognize papers “like those focused on GANs, ImageNet, and the transformer architecture. These are research efforts that were widely cited, changed directions of entire research domains, and had enduring practical impact and relevance.” Another emphasizes that “the gap between where AI papers are now and Test-of-Time caliber is pretty enormous!” Many early-resolution respondents, however, believe AI can nevertheless meet this bar relatively soon based on evolving capabilities or that the resolution panel is likely to apply generous standards. One suggested that “people will be tempted to give some AI paper a ‘Test of Time’ Award just for the frisson of novelty,” and another that “the act of judging an AI authored paper as award worthy becomes a way to signal technological progress or generate publicity.” A third argued that specific topics like “‘AI consciousness’ or chain-of-thought research seem like they would particularly plausibly merit recognizing the contribution of the AI system itself.”