Catastrophic and Sub-Catastrophic AI Risk by 2050

Most respondents assess the probability gap between the two outcomes as significant: “A million…deaths is a far weaker bar than a 10% population loss…reachable by routes that do not require loss of control: AI-enabled bio misuse by a small group, AI-directed military systems used at scale in a great-power or regional war, or cascading failure in AI-managed critical infrastructure.”; “Events killing a million people are fairly common historically, and most future ones will [involve] AI systems.”

A minority argues the gap is small; these respondents tend to focus on loss-of-control scenarios: “If we have lost control enough that the AI can kill a million people, we have lost control.”; “The window between AI systems being sufficiently capable to cause 1,000,000 deaths and AI systems being sufficiently capable to cause 10% of the global population is probably very small.” One respondent writes, “there is no guarantee that AI lethal risks follow a power-law distribution. This is exemplified by the different scenarios considering the existential risks of AI, in which a misaligned AI…waits till its success is guaranteed, which is when the AI takeover or human extinction event happens.”

Most respondents believe an extension of the status quo — no major policies from 2026 to 2050 — would result in more risk than they assigned to their unconditional forecast: “My status-quo estimates are higher [than my unconditional] because freezing policy through 2050 excludes future safeguards that I anticipate.”; “The status quo rules out reactive regulation after a warning shot, which is the mechanism I'd otherwise expect to do most of the work between now and 2050.” A smaller group assigns equal risk to their unconditional and status quo scenarios, generally because they expect little policy movement in either case: “Because I am not optimistic about any of these policies being implemented, my unconditional forecast matches the status quo scenario.”

Most respondents think that a federal law barring states from regulating AI more strictly than the federal government would raise the level of risk. As one writes, “The direct effect is that SB 53 [California’s Transparency in Frontier Artificial Intelligence Act] and other state laws would no longer be enforceable, and future state AI regulation would be blocked entirely. This matters more than it might seem, because…companies often choose to comply with the strictest state standard across the entire country [and] the strictest state law effectively becomes a national floor.” Another adds that a federal ceiling preemption could “remove incentives to strengthen federal frameworks (in response to stricter state-level laws that end up proving beneficial) [and] signal to China and other states a more hands-off approach, incentivizing similar behavior.”

Several forecasters, however, argue any increase in catastrophic risk would be incidental because the regulations being preempted largely don’t address catastrophic risk: “This is much less likely to affect the probability of catastrophes as opposed to other regulatory priorities (bias, data exploitation and price discrimination, demonstrated or speculative effects on physical/psychological health).”; “State AI laws...address discrimination, disclosure, and synthetic media. A federal ceiling on them [does not matter] for mass casualties.”

Most respondents agree that slowing AI development is the policy with the most potential, in theory, to mitigate loss-of-control risk, with optimists arguing it can buy time “for safety to keep pace with capability progress.” But many worry that slower development will merely postpone the emergence of dangerous capabilities: “Slowing progress down is still likely to lead to very powerful systems being developed and deployed by 2050.”

US only. Respondents frequently argue a unilateral slowdown is unlikely to reduce risk substantially: “I don’t think unilateral slowdown will do much. Even assuming it would hold, China would catch up within ~6 months and then the risk would come from them.”; “Barely helps, because the frontier just advances somewhere with less safety culture.” A significant minority disagree, viewing Chinese progress as downstream of US progress: “Chinese labs rely heavily on distilling US work, so US slowdowns in fact cause Chinese slowdowns.” Many argue that even if it does little alone, it raises the odds of a bilateral agreement: “Once the US is out of the race, that takes the pressure off the Chinese and makes them likely to have their own slowdown.”

US and China. Most see substantially greater benefits if a slowdown covers both leading jurisdictions, ranking it at or near the top because “the two countries hosting essentially all frontier training are inside it.” One writes, “A binding speed-limit agreement between the two removes the prisoner’s dilemma of being forced to sacrifice safety testing out of fear that the opponent will break through first,” and another that it “seems like the only policy on the menu with a realistic chance of meaningfully reducing catastrophic risk from loss of control.” Pessimists emphasize evasion and covert programs: “Residual risk remains from compliance and verification doubts, the possibility that development migrates to other jurisdictions…or even hidden military programs.”

Respondents who focus on risk from the misuse of released models often rate this policy highly, with one noting “there are quite a few safety regimes like this, in aviation and medicine, where it obviously has costs but seems to work out quite well for safety.” For many, however, the Hugging Face incident and other loss-of-control concerns temper enthusiasm: “Models are now dangerous in training. Even the strictest pre-release testing regime wouldn’t stop the RSI sprint to ASI with internal models.”; “I expect most risk comes from unreleased models.”

US only. Supporters emphasize catching “dangerous models at the gate before release, whereas a slowdown only changes the pace at which dangerous capabilities arrive.” One writes, “That seems valuable not because I have great faith in regulatory foresight, but because it creates one more place where new evidence can interrupt deployment.” Doubters argue the US government has “very weak state capacity to actually understand and make coherent decisions about pre-release vetting,” and that “a system can pass the test and still be more dangerous than the alternative” — particularly since “the main threat model is that AIs are scheming and are able to hide this, so they would just pass the tests.”

International. As with a joint slowdown, most view this version as a significant improvement, largely “because it would reduce regulatory arbitrage and cover frontier development across both major AI powers,” and would create a “strong gravitational pull for other jurisdictions to join.” Many point to the potential for “an IAEA-like admission network.” Skeptics question the competence or independence of international institutions — “its standards will tend toward the lowest common denominator” — and note international coverage would still miss internal deployment and already-diffused capabilities.

Many respondents are optimistic this policy could change developer behavior by “[aligning] AI Lab economics with public safety” and “[leveraging] the labs' own understanding of their products’ capabilities.” One argues, “pricing tail risk forces developers to quantify and reduce it continuously, without a legislature having to specify in advance what ‘dangerous’ means — which is exactly the property you want when capabilities are moving faster than rulemaking.”

Skeptics think developers will treat liability “as a cost of doing business,” doubt courts’ ability to attribute harms, or worry attention will shift toward smaller, insurable risks: “Liability deters developers who can be identified, sued, and made to pay. That describes a traceable harm from a deployed commercial model. It does not describe a world where eight hundred million people have died.”

There is broad agreement that this is the lowest-risk condition. Optimists describe complementary layers and a ‘Swiss-cheese effect’. One argues that “each policy acts on a different part of the failure chain, and modest improvements at several points can produce a meaningful reduction in overall probability.”

Skeptics focus on the risks that persist regardless: “The bundle’s components are complementary but their failure modes are correlated: the international slowdown and authorization regimes share a common verification and compliance fate, and all three leave the same residual core untouched (covert and military development, deceptive systems that pass evaluation, and proliferation outside member jurisdictions).”