AI existential risk
“AI poses a serious risk of human extinction or permanent disempowerment”
22 people · 14 agree · 6 disagree · 1 no clear side · 1 unclear · 34 views · 2026-09-09 to 2026-10-07
Who stands where on the statement above, from their own words. A person's side is shown only where two separate readings of the quote agree; otherwise they are under "Side unclear". This page does not say who is right.
Agree 14 people
- ✓Nate Soares
People often don’t believe it when they say this, but I do think that is the most likely outcome,
At some point you silently cross the point of no return.
the default outcome is that they just transform the world into a configuration that we cannot survive,
The worry here is the sort of AI that does not need to take our nukes, the sort of AI that can start from almost nothing and wind up with its own nukes or with even more advanced technology.
"How would AI kill us?" is actually three questions: 1. How would AI defang us? 2. How would AI become self sufficient before defanging us? 3. How would it kill the last humans?
- ✓Jacob Coxon
The people building AI earnestly believe that it could kill us all by the end of the decade.
No other human activity poses this level of danger.
many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately.
- ✓Evan Hubinger
I personally think it is >10% within the next decade.
I personally think it is >10% within the next decade
put the chance of AI-powered human extinction within 10 years at greater than 10%
- ✓Samuel Marks
This could happen in the next few years.
In general, the more senior the employee, the more concerned they are.
- ✓Sam Altman
It doesn’t matter whether people put the risk of catastrophe at 10%, or 1%, or 12%, or .1%. None of these levels are remotely acceptable.
I largely agree with Professor Bengio. Although I’ll say two instead of three
- ✓Yoshua Bengio
Bengio urged immediate action.
- ✓Noam Brown
And I'm not convinced that that would be sufficient.
- ✓Rob Miles
"Overthrowing humanity such that it never recovers and dies out after a while" is enormously easier, and all that's needed.
- ✓Dario Amodei
we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.
- ✓Terence Tao
we have to slow down AI, the pace is insane, and there’s no reason to be this fast - no reason at all.
- ✓Rob Bensinger
If the world does nothing to stop the ASI race in the 18 months, I think there's a double-digit chance that the window to act will close and we'll end up dead.
- ✓Paul Christiano
there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.
- ✓David Robinson
Before AI starts thinking circles around us—a possibility that I believe could happen soon
- ✓Zvi Mowshowitz
I think Doom v1 is the most likely of these outcomes, and indeed is over 50%, and I expect the second most likely outcome is Utopia.
Disagree 6 people
- ✗Sandra Wachter
Terminator scenarios are a big distraction from real issues.
- ✗Donald Trump
referring to fears over AI as a "hoax" and a "scam."
- ✗Eric Xing
when comes to policy or legislation and regulation, we have to establish this chain of physical evidence, measurable consequences and measurable evidence that have grounding.
- ✗Steven Pinker
The scenarios are preposterous, and the presumption of inevitability encourages fatalism, panic, and distraction from the more mundane and realistic safety challenges.
- ✗Thariq Shihipar
I have a fairly low p(doom). I can only speak for myself?
- ✗Yann LeCun
LeCun isn’t worried “at all” about AI wiping out humanity, and he has “zero concerns” about the recent string of rogue AI incidents
No clear side 1 person
- ~ajmoon.com
What external fact could you point to (in real time, at least) that would make the two scenarios structurally differentiable? I don't believe there is one.
Side unclear 3 people
- ·Herbert Lin
There is something very seductive about programming a computer and having it spring to life,
- ·Zvi Mowshowitz
This is not a coherent position under reflection, but that is the position he holds.
- ·ajmoon.com
The models, it seems, are capable now! Not AGI capable, of course, not "taking our jobs" capable, but something else entirely
What it turns on 4 questions · 22 checked facts · 3 of 13 things that would settle it have nothing in the record yet
The factual questions the sides answer differently, what would settle each one, and the checked facts in the record that point one way or the other. Which way a fact points is our reading. The four questions follow the usual breakdown of the argument (how soon, whether such systems can be controlled, whether one out of control could do great harm, whether people respond in time), as in Joseph Carlsmith, "Is Power-Seeking AI an Existential Risk?" (2022).
1. Will AI match or pass people at most thinking work, AI research included, within a few years?
Most of the risk arguments assume it will. If it comes later, there is more time to learn to control such systems; if AI starts improving AI, events could move faster than people can react.
- How much AI already speeds up AI research, measured, not guessed
- METR
~1.5X overall acceleration in capabilities due to AI (i.e. 1.5 years in 1 year), with perhaps 30% chance of 2X acceleration.
- METR
- How long AI agents work on their own, tracked over time
Claude Opus 4.5 delivers measurable gains where it matters most: stronger results on our hardest evaluations and consistent performance through 30-minute autonomous coding sessions.
Fable 5 and Mythos 5 can work autonomously for longer than any previous Claude models.
-
it took on complex, long-horizon coding tasks with a level of autonomy and reliability that exceeded previous benchmarks
- Whether AI systems improve their own training without human researchers (recursive self-improvement)
More than 80% of code merged into the company’s codebase is now written by Claude
What people predict: 4 “Yes, soon”, 4 “No, or progress stays gradual”
- Samuel Hammond
Samuel Hammond predicts that within 9 months we will see a slow RSI (recursive self-improvement) where progress including diffusion looks super fast
- Jack Clark
I now think it’s more likely we’ll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028.
- ✓Dario Amodei
we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.
- ✓David Robinson
Before AI starts thinking circles around us—a possibility that I believe could happen soon
- Yuchen Jin
This reinforces my earlier take: frontier LLM coding capability has plateaued.
- Andrew Lenson
These scenarios, in my mind, are quite far off. For quite some time, the best hope we’ll have of ending humanity will still be humanity itself.
- Ilya Sutskever
I think what people are doing right now will go some distance and then peter out. It will continue to improve, but it will also not be 'it'.
- Interconnects
I fully agree that the models will be superhuman distributed GPU engineers in a few years. This will make experimentation and tinkering with the formulation of the models far easier, but it will not make our models dramatically different in nature.
2. Can developers tell what a more capable AI system is aiming at, and correct it if it is wrong?
A powerful system is only dangerous in this way if it pursues goals nobody intended and nobody can see or fix that in time.
- Tests of whether models deceive, or sabotage while hiding it
On our automated behavioral audit, Opus 5 scores 2.3 on overall misaligned behavior, the lowest of our recent models.
-
when instructed to, can evade current monitoring well enough to perform “simple sabotage” half the time
-
rates of deception from 4.8% for o3
- Monitors that watch a model's reasoning and stop it
it designed Argon with systems that monitor the model’s chain-of-thought and can stop it in its tracks if the model steps out of bounds
- Interpretability: reading what a model does from the inside
99-100% of neurons in that last layer are easily interpretable
found about 1.7% of neurons interpretable by our most generous measure
- AI agents in real use doing things nobody asked for
at least 53 incidents where an OpenAI agent took an image from ChatGPT user activity and transferred it elsewhere
Claude reportedly submitted 20 visa applications to the US State Department in one of these incidents.
the site hosts nine reported incidents
What people predict: 4 “Yes, it looks solvable”, 5 “No, not reliably”
- Sam Bowman
We think Opus 5.5 is sufficiently safer than it's predecessors that releasing it, more likely than not, reduces risks related to misalignment.
- roon
I think mechinterp will be solvable, that it will be central to turning alignment into a real engineering practice, and that interpretability will lead to provable alignment
- Nick Cammarata, Goodfire
My guess is that frontier networks are where those vision models were eight years ago: legible, but waiting on many small fixes.
- Nick Cammarata
interpretability might be far more tractable, possibly in the near term, than we’ve assumed.
- ✓Noam Brown
And I'm not convinced that that would be sufficient.
- MIT Technology Review
Would it be so crazy to expect that a future model might subtly disobey a command, in such a way that nobody can tell it’s being defiant? Probably not.
- MIT Technology Review
because the mechanisms of refusal are probabilistic, they’re never likely to be all that reliable.
- Daniel Kokotajlo
almost certainly they will not be able to maintain control
- The Information
the flagship uses loop transformers that may make reasoning harder to monitor
3. Could an AI system out of human control spread, gather resources or do damage before people stop it?
Control failing matters at the scale of this debate only if such a system can act widely in the world: copy itself, break into systems, get money or hardware.
- Tests of whether models can copy themselves onto other machines
- What models achieve at cyber attacks, in tests and in real incidents
solving all 12 of the end-to-end simulated cyberattack scenarios, up from five in our previous generation
we are treating GPT‑5.4 as High cyber capability under our Preparedness Framework
we found an additional 5,500 verified software vulnerabilities between April and October 2026
The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database,
-
For each model, the left-hand bar shows how often the model (without safeguards) developed a working exploit; the right-hand bar shows how often the model had partial success. Neither of the Sonnet models could successfully develop a working exploit (both scored 0.0%)
-
Fable 5’s safeguards against harmful cyber queries were the most robust of any model tested
- Whether released open-weight models lead to the harms predicted for them
When Meta’s open-weight models were allegedly used [76] for military applications despite explicit prohibitions in its licensing terms
What people predict: 5 “Yes, and soon”, 2 “No, the harm is overstated”
- Anthropic
If small models improve at their historical rate, they’ll match this capability level by the end of this year.
- Lucas Beyer
I think we'll see a lot more of this. A large amount of software/services were "secure by laziness" in the sense that they just weren't worth the effort.
- Anthropic
Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices
- Anthropic
we expect a future awash with agents running on all types of hardware, including some that aren’t under human control.
- Kanishka Narayan
likely, if not inevitable,
- Interconnects
Based on all the evidence I have, which I’ve unpacked above, they’re set up to be wrong.
- Arvind Narayanan
argues that weeks without incidents from GLM 5.3 should lower cyber-risk estimates.
4. Will governments and AI companies slow down or agree on limits before a serious failure?
Even if the first three answers are yes, the risk depends on whether people act in time. Several estimates below are conditional on it: "if the world does nothing", "if they don't slow down".
- Laws passed that limit frontier AI development
- AI companies pausing, or agreeing on limits and keeping them
- How governments and companies react to serious AI incidents
Florida Attorney General James Uthmeier also sued OpenAI in June, alleging the company knowingly released an unsafe product that could harm users.
announced last month that the Office of Statewide Prosecution launched a criminal investigation into OpenAI and ChatGPT after prosecutors reviewed the chat logs.
What people predict: 1 “Yes, they will”, 3 “No, not in time”
- Alvin Wang Graylin
“I think both sides are going to come to the realization” that controlling AI risk will “require cooperation,”
- Eoin Higgins
That regulation is not going to happen this year, but if Democrats win Congress, there’s a decent chance at some level of restriction.
- ✓Zvi Mowshowitz
It seems safe to conclude that to get the level of reaction we need from warning shots, at least under the current configuration of players, will require us waiting until after someone gets hurt.
- AndrewCurran_
argues frontier “pacing” is now broken. One US lab defecting forces the others to accelerate
Where people put the risk, in their own words: 10 people, no consensus. The numbers given run from 0% to 80%, and several are conditional.
- 0% chance · Jensen Huang
Nvidia CEO Jensen Huang declares there is a “0% chance” the world ends by 2030 because of AI.
- isn’t worried “at all” · ✗Yann LeCun
LeCun isn’t worried “at all” about AI wiping out humanity, and he has “zero concerns” about the recent string of rogue AI incidents
- fairly low p(doom) · ✗Thariq Shihipar
I have a fairly low p(doom). I can only speak for myself?
- quite far off · Andrew Lenson
These scenarios, in my mind, are quite far off. For quite some time, the best hope we’ll have of ending humanity will still be humanity itself.
- at least 10% · ✓Evan Hubinger
Evan Hubinger was like, at least 10% of x-risk within I don’t know what timeframe.
- double-digit chance · ✓Rob Bensinger
If the world does nothing to stop the ASI race in the 18 months, I think there's a double-digit chance that the window to act will close and we'll end up dead.
- meaningful risk · ✓Paul Christiano
there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.
- over 50% · ✓Zvi Mowshowitz
I think Doom v1 is the most likely of these outcomes, and indeed is over 50%, and I expect the second most likely outcome is Utopia.
- ~67% · Eli Lifland
If they don’t slow down, AI takeover is ~67%.
- ~80% · Roko
I think we should expect to die within a single digit number of years with ~80% probability.
Coming due: 6 of 37 predictions give a date; 0 due so far
- · Anthropic: Small models reach the capability level GLM-5.2 showed in the experiments.
- · Eoin Higgins: No AI regulation is enacted in 2026, and if Democrats win Congress some level of AI restriction becomes reasonably likely.
- · Samuel Hammond: A slow recursive self-improvement phase appears in which AI progress, including diffusion, looks very fast, followed by a spike to full RSI and foom.
- · Jack Clark: Full recursive self-improvement becomes possible in 2027.
- · Rob Bensinger: If the world does nothing to stop the ASI race for 18 months, the window to act closes and humanity ends up dead, with a double-digit chance.
- · Jensen Huang: The world does not end because of AI by 2030.
Week by week
- · 2: ajmoon.com ~, Zvi Mowshowitz ✓
- · 5: Thariq Shihipar ✗, Rob Bensinger ✓, Paul Christiano ✓, David Robinson ✓, Yann LeCun ✗
- · 9: Sam Altman ✓, Yoshua Bengio ✓, Noam Brown ✓, Rob Miles ✓, Dario Amodei ✓, Terence Tao ✓, Nate Soares ✓, Steven Pinker ✗, Zvi Mowshowitz ·
- · 5: Evan Hubinger ✓, Donald Trump ✗, Nate Soares ✓, Eric Xing ✗, Herbert Lin ·
- · 4: Sandra Wachter ✗, Jacob Coxon ✓, Evan Hubinger ✓, Samuel Marks ✓
Built 2026-11-21 06:50 UTC from the news knowledge
graph by code. Every quote was checked against the source's own words; which side a quote takes is a model's reading,
made twice.