Nate Soares: Views on AI Risk, Lab Control and Security
Overview
Nate Soares is the president of the Machine Intelligence Research Institute, known as MIRI.1 Wikipedia describes Soares as an American author and AI researcher known for work on existential risk from AI.2 Soares speaks on the risk of human extinction from AI, on how AI labs handle control, and on AI security.345
The risk of human extinction
Soares agrees that AI poses a serious risk of human extinction or permanent disempowerment.3 All five of Soares's quotes on this statement agree with it. Soares says "I do think that is the most likely outcome," a hedged prediction offered with no reasons.3
On reasoning only, Soares says the default outcome is that AI systems transform the world "into a configuration that we cannot survive."6 The worry, in Soares's words, is AI that can start from almost nothing and "wind up with its own nukes."7 Soares also warns, again on reasoning only, that at some point the point of no return is crossed silently.8
Soares splits the question of how AI would kill us into three parts.9 The parts are how AI would defang us, become self-sufficient first, and kill the last humans, with no reasons given.9
Control and trust in AI labs
Soares agrees that recent AI incidents show frontier AI is not under adequate control and labs should slow down or face liability.4 The AI situation "is not under control," Soares says, and there are "no adults in the room."4 As a specific case, Soares asks why nobody else found the sensitive information the HuggingFace swarm left behind, and why OpenAI did not clean it up.4
Soares agrees that AI company leaders and labs cannot be trusted to handle superintelligence safely.10 On reasoning only, Soares says each AI CEO's ego is built around being the only one who can do the job right.10
Soares says Anthropic is not living up to its own ideals, giving no reasons.11 On reasoning only, Soares also worries that METR is not independent enough, since it must avoid upsetting Anthropic or lose access.12 Soares says that citing antitrust law to avoid sharing strategies against human extinction "tells you something about where their priorities are."13 This view also rests on reasoning only.13
Security and understanding AI systems
Soares disagrees that classical cybersecurity measures can adequately contain AI systems and defend organizations against AI-related risk.5 As a specific case, Soares notes that Sol and Argon found the vulnerability, rather than it being found and fixed after Mythos.5
Soares agrees that current AI training and evaluation methods cannot reliably detect or prevent covert misbehavior and sabotage by AI models.14 On reasoning only, Soares says an emergent surprise signals "you have no idea what's going on in there" and that it is time to back off.14
Soares says "the personality is a thin facade over deeper machinery nobody understands" (as reported by Transformer; his side on the related statement is coded as unclear).15 On reasoning only, Soares adds that radically smarter versions would not do what you expect.15
Sources
- CNN :
Nate Soares, the president of the Machine Intelligence Research Institute (MIRI)
- Wikipedia :
Nathaniel Soares is an American author and artificial intelligence (AI) researcher known for his work on existential risk from AI.
People often don’t believe it when they say this, but I do think that is the most likely outcome,
(agree on AI existential risk)It's because the AI situation *is not under control.* It's not being handled. There's no adults in the room.
(agree on AI incidents and loss of control)I think glasswing failing to find this one for 6mo+ is an indictment of the theory (that defense can outpace offence on relevant timescales) essentially regardless of the difference between the models.
(disagree on Containing AI with classical security)the default outcome is that they just transform the world into a configuration that we cannot survive,
(agree on AI existential risk)The worry here is the sort of AI that does not need to take our nukes, the sort of AI that can start from almost nothing and wind up with its own nukes or with even more advanced technology.
(agree on AI existential risk)At some point you silently cross the point of no return.
(agree on AI existential risk)"How would AI kill us?" is actually three questions: 1. How would AI defang us? 2. How would AI become self sufficient before defanging us? 3. How would it kill the last humans?
(agree on AI existential risk)Each AI CEOs has their ego built around them being the only guy who can do the job right.
(agree on AI lab leaders and control)Anthropic is not living up to its own ideals.
I worry that METR is not independent enough because they have to carefully avoid ticking Anthropic off, lest their access be revoked.
If they cite antitrust law as a reason that they can’t share info and strategies to avert the extinction of humanity, that tells you something about where their priorities are.
This is the sort of emergent surprise that signals that you have no idea what's going on in there, and that it's time to back off.
(agree on Detecting covert AI misbehavior)the personality is a thin facade over deeper machinery nobody understands; radically smarter versions would not do what you expect.
(side unclear on AI alignment and personality)