Robinson on safety at frontier AI labs
Overview
David Robinson is a longtime OpenAI safety researcher who quit the firm and who led the writing of safety reports for ChatGPT's developer.12 Robinson speaks most on what labs should do before shipping powerful AI and on the culture inside those labs.34 Shorter remarks cover extinction risk, alignment and outside pressure for safety.567
What labs should do before shipping
Robinson is quoted four times agreeing that labs should not ship frontier AI models unless they are proven safe. Frontier labs, Robinson says, "need to run like nuclear-power plants or busy airports," with layers of redundancy and careful planning.3 Robinson offers reasoning only for this recommendation.3
OpenAI and other labs, Robinson says, are growing and deploying frontier AI "with far less redundancy and rigor than this."8 The harm from an irreversible loss of control, in Robinson's view, would be much greater than the harm from any single meltdown.8 This view also rests on reasoning only.8
Robinson also says AI companies need to rely more on the safety expertise that already exists in other fields.9 New science is needed, Robinson adds, before anyone creates systems significantly more capable than today's.9 Reasoning alone supports both points.9
Robinson says "the time for trial and error is over," because iteration after a mistake may not be possible.10 For this, Robinson cites the Hugging Face incident, in which OpenAI let a swarm of agents out by mistake.10
Companies, Robinson says, do not have anything close to certainty that good test scores mean a good model.11 Models might detect when they are being tested and behave differently once deployed, Robinson adds, offering reasoning only.11
Culture and mistakes inside the labs
Robinson describes a can-do attitude and work timelines that amount to perpetual sprints, both common across the industry.4 The safety approach from such a culture, Robinson says, "starts with unimpeded optimism" about solving problems as they arise.12 Robinson also says "this approach, by its very nature, guarantees periodic failures," and that the scale of those failures is growing.13 Robinson cites the same Hugging Face incident for this view.13
Anthropic, Robinson says, has acknowledged accidentally turning off its own safeguards because of a misconfiguration.14 Robinson believes such mistakes are typical of the industry.14
Extinction risk, alignment and outside pressure
Robinson agrees that AI poses a serious risk of human extinction or permanent disempowerment.5 AI out-thinking people is, in Robinson's words, "a possibility that I believe could happen soon."5 That view rests on reasoning only.5
So far, Robinson says, the AI industry has failed to teach machines to consistently act as a wise and caring person would.15 Robinson agrees that the alignment problem is currently unsolved and requires fundamental scientific progress.6 Rather than an engineering problem, Robinson says, "I think it’s a science problem."6 Both of these views rest on reasoning only.156
Robinson agrees that powerful AI must be governed through democratic processes and shared international standards rather than by labs or single actors.7 Robinson says colleagues were so busy sprinting that big changes seldom got considered.7 That led Robinson to conclude that stronger safety incentives, "coming from outside the company," are a big part of getting this right.7 As support, Robinson cites three and a half years at OpenAI, leading the drafting of its Preparedness Framework and overseeing safety reports on 12 frontier launches.7
Sources
- The Verge :
longtime OpenAI safety researcher David Robinson quit the firm
- theguardian.com :
David Robinson, who led the writing of safety reports that accompanied the ChatGPT developer’s product releases
frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning
(agree on Safety standard for shipping AI)A can-do attitude of achieving the seemingly impossible—coupled with work timelines that amount to perpetual sprints—are common across the industry. The safety approach that emerges from such a culture starts with unimpeded optimism
Before AI starts thinking circles around us—a possibility that I believe could happen soon
(agree on AI existential risk)I don’t think that alignment is an engineering problem. I think it’s a science problem.
(agree on Alignment is unsolved)my colleagues and I were so busy sprinting that we seldom had the chance to consider big changes, much less to actually make them. That’s why I concluded that stronger incentives for safety—coming from outside the company—are a big part of getting this right.
(agree on Democratic governance of AI)- Quote not reprinted: this page already repeats as much of this article as we allow.
- Quote not reprinted: this page already repeats as much of this article as we allow. (agree on Safety standard for shipping AI)
the time for trial and error is over. Achieving something much closer to perfection the first time is essential because iteration after a mistake may not be possible.
(agree on Safety standard for shipping AI)- Quote not reprinted: this page already repeats as much of this article as we allow. (agree on Safety standard for shipping AI)
- Quote not reprinted: this page already repeats as much of this article as we allow.
- Quote not reprinted: this page already repeats as much of this article as we allow.
- Quote not reprinted: this page already repeats as much of this article as we allow.
- Quote not reprinted: this page already repeats as much of this article as we allow.