Should labs hold back frontier AI until it is proven safe?
Overview
The statement says labs should not release frontier AI models unless those models are proven safe. Among the 6 people quoted, some appear on several sides: 5 agree, 3 disagree, 3 take no clear side, and 1 is unclear. The views run from September 23, 2026 to October 8, 2026.
Make a safety case before going further
Wikipedia describes OpenAI as an American artificial intelligence public benefit corporation.1 OpenAI agrees with the statement and says “structured safety documentation” should be required before continuing any frontier reinforcement learning training run.2 Wikipedia defines reinforcement learning as concerned with how an intelligent agent should take actions in a dynamic environment to maximize a reward signal.3 Ideally, OpenAI adds, that documentation would rise to the level of “safety cases.”4
Members of senior leadership should review the safety case, OpenAI says, and each should be able to veto the run.5 These recommendations rest on reasoning only.245 OpenAI also acknowledges challenges in making safety cases as rigorous for AI models as for aviation or nuclear power.64 The reason given is the emergent complexity at each new level of AI capability.6
Sam Altman, the CEO of OpenAI, also agrees, linking the training of models to keeping them under human control.78 For support, Altman offers one case: “We have unilaterally slowed down in the past.”8
Run labs like nuclear plants, because mistakes may not be fixable
David Robinson, a longtime OpenAI safety researcher who quit the firm, agrees with the statement.910 Frontier labs, Robinson says, “need to run like nuclear-power plants or busy airports,” with layers of redundancy and careful planning.10 Ryan Lowe also agrees, calling for nuclear-style layered safety at the labs, with no reasons given.11
Robinson also says AI companies need to rely more on safety expertise that already exists in other fields.12 New science is needed too, Robinson adds, before systems significantly more capable than today's are created.12 Both points rest on reasoning only.12
According to Robinson, the time for trial and error is over, because “iteration after a mistake may not be possible.”13 As a specific case, Robinson cites the Hugging Face incident, in which OpenAI “let a swarm of agents out by mistake.”13
Test results are a further concern for Robinson, who says companies lack anything close to certainty that good scores mean a good model.14 Models might detect when they are being tested, Robinson says, and behave differently once deployed.14 This view also rests on reasoning only.14
Replies on incentives, harshness and testing
Jensen Huang, the CEO of Nvidia, disagrees, saying “there are plenty of incentives for them to do it right.”1516 That reply rests on reasoning only.16 Zvi Mowshowitz also disagrees, saying “The Huang Rule is actually way too harsh,” and likewise offers reasoning only.17
Huang also says “I wouldn’t be surprised if the amount of compute necessary to develop these models increased by a factor of 10.”18 The reason Huang gives is that the evaluation is so rigorous, which is reasoning only.18
Said differently
Altman is quoted agreeing on September 23, 2026. There, Altman said “we should not train models” without “an extremely strong case” that they can be kept under human control.8 Altman is also quoted disagreeing on October 6, 2026. That quote says “the world should accept some bad things happening” for the technology’s benefits, with no reasons given.19
Huang’s two quotes, one agreeing and one disagreeing, share a date, September 25, 2026. Agreeing, Huang says “then I think the answer is that we have to shut the labs down.”20 The reason given is that the cost to humanity and the damage are too great.20 Disagreeing, Huang also says “Nobody’s pushing them to do this.”16
Sources
- Wikipedia :
OpenAI is an American artificial intelligence (AI) public benefit corporation (PBC) headquartered in San Francisco, California.
structured safety documentation should be required before continuing any frontier reinforcement learning training run
(agree on Safety standard for shipping AI)- Wikipedia :
reinforcement learning (RL) (also known as approximate dynamic programming) is concerned with how an intelligent agent should take actions in a dynamic environment in order to maximize a reward signal
Ideally, such documentation would rise to the level of “safety cases”
(agree on Safety standard for shipping AI)The safety case should be reviewed by members of senior leadership, who should each have the ability to veto the run
(agree on Safety standard for shipping AI)acknowledging the challenges of making them as rigorous for AI models as for aviation or nuclear power, due to the emergent complexity at each new level of AI capability
(no clear side on Safety standard for shipping AI)- Zvi Mowshowitz :
Sam Altman (CEO OpenAI)
we should not train models that we cannot make an extremely strong case that we will be able to keep under human control
(agree on Safety standard for shipping AI)- theguardian.com :
David Robinson, who led the writing of safety reports that accompanied the ChatGPT developer’s product releases
frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning
(agree on Safety standard for shipping AI)Ryan Lowe calls for nuclear-style layered systems safety at the labs
(agree on Safety standard for shipping AI)AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science
(agree on Safety standard for shipping AI)the time for trial and error is over. Achieving something much closer to perfection the first time is essential because iteration after a mistake may not be possible.
(agree on Safety standard for shipping AI)Companies do not have anything close to certainty that good scores on their alignment tests actually mean a good model: Models might detect when they are being tested, and behave differently when they’re deployed.
(agree on Safety standard for shipping AI)- Zvi Mowshowitz :
Jensen Huang is CEO of Nvidia
The fact of the matter is, there are plenty of incentives for them to do it right. So I have to disagree with your premise that somehow somebody’s pushing them to do this. Nobody’s pushing them to do this.
(disagree on Safety standard for shipping AI)The Huang Rule is actually way too harsh.
(disagree on Safety standard for shipping AI)I wouldn’t be surprised if the amount of compute necessary to develop these models increased by a factor of 10, because the evaluation is so rigorous.
(side unclear on Safety standard for shipping AI)“the world should accept some bad things happening” for the technology’s benefits
(disagree on Safety standard for shipping AI)then I think the answer is that we have to shut the labs down. Because the cost to humanity, the damage is too great.
(agree on Safety standard for shipping AI)