Jacob Coxon leaves Anthropic
Anthropic researcher Jacob Coxon announced his resignation with a warning that his employer and OpenAI were racing toward self-improving superintelligence and gambling with people's lives.1 The Atlantic dates the departure to September, saying Coxon quit in order to argue that AI companies might kill us all.2 According to IEEE Spectrum, Anthropic's head of alignment science Evan Hubinger replied that Anthropic earnestly believes AI could kill all humans, and CEO Dario Amodei later told CNN he agreed with Coxon much more than he disagreed.34
Coxon says a slide from Anthropic's IPO roadshow was a big factor in his decision to leave; he says it shows continued superexponential algorithmic progress in pretraining compute efficiency.56
Coxon says he committed to donating all of his OpenAI equity to AI safety causes, in part to back his message that the race to superintelligence must be paused or drastically slowed.789 The Atlantic says he had earlier seemed to laugh off the idea of donating the shares in an interview, saying "I like the money, bro," and calls this naked hypocrisy before noting he has since agreed to donate.1011 Zvi Mowshowitz says it makes sense for Coxon to make the pledge because people see it that way, though he would ideally prefer Coxon keep his stock, and he argues that holding equity makes Coxon's statements admissions against interest and so more credible.1213
Coxon is due to speak at a protest outside Anthropic's San Francisco offices on the day of the IPO, which Mowshowitz reports was priced at $83 per share, about a $1.5 trillion valuation.1415
What each sentence rests on
Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.”
— spectrum.ieee.org, The AI Superintelligence Debate Has Serious FlawsIn September, an Anthropic researcher named Jacob Coxon quit the company in order to argue that AI companies might kill us all.
— The Atlantic, Silicon Valley Is Suffering From Model CollapseEvan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!”
— spectrum.ieee.org, The AI Superintelligence Debate Has Serious Flaws“I agree with Jacob much more than I disagree with him.”
— spectrum.ieee.org, The AI Superintelligence Debate Has Serious FlawsThis slide from the Anthropic IPO roadshow was a big factor in my decision to leave.
— Zvi Mowshowitz, AI #195: Opening Bellthey are seeing continued superexponential algorithmic progress.
— Zvi Mowshowitz, AI #195: Opening BellI committed to donating the equity I hold in OpenAI. I plan to give all of it to AI safety causes, both because I think donations are impactful and because I want to back my message.
— Zvi Mowshowitz, AI #195: Opening BellWe need to pause or drastically slow down the race to superintelligence.
— Zvi Mowshowitz, AI #195: Opening BellI plan to give all of it to AI safety causes, both because I think donations are impactful and because I want to back my message. We need to pause or drastically slow down the race to superintelligence.
— Zvi Mowshowitz, AI #195: Opening Bell“I like the money, bro,” he told the audience. He has since agreed to donate his equity.
— The Atlantic, Silicon Valley Is Suffering From Model Collapseit looks like naked hypocrisy
— The Atlantic, Silicon Valley Is Suffering From Model CollapseSo in Coxon’s specific case, it makes sense for him to pledge to give his shares away to promote AI safety.
— Zvi Mowshowitz, AI #195: Opening Bellwhich if anything makes them more credible, not less
— Zvi Mowshowitz, AI #195: Opening Bell- Quote not reprinted: this page already repeats as much of this article as we allow. — Zvi Mowshowitz, AI #195: Opening Bell
The final price will be $83 per share, or about a $1.5 trillion valuation
— Zvi Mowshowitz, AI #195: Opening Bell