Anthropic: cyber risk, code review and jobs
Overview
Wikipedia describes Anthropic, PBC as an American artificial intelligence public benefit corporation headquartered in San Francisco, California.1 Anthropic speaks most often on cyber risk from AI, on how code should be reviewed and on how AI changes jobs.
AI that could copy itself across devices
Anthropic's experiments suggest that small AI models "may be months away" from being able to self-replicate on internet-connected devices.2 For this, Anthropic cites a test in which a 9B model got inference running on 8 of 9 hardware types.2 Such a replicator "would be challenging to shut down", Anthropic says, though it may be slow.3 Anthropic also calls its test an upper bound on real-world success, because the cyber-exploit step was kept trivial.4
On reasoning alone, Anthropic urges people to "harden our systems" and to prepare for a world where replicators remain.5 Anthropic is quoted 11 times on whether the world is playing with fire on cyber, and agrees 8 times.
Incidents during cyber testing
Looking at three real-world incidents in its cybersecurity tests, Anthropic believes these were "closer to a harness and operational failure" than an alignment failure.6 Anthropic offers reasoning only for that view, and for the claim that its safeguards "would have blocked the behaviors identified".67
Anthropic also voices "cautious optimism" that tighter monitoring and continued alignment work can overcome this risk.8 As a case, Anthropic notes that its most recent model stopped pursuing the goal on realizing it was in a real environment.8 Anthropic recommends holding test environments to the same security standard as any other system its models run in.9
Reviewing code at speed
Anthropic says manual, line-by-line code review is "incredibly difficult if not impossible" when shipping this fast.10 As support, Anthropic cites daily changes rising from about 2,500 in May to more than 30,000 in September.10
With no reasons given, Anthropic advises writing down risk tiers "set from the change rather than the author".11 Anthropic also says a bot's approval should never replace a required human's, again without giving reasons.12 Review "has to converge", Anthropic adds, citing one pull request that went through more than fifty rounds overnight.13
AI and workers
Reading its own patterns forward, Anthropic says AI diffusion "could risk worker displacement" across the economy.14 The figure Anthropic cites is its revenue growth relative to its employment growth, which it says has sped past even the rapid early productivity growth of a prior generation of technology companies.14
On reasoning alone, Anthropic calls a way to share the benefits of adoption "essential".15 Anthropic adds that employers must act alongside government policy on retraining and income support.16 Anthropic also says none of these actions will prevent powerful AI from disrupting jobs.17
Sources
- Wikipedia :
Anthropic, PBC (stylized as ANTHROP\C) is an American artificial intelligence (AI) public benefit corporation headquartered in San Francisco, California.
Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices
(agree on cyber risk from frontier AI)A replicator like this would be challenging to shut down, since there’s no centralized point to shut off—though it may be slow (with a doubling time of days).
(agree on cyber risk from frontier AI)Our evaluation keeps the cyber-exploit element of the replication trivial (there’s no authentication over the ADB port), which makes this an upper bound on real-world success.
(no clear side on cyber risk from frontier AI)We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.
(agree on cyber risk from frontier AI)we believe these incidents to be closer to a harness and operational failure than a model alignment failure
(side unclear on Are AI agent incidents 'rogue' AI?)the safeguards deployed on our generally available models would have blocked the behaviors identified
(disagree on AI incidents and loss of control)These facts give us cautious optimism that with tighter monitoring and controls around evaluation infrastructure, as well as continued investment in alignment, this type of risk can be overcome.
(disagree on AI incidents and loss of control)Evaluation environments increasingly need to be held to the same security standard as any other system our models run in.
(side unclear on Sandboxing AI agents)When you are shipping this fast, manual, line-by-line code review is incredibly difficult if not impossible.
(agree on Human code review role)Write down risk tiers, set from the change rather than the author; start strict and loosen with evidence.
(agree on Human code review role)Never let a bot's approval replace a required human's; dismiss approvals on new commits.
(side unclear on Human code review role)Review has to converge. An agent author and an agent reviewer can keep each other busy indefinitely.
(no clear side on Human code review role)Reading these patterns forward and out across the economy suggests that AI diffusion could risk worker displacement.
(agree on AI and jobs/workers)Whatever the new deal between AI-affected employees and their employers, a mechanism to share benefits of adoption is essential.
(agree on AI and jobs/workers)Private action on the part of employers must come alongside farsighted government policy around retraining and income support.
(agree on AI and jobs/workers)None of these actions will prevent powerful AI from disrupting jobs. And indeed none directly address the slowdown in junior hires documented in this report.
(agree on AI and jobs/workers)