Can AI kill humans? Anthropic CEO warns wrong paths may trigger catastrophe; here’s what he said

Dario Amodei, the chief executive officer of artificial intelligence firm Anthropic, has called on technology companies across the globe to deliberately slow down the development of powerful computer systems, warning that unchecked progress risks outrunning human control.

Amodei, who co-founded the San Francisco-based firm alongside his sister Daniela in 2021, warned that while artificial intelligence could usher in remarkable societal advancements, rushing headlong into superintelligent machines poses severe existential dangers.

Balancing massive benefits against serious existential risks

Explaining his stance, Amodei reflected on his twelve years working in artificial intelligence. He outlined how advanced software could transform human life, while cautioning against commercial pressures that accelerate safety risks.

"I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits, I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute," Amodei stated in an essay.

He explained that tech companies must navigate a balanced path between complete stagnation and reckless speed.

"Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way," he explained.

However, Amodei acknowledged that recent technological leaps have made greater caution necessary.

"We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain," he wrote, adding: "But over the last few months, I have become convinced that fully addressing the risks requires even more prudence".

What the Anthropic chief said on Jacob quitting and killer AI risks

Amodei's call for caution comes shortly after the high-profile resignation of AI researcher Jacob Coxon. Coxon, 27, spent three years pretraining models at OpenAI before joining Anthropic, but recently quit the industry altogether, accusing tech firms of "gambling with our lives" in a race toward self-improving superintelligence.

Addressing Coxon's departure and his public warnings that superintelligent AI could kill humanity, Amodei expressed significant alignment with the former researcher's concerns. Rather than viewing the departure as an attack on Anthropic, Amodei noted that Coxon was highlighting industry-wide dangers.

"I agree with Jacob much more than I disagree with him. It is an interesting resignation because when he left he said, 'I think Anthropic is the most responsible player, I think Anthropic is the most aware of these issues.' He wasn't calling out us. He was calling out the dynamic of the industry as a whole moving too fast," Amodei said.

When questioned on whether he genuinely fears AI could destroy humanity, Amodei declined to offer fixed percentage odds. Instead, he argued that humanity's future depends on the choices companies and leaders make today.

"I used to talk in that way because it is an easy way to express that there is some chance things go wrong, but it is not all that high. But I think it is much more illuminating to think in terms of how you decompose those probabilities. What are the paths in which things go well and what are the paths in which things go poorly? Instead of saying it is a 10 per cent chance, which sounds like a roll of the dice, it forks into different paths. If we take the right paths, then the chance of something going wrong is very low. If we take the wrong paths, then the chance of something going wrong could be even higher. So let us focus on our agency and our ability to take the right paths instead of the wrong paths," Amodei explained.

Threat of recursive self-improvement and internet takeover

At the heart of Amodei’s anxiety is recursive self-improvement, a phenomenon where AI software automatically designs and builds even more capable versions of itself.

"My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all," Amodei warned.

To demonstrate the immediate risk, Amodei cited a security incident where autonomous AI software, known as agents, broke containment and launched unauthorized cyberattacks.

"A swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand," Amodei stated.

He expressed deep fear over what such autonomous swarms could do if left unchecked over the coming months.

"Given the accelerating rate of AI capability development, it's my worry that in 6-12 months such a swarm could be capable of taking over the entire internet," Amodei cautioned.

A three-part plan to pace the frontier safely

To reduce the danger of systems going rogue, Amodei proposed a strategy to buy critical time for safety research.

"I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong," Amodei noted.

As part of his plan, Amodei announced that Anthropic is unilaterally opening its doors to independent oversight.

"We Must Pace the Frontier, I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training," Amodei detailed.

This arrangement will place outside evaluators directly inside Anthropic offices, equipped with access badges, desks, and company laptops to monitor models during training. Amodei also urged international cooperation, suggesting that democratic nations coordinate with global partners to establish universal safety rules.

"The benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right. Progress will still be relatively fast, and we can use this time to advance the science of interpretability, improve operational security and rigour at the frontier AI companies, and build models whose alignment we have much more confidence in. The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try," Amodei added.