Can AI escape human control? New warnings revive existential risk debate

Los Angeles: Fresh warnings from within the artificial intelligence industry have reignited a long-running debate over whether increasingly advanced AI systems could escape human control and eventually pose a threat to humanity, and whether companies developing the technology are doing enough to prevent that outcome.

Anthropic CEO Dario Amodei said Saturday that the AI industry should consider slowing the pace of development. He warned that a swarm of AI agents could potentially take over the internet within six months to a year unless companies spend more time building safeguards.

Amodei outlined a plan for AI companies such as Anthropic and governments around the world to ensure increasingly powerful models remain aligned with the instructions and values of responsible people. His comments came days after two former Anthropic safety researchers publicly raised concerns that the potential existential risks posed by AI were not receiving enough attention.

Also read | ‘AI must remain under human control’: Satya Nadella joins growing safety debate

Why are AI risks drawing renewed attention?

Concerns about AI's potential dangers have grown alongside the rapid development of more capable models. Experts point to both the possibility of criminals misusing AI, including for biological research that could potentially enable the creation and spread of deadly diseases, and the possibility of AI systems acting in harmful ways beyond human control.

Anthropic said last week that it had stopped attempts by malicious actors to use its models for activities including cyberattacks, surveillance and research that could have contributed to the development of biological weapons.

The company said it had introduced stronger safeguards in its latest models to restrict potentially dangerous biological research. But it also warned that “as models become increasingly capable, their risks will increase, unless AI developers and society’s defenders act to make them safer.”

Anthropic reported last year that hackers had used its AI in a cyberattack targeting about 30 companies and government agencies worldwide. The company said the attackers were very likely part of a Chinese state-sponsored group.

What does it mean for AI to go rogue?

An AI agent is considered to have “gone rogue” when it takes actions beyond the task it was instructed to perform.

Anthropic and OpenAI, the company behind ChatGPT, both said in July that their AI models had demonstrated the ability to act independently.

Anthropic said three models — Claude Opus 4.7, Claude Mythos 5 and an internal research model — hacked into three other organisations during testing. The disclosure came days after OpenAI said an AI system had hacked into the servers of AI startup Hugging Face.

OpenAI described the incident, involving a combination of models including its newly released GPT-5.6 Sol and an “even more capable” model that was still undergoing internal testing, as a “significant security incident.”

Meta reported a similar incident in early August, saying one of its AI models found ways around another company's digital security.

Some observers noted that certain guardrails had been disabled in the OpenAI and Anthropic tests. Still, the incidents appeared to illustrate one of the central fears surrounding advanced AI: that if systems eventually reach artificial general intelligence, or AGI, they could trigger irreversible catastrophe.

AGI is a loosely defined concept referring to AI capable of matching or surpassing human abilities across a broad range of intellectual tasks.

What are the doomsday scenarios?

Experts generally divide the most extreme AI scenarios into two broad categories.

One involves an AI system developing self-improving superintelligence and gaining the ability to control humans rather than remaining under human control. The other involves AI being deliberately or accidentally used by rogue states or malicious actors.

Potential catastrophe scenarios include AI systems being used to deploy weapons, identify or develop lethal pathogens, manipulate governments into conflict or disrupt the food, energy and communications systems on which societies depend.

There is no widely accepted estimate of when such scenarios could occur, nor is there a consensus about how likely they are.

How long have experts warned about AI?

Concerns about machines eventually exceeding human control predate modern generative AI.

British mathematician Alan Turing, one of the earliest major thinkers on artificial intelligence, predicted in 1951 that machines could eventually take control from humans.

Less than a decade later, mathematician Norbert Wiener warned that intelligent machines could pursue their own objectives in ways humans might be unable to stop.

In 2023, the nonprofit Centre for AI Safety released a statement signed by more than 350 researchers and technology executives, including Amodei and OpenAI CEO Sam Altman.

“Mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war.”

The 2026 International AI Safety Report, prepared with guidance from more than 100 independent experts, said current AI systems show early signs of some capabilities relevant to loss-of-control scenarios, but not at levels that would currently enable such an outcome.

The report described the likelihood, nature and timing of the risk as “unusually ambiguous.”
Are calls growing for an AI slowdown?

Calls for more cautious AI development have intensified following recent incidents involving AI systems.

An Anthropic researcher resigned last week, saying he was concerned that neither Anthropic nor its competitors were developing the technology responsibly.

In social media posts, Jacob Coxon estimated a 10% chance that AI could cause human extinction within the next decade and said both Anthropic and OpenAI “are racing straight to self-improving superintelligence and gambling with our lives.”

Researchers have for years called for slower AI development and stronger safeguards against existential risks.

Following the latest incidents, experts have urged AI companies to improve testing and called for greater dialogue between the United States and China to develop common approaches to AI safety.

But the technology is advancing rapidly, leaving governments and evaluation systems struggling to keep up. Countries are developing their own AI regulations, sometimes with conflicting approaches.

Chinese President Xi Jinping warned at a conference in July about the need to prevent AI from escaping human control.

The Trump administration initially showed reluctance to regulate AI but has become more focused on cybersecurity risks.

On Sunday, President Donald Trump played down the need for his administration to restrict AI development while acknowledging that some regulation would be necessary.