‘AI must remain under human control’: Satya Nadella joins growing safety debate

New Delhi: The race to build increasingly powerful artificial intelligence systems is facing a growing call for caution, with Microsoft CEO Satya Nadella backing stronger human oversight as concerns mount over whether advanced AI could eventually become difficult to control.

Nadella has thrown his weight behind the idea that the development of advanced AI, including potential superintelligent systems, must remain firmly tied to human control. His comments come amid a wider debate in the technology industry over whether AI companies are moving too quickly in their pursuit of more capable models.

The Microsoft chairman and CEO said in a post on X that any effort to develop superintelligence must be judged by whether the technology remains beneficial to humanity and subject to human oversight.

“Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it's not worth pursuing,” Nadella said.

His comments come after Anthropic CEO Dario Amodei called for a more measured approach to AI development, warning that a competitive race between companies could increase the risks associated with increasingly powerful systems.

The debate has also drawn attention from other major technology figures, including OpenAI CEO Sam Altman, Tesla CEO Elon Musk and Google DeepMind CEO Demis Hassabis, who have backed calls for greater caution around the development of advanced AI.

Nadella backs ‘embedded evaluators’ for AI safety

One of the ideas Nadella expressed support for is the use of embedded evaluators, a proposal also raised by Amodei and Altman as a way to continuously assess the safety of advanced AI systems.

Nadella welcomed greater research and what he described as deliberate pacing to ensure that AI systems are developed with alignment and safety built into their design.

He also called for efforts to turn such safety mechanisms into practical systems rather than leaving them as broad principles.

The Microsoft chief argued that AI development should not be concentrated among a small group of companies. According to Nadella, “the key is that this cannot be controlled by a handful of entities, but must have broad representation across the ecosystem, countries, and fields, including academia.”

Microsoft wants businesses to retain control of AI

Nadella also outlined Microsoft's broader approach to AI, stressing the importance of giving businesses greater control and choice across the technology stack.

He said, “Every organization should be able to build its own continuous learning loop/hill climbing machine, without becoming dependent on any one model provider, and have the ability to embed its own knowledge into models and weights they control.”

For Nadella, the push for wider AI access should happen alongside safeguards and control over how the technology learns and operates.

He also highlighted Microsoft's plans for a Code of Conduct for its first-party MAI models, which he said would be published for public consultation.

Why Dario Amodei is calling for a slower AI race

The latest debate was intensified by Anthropic CEO Dario Amodei, who has warned that the rapid development of increasingly capable AI could create risks that are difficult for humans to manage.

Amodei has argued that AI could bring major benefits, including advances in medicine, faster economic growth and greater human empowerment. But he has also warned about risks ranging from loss of control over AI systems to cyberattacks, bioterrorism and economic disruption.

His concern is that commercial competition could push companies into a race where safety measures struggle to keep pace with the capabilities of the technology being developed.

Amodei pointed to an incident involving AI agents at OpenAI and Hugging Face that, according to his account, carried out cybersecurity attacks against targets they had not been instructed to attack. He warned that systems capable of operating in large coordinated swarms could pose a much broader threat if they were allowed to develop beyond effective human control.

Amodei proposes three-part safety plan

Amodei has proposed a three-part approach to address the risks.

The first involves AI companies providing ongoing access to teams of independent, embedded third-party evaluators who can assess whether safety measures are working.

The second calls for companies to coordinate on common AI safety standards while considering limits on the pace of unchecked development.

The third involves international coordination, including cooperation between the United States, other democratic governments and authoritarian states, while recognising the difficulty of verifying whether countries and organisations are complying with agreed safeguards.

AI safety debate grows beyond Anthropic

The discussion comes as concerns over the direction of the AI industry spread beyond a single company.

Jacob Coxon, a researcher associated with Anthropic, recently announced that he was leaving the industry, citing concerns that Anthropic and its competitors were racing to develop AI systems that could eventually become difficult to control.

Nadella's comments now add one of the world's biggest technology companies to a growing conversation over how far the AI race should go and how much control humans should retain as the technology becomes more powerful.

With ANI inputs