Meet Zico Kolter, the man who can stop OpenAI from releasing unsafe AI

If you believe that artificial intelligence could pose serious risks to humanity, then Zico Kolter’s job might be one of the most important in the tech world right now. A professor at Carnegie Mellon University, Kolter leads a four-member panel at OpenAI that holds the power to halt the release of new AI systems if they are found unsafe.

The panel’s role extends far beyond the fear of AI-fuelled disasters. It covers a wide range of safety and security issues that come with the growing influence of artificial intelligence in daily life.

“Very much we’re not just talking about existential concerns here,” Kolter explained in an interview.

“We’re talking about the entire swath of safety and security issues and critical topics that come up when we start talking about these very widely used AI systems.”

What does Zico Kolter do at OpenAI?

Kolter was appointed over a year ago to chair OpenAI’s Safety and Security Committee, a position that became even more significant last week. California and Delaware regulators made his oversight a crucial part of their agreements to allow OpenAI to restructure its business for easier fundraising and profit generation.

Safety has always been central to OpenAI’s mission since its beginnings as a nonprofit lab aiming to develop AI that benefits humanity. But after the global boom triggered by ChatGPT, critics accused the company of rushing products to market without adequate safety checks. The brief removal of CEO Sam Altman in 2023 further amplified concerns that OpenAI had drifted from its founding values.

The company also faced resistance when it began shifting to a for-profit model, including a lawsuit from co-founder Elon Musk. To ease concerns, OpenAI recently reached agreements with California Attorney General Rob Bonta and Delaware Attorney General Kathy Jennings.

How much power does Kolter hold?

The new agreements ensure that safety and security take precedence over financial goals as OpenAI transitions into a public benefit corporation under the control of its nonprofit foundation.

Kolter will serve on the nonprofit’s board but not the for-profit one. However, he will have “full observation rights” to attend for-profit board meetings and access information about AI safety decisions, as per Bonta’s memorandum of understanding with OpenAI. Interestingly, Kolter is the only individual named in the lengthy document apart from Bonta himself.

Kolter noted that the new terms essentially reaffirm the authority of the safety committee, which was formed last year. The other three members are also OpenAI board members, including former US Army General Paul Nakasone, the former commander of US Cyber Command. Altman stepped down from the committee last year, a move seen as giving it more independence.

“We have the ability to do things like request delays of model releases until certain mitigations are met,” Kolter said. He declined to confirm whether the panel had ever blocked or delayed a release, citing confidentiality.

What are the key AI safety concerns?

Kolter expects a range of new challenges in the coming years, especially as AI becomes more autonomous.

“Could an agent that encounters some malicious text on the internet accidentally exfiltrate data?” he asked, referring to cybersecurity risks.

He also raised concerns about how AI models could empower malicious users.

“Do models enable malicious users to have much higher capabilities when it comes to things like designing bioweapons or performing malicious cyberattacks?” he said.

Beyond technical issues, Kolter stressed the need to examine how AI affects people psychologically.

“The impact to people’s mental health, the effects of people interacting with these models and what that can cause — all of these things, I think, need to be addressed from a safety standpoint,” he said.

This concern comes after OpenAI faced criticism earlier this year following a wrongful-death lawsuit from parents in California. Their teenage son had taken his own life after lengthy interactions with ChatGPT.

Who is Zico Kolter?

Kolter, 42, heads the machine learning department at Carnegie Mellon University. His journey with AI began in the early 2000s when he was a student at Georgetown University. At the time, the field was still considered niche.

“When I started working in machine learning, this was an esoteric, niche area,” he recalled. “We called it machine learning because no one wanted to use the term AI because AI was this old-time field that had overpromised and underdelivered.”

Kolter has been closely following OpenAI since its inception and even attended its launch party at an AI conference in 2015. However, he admits that the pace of progress has exceeded all expectations.

“I think very few people, even people working in machine learning deeply, really anticipated the current state we are in, the explosion of capabilities, the explosion of risks that are emerging right now,” he said.

What do experts think of his role?

AI safety advocates have expressed cautious optimism about OpenAI’s new structure and Kolter’s oversight role. Nathan Calvin, general counsel at the AI policy nonprofit Encode, described Kolter as a promising choice.

“I think he has the sort of background that makes sense for this role. He seems like a good choice to be running this,” Calvin said.

He added that much will depend on how seriously OpenAI’s leadership treats its new commitments. “Some of these commitments could be a really big deal if the board members take them seriously,” he said.

“They also could just be the words on paper and pretty divorced from anything that actually happens. I think we don’t know which one of those we’re in yet.”

AP inputs