A group of insiders at San Francisco artificial intelligence company OpenAI is accusing the company of a culture of recklessness and secrecy as it focuses on building the most powerful AI system ever.
The group includes nine current and former OpenAI employees, who have come together in recent days over shared concerns that the company is not doing enough to prevent AI systems from becoming dangerous.
Members say OpenAI, which began as a non-profit research lab and attracted public attention in 2022 with the release of ChatGPT, is prioritizing profits and growth as it strives to build artificial general intelligence (AGI), an industry term for computer programs that can do anything humans can do.
They also claim that OpenAI has used heavy-handed tactics to stop employees from raising concerns about the technology, including restrictive non-disparagement agreements it requires departing employees to sign.
“OpenAI is very ambitious about building AGI and is in a race to be first in the field,” said Daniel Kokotajiro, a former researcher in OpenAI's governance department and one of the group's organizers.
The groups published an open letter on Tuesday calling on major AI companies, including OpenAI, to increase transparency and strengthen whistleblower protections.
Other members include William Sanders, a research engineer who left OpenAI in February, and three former OpenAI employees: Carol Wainwright, Jacob Hilton, and Daniel Ziegler. Several current OpenAI employees signed the letter anonymously for fear of retaliation from the company, Kokotajiro said. One current and one former employee of Google DeepMind, Google's central AI lab, also signed the letter.
OpenAI spokesperson Lindsay Held said in a statement: “We are proud of our track record of delivering the most capable and safe AI systems, and we believe in a science-based approach to addressing risks. Given the importance of this technology, we agree that rigorous discussion is essential, and we will continue to engage with governments, civil society, and other communities around the world.”
A Google spokesman declined to comment.
The campaign comes at a tough time for OpenAI, which is still recovering from an attempted coup last year, when the company's board voted to fire CEO Sam Altman over concerns about his outspokenness. Altman returned a few days later, and the board has been reshuffled with new members.
The company also faces legal battles with content creators who accuse it of stealing copyrighted works to train its models. (The New York Times sued OpenAI and its partner Microsoft for copyright infringement last year.) And its recently unveiled ultra-realistic voice assistant was marred by a public spat with Hollywood actress Scarlett Johansson, who claimed OpenAI imitated her voice without her permission.
But none is more persistent than the accusation that OpenAI is too careless about safety.
Last month, two senior AI researchers, Ilya Sutskever and Jan Reicke, left OpenAI amid skepticism. Dr. Sutskever, who served on the OpenAI board and voted for Altman's firing, had warned about the potential risks of powerful AI systems. Some safety-conscious employees saw his departure as a setback.
So did the departure of Dr. Leike, who, along with Dr. Sutskever, led OpenAI's “superalignment” team, which focuses on managing risk for powerful AI models. In a series of public posts announcing his departure, Dr. Leike said he believes “safety culture and process have taken a backseat to flashy products.”
Neither Dr. Sutskever nor Dr. Leike signed the open letter written by the former employees, but their departures have inspired other former OpenAI employees to speak out.
“When I joined Open AI, I didn't come in with the attitude of, 'Let's put something out there and see what happens, and then we'll fix it,'” Sanders said.
Some of the former employees are associated with Effective Altruism, a utilitarianism-inspired movement that has become interested in preventing existential threats from AI in recent years. Critics accuse the movement of promoting doomsday scenarios about technology, including the idea that out-of-control AI systems could take over and wipe out humanity.
Kokotajiro, 31, joined OpenAI as a governance researcher in 2022 and was asked to predict the progress of AI. He was, to say the least, not optimistic.
In his previous work at an AI safety organization, he predicted that AGI might arrive as soon as 2050. But seeing how quickly AI is advancing, he's shortened that timeline. Now, he believes there's a 50 percent chance that AGI will arrive by 2027, just three years from now.
He also believes that the chances that advanced AI will destroy or cause catastrophic damage to humanity — a dire statistic often abbreviated in AI circles as “p(doom)” — are 70 percent.
At OpenAI, even though the company had safety protocols in place — including a collaboration with Microsoft called a “Deployment Safety Committee” that was supposed to vet new models for any major risks before releasing them to the public — that hardly seemed to slow anything down, Kokotajiro noted.
For example, Microsoft began quietly testing a new version of its Bing search engine in India in 2022, which some OpenAI employees believed included a then-unreleased version of GPT-4, OpenAI's cutting-edge large-scale language model, Kokotajiro said. Microsoft reportedly did not get safety committee approval before testing the new model, and the committee did nothing to stop Microsoft from rolling it out more broadly after it learned of the tests through a series of reports that Bing was behaving oddly with users, Kokotajiro said.
Microsoft spokesman Frank Shaw disputed those claims. He said the tests in India did not use GPT-4 or OpenAI models. He said Microsoft's first release of technology based on GPT-4 was in early 2023 and that it had been reviewed and approved by the safety committee's predecessor.
Kokotajiro said he eventually grew so concerned that he told Altman last year that the company should “play it safe” and spend more time and resources on protecting against AI risks rather than rushing ahead to improve models. Altman claimed to agree with him, but not much changed, Kokotajiro said.
He resigned in April. In an email to his team, he said he was stepping down because he had “lost confidence that OpenAI will act responsibly” as the system approached human-level intelligence.
“The world is not ready, and neither are we,” Kocotaijo wrote, “and I fear we are forging ahead regardless, justifying our actions.”
OpenAI announced last week that it has begun training a new flagship AI model and is forming a new safety and security committee to investigate risks associated with the new model and other future technologies.
In leaving the company, Kokotajiro refused to sign OpenAI's standard documents for departing employees, which contain strict non-disparagement clauses that forbid him from saying anything negative about the company or risk losing his vested interests.
Refusing to sign could leave many employees with millions of dollars to lose. Kokotajiro said he is prepared to give up about $1.7 million in vested stock, a large portion of his net worth, all of which he said he is willing to give up.
(A mini-furor erupted after Vox reported on those contracts last month, in which OpenAI responded by claiming that it had never recovered any vested interests from former employees and had no plans to do so in the future. Altman said he was “deeply embarrassed” that he had not known about the contracts, and that the company would remove the non-disparagement clauses from its standard documents and release former employees from their contracts.)
In an open letter, Kokotajiro and other former OpenAI employees called for an end to the use of non-disparagement and non-disclosure agreements at OpenAI and other AI companies.
“Extensive confidentiality agreements prevent us from raising our concerns with anyone other than the companies, who may not be able to address these issues,” they wrote.
They also call on AI companies to “support a culture of open criticism” and establish a process for employees to anonymously report safety concerns.
The company has hired pro bono legal adviser Lawrence Lessig, a prominent legal scholar and activist who also advised Frances Haugen, a former Facebook employee who turned whistleblower and accused the company of putting profits over safety.
In an interview, Lessig said that while traditional whistleblower protections typically apply to reporting misconduct, given the importance of AI technology, it's important that employees of AI companies be able to freely discuss risks and potential harm.
“Employees are a critical line of defense for safety, and if they cannot speak freely without retaliation, those channels will be closed,” he said.
OpenAI spokesman Held said the company has “resources for employees to voice their concerns,” including an anonymous integrity hotline.
Kokotajiro and his group are skeptical that self-regulation will be enough to prepare for a world with more powerful AI systems, so they're calling on lawmakers to regulate the industry.
“We need some kind of governance structure to manage this process that is democratically accountable and transparent,” Kokotajiro said, “instead of having several different private companies competing with each other and keeping everything secret.”