A group of OpenAI insiders is calling out a culture of recklessness and secrecy at the San Francisco artificial intelligence company as it races to build the most powerful AI system ever created.
The group, made up of nine current and former OpenAI employees, recently came together over shared concerns that the company wasn't doing enough to prevent its AI systems from becoming dangerous.
OpenAI, which started as a non-profit research lab and opened to the public with the launch of ChatGPT in 2022, is prioritizing profit and growth as it attempts to build artificial general intelligence (AGI, a computer industry term), members say. A program that can do anything a human can do.
They also claim that OpenAI used hard-line tactics to prevent employees from voicing concerns about the technology, including restrictive non-disparagement agreements it asked departing employees to sign.
“OpenAI is very interested in building AGI and is racing recklessly to be the first,” said Daniel Kokotajlo, a former researcher in OpenAI’s governance department and one of the group’s organizers.
The group published an open letter on Tuesday calling on major AI companies, including OpenAI, to increase transparency and protections for whistleblowers.
Other members include William Saunders, a research engineer who left OpenAI in February, and three former OpenAI employees: Carroll Wainwright, Jacob Hilton, and Daniel Ziegler. Several current OpenAI employees supported the letter anonymously for fear of retaliation from the company, Kokotajlo said. One current and former employee of Google DeepMind, Google's central AI laboratory, also signed.
“We are proud of our track record of delivering the most capable and safe AI systems, and we believe in our scientific approach to addressing risk,” OpenAI spokesperson Lindsey Held said in a statement. “We agree that rigorous debate is important given the importance of this technology, and we will continue to engage with governments, civil society and other communities around the world.”
A Google spokesperson declined to comment.
This campaign comes at a difficult moment for OpenAI. It's still recovering from an attempted coup last year, when members of the company's board decided to fire CEO Sam Altman over concerns about his candor. Mr. Altman returned a few days later and the board was reconstituted with new members.
The company is also facing a legal battle with content creators who have accused it of stealing copyrighted work to train models. (The New York Times sued OpenAI and its partner Microsoft last year for copyright infringement.) And the recent unveiling of a hyper-realistic voice assistant was marred by a public spat with Hollywood actress Scarlett Johansson, whom OpenAI claims to have copied. . Her voice without permission.
But nothing has stuck quite like accusations that OpenAI has been too cavalier about safety.
Last month, two senior AI researchers, Ilya Sutskever and Jan Leike, left OpenAI for the cloud. Dr. Sutskever, who sits on the OpenAI board and voted to fire Mr. Altman, warned of the potential risks of powerful AI systems. His departure was seen as a setback by some safety-minded employees.
So did Dr. Leike, who together with Dr. Sutskever led OpenAI's “Superalignment” team, which focused on risk management for powerful AI models. In a series of public posts announcing his departure, Dr. Leike said he believed “safety culture and processes have taken a back seat to shiny products.”
Neither Dr. Sutskever nor Dr. Leike signed the open letter written by their former employees. But their departure has allowed other former OpenAI employees to speak out.
“When I signed up for OpenAI, I didn’t sign up for the attitude of ‘let’s put things out there, see what happens and fix it later,’” Mr. Saunders said.
Some of the former employees are associated with effective altruism, a utilitarianism-inspired movement that has become concerned in recent years with preventing the existential threat of AI. Critics have accused the movement of promoting apocalyptic scenarios about technology. A controllable AI system could take over and annihilate humanity.
Cocotylo (31 years old) joined OpenAI as a governance researcher in 2022 and was asked to predict AI developments. He was not an optimistic person, to put it mildly.
Previously working at the AI Safety Institute, he predicted that AGI would arrive by 2050. But after seeing how quickly AI was improving, he shortened his schedule. Now he believes there is a 50% chance that AGI will arrive by 2027 in just three years.
He also believes there is a 70% chance that advanced AI will destroy or catastrophically harm humanity (a grim statistic often shortened to “p(doom)” in the AI industry).
At OpenAI, Mr. Kokotajlo rarely slowed down despite the company having safety protocols in place, including a joint effort with Microsoft known as the “Deployment Safety Committee,” which is supposed to review new models for key risks before they are released to the public. .
For example, he said that in 2022, Microsoft quietly began testing a new version of its Bing search engine in India, which some OpenAI employees believed included a then-unreleased version of GPT-4, OpenAI's cutting-edge large-scale language model. . Mr. Kokotajlo said he was told that Microsoft did not get approval from the safety board before testing the new model, and that it failed to stop Microsoft even after the board learned of the testing following a series of reports that Bing was behaving strangely with users. Please don't roll it out more broadly.
Microsoft spokesman Frank Shaw disputed this claim. He said the Indian tests did not use GPT-4 or OpenAI models. He said Microsoft first released GPT-4-based technology in early 2023 and that it had been reviewed and approved by his predecessor in the Safety Committee.
Ultimately, Mr. Kokotajlo was so worried that he told Mr. Altman last year that the company should invest more time and resources into preventing AI risks rather than charging up front to improve its models “around safety.” He said Mr. Altman insisted he agreed with him, but that little had changed.
In April he quit. In an email to his team, he said he was leaving because he had “lost confidence that OpenAI will act responsibly” as its systems approach human-level intelligence.
“The world is not ready yet, and we are not ready yet,” Mr. Kokotajlo wrote. “And I worry that we are rushing forward, rationalizing our actions.”
OpenAI said last week it began training a new flagship AI model and is forming a new safety and security committee to explore risks associated with the new model and other future technologies.
On his way out, Mr. Kokotajlo refused to sign OpenAI's standard paperwork for departing employees. It contains a strict anti-disparagement clause that prohibits employees from saying anything negative about the company. Otherwise, you risk losing your vested rights.
If many employees refuse to sign, you could lose millions of dollars. Mr. Kokotajlo's vested interests amount to roughly $1.7 million, the majority of his net worth, and he is prepared to confiscate everything, he said.
(A minor controversy erupted last month after Vox reported the news of these contracts. In response, OpenAI claimed it had never recovered vested rights from former employees and would not do so in the future. Mr. Altman said that failure to do so would be “unreasonable”. “I am truly embarrassed,” he said. The company said it would remove the disparagement clause from its standard documents and release the former employee from his contract.)
In an open letter, Mr. Kokotajlo and other former OpenAI employees called for an end to the use of disparagement and non-disclosure agreements at OpenAI and other AI companies.
“Extensive confidentiality agreements block us from voicing our concerns except to companies that fail to address these issues,” they wrote.
They also called on AI companies to “support a culture of public criticism” and establish reporting processes where employees can raise safety concerns anonymously.
They hired pro bono lawyer Lawrence Lessig, a prominent legal scholar and activist. Mr. Lessig also advised Frances Haugen, a former Facebook employee who turned whistleblower and accused the company of putting profits over safety.
In an interview, Mr. Lessig said that while traditional whistleblower protections typically apply to reporting illegal activity, it is important that AI company employees can freely discuss risks and potential harms, given the importance of the technology.
“Employees are an important line of safety defense, and if they cannot speak freely without retaliation, their channels will be shut down,” he said.
Mr. Held, the OpenAI spokesman, said the company has “ways for employees to express concerns,” including an anonymous integrity hotline.
Mr. Kokotajlo and his group are skeptical that self-regulation alone will be enough to prepare for a world with more powerful AI systems. So they're calling on lawmakers to regulate the industry, too.
“There needs to be some kind of democratically accountable and transparent governance structure in charge of this process, rather than just two different private companies competing against each other and keeping everything secret,” Kokotajlo said.