When Jacob Coxon departed Anthropic in September, his concerns about AI development stirred fears nationwide. The issue came into focus on Thursday, coinciding with the public debut of Anthropic stock, when Coxon urged attention to AI safety.
“It doesn’t matter whether they invest,” Coxon told Business Insider. “We have to stop RSI.” RSI, or recursive self-improvement, involves advanced AI training future AI generations. Coxon joined a “Ban RSI” protest outside Anthropic’s headquarters in San Francisco.
The company’s initial public offering valued it at a remarkable $1.7 trillion. Protesters communicated their message through chants, signs, and speeches from Coxon, ex-OpenAI researcher Daniel Kokotajlo, and YouTuber Rob Miles. Later, a group from Stop AI obstructed the office entrance.
An independent report by the AI safety organization METR noted AI’s role in accelerating Anthropic’s research. Anthropic’s IPO prospectus acknowledged potential risks if AI fully designs future models unsupervised, stating potential policy changes to maintain human control.
Coxon advocated for government lobbying and intercompany coordination to fully ban RSI. “Companies are the problem,” he stated, emphasizing that recursive self-improvement is feasible in very few global locations, including Anthropic’s office.
Prominent AI safety advocates joined the protests. Both Coxon and Kokotajlo shared how AI researchers privately voice concerns about technology risks. These apprehensions have reached Congress, where restrictions on RSI are suggested.
“It’s the sort of thing a comic book supervillain would do,” Kokotajlo commented.
Oliver Habryka of LessWrong expressed relief over the legal requirements for post-IPO information disclosure from Anthropic. Yet he worried about competitive pressure to utilize RSI.
David Krueger from the nonprofit Evitable pointed out the widespread agreement on RSI’s risks, terming it “insane.” Despite appreciating protests, he noted their limited impact inside the company. “Anthropic employees have embraced the risks,” he observed.
Anthropic declined to comment. Stephen Council of Business Insider reported on this unfolding scenario, highlighting industry implications and ongoing discussions around AI safety.
