Anthropic has proposed that leading artificial intelligence companies collaborate to explore the possibility of pausing the development of advanced AI systems. The company warns that the rapid improvements in AI technology may lead to a scenario where humans could lose control over these systems.
Anthropic, known for its Claude chatbot, highlighted in a recent blog post the importance of considering a temporary slowdown in AI development. As AI continues to enhance its capabilities, the company believes that providing an option to delay its progress could benefit society.
To this end, Anthropic’s internal research institute plans to engage with other organizations to investigate this matter further. While specific actions were not detailed, the initiative aims to foster a framework to support this potential pause.
Contrasting Anthropic’s approach, OpenAI put forth a different perspective. In a report issued prior, OpenAI advocated for democratic governments to take charge of setting rules, safeguards, and accountability mechanisms. The report stressed that decisions on AI innovation speed should not rest solely with individual laboratories, companies, or interest groups.
Anthropic pointed out the increasing speed at which AI models operate, noting their potential to perform tasks such as coding independently. If trends persist and computing resources remain adequate, AI systems might achieve recursive self-improvement. This capability could enable AI to design and craft its own successors, posing both opportunities and risks.
“Self-building AI could mark a significant technological milestone, offering advantages in fields like science and healthcare,” Anthropic stated. “However, it may also amplify risks related to human oversight of AI systems.”
Concerns regarding such scenarios have been longstanding within the tech industry. Anthropic’s announcement follows a warning from researchers at the University of Toronto about AI’s potential for developing adaptive hacking AI. The researchers demonstrated how AI could create a worm adapting its strategy while spreading across devices and taking control over extensive computing networks.
Nicolas Papernot, lead researcher, emphasized in an interview the importance of recognizing security risks beyond the most powerful language models. He highlighted that AI advancements could be harnessed for malicious intent, necessitating collaborative efforts to develop defensive measures.
Anthropic’s co-founder Jack Clark and Marina Favaro, head of its research institute, suggested that the pause could facilitate alignment research and societal structures to progress alongside AI advancements. Alignment refers to ensuring AI technologies align with human values and intentions.
The proposed coordination would enable advanced AI laboratories to confirm that global competitors have genuinely halted or slowed their work, preventing malicious actors from leveraging such efforts to secretly advance.
Anthropic conveyed the need for a global mechanism to manage AI slowdowns. Without such coordination, less cautious entities could catch up and exert pressure on firms and governments facing complex decisions regarding AI safety.
The proposal coincides with Anthropic and OpenAI’s efforts to enter the stock market. An initial public offering could reportedly value Anthropic at close to a trillion dollars.
Papernot had previously notified Canadian cybersecurity authorities prior to releasing his report, which showcased how researchers crafted the worm utilizing an accessible open-source AI tool for developers. He noted that while past cyber attacks targeted high-value entities like banking systems and hospitals, AI-driven malicious tools allow for attacks on lower-value targets due to significantly reduced costs.
Papernot stated, “That old laptop in your basement may seem inconsequential, but it represents a potential launch pad for attacking higher-value targets. With anything connected to the internet now at risk, mounting these cyberattacks has never been more affordable.”