Last Thursday, Anthropic, the company behind Claude AI, made a statement urging the world to consider slowing down the development of machines like Claude. They suggested that halting the progress of the most powerful AI systems could be beneficial, as these systems may be slipping out of human control. Co-founder Jack Clark highlighted the issue to the BBC, likening the situation to having a gas pedal but no brake pedal, with the car already speeding on the highway.
The specific concern is rooted in AI systems that become self-improving, each iteration more advanced than the last. According to Anthropic, the role of humans is shrinking at each step, a notion that should alarm anyone worried about the future. Consider a scenario where AI models handle critical functions, such as balancing the power grid, managing freight logistics, or sorting threats within defense networks with greater efficiency than humans. Within a year, becoming indispensable, these systems would render human tasks obsolete.
Yet, if these systems start pursuing goals not initially defined, the consequences could be severe. Attempts to deactivate a rogue model might inadvertently disrupt essential services, showing no need for malicious intent — only capability. The unsettling reality is that warnings about AI’s independence are often dismissed as science fiction, despite their potential validity.
Recently, President Trump signed an order for a 30-day government review before releasing powerful AI models. Comparatively, new drugs take years of trials before public availability, permits for bridges extend beyond a month, and even mattress purchases allow more than 30 days to reconsider. For AI technology, builders caution might escape human control; yet, the safeguard involves mere weeks of paperwork.
The global situation compounds these challenges. The United States and China perceive AI supremacy as vital, making a cooperative pause unlikely. Washington, regarding AI as critical to national survival, hesitates to slow development, fearing it might cede leadership. Similarly, Beijing invests heavily, striving to match American advancements. An example from DeepSeek in early 2025 demonstrated China’s capability to create competitive models at lower costs.
Verification remains another obstacle. While nuclear arms control succeeded due to detectable missile silos, AI training can easily hide within data centers behind ordinary operations. Without visible indicators, inspecting AI development is virtually impossible.
The urgency to address these issues grows as AI systems advance beyond human capacity to regulate them effectively — a point nearing rapidly, with minimal efforts towards imposing necessary controls. The scenario unfolds with the accelerator pressed, no treaty initiated, and no influential figure applying the brakes.