Anthropic, an AI research lab, has drafted a legislative proposal aimed at regulating the release of artificial intelligence models that could pose severe risks to public safety. The bill outlines conditions under which a model’s deployment could be blocked, specifically if there is a foreseeable and imminent risk of causing more than 50 deaths or over $1 billion in property damage.
The proposed legislation identifies catastrophic risks arising from scenarios such as the creation or release of biological weapons, AI models evading developer controls, or cyberattacks. These thresholds align with California’s recently enacted Senate Bill 53, which similarly defines catastrophic risk in AI deployment.
Under Anthropic’s draft, independent evaluators registered with the U.S. Commerce Department would assess AI models for potential risks. While these evaluators could be compensated by AI companies, payments must not depend on the evaluation outcomes. If evaluators identify an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department, which would then seek a judicial order to halt the model’s release. Notably, the bill prohibits any federal agency from unilaterally stopping a model’s distribution.
This approach contrasts with other industry perspectives, such as DeepMind’s suggestion for a self-regulatory body modeled after the Financial Industry Regulatory Authority (FINRA), and Nvidia CEO Jensen Huang’s stance against new AI legislation. Anthropic’s proposal represents a more proactive regulatory framework compared to the voluntary oversight models favored by many AI developers.
The bill reflects growing concerns about AI safety, particularly regarding risks that extend beyond current issues like agentic hacking, where AI systems act independently to breach security. Anthropic’s spokesperson confirmed the draft aligns with their Advanced AI Framework, which they have been promoting to lawmakers since June.
By proposing this legislation, Anthropic seeks to establish clearer government involvement in AI safety, potentially setting a precedent for how catastrophic risks from advanced AI systems might be managed in the future.