Anthropic Proposes AI Safety Bill That Could Block Models if 50 People or a Billion Dollars

Anthropic has released a draft AI‑safety bill that would require a federal judge to halt a model’s release if independent evaluators flag a foreseeable risk to more than 50 people or a billion dollars in property damage.

Key points

• Draft bill proposes 72‑hour notification from evaluators to Commerce Department.
• Commerce must request a DC judge to stop distribution; agencies cannot act alone.
• Evaluators must be registered with Commerce and cannot be paid contingent on results.
• Thresholds mirror California’s Senate Bill 53, defining catastrophic risk.
• Anthropic cites its Advanced AI Framework and has been lobbying lawmakers since June.

What happened

Politico reported that Anthropic is shopping the draft bill to Congress. The legislation centers on scenarios where a model’s release could create a “foreseeable, imminent risk” involving either more than 50 potential deaths or a billion dollars in damage. The risk must stem from a biological weapon, the model evading its developer’s controls, or a cyberattack.

In the draft, independent evaluators—registered with the U.S. Commerce Department—would assess a model’s potential for disaster. If they determine an imminent catastrophic risk, they have 72 hours to notify Commerce. Commerce then must seek a judge’s order to halt the model’s release; no federal agency may exercise unilateral discretion to stop distribution.

Anthropic’s spokesperson confirmed the draft to Politico, noting, "This bill represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June."

The backstory

The proposal follows a year of heightened scrutiny over AI safety and the industry’s push for voluntary oversight. Companies like Anthropic have long advocated for embedded evaluators—neutral third‑party observers within AI firms—to monitor risk. In contrast, industry leaders such as Nvidia’s Jensen Huang have dismissed the need for new laws. The draft bill attempts to formalize the evaluator concept while adding judicial oversight, stepping beyond the voluntary framework previously promoted.

California’s Senate Bill 53, recently enacted, set similar thresholds for catastrophic risk, using 50 deaths or a billion dollars in damage as benchmarks. Anthropic’s bill mirrors those metrics, suggesting a national standard could follow.

Why the law matters for AI developers

The bill forces developers to submit models to external assessment before release, potentially delaying innovation cycles. It also places a legal burden on firms to document risk assessments and maintain independent evaluator relationships. The requirement that evaluators be paid only for their work—without contingent compensation—introduces a new compliance layer that could be costly to implement.

The proposed judicial gatekeeping could create a chilling effect on rapid deployment of large language models. Developers may opt to postpone releases or invest in more robust internal controls to avoid triggering the 50‑person or billion‑dollar thresholds.

What builders and businesses should do

• Review your model‑release pipeline to identify potential risks that could trigger the thresholds.
• Consider setting up independent evaluator relationships early to meet the bill’s registration requirements.
• Document all risk‑assessment processes to satisfy potential court scrutiny.
• Monitor legislative developments in both federal and state arenas for updates or amendments.
• Engage with industry groups to shape the final wording of the bill and advocate for balanced oversight.

The bottom line

Anthropic’s draft bill could become a cornerstone of federal AI regulation if adopted. Developers should prepare for increased scrutiny of model safety and the possibility of judicial review before public release. The bill’s fate will be decided in Congress, with implications for the entire AI ecosystem within the next 12 to 18 months.