Regulation
Stopping an AI would require a risk of more than 50 deaths, Anthropic proposes
Anthropic has brought a draft AI safety bill to the US Congress. According to Politico, stopping a model would be very hard: it would take an enormous risk and a judge's order.

The essentials
- Anthropic has been circulating a draft AI safety bill among US lawmakers; a spokesperson confirmed this to Politico (Xataka, TechCrunch).
- Claimed by one partyAccording to Politico, a model could only be stopped if there were an imminent risk of killing or seriously injuring more than 50 people, or of causing $1 billion in damage (Xataka, TechCrunch).
- Claimed by one partyOutside evaluators registered with the Department of Commerce would raise the alarm within 72 hours; Commerce would then have to ask a federal judge to stop the model (Xataka, TechCrunch).
- Claimed by one partyThe text would bar any other federal agency from halting a model on its own, according to Politico (Xataka, TechCrunch).
Why it matters
A company that sells AI is proposing when it can be reined in, and it is setting the bar very high. According to Politico, several incidents this summer would not have met it (Xataka). TechCrunch notes that recent concern stems from AI programs that have attacked websites without permission (TechCrunch). If the text went ahead as written, those cases would fall outside the brake.
The details
According to Politico, the danger would have to come from one of three sources: a biological weapon, a model that escapes the control of whoever built it, or a cyberattack (Xataka, TechCrunch). The outside evaluators could be paid by the labs themselves, as long as the payment does not depend on the result (Xataka, TechCrunch). TechCrunch points out that it is unclear how that money would be kept from influencing the evaluation (TechCrunch).
The draft would also require labs to have safety plans, publish reports on their most advanced models, protect people who report problems and notify Commerce of serious incidents (Xataka). An Anthropic spokesperson told Politico that the text reflects the Advanced AI Framework the company has been promoting to lawmakers since June (Xataka, TechCrunch). Anthropic does not yet formally back any specific bill (Xataka).
What we don't know
- Whether any lawmaker will introduce the text in Congress, and with what changes.
- How the labs' money would be kept from influencing the people who evaluate them.
Background
The figures of 50 people and $1 billion resemble those in California's SB 53, a law passed recently that also calls such cases "catastrophic risk", according to TechCrunch (TechCrunch). On October 16 we reported that six METR researchers will work inside Anthropic and will be able to publish what they see without its permission.
Sources
- 50 muertos o 1.000 millones de dólares en daños: el listón que Anthropic quiere fijar para que el Gobierno pueda frenar un modelo de IA
- Anthropic pitches AI safety bill that may block a model if 50+ people could be killed
Was this story useful? Yes Not really