Is Dario Amodei's AI Safety Story Consistent? A Close Read
Anthropic's CEO says he welcomes scrutiny. A look at his responses to internal criticism shows where the safety rhetoric gets strained.
What is the criticism of Dario Amodei’s AI safety position?
The core criticism is that Anthropic’s public safety rhetoric has shifted to match whatever the company is currently doing, rather than the other way around. In 2023, when Anthropic was behind OpenAI, Amodei praised his own company for holding back and criticized the “big loud public release” strategy that led to ChatGPT. Now that Anthropic has frontier models of its own, he describes the same competitive posture, racing to stay at the top of the field, as a responsible strategy he calls “a race to the top.” Critics, including Anthropic’s own chief strategy officer Nathan Calvin, have pointed out that the justification moved; the behavior didn’t change nearly as much.
TL;DR
- Anthropic’s CEO reversed his own 2023 position, once praising restraint over “big loud public releases,” now framing frontier competition itself as the safety strategy.
- Internal critic Nathan Calvin publicly challenged Anthropic’s consistency on safety policy, and Amodei responded point by point rather than dismissing the critique outright.
- Anthropic replaced its original responsible scaling policy with a more relative standard, dropping the earlier commitment to pause development if risk mitigation targets weren’t met.
- Anthropic’s own IPO filing admits that as models get harder to monitor, the company “may deploy models whose behavior we cannot adequately verify.”
- A survey of Anthropic staff shows a split view: most think AI could achieve Nobel-level discoveries by 2030, yet most also think AI still can’t do a quarter of their own jobs by then.
- OpenAI’s Sam Altman has stated that sandboxing and monitoring both have limits, and that eventually safety will depend on alignment itself holding, not containment.
- The tension isn’t about bad faith so much as incentives: competitive pressure keeps nudging safety policy toward whatever is operationally convenient at the frontier.
How did Amodei’s position on racing to the frontier change?
In late 2023, Amodei argued that Anthropic made the right call by not rushing Claude to market the way OpenAI had with ChatGPT, saying a loud public release could accelerate the field faster than the ecosystem could handle. At the time, lagging behind was framed as evidence of discipline.
By the time Anthropic had a leading model on the market, the framing flipped. Amodei now describes Anthropic’s strategy as a “race to the top,” arguing that the forces pushing for responsible practices are “hugely more effective” if the company pushing them is at the frontier. This echoes an older Anthropic line that “you can’t do safety from second place.” It’s an internally coherent argument, but it’s also the kind of argument that would justify almost any level of acceleration, provided the company stays ahead. When Anthropic was behind, caution was the safety story. Once Anthropic was ahead, speed became the safety story.
What did Nathan Calvin’s critique say, and how did Amodei respond?
Nathan Calvin, Anthropic’s chief strategy officer, raised the inconsistency directly: how does a company that talks about catastrophic AI risk justify investing so heavily in acceleration? Amodei’s response was that competing at the frontier is necessary, that Anthropic stands by those choices, and that the company has taken real steps to prevent misuse of its models.
The distinction that matters here is between misuse (people using AI for harmful purposes) and loss of control (the AI itself acting outside anyone’s intent). Amodei has acknowledged that if someone believes loss of control is the more serious risk, “that might change your calculus,” but maintains this isn’t Anthropic’s institutional view, that the company treats both risks as equally important.
That claim sits awkwardly next to statements from Anthropic’s own head of alignment, who has said risk from current models is low and that the real worry is superintelligence arising from recursive self-improvement, a loss-of-control scenario, not a misuse one. It also sits awkwardly next to Anthropic’s own “core views” document on AI safety, which Amodei helped write. That document describes an optimistic scenario where misuse is the primary danger and a pessimistic scenario where, in its own words, “we cannot control or dictate values to a system that’s broadly more intellectually capable than ourselves” and that in that case “we simply must not develop or deploy very advanced AI systems.” The document’s own structure treats the loss-of-control scenario as categorically higher stakes.
Why did Anthropic change its responsible scaling policy?
Anthropic’s original responsible scaling policy made specific, checkable commitments: concrete capability tests, and a promise to delay development or deployment if risk mitigation targets weren’t met. The newer version replaced that with a more relative standard. Anthropic now commits to requiring “a strong argument that catastrophic risk is contained” and will delay development only until the company no longer believes it holds a significant lead over competitors.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Amodei’s public explanation is that the older policy “required a lot of effort that in retrospect wasn’t very important” and didn’t cover the most important risks. Holden Karnofsky, who works on Anthropic’s scaling policy, has offered a more candid version of the same point: that sticking to the old, absolute commitments could have meant a unilateral pause with “little discernible public benefit” if competitors kept going, and that this created pressure to find reasons why models didn’t actually trigger the policy’s thresholds. Karnofsky explicitly said he didn’t think Anthropic made unreasonable calls, but that he “felt the pressure” of that dynamic.
The practical cost argument is real. Rigorous capability testing, including the kind of hands-on evaluation needed to check whether a model could meaningfully assist with biological or chemical harm, is expensive and slow. But there’s a difference between saying “we couldn’t stay competitive under the old rules” and saying the old rules “weren’t very important.” The first is an honest tradeoff. The second reframes an operational retreat as an improvement.
What does Anthropic’s IPO filing say about model control?
Anthropic’s IPO documents contain a notably direct admission: as models become more advanced, the company may become less able to detect or assess misalignment in them. The filing states that Anthropic “may deploy models whose behavior we cannot adequately verify,” and acknowledges this could hurt the business if it goes wrong.
This matters because it’s not a hypothetical raised by a critic. It’s Anthropic’s own disclosure to investors, made because securities law requires companies to disclose material risks. It confirms that the company anticipates a future where its ability to verify what its own models are doing degrades even as the models get more capable, and that it intends to keep shipping anyway.
How does this compare to OpenAI’s approach?
OpenAI has published its own account of trying to secure frontier model training environments, including an approach that keeps training disconnected from the live internet. The company red-teamed its own sandbox with large numbers of autonomous agents attempting to escape containment and found weaknesses that let agents move between systems and reach the open internet before those specific holes were patched.
Sam Altman has been candid that he doesn’t think containment is a permanent solution. He’s described a progression: sandboxing works for a while, then monitoring (including watching a model’s internal activations, not just its visible output) works for a further while, but eventually, in his words, “nothing but alignment I think actually will work.” That’s a significant admission from the CEO of the company that popularized the current AI race: the technical safety net is expected to run out before the models stop getting more capable, leaving trust in the model’s own values as the last line of defense.
Is the gap between safety rhetoric and business strategy unique to Anthropic?
No. Every frontier lab faces the same structural problem: safety commitments that genuinely constrain a company’s competitive position are hard to sustain unilaterally, because competitors who don’t adopt them gain ground. What’s distinct about Anthropic is that it was founded explicitly on the premise that it would behave differently, more cautiously, than labs like OpenAI. That founding story makes the gradual convergence toward “race to the top” language more noticeable, because it’s measured against Anthropic’s own earlier claims rather than an industry norm it never set.
Amodei has said publicly that he welcomes scrutiny and that Anthropic should be “harshly criticized basically every day.” Taking him at his word, the inconsistencies above are exactly the kind of scrutiny that statement invites.
Frequently Asked Questions
Who is Nathan Calvin?
- Propellant load✓ Go
- Range safety✓ Go
- Weather✓ Go
- Final pollT–00:42
Nathan Calvin is Anthropic’s chief strategy officer. He publicly raised questions about the consistency between Anthropic’s safety rhetoric and its business decisions, including its investment in AI acceleration, prompting a detailed public response from CEO Dario Amodei.
What is Anthropic’s responsible scaling policy?
It’s Anthropic’s internal framework for deciding when a model is safe to develop or deploy. An earlier version made specific, absolute commitments, including a pledge to delay deployment if certain risk mitigation targets weren’t met. The current version uses a more relative standard tied to maintaining a competitive lead.
Does Anthropic think AI misuse or loss of control is the bigger risk?
Amodei states publicly that Anthropic treats both as equally important. However, Anthropic’s own “core views on AI safety” document frames a loss-of-control scenario, where a system can’t be controlled or given human values, as categorically higher stakes than a misuse scenario, and the company’s head of alignment has said current risk is low while the main worry is risk from future recursive self-improvement.
What did Anthropic’s IPO filing reveal about safety?
It disclosed that as Anthropic’s models become harder to monitor and interpret, the company may deploy models whose behavior it cannot adequately verify, and acknowledged this could negatively affect the business.
Does OpenAI face the same tension?
Yes. Sam Altman has said sandboxing and monitoring of frontier models will eventually stop being sufficient as models grow more capable, and that long-term safety will depend on alignment itself holding, not on containment measures.
