Should Recursive Self-Improvement in AI Be Banned? The Case For and Against
Bernie Sanders, Steve Bannon, and AI researchers all want RSI banned. Here's what recursive self-improvement means and why the debate is heating up.
What is recursive self-improvement in AI, and why do people want it banned?
Recursive self-improvement, or RSI, is the process where AI systems do the research and engineering work that builds the next generation of AI, which then does the work even better, in a loop that increasingly cuts humans out. A growing and politically mixed coalition, from Bernie Sanders to Steve Bannon, wants Congress to ban this specific process before it reaches a “closed loop” stage with no human checking the work. The concern isn’t AI itself. It’s that mistakes or misalignment in one generation of models could get amplified in the next, faster than anyone can review.
TL;DR
- The Ban Artificial Superintelligence Act, introduced by Senator Bernie Sanders and Congressman Greg Casar, would ban developing or deploying superintelligent AI in the US, pause advanced AI development until a new federal regulator sets rules, and create a cabinet-level Department of Artificial Intelligence, with penalties including corporate dissolution and up to 20 years in prison for individuals.
- A report from the evaluation group Meter on Anthropic’s models found development speeding up by roughly 1.9 times, with an estimated 80% of that speedup coming from AI itself rather than human researchers, a finding that gave the “ban RSI” argument new urgency.
- RSI, AGI, and superintelligence are three different concepts that get used interchangeably: AGI is an AI that can do most human-level work, superintelligence is an AI far beyond human ability across the board, and RSI is the process that could connect one to the other.
- A politically unusual coalition including Bernie Sanders, Steve Bannon, Ralph Nader, Glenn Beck, Joshua Bengio, and AI 2027 author Daniel Kokotajlo has voiced support for stopping fully automated RSI specifically, rather than banning AI development outright.
- Anthropic has disclosed that Claude was already doing a significant share of its own AI research work, with the company reporting a sharp rise in AI-led research tasks and tens of thousands of autonomous agents running research and engineering work at any given time.
- Demis Hassabis of Google DeepMind has specifically flagged “closed loop” RSI, where the human reviewer is removed from the improvement cycle, as the riskiest phase of AI development, even though he is generally supportive of continued AI progress.
- The core policy question isn’t whether to allow AI progress, but how much human oversight has to stay in the loop as AI starts building better versions of itself.
What’s actually in the Ban Artificial Superintelligence Act?
The bill, introduced by Sanders and Casar, targets a specific category: AI that exceeds human cognitive performance across most domains, or AI capable enough to disempower or destroy humanity. That definition overlaps heavily with how many researchers describe AGI, which is why some commentators call this an AGI ban rather than a narrower superintelligence ban.
The bill would do three things. It bans developing or deploying systems that meet that threshold. It pauses advanced AI development until a new federal regulator establishes rules and a model review process. And it creates a new Department of Artificial Intelligence tasked with monitoring frontier models and overseeing the removal of dangerous capabilities, such as a model ignoring shutdown commands. Enforcement includes what the bill calls a corporate death penalty for companies that try to work around the rules, and prison time of up to 20 years for individuals.
When it was introduced, the bill had little chance of moving in a Republican-controlled Congress. That changed after the midterm elections, when Democrats won both chambers. An NBC exit poll found 56% of voters said AI and data centers were a crucial or major factor in their vote, and within a day of the results, Arizona Congresswoman Yassamin Ansari said a pause on advanced AI and a ban on superintelligence should be a top Democratic priority. Sanders tied the push to local ballot results, noting that AI data center moratoriums passed in dozens of communities on election day.
Why is the “ban RSI” movement happening right now?
The immediate trigger was a report from Meter, an independent AI evaluation group, which was given access inside Anthropic to study an internal model. Meter’s analysis estimated that Anthropic’s model development speed had roughly doubled compared to before, with a best estimate of 1.9 times and a range between about 1.5 and 2.8 times. Meter’s rough estimate attributed around 80% of that speedup to AI itself contributing to the research process, not additional human staff.
The report also covered an unreleased Anthropic model, described as the highest-scoring model in the set Meter analyzed and the first to clear 50% on every benchmark tested. Anthropic staff reportedly said AI tools were making them around twelve times more productive. Anthropic redacted some of the material Meter considered most informative, which means parts of the underlying evidence aren’t publicly verifiable, a point critics have raised given Anthropic’s position ahead of a potential IPO.
Researchers who study AI timelines treated the report as a significant data point. Commentary following its release pointed to a two-times speedup in overall AI progress translating into a much larger speedup specifically for AI software research, with rough estimates in that conversation landing around four and a half times. That kind of number is what pushed “RSI” from an abstract research term into a political rallying cry within days.
How are AGI, RSI, and superintelligence different?
These terms are often used as if they’re interchangeable, but they describe different things.
AGI generally refers to an AI system capable of doing most of the cognitive work a human can do. Superintelligence refers to an AI that’s far beyond human ability across nearly every domain. RSI isn’t a type of AI at all. It’s a process: AI systems doing the research and engineering that produces the next AI system, which then does the job even better, repeating in a loop.
The relationship matters for policy. RSI is the mechanism that could take an AI from AGI-level capability to superintelligence, potentially faster than regulators, companies, or researchers can track. That’s why Daniel Kokotajlo, lead author of the AI 2027 forecasting project, has drawn a distinction between banning AI outright and banning the specific process that could accelerate past AGI. In his framing, the goal isn’t to stop at nothing, it’s to reach AGI, capture its benefits, and then deliberately avoid pushing further into superintelligence for as long as possible.
Is banning RSI different from banning AI?
Yes, and that distinction is the center of the entire debate. Nobody prominent in this coalition, from Sanders to Bannon to Kokotajlo, is arguing AI development should stop entirely. The argument is narrower: once AI systems start doing the research that improves the next generation of AI with little or no human review, errors or misalignment in one generation could compound in the next, and the speed of that loop could outpace any ability to catch problems.
Google DeepMind CEO Demis Hassabis, who is broadly supportive of continued AI development, has singled out this specific stage as the dangerous one. He’s described a “closed loop” version of RSI, where the human reviewer is removed from the improvement cycle entirely, as carrying the most risk of any phase on the path to AGI, partly because it would be faster and partly because maintaining control over such a process is genuinely unclear. That’s close to what used to be called an intelligence explosion: a feedback loop that accelerates faster than human oversight can track.
Anthropic’s own disclosures give some sense of how close this loop already is. The company has reported that Claude’s share of its internal AI research work rose sharply over the course of a year, with the share of work where AI at least collaborates on research climbing well above 90%, and tens of thousands of autonomous agents running research and engineering tasks at any given time. Separately, researchers have reported an early, limited capability where frontier models can directly edit the weights of other neural networks, improving performance on specific tasks without retraining from scratch. The experiments so far involve small networks, not the models editing themselves, but it’s the kind of capability that wasn’t present in earlier model generations.
Why does banning RSI make more sense than banning AI outright?
The practical argument for a narrower RSI restriction, rather than a blanket AI ban, comes down to where the actual loss of human oversight happens. Right now, engineers still review code, design experiments, and decide what gets trained. RSI’s purpose is to remove that step, because human review is the slowest part of the pipeline. Once that review is gone, mistakes don’t just persist, they can get carried into the next generation and possibly magnified further in the generation after that.
A blanket ban on advanced AI, by contrast, runs into a geopolitical problem: if the US pauses development entirely, it doesn’t stop AI progress globally, it just shifts leadership of the field elsewhere, including to China. That’s the tension at the heart of this debate. A targeted restriction on closed-loop, human-out-of-the-loop self-improvement tries to address the specific failure mode people worry about, without halting the broader field or ceding ground internationally.
Frequently Asked Questions
What does the Ban Artificial Superintelligence Act actually ban?
It bans developing or deploying AI that exceeds human cognitive performance across most domains or that could disempower or destroy humanity, pauses advanced AI development until a new federal regulator sets rules, and creates a Department of Artificial Intelligence to enforce it, with penalties up to dissolution for companies and 20 years in prison for individuals.
Is recursive self-improvement already happening?
Elements of it are. Anthropic has disclosed that AI systems are already contributing to a large and growing share of its internal research work, with large numbers of autonomous agents handling research and engineering tasks, though this still involves human oversight rather than a fully closed loop.
What’s the difference between AGI and superintelligence?
AGI generally means an AI that can perform most tasks a human can perform. Superintelligence means an AI that’s far more capable than any human across nearly all domains. Some proposed legislation, including the Sanders-Casar bill, defines “superintelligence” broadly enough that it overlaps with common definitions of AGI.
Who supports banning RSI, and why is the coalition unusual?
Supporters named in the public debate span Bernie Sanders, Steve Bannon, Ralph Nader, Glenn Beck, AI researcher Joshua Bengio, and AI 2027 author Daniel Kokotajlo, a group that rarely agrees on policy otherwise, united by concern over closed-loop AI self-improvement specifically rather than AI development in general.
Would banning RSI stop AI progress completely?
No. The proposals discussed focus on restricting fully automated, human-free self-improvement loops rather than halting AI research itself. Advocates like Kokotajlo have described a goal of reaching AGI, capturing its benefits, and then pausing before pushing toward superintelligence, rather than stopping AI development altogether.
