What Could Stop AI Recursive Self-Improvement by 2036?
AI researchers debate whether generalization limits, objective-specification problems, or bottlenecks could delay recursive self-improvement past 2036.
What is recursive self-improvement, and why does 2036 matter?
Recursive self-improvement (RSI) is the idea that an AI system capable enough to do AI research could improve itself, produce a better version, and repeat that cycle faster than humans can. Once that loop starts turning without human bottlenecks, capabilities could compound quickly rather than progressing at the pace of human-led research cycles. The year 2036 shows up in this debate simply as a stand-in for “over a decade from now,” a long enough horizon that if superintelligence hasn’t arrived by then, something specific must have blocked it. Three researchers, Beren Millidge (CTO of Zyphra), John Schulman (chief scientist at Thinking Machines, formerly of OpenAI), and Charlie O’Neill (head of model training at Baseten), worked through the technical reasons that block, rather than accelerate, that outcome.
TL;DR
- The central risk to fast takeoff isn’t compute or scale, it’s generalization: models could get extremely good at benchmarks and RL environments while failing to transfer that skill to open-ended, real-world research.
- A recurring pattern the researchers describe is a hype-then-plateau cycle: each new model looks like AGI for a few weeks, then its weaknesses in judgment and self-checking become obvious, and progress feels bottlenecked again.
- The core paradigm since 2012, roughly self-attention plus reinforcement learning plus scaling, has needed repeated discontinuities (new scaling laws, RLHF, RL on verifiable environments) to keep progress looking linear, and it’s unclear whether the next discontinuity is discoverable by the current recipe.
- Objective specification is flagged as the deepest problem: AI systems are good at optimizing a clearly defined objective, but generating the right objective for open-ended science is a different and much harder skill.
- A chess-Elo analogy suggests that once AI crosses human-level performance in a domain, gains can look sudden even though the underlying improvement curve was smooth all along, meaning a slow linear trend could still produce a sharp-feeling transition.
- One researcher argues current AI research is nowhere near the ceiling of possible efficiency because so little compute is spent on automated theorizing and experiment design compared to raw experimentation.
- The most likely non-technical blocker raised isn’t a capabilities wall at all, it’s external regulation or a political shock that slows deployment regardless of what the models can do.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
Why hasn’t AI capability translated into explosive economic impact yet?
The researchers point to a specific pattern: AI systems solve tasks that once looked like clear markers of general intelligence (chess, competition math, coding benchmarks) without producing proportional real-world transformation. They compare this to Moravec’s paradox, the classic observation that tasks humans find hard (advanced math, chess) turned out to be easy for computers, while tasks humans find effortless (locomotion, common-sense judgment) turned out to be extremely hard to automate.
The concern is a second-order version of that paradox: AI might become extremely good at anything that can be turned into a benchmark or training environment, while a persistent gap remains between simulated performance and messy, real-world usefulness. Even when a model can write far more code than a person, that hasn’t translated into proportional productivity gains, because humans and organizations get bottlenecked wherever the model’s judgment is weakest, or wherever it can’t reliably check its own work. New models arrive, impress people for a few weeks, and then start “feeling dumb” again once users push past the polished demo cases. The open question is how many more of these cycles happen before, or instead of, an actual takeoff.
What would have to be true for a fast takeoff to actually happen?
The argument for a fast takeoff rests on a scaling logic: if you can build an AI agent that is even marginally better than the best human AI researchers, you can run thousands or millions of copies of it in parallel, and run them faster as hardware improves. That combination, the reasoning goes, eventually overwhelms every other bottleneck in the research process.
The counterargument is about distance from a hypothetical optimum. If there’s a fundamentally better learning architecture waiting to be discovered, the question becomes whether that discontinuity is reachable by scaling the current paradigm (transformers, self-attention, RL on environments) or whether it requires abandoning gradient descent and neural networks in something like their current form. The researchers draw a parallel to Moore’s Law: it looked like a smooth exponential from a distance, but sustaining it required constant, non-obvious engineering discontinuities. Something similar has happened with large language models: pretraining scaling hit diminishing returns, and reinforcement learning stepped in to produce a new curve. Whether the current paradigm can discover its own next discontinuity, the way researchers discovered RLHF and RL-on-verifiable-environments, is treated as an open and unresolved question.
Is generalization the real bottleneck?
Generalization is the throughline of the entire discussion. The field’s biggest wins, from raw next-token prediction learning to produce broad reasoning ability, to skills learned on verifiable tasks transferring to less verifiable, more open-ended ones, are described as forms of generalization nobody had strong a priori reason to expect. It worked, but it wasn’t predictable in advance.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
John Schulman recalls an early intuition from OpenAI’s history: minimizing log loss on next-token prediction seemed like it shouldn’t produce intelligence, because the “important bits” of understanding are a tiny fraction of the total prediction loss, easily swamped by noise. The expectation was that better-crafted objectives would be needed. Instead, the naive objective worked far better than expected. That history cuts both ways: it shows the field has been repeatedly surprised by generalization succeeding beyond expectations, but it also means nobody has a reliable method for predicting when the next generalization jump will or won’t happen.
The sharpest form of this concern is about objective specification. Optimizing a well-specified objective (lower pretraining loss, higher benchmark reward) is something current systems are already good at, and there are concrete examples in the discussion of AI-assisted analysis potentially catching flawed methodology in published scaling-law research years earlier than it was actually caught. But proposing the right objective in the first place, especially for the kind of open-ended science that produces paradigm shifts, is a different capability. A durable self-improvement loop requires an AI to propose an objective, optimize it, evaluate the result, propose the next objective, and keep that chain from derailing over long stretches of time. Whether that specific skill, self-directed objective generation, generalizes as well as narrower skills have is treated as the key unresolved question for any rapid RSI scenario.
Could a slow, linear trend still produce a sudden-feeling takeoff?
One argument suggests yes. Chess engines improved along a fairly steady Elo curve for decades, but the human experience of that improvement was not steady at all: it went from humans always beating machines to humans never beating machines, and that transition can look abrupt when a system crosses from below human-level to above it, even though the underlying trend never changed shape. Applied to AI research capability broadly, this implies capabilities could keep rising in a smooth, unremarkable-looking way and then cross into “better than the best human researchers” territory in a way that feels sudden even without a hidden discontinuity in the mathematics. Under this framing, the researchers argue that a scenario where 2036 looks technologically unremarkable is only plausible if progress asymptotes before crossing that human-level threshold, since current trends already appear close to it.
What role does automated research and theory-building play?
A separate argument for continued fast progress focuses on underused compute: right now, very little compute is spent on deep theoretical analysis before and after each experiment, relative to the compute spent running the experiment itself. The proposal is that an AI system could spend an amount of compute equivalent to a “century” of human theorizing designing a small-scale ablation before committing to an expensive large-scale run, and then spend comparably heavy analysis afterward figuring out what to run next. Concrete historical examples are cited where this kind of scrutiny plausibly would have caught mistakes early, such as flawed handling of intermediate checkpoints in early scaling-law research, or the realization that model width and learning-rate scaling needed different treatment than initially assumed. The claim is that a tenfold speedup on “maximize the objective we already have” seems achievable through better automated theorizing. What this approach doesn’t obviously solve is generalizing to inventing the next objective from scratch, which loops back to the objective-specification problem as the harder ceiling.
Frequently Asked Questions
What does “recursive self-improvement” mean in AI?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
It refers to an AI system reaching a capability level where it can meaningfully improve its own design, training process, or successor models, creating a feedback loop where each improved version accelerates the next round of improvement, without requiring proportional human research effort.
Why do some researchers doubt a fast AI takeoff will happen by 2036?
The main doubts center on generalization limits: models might get excellent at narrow, well-specified benchmarks and RL environments without developing the ability to propose good research objectives themselves, do open-ended science, or transfer skill into messy real-world settings where feedback isn’t clean and verifiable.
What is the “objective specification” problem in AI research?
It’s the difficulty of defining a clear, optimizable goal for open-ended tasks. Current systems are good at optimizing objectives humans already specify clearly (lower loss, higher reward), but generating the right new objective for a genuinely novel research direction is a separate and much less understood capability.
Does the chess Elo analogy mean progress could look sudden even if it’s actually gradual?
Yes, that’s the argument. A steadily improving system can look unremarkable for a long stretch and then appear to cross a threshold abruptly once it passes human-level performance, even though the underlying improvement curve never changed shape.
What’s the most likely non-technical reason superintelligence wouldn’t arrive by 2036?
The researchers flagged dramatic AI regulation, or some other political or exogenous shock, as arguably more likely than a purely technical wall, though the discussion focused mainly on technical bottlenecks like generalization and objective specification.
