Confronting the Limits of Recursive Self-Improvement
Eryk Salvaggio / Nov 17, 2026
The logo of Anthropic is visible on a laptop screen against a Nasdaq-100 candlestick chart. The company is scheduled to begin trading on Thursday, Nov. 19, 2026. (Photo by Silas Stein/picture-alliance/dpa/AP Images)
In 1965, Irving John Good, a mathematician who worked alongside Alan Turing, proposed the concept of the “ultraintelligent” machine. “Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man, however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind,” he wrote.
Today, the prospect of an “intelligence explosion” is attached to another speculative concept in the field of artificial intelligence, recursive self-improvement (RSI). This term is tied to the increasing capacity for AI systems to produce code without human supervision.
In May, Anthropic claimed that its large language model, Claude, had written 80% of the company’s merged code. METR, an independent organization with close ties to the industry it monitors, recently published a report commissioned by Anthropic. The report estimated that Anthropic’s benchmark scores improved last year at a rate that would have taken two years in the years before, and claimed (“with substantial uncertainty,” they note) that 80% of the acceleration is due to AI assistance in code development. METR reports that the most useful evidence it observed to arrive at this estimate was redacted by Anthropic, since supposedly it might help other companies develop similar techniques. Anthropic is slated to launch its initial public offering, aiming for a valuation of $1.4 trillion, on Nov. 18.
The “intelligence explosion” is one of the promises of the AI boom, and “boom” is the key word here: Eliezer Yudkowsky, who popularized the term RSI, suggests that a “seed AI” system would serve as a catalyst to an explosive powder keg of exponential capabilities. As the phrase implies, once the system can improve, it can beget an improved version of itself, and so on. This belief in recursion, paired with METR’s report, has led influential AI observers to expect accelerated timelines for RSI. For example, Dwarkesh Patel predicted last week that the world is “a couple of months away ... from AIs that are as good as the best humans at AI research,” and that this, in turn, will lead to AI that is “at least as good the best human experts in every field” by 2030, or earlier.
Verifiable benchmarks
The reality is likely more complicated. Many of the measures in the METR report were based on specific benchmarks, which can limit the kind of problems that can be assessed to the kinds of problems that can be included on a test. Additionally, METR does not report on the kinds of tests that were evaluated. But a hallmark of these assessments is that the answers are automatically verifiable: they can be checked as true or false, right or wrong, following an automatable logic.
The increase in such capacities is specifically trained into the models through Reinforcement Learning with Verifiable Rewards (RLVR), a sleeper innovation that accounts for a number of capacity improvements in the past year, particularly in coding and math. RLVR has accounted for what many users experienced as a shift in capability in early 2025 as it was integrated well across a number of systems. RLVR is a form of post-training that optimizes models to do precisely the kind of writing that results in verifiable answers.
RLVR takes a base model and measures the text it produces to fetch a verifiable answer, like those on the tests. When the model succeeds, the text it produces is understood as a desirable pattern, and steps are taken to ensure more of that kind of text is produced in the next stage of training. This produces longer text that expands the likelihood of correct answers, but also emphasizes particular tokens and language that can cause the model to rewrite text from different frames.
RLVR explains these capabilities rather than diminishing them, and code — the topic at the heart of the RSI push — is a verifiable task. But if the models are showing stronger capacities for these kinds of tasks, it does not follow that the models are advancing in other ways needed to facilitate their own exponential improvement.
Recursion, or noise?
A fully enclosed, self-perpetuating recursive system, without a capacity for redirection, becomes a feedback loop akin to the sound leaving a speaker and entering the same system through a microphone, creating echoes that break apart into a wall of horrid sound. That sound you’re hearing may not be the sonic boom of AI growth hitting the speed barrier, but a breakdown in how the system selects tasks: they converge.
AI agents currently operate as multiple instances running on the same training. For recursive amplification to work, you need diversity in the input. It’s not an impossible task, but it’s a strikingly different setup from setting a stochastic flock (or “agentic swarm,”) loose on a series of tasks that can be automatically graded.
An important question to ask, then, is what, precisely, is the “self” in recursive self-improvement? A recent technical paper suggests that “every improvement loop is a claim that some signal can substitute for human judgment.” Self-improvement is tied to the quality of a specific automated replacement installed for a specific automated task. How we draw the boundaries around a system can help us understand what it is able to do. Automate variety well, and you can create better systems. But something has to generate a signal, and something has to change that signal. Right now, a model cannot generate signals of the same quality across different layers of its own architecture.
A research paper from this summer articulates the consequences of these limits well. The question of the paper was whether AI agents could conduct AI research. The authors gave models research questions from papers written by human researchers that had been submitted to NeurIPS 2026 but not yet published. They asked the models to tackle the same research question, without access to those papers, and then had the human researchers grade the agents on the research. The models were left to scaffold an experimental design that could answer the question, execute it and evaluate the results.
Beyond engineering the scaffolding, the models failed spectacularly. They did not take feedback and did not recalibrate when paths became obviously fruitless, supporting the hypothesis that “[a]gents can’t hill-climb their way into choosing a set of candidate hypotheses, deciding what evidence would settle a research question, or incorporating feedback effectively and recognizing that an approach has failed and the right move is to start over,” as the authors wrote. Left unattended, the machines kept repeating themselves into a microphone, rather than trying to find a new source of information or shift to a new set of ideas.
It could be tempting for advocates who want to believe that RSI is around the corner to dismiss this as a failure of existing models when nobody has suggested RSI is in these models. I take the experiment as a measure of the gaps between signals of strength and weakness. Machines can become better at narrow tasks that benchmarks excel at capturing, but the same models consistently fall apart in an open world of vast possibilities.
‘I’m afraid I can’t do that’
Prediction is a fool’s game, and even as a critical observer of large language models and image models I am continually surprised by them. But I am unconvinced when I hear predictions of future capacities based on following trendlines unrelated to the capacities being promised. Benchmarks are a particularly challenging trajectory to rely upon, as they are easily gamified, and often test for precise forms of problems that human minds do not intuitively isolate.
As a result, it can be easy to conflate a model’s capacity to write code or tackle math problems with a model’s capacity to plan a research paper. It is more likely to me that attempts to automate human decisions reveals a more nuanced interplay between ways of thinking, rather than rendering it obsolete.
Famously, John Good was a consultant advising Stanley Kubrick on how to depict the supercomputer HAL 9000 in the 1968 film 2001: A Space Odyssey. It’s telling that we describe that machine as super intelligent because it spoke and listened, despite that the major plot point of the film was that it broke. HAL 9000 excelled at running a space ship, and yet, failed spectacularly at opening pod bay doors.
Authors

