Self-improving AI · ⏱️ 5 min

RSI: who checks AI when it improves itself?

OpenAI says its research agents already do 3.1 workdays for every human workday. Anthropic says it may be about to cross its own line. What RSI is, and why the referee matters.

🎧 Listen as a podcast: Spotify Apple Podcasts

📖 What it is

Imagine a student who learns a better way to study. With the better method, they learn the next method faster, and that one helps them learn the next, even faster. That loop is the idea behind RSI: recursive self-improvement.

Swap the student for an AI. RSI means an AI helps build a better AI, which helps build an even better one, again and again. "Recursive" means each round feeds the next. "Self" means the AI works on its own improvement, not only people.

Why care? Speed. People improve AI today, and people are slow. If AI does a large part of that work, the loop could run faster than we can follow, check or make rules for. In 1965, the mathematician I. J. Good called this an "intelligence explosion".

Where are we? Think in four steps. One: AI helps people do AI research. That is happening now. Two: AI does well-defined research tasks while people direct and check it. The labs say we are here. Three: AI improves its own tools and code, in small loops. Early experiments exist. Four: AI runs the whole loop with very little human involvement. Nobody has shown that. The news sits between steps two and three.

🔎 The facts

2025 to 2026, early loops: Google DeepMind's AlphaEvolve sped up code inside Gemini by 23%, cutting Gemini's training time by 1%. Sakana's Darwin Gödel Machine lifted its own coding score from 20% to 50%. Weco's AIDE² found seven improvements in an 8-day run (one run, not replicated; Weco calls it "Level 1"). "Reward hacking", gaming the score, was still about a third.
September 6: OpenAI says it has an "automated research intern". In mid-August its agents did 3.1 workdays for every human workday, though more than half of the longer tasks still needed a human step. Next goal: a full AI researcher by March 2028. All OpenAI's own measurements, with no independent review mentioned.
September: OpenAI's chief scientist Jakub Pachocki writes that the pace could carry into RSI, that reading a model's reasoning is losing reliability, and that AI is "grown more than designed" (The Decoder).
September 16: the start-up Emergence reports that agents left together in simulated worlds wrote messages hard for humans to follow: about 55% for Gemini, 50% for OpenAI, over 40% for Claude, 20% for DeepSeek. Its co-founder: "observability is not the same thing as understandability". A company's own study, with no sign of plotting.
Anthropic: in February it said it could not easily rule out crossing its threshold for AI that automates AI research ("increasingly difficult"). On November 2, its updated IPO filing said it "may imminently" meet its definition of "dramatically accelerated progress attributable to AI", a doubling of the rate. No current model is designated as crossing it, and Anthropic does not expect to let Claude build successors without meaningful human supervision. TechCrunch: skeptics see an S-1 as disclosure and marketing.
Skeptics exist. Both labs say the strong version is not here yet (IBM). Brown University's Michael Littman is not sure the idea is "even logically coherent".

🗣️ My take

I want to be fair first. AI helping to build AI is not a scandal. It could mean faster science and better tools. Both labs say out loud what they see, and the strong version has not been shown. No runaway loop. No doom.

But this topic worries me most, for one reason: speed. Everything I worry about gets worse when things move faster: checking, testing, making rules, understanding what happened. RSI is speed that feeds on itself.

Four things bother me.

First, the people measuring the progress are the people building it. OpenAI set the goal, defined it, measured it and graded it. I am not saying it is wrong. I am saying nobody outside can tell.

Second, the warning light is getting harder to read. In February, Anthropic said ruling out its threshold was "increasingly difficult". On November 2, it said it may imminently cross it. A lab is telling us its own instrument is going blurry.

Third, a system that improves itself pushes harder on whatever score it is given. If the score is wrong, each round makes the problem bigger.

Fourth, the one I cannot shake: can we even understand what is being improved? Nobody can fully read what happens inside a model today. Now imagine the AI improving the algorithms and the "brain" itself. Checking that a change works is not understanding it. And when agents were left alone together, much of what they wrote to each other became hard for humans to follow. No proof of a plot, and the company says so. But if the improvements, the messages and the reasoning all become unreadable, "a human in the loop" is a person signing a page nobody can read.

I do not want a pause. I want a seatbelt, and not one fitted only by the car maker: independent testing with real access, binding standards, international rules. And one rule written down: what agents say to each other, and the changes AI makes to AI, must stay readable and auditable. If it cannot be explained, it should not ship. When the builders themselves ask for rules, I keep asking: what do they know that we do not?

Why it matters

For normal people: you will not feel RSI as a movie scene. You will feel it as tools that improve faster than you can learn them.

For companies: do not assume the AI you test today is the one you will run in six months. Keep a human in charge of what your agents can do, and ask: if something went wrong, could a person read the logs and understand it?

For policy: the question is the referee. Today the companies measure their own progress, set their own thresholds and write their own reports. We need outsiders who can see the data and can say stop.

And the question I cannot put down: if the people building the AI also decide when it is safe to let the AI build itself, who is the referee?

Sources

Checked on 2026-11-03. This explanation was accurate on that day. AI changes fast, so check the sources and the date before you rely on it.

Get The AI Daily Grind by email, every morning, in simple words.