The AI coding paradox
Developers feel faster with AI. Controlled trials often measure the opposite. Then December 2025 arrived and four independent practitioners named the same inflection — from Karpathy to Benioff.
The headline empirical finding is a gap, not a triumph. METR’s July 2025 randomized trial put 16 experienced open-source developers on 246 real issues in repos they had maintained for years. Issues were randomly assigned to allow or block AI. Developers forecast a 24 percent speedup. The measured outcome was a 19 percent slowdown. After the study they still believed they were 20 percent faster. Context switching, prompt iteration, integration friction, and QA overhead drove the drag. METR explicitly did not claim the result generalizes beyond that setting.
February 2026 brought a walk-back, not a victory lap. Severe selection bias had broken the replication — developers refused to participate without AI, picked issues where AI would not help, and compensation fell from $150 an hour to $50. Preliminary 2026 numbers were weak and wide: returning developers −18 percent speedup (confidence interval −38 to +9); new recruits −4 percent (−15 to +9). METR’s current line: developers are “more sped up” in early 2026 than early 2025, but the evidence is thin. In late August Anthropic said METR is estimating real-world productivity gains from coding agents, and that both would share more soon. Ongoing. No numbers. It does not resolve the RCT or the walk-back.
Asymmetric on novel code
Karpathy’s October 2025 reconciliation still holds in structure if not in magnitude. Agents excel at boilerplate and at code with many public training examples. They struggle on unique repositories — his nanochat project is the example he gives. Uvik’s field report sees engineers stacking inline and agentic tools beat single-tool engineers on time-to-first-merged-PR by two to three times. Different population, different task mix, different tools. Bug rate is a first-class cost Uvik flags at 1.5–2× on legacy work when only time-to-PR is watched. A first-party August session on the wiki stacks the other way: Cursor plus Paper for overlay copy, a Paper-blind coder for physics. That is surface-stacking on one project, not Uvik’s two-to-three-times survey.
December 2025 is where the story forks. Karpathy, in March 2026, said he flipped from roughly 80/20 self-coding to 20/80 delegation — “I don't think I've typed like a line of code probably since December.” Parallel agents: one researching, one planning, one implementing. Marc Benioff named the same inflection from the enterprise side: “when anthropic 46 hit, boom, everyone could code.” Nick Turley at OpenAI said engineers “don't open their IDE like ever.” Four vantages, one date window. It does not settle METR’s mean; it narrows where slowdown still applies.
Some projects you get a 20x improvement … on other things, it just falls flat.
James Liu, Mediaocean, MIT Technology Review, December 2025
Plan-then-execute coding, parallel Claude workflows, agent output verification, and tool stacking are the operational responses the wiki tracks — steer the plan, run several sessions, give the agent tests and a browser, stack inline completion with agent passes. Autoresearch-style recursive self-improvement is the extreme case: the tool improving its own research loop. Satya Nadella’s claim that 25 percent of Microsoft’s code is AI-generated is an input metric, not shipped output. The empirical record disagrees on what that implies.
September 5 and 6 produced lab language that sounds like that extreme case and is not a demonstration. Meta’s AIRA₃ thread closed on “a system that compounds its own knowledge” aimed at unlocking recursive self-improvement — framing on a swarm that still runs human-named graded tasks. Jakub Pachocki’s essay An Alien Mind carried a “strong expectation” that progress could sustain into RSI, plus “extreme caution” and “no one is prepared.” OpenAI’s September 6 research-acceleration post claimed an automated research intern for well-defined tasks under human direction, 3.1 agent-workdays per human workday as of mid-August, and a March 2028 automated-researcher target. The same post: “We do not yet know how to safely get all the way to aligned, full RSI.” Karpathy’s March nanochat result remains the wiki’s most concrete personal-scale demonstration. Do not collapse intern, 2028 researcher, and aligned full RSI into “RSI achieved.”
September 8 attached process color to that intern snapshot and did not replace it. OpenAI’s forced Navier–Stokes issuer page named about 10,000 concurrent agents, about 88 hours, and about 2.7 million messages on a Millennium-adjacent eval — a dated lab-scale multi-agent run, not a rewrite of 3.1 agent-workdays per human workday. Sam Altman, on the same afternoon, called the magnitude sooner than expected and “the strongest evidence yet” of urgency to pace for safety. Chronological color on Pachocki’s caution. Not a rewrite of the essay. Not RSI achieved.