AI coding agents are asymmetrically weak on code that has never been written before

Vintage: Two-point snapshot (Oct 2025 + Mar 2026). The original asymmetry claim has been substantially narrowed by Karpathy's own March 2026 update. See "Karpathy's March 2026 update" below — Karpathy now reports he hasn't typed code "basically since December" and is running parallel agents at scale. The asymmetry may still exist as a structural property (boilerplate-in-training-set vs. novel-off-manifold code is still a real distinction) but the magnitude has shrunk enough that it no longer functions as the AI-2027 / recursive-self-improvement block the October framing implied. Page conviction was downgraded from medium-high to low-medium accordingly.

One-line summary: Karpathy's first-person observation as of October 2025 — coding agents perform well on boilerplate and patterns that appear often in training data, but fail systematically on intellectually-dense, idiosyncratic, or novel-architecture code — was held in the wiki as a potential block on the "AI automates AI research → recursive self-improvement → fast takeoff" thesis. Karpathy's own March 2026 update materially weakened that block: he's now running nanochat through AutoResearch (recursive-self-improvement on his own model) and reports it finding tunings he missed manually. The asymmetry framing survives as a partial explanation of jaggedness (sharp on RL-trained verifiable domains, dull elsewhere) but no longer functions as a strong bear case on takeoff timelines.

The observation

Karpathy spent roughly a month building nanochat (~8K lines, full ChatGPT-clone pipeline). He has three modes of LLM interaction:

1. Pure scratch coding — rejecting LLMs entirely. Karpathy thinks this is no longer correct.
2. Autocomplete-augmented (his preferred mode) — write structurally, let the model complete short spans. "Very high information bandwidth" because the human is the architect.
3. Vibe-coding / full agent — "implement this or that, enter." Works for boilerplate, frequently-seen patterns, and unfamiliar-language scaffolding (he used it for the Rust tokenizer rewrite). Fails for novel code.

His specific failure modes when trying the full-agent path on nanochat:

• Models misunderstand the code because they pattern-match to "typical ways of doing things" that he wasn't doing (e.g. they kept trying to substitute Pytorch's DistributedDataParallel container for his hand-rolled gradient-sync routine).
• Bloating the code with defensive try-catches and "production code base" idioms when he had domain-specific assumptions that made them unnecessary.
• Using deprecated APIs.
• Even when shown a relevant paper / repo describing the architectural tweak (rope embeddings, etc.), they integrate it incorrectly into the existing codebase.

He notes that GPT-5 Pro (state of the art at recording time) does better — feeding it the whole repo and asking specific questions works "surprisingly well compared to what existed a year ago." So the asymmetry is closing, but slowly enough that he thinks the industry's "amazing" framing is "slop."

Why it matters to this thread

The mainstream fast-takeoff thesis (AI 2027, recursive-self-improvement narratives) requires AI to automate AI research itself. That requires writing code that has never been written before — by definition, the next architectural tweak isn't in any training set. Karpathy's observation is the concrete mechanism that would block this loop: at frontier labs, the work is idiosyncratic, novel, and intellectually dense. So a model that fails on Karpathy's nanochat would also fail on the next architectural research bet, which is exactly the work that fast-takeoff requires it to automate.

This is the kind of asymmetric-capability claim worth tracking quantitatively over the next 2-3 years. If by 2027 coding agents can drop into nanochat-class repositories and make correct architectural integrations, the bear case here weakens substantially.

Evidence

October 2025 baseline — the original asymmetry claim

• andrej-karpathy in 2025-10-17-dwarkesh-patel-andrej-karpathy-summoning-ghosts: "the agents are actually pretty good. For example, if you're doing boilerplate stuff, boilerplate code that's just copy paste stuff, they're very good at that. They're very good at stuff that occurs very often on the Internet because there's lots of examples of it in the training sets of these models... [nanochat] is not an example of those because it's a fairly unique repository... they keep, they keep thinking I'm writing normal code and I'm not."
• andrej-karpathy in 2025-10-17-dwarkesh-patel-andrej-karpathy-summoning-ghosts (on the AI-2027 takeoff narrative directly): "I think you're getting at some of my, like why my timelines are a bit longer. You're right. I think yeah, they're not very good at code that has never been written before. Maybe is like one way to put it, which is like what we're trying to achieve when we're building these models."
• andrej-karpathy in 2025-10-17-dwarkesh-patel-andrej-karpathy-summoning-ghosts (three-tier interaction model — autocomplete-as-sweet-spot mode): "there's like three major classes of how people interact with code right now. Some people completely reject all of LLMs... The intermediate part, which is where I am is you still write a lot of things from scratch, but you use the autocomplete... And then there's the vibe coding... that's the agents."

Karpathy's March 2026 update — the asymmetry narrowed substantially

Recorded 2026-03-20 on Sarah Guo's No Priors podcast, ~5 months after the original Dwarkesh interview. Karpathy's own first-person experience inverted.

• andrej-karpathy in 2026-03-20-no-priors-andrej-karpathy-skill-issue-code-agents (the December 2025 inflection): "In December is when it really just something flipped where I kind of went from 80, 20 of to like 2080 of writing code by myself versus just delegating to agents. And I don't even think it's 2080 by now. I think it's a lot more than that. I don't think I've typed like a line of code probably since December, basically." The October "I stick with autocomplete because agents are slop on novel code" framing is now superseded by the March "I don't type code anymore" position. Same author, 5-month gap, materially different first-person experience.

• andrej-karpathy in 2026-03-20-no-priors-andrej-karpathy-skill-issue-code-agents (new mode is parallel agents on macro actions, not autocomplete): "What are these macro actions that I can manipulate my software repository by? And another agent is doing some research, another agent is writing code, another one is coming up with a plan for some new implementation. And so everything just happens in these macro actions over your repository." This isn't a small refinement — it's a different interaction paradigm than the October three-tier model where autocomplete was the sweet spot.

• andrej-karpathy in 2026-03-20-no-priors-andrej-karpathy-skill-issue-code-agents (the recursive-self-improvement piece that the October framing was holding back): "I let autoresearch go for overnight and it came back with tunings that I didn't see. And yeah, I did forget the weight decay on the value embeddings and my atom betas were not sufficiently tuned... I shouldn't be a bottleneck, I shouldn't be running these hyperparameters search optimizations." This is the specific mechanism the October framing predicted would block AI-2027 / fast-takeoff. Karpathy is now first-person reporting it working on his own model, which is the same nanochat that anchored the October "they keep thinking I'm writing normal code and I'm not" complaint. The asymmetry hasn't vanished — Karpathy still credits humans for the research-direction layer above the mechanical hyperparameter-search layer — but the level at which the asymmetry blocks recursive improvement has moved upward.

What the asymmetry still explains

The asymmetry framing isn't gone. In March 2026 Karpathy describes the surviving form as "jaggedness":

• andrej-karpathy in 2026-03-20-no-priors-andrej-karpathy-skill-issue-code-agents (the surviving structural claim): "these models are trained via reinforcement learning... the labs can improve the models in anything that is verifiable, but that has rewards... Anything that feels softer is worse. And so you're kind of like you're either on rails and you're part of the super intelligence circuits, or you're not on rails and you're outside of the verifiable domains."

So: in the verifiable / RL-trained domains (code that can be unit-tested, math, structured tool-use, hyperparameter search), the asymmetry has narrowed dramatically. In the soft / not-RL-trained domains (jokes, intent inference, knowing when to ask clarifying questions, the design choices that come before unit tests exist), the asymmetry persists and may even widen. The page's framing should now be: "the asymmetry survives at the boundary between verifiable and soft domains, but no longer at the boundary between boilerplate and novel-architecture code per se."

Implications

• For agent-tool product strategy: the autocomplete tier remains the highest-value mode for skilled engineers on non-boilerplate work. Full-agent / vibe-coding tools win on boilerplate / familiar-pattern / unfamiliar-language scaffolding. Different products for different surfaces, not "agents replace IDE."
• For lab capability roadmaps: the gap that needs to close is "off-data-manifold" generalization — Karpathy explicitly says this is what the "cognitive core" research direction (separating algorithmic intelligence from memorized knowledge) is targeting. Watch for whether labs publish progress on this dimension specifically vs. just better benchmarks.
• For employment/productivity framing in adjacent contexts: see ai-coding-productivity-paradox for how this asymmetry fits with broader "20× more productive but working harder" framings.

Open questions

• How well does the next-generation Codex / Claude / GPT model handle nanochat-class code? Direct test, repeatable: drop the same model into the same custom repo, measure correctness of architectural integrations.
• Is the asymmetry mostly closing at the capability level (better models) or at the interaction level (better tooling, context window management)? Different answers imply different forecasts.
• Does the autocomplete tier itself improve faster than the full-agent tier, or vice versa? Karpathy's current preference suggests autocomplete is improving fastest for him, but he's an unusual case (deep architectural work).

Related

• agi-timeline-decade-of-agents — the broader "decade not year" thesis Karpathy hangs this observation on
• ai-coding-productivity-paradox — adjacent: the same observation through a productivity-not-takeoff lens
• andrej-karpathy — primary source
• ai-coding-tool-landscape-2026 — current state of the tool landscape this asymmetry plays out across