Distillation and iterated amplification (ITAD)
Distillation and iterated amplification (ITAD)
One-line summary: A large teacher model generates traces; a smaller student trains on them; the student is then amplified and distilled again. This loop — not architecture — is now described as one of the innermost loops of model training, and it means frontier capability leaks downstream by construction, which is why "they distilled off us" is an explanation of open-weight parity rather than a defence against it.
What it is
The clearest definition in the vault, from alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why:
"distillation is a process in machine learning whereby a usually larger, more expensive model is used as a teacher to train a usually smaller student model… You take a large model, you have it generate lots of traces, lots of outputs, and then you use those outputs as training data for a smaller model"
And the loop it sits inside — the reason it is structural rather than incidental: "iterated amplification and distillation or itad, is… one of the innermost loops of model training that we now find ourselves in."
Why it matters
It reframes the open-weight parity debate. will-marshall offers distillation as the deflationary explanation for chinese-open-weight-frontier-parity: "They did distillation almost certainly on the best models." — i.e. GLM 5.2's performance is derivative, so the frontier is still the frontier.
alexander-wissner-gross rebuts the asymmetry rather than the mechanism: "this is not just the Chinese who've been distilling off western models. Google DeepMind was this is public information was found to have done this earlier. Grok infamously doing it. Elon admitted it and then also purchased cursor which had been fine tuning off of traces on top of Claude."
If everyone distills from everyone, then distillation is not a Chinese shortcut but the industry's shared training loop — and the implication flips. A frontier lab's lead becomes a time advantage measured in months of trace generation, not a moat in the durable sense.
It bounds the coding-data moat. proprietary-coding-data-as-moat argues that trace data is the scarce input. But traces are exactly what distillation harvests. Cursor's own traces were reportedly built on Claude's outputs before Cursor itself was acquired. The moat and the leak are the same substance.
Evidence
- alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "distillation is a process in machine learning whereby a usually larger, more expensive model is used as a teacher to train a usually smaller student model"
- alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "iterated amplification and distillation or itad"
- alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "this is not just the Chinese who've been distilling off western models."
- will-marshall in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "They did distillation almost certainly on the best models."
- peter-diamandis in 2026-06-29-podcast-moonshots-why-the-us-government-is-blocking-model-releases (June 2026), relaying Anthropic's accusation: "using 28.8 million fraudulent exchanges across 25,000 fake accounts to securely extract and copy Claude's capabilities if the claim holds up, it's the single largest AI model theft accusation ever made."
- alexander-wissner-gross (same source): "the shoe is ironically on the other foot. Anthropic sued multiple times for using improperly copyrighted books and other media for pre training its own models. Now Anthropic is turning around and accusing Alibaba for using Anthropic outputs to post train Alibaba's own models."
- emad-mostaque (same source): API-trace distillation ("how do you answer this really hard question") is "incredibly difficult to lock down"; he treats the Alibaba sock-puppet claim as plausible ("Alibaba probably did have these sock puppets") and reads GLM 5.1→5.2 as the RSI/distillation loop in the open.
Adjacent Anthropic Sep 2026 volume table — not an ITAD rewrite
- From 2026-09-11-anthropic-sep-2026-threat-intelligence-autonomous-cyber (issuer threat report): dated per-lab illicit-distillation assessments (Alibaba >151M May–Jul; Moonshot >23M; DeepSeek >12.1M in 14 July days). Chronological later print than the June Moonshots 28.8M / 25k-account relay above — not a rewrite of the ITAD-as-shared-loop thesis on this page. Named-lab replies not retrieved. Full treatment: anthropic-illicit-distillation-sep-2026.
Contradictions / tensions
- Does distillation cap the student below the teacher? The received view is that a student cannot exceed its teacher on the distilled distribution. GLM 5.2 reportedly matches or exceeds Western models on some benchmarks — which either falsifies the cap, or means GLM 5.2 is not purely distilled, or means the benchmarks are saturated. None of these is settled in-source. This is the sharpest open question here.
- The accusation is made without citation of a specific detection method; "almost certainly" is Marshall's own hedge.
Open questions
- If ITAD is genuinely an inner training loop, is any frontier lead durable beyond one trace-generation cycle?
- Can a student exceed its teacher, and if so by what mechanism (amplification search? verifier signal?) — this is the load-bearing question for chinese-open-weight-frontier-parity.
Sources
- 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why
- 2026-06-29-podcast-moonshots-why-the-us-government-is-blocking-model-releases
- 2026-09-11-anthropic-sep-2026-threat-intelligence-autonomous-cyber — Sep 10 Anthropic per-lab volume table; not an ITAD rewrite