Distillation and iterated amplification (ITAD)
Distillation and iterated amplification (ITAD)
One-line summary: A large teacher model generates traces; a smaller student trains on them; the student is then amplified and distilled again. This loop — not architecture — is now described as one of the innermost loops of model training, and it means frontier capability leaks downstream by construction, which is why "they distilled off us" is an explanation of open-weight parity rather than a defence against it.
What it is
The clearest definition in the vault, from alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why:
"distillation is a process in machine learning whereby a usually larger, more expensive model is used as a teacher to train a usually smaller student model… You take a large model, you have it generate lots of traces, lots of outputs, and then you use those outputs as training data for a smaller model"
And the loop it sits inside — the reason it is structural rather than incidental: "iterated amplification and distillation or itad, is… one of the innermost loops of model training that we now find ourselves in."
Why it matters
It reframes the open-weight parity debate. will-marshall offers distillation as the deflationary explanation for chinese-open-weight-frontier-parity: "They did distillation almost certainly on the best models." — i.e. GLM 5.2's performance is derivative, so the frontier is still the frontier.
alexander-wissner-gross rebuts the asymmetry rather than the mechanism: "this is not just the Chinese who've been distilling off western models. Google DeepMind was this is public information was found to have done this earlier. Grok infamously doing it. Elon admitted it and then also purchased cursor which had been fine tuning off of traces on top of Claude."
If everyone distills from everyone, then distillation is not a Chinese shortcut but the industry's shared training loop — and the implication flips. A frontier lab's lead becomes a time advantage measured in months of trace generation, not a moat in the durable sense.
It bounds the coding-data moat. proprietary-coding-data-as-moat argues that trace data is the scarce input. But traces are exactly what distillation harvests. Cursor's own traces were reportedly built on Claude's outputs before Cursor itself was acquired. The moat and the leak are the same substance.
Evidence
- alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "distillation is a process in machine learning whereby a usually larger, more expensive model is used as a teacher to train a usually smaller student model"
- alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "iterated amplification and distillation or itad"
- alexander-wissner-gross in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "this is not just the Chinese who've been distilling off western models."
- will-marshall in 2026-06-26-podcast-moonshots-the-10b-satellite-empire-putting-ai-in-orbit-why: "They did distillation almost certainly on the best models."
Contradictions / tensions
- Does distillation cap the student below the teacher? The received view is that a student cannot exceed its teacher on the distilled distribution. GLM 5.2 reportedly matches or exceeds Western models on some benchmarks — which either falsifies the cap, or means GLM 5.2 is not purely distilled, or means the benchmarks are saturated. None of these is settled in-source. This is the sharpest open question here.
- The accusation is made without citation of a specific detection method; "almost certainly" is Marshall's own hedge.
Open questions
- If ITAD is genuinely an inner training loop, is any frontier lead durable beyond one trace-generation cycle?
- Can a student exceed its teacher, and if so by what mechanism (amplification search? verifier signal?) — this is the load-bearing question for chinese-open-weight-frontier-parity.