medium convictionactive · updated 2026-09-03T00:00:00.000Z
Pure-SSM ICL/copy gap → small attention budget → hybrid viability (not transformer replacement)
Controlled same-data 8B work says pure SSMs lose on copy/ICL/long-context retrieval; hybrids with a small softmax budget match or beat Transformers on quality while advertising decode/KV-cache wins. Falcon Mamba is a leaderboard counter-claim, not a same-data refutation. Abstracts-only; author-reported speedups.
The chain
1
Softmax attention's O(n²) time/memory is the stated motive for sub-quadratic alternatives; exact softmax is SETH-hard to approximate in sub-quadratic time.
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "Softmax self-attention is O(n²) in sequence length for both time and memory, which is the stated motive for almost every alternative in this literature."
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "approximating softmax-attention output in sub-quadratic time is hard under the Strong Exponential Time Hypothesis"
2
Selective SSMs (Mamba / Mamba-2) closed the historical small-to-medium quality gap in author-reported results, but that is not a same-data replacement proof.
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "Mamba-3B outperforms same-size Transformers and matches Transformers twice its size on pretraining and downstream language modeling; inference is reported as 5× higher throughput."
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "These are author-reported results from the introducing paper, not an independent replication."
3
At controlled 8B / 3.5T tokens, pure SSMs lag on copying, ICL, and long-context reasoning; a ~7% attention hybrid beat the Transformer on all 12 standard tasks.
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "Pure SSMs match or exceed the Transformer on many tasks but lag on strong copying, in-context learning (e.g. 5-shot MMLU, Phonebook), and long-context reasoning."
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "The 8B hybrid exceeded the 8B Transformer on all 12 standard tasks (+2.65 points average) and was predicted up to 8× faster at token generation."
4
The advertised serving win is inference memory / decode (fixed-size state vs growing KV cache), not extra expressivity — SSMs share Transformers' TC⁰ limits.
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "the inference-memory advantage is the most consistently advertised, and it grows with context length."
From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "SSMs, like Transformers, cannot express computation outside TC⁰, so they cannot solve permutation composition"
What would falsify this
- Step 3: A same-data 8B+ study shows a pure SSM matching the Transformer on copy / Phonebook / needle, not just Open LLM Leaderboard.
- Step 4: A result showing SSMs outside TC⁰ / solving permutation composition under the same complexity assumptions Merrill uses.
Contradictions / tensions
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: Falcon Mamba 7B (pure Mamba, 5.8T tokens) claims to beat Llama 3.1 8B / Mistral 7B on the Open LLM Leaderboard — not a same-data control.
- RetNet's introducing paper claims a successor with no performance sacrifice; later work treats linear/recurrent layers as components.
Implications
- Practitioner 'post-transformer' talk (Liquid / linearized attention) should be read against this academic grain: hybrids, not replacement. See post-transformer-architectures.
- Do not treat Falcon Mamba leaderboard claims as a same-data refutation of the copy/ICL gap.
Companies
noneConcepts
Sub-quadratic models vs transformers — efficiency-viable, not a drop-in replacementPost-transformer architectures (and why the transformer keeps winning anyway)Frontier-class intelligence is escaping the datacenter to the edge
Open questions