Sub-quadratic models vs transformers — efficiency-viable, not a drop-in replacement
Sub-quadratic models vs transformers — efficiency-viable, not a drop-in replacement
Vintage: 2026-08. Abstracts-only
/academic-researchsynthesis generated 2026-08-18 (15 papers; Consensus abstracts, not full PDFs). Architecture claims age; treat numerical speedups as author-reported until PDFs are clipped.
One-line summary: Peer-reviewed / ArXiv abstracts support sub-quadratic sequence models as an efficiency architecture, not (in controlled same-data studies) a drop-in quality replacement for softmax Transformers; the live path is a small attention budget on a linear-time backbone.
The insight
2026-08-18-academic-research-sub-quadratic-model-viability-vs synthesizes three rounds of Consensus abstracts into a narrower viability claim than "SSMs replace Transformers":
- Selective SSMs closed a small-scale quality gap (Mamba / Mamba-2 author-reported).
- At controlled 8B / 3.5T, pure SSMs lag on copying, in-context learning, and long-context reasoning; a hybrid with ~7% attention beat the Transformer on all 12 standard tasks (Waleffe et al. 2024).
- Independently, hybrids (Jamba, Samba, Griffin / RecurrentGemma, Transformer-to-Mamba distillation) are the architecture that claims Transformer-level quality plus sub-quadratic serving.
- Falcon Mamba 7B is the live contradiction — a pure SSM that claims to beat Llama 3.1 8B / Mistral 7B on the Open LLM Leaderboard after 5.8T tokens — but it is not a same-data control.
- Expressivity does not rescue the replacement story: SSMs sit in the same (\mathsf{TC}^0) class as Transformers (Merrill et al. 2024). Recurrent "state" is not extra state-tracking power.
This is the academic-layer companion to the practitioner tension on post-transformer-architectures (Liquid / Kimi K3, July 2026). Different grain: abstracts vs Moonshots guests. Canonical chain: ssm-icl-gap-to-hybrid-attention-budget.
The chain
The 2026-08 academic synthesis frames the practical path as SSMs for sequence modeling plus sparse attention for in-context learning — not a wholesale transformer replacement. That chain is recorded as ssm-icl-gap-to-hybrid-attention-budget.
Canonical: ssm-icl-gap-to-hybrid-attention-budget.
Evidence
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "Sub-quadratic sequence models are viable as efficiency architecture, not yet shown — in controlled same-data studies — to be a drop-in replacement for softmax Transformers on the tasks Transformers are best at."
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "A controlled 8B / 3.5T-token comparison then found that pure SSMs lag on copying, in-context learning, and long-context reasoning, while a hybrid with only ~7% attention beat the Transformer on all 12 standard tasks and was predicted up to 8× faster at decode."
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "Taken together, the abstracts support a narrower viability claim than 'SSMs replace Transformers': a small attention budget plus a linear-time backbone is enough to match or beat a dense Transformer at 8B–14B class, with large decode and long-context memory wins."
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "The live contradiction is Falcon Mamba 7B, a pure SSM that claims to beat Llama 3.1 8B / Mistral 7B on the Open LLM Leaderboard after 5.8T tokens — a result that is not a same-data control and sits against Waleffe’s controlled finding."
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "SSMs themselves sit in the same (\mathsf{TC}^0) expressivity class as Transformers — their recurrent “state” does not buy extra state-tracking power."
Contradictions / tensions
- Pure SSM vs hybrid after controlling data. Waleffe (same-data 8B) vs Falcon Mamba (leaderboard, not same-data). The synthesis keeps both; it does not pick a winner.
- Author “successor” claims vs later consensus. RetNet is introduced as a Transformer successor; later large-scale work converges on hybrids. The successor claim is not independently established at frontier scale in this corpus.
- Abstracts-only limitation. Speedups are author-reported and not commensurate across papers. Full-text tables were not fetched.
Open questions
- how-little-attention-is-enough — 7% (Waleffe), a quarter of attention layers after distillation (Wang), sliding-window / local (Samba, Griffin), interleaved blocks (Jamba). No agreed minimal budget.