Academic research: sub-quadratic model viability vs traditional transformers

Generated by /academic-research on 2026-08-18. Synthesized across 3 rounds from 15 peer-reviewed / ArXiv papers (see Provenance). Treat as raw material — review before promoting into a project or thread. Context: none Note: this literature lives primarily on ArXiv (CS/ML). Abstracts often omit exact effect sizes, data mixes, and failure cases. A complementary /autoresearch pass would capture 2025–2026 practitioner evidence (Kimi linearized attention, Liquid AI, production serving) that this academic pass does not cover. Abstracts-only limitation (v1): claims below are synthesized from Consensus abstracts + metadata, not full papers.

Summary

Sub-quadratic sequence models are viable as efficiency architecture, not yet shown — in controlled same-data studies — to be a drop-in replacement for softmax Transformers on the tasks Transformers are best at. Selective SSMs (Mamba) closed the historical quality gap at small-to-medium scale and can match or beat same-size Transformers on many language-modeling and multimodal benchmarks (Gu et al. 2023; Dao et al. 2024). A controlled 8B / 3.5T-token comparison then found that pure SSMs lag on copying, in-context learning, and long-context reasoning, while a hybrid with only ~7% attention beat the Transformer on all 12 standard tasks and was predicted up to 8× faster at decode (Waleffe et al. 2024). That hybrid pattern — a little softmax or local attention on top of a linear-time recurrent/SSM stack — is independently reported for Jamba, Samba, Griffin / RecurrentGemma, and Transformer-to-Mamba distillation (Lieber et al. 2024; Ren et al. 2024; De et al. 2024; Botev et al. 2024; Wang et al. 2024). Theory cuts both ways: approximating softmax attention in sub-quadratic time is SETH-hard (Kacham et al. 2023), and SSMs themselves sit in the same (\mathsf{TC}^0) expressivity class as Transformers — their recurrent “state” does not buy extra state-tracking power (Merrill et al. 2024). The live contradiction is Falcon Mamba 7B, a pure SSM that claims to beat Llama 3.1 8B / Mistral 7B on the Open LLM Leaderboard after 5.8T tokens (Zuo et al. 2024) — a result that is not a same-data control and sits against Waleffe’s controlled finding.

Findings

The quadratic bottleneck, and two families of response

Softmax self-attention is (O(n^2)) in sequence length for both time and memory, which is the stated motive for almost every alternative in this literature (Gu et al. 2023; Sun et al. 2025). A 2025 survey groups the responses into two principal families: linear attention (kernel approximations, recurrent / fast-weight forms, linear-time inference) and sparse attention (fixed patterns, block routing, clustering) (Sun et al. 2025). Selective SSMs, gated recurrences, and hybrids sit alongside those families as a third practical cluster — they replace or interleave attention rather than approximating the softmax matrix itself (Gu et al. 2023; Sun et al. 2025).

A theoretical barrier matters for anyone hoping to keep exact softmax and still go sub-quadratic: approximating softmax-attention output in sub-quadratic time is hard under the Strong Exponential Time Hypothesis (Kacham et al. 2023). One 2023 workaround replaces softmax with high-degree polynomial attention plus kernel sketching; the resulting PolySketchFormer is claimed linear-time with approximation guarantees, and one set of GPT-2-style / 32k-context TPU experiments reported 2.5–4× training speedup versus FlashAttention with no observed quality drop (Kacham et al. 2023). That is a single primary-study claim at modest scale, not a frontier-LLM result.

Selective SSMs closed the small-scale quality gap

Pre-Mamba sub-quadratic lines (linear attention, gated convolution, structured SSMs) had “not performed as well as attention on important modalities such as language,” which Gu et al. 2023 attribute to missing content-based reasoning. Making SSM parameters input-dependent (selection) plus a hardware-aware parallel scan is the proposed fix. The abstract’s headline quality claims: Mamba-3B outperforms same-size Transformers and matches Transformers twice its size on pretraining and downstream language modeling; inference is reported as 5× higher throughput; quality continues to improve on real data out to million-length sequences (Gu et al. 2023). These are author-reported results from the introducing paper, not an independent replication.

Dao et al. 2024 then argue Transformers and SSMs are closely related via structured semiseparable matrices (state-space duality). That framework yields Mamba-2, described as 2–8× faster than Mamba’s selective SSM while remaining competitive with Transformers on language modeling, and as matching or outperforming Transformers at small-to-medium scale (Dao et al. 2024). The duality framing is useful for the viability question: “sub-quadratic vs Transformer” is partly a computational view of related operators, not always a wholly different function class.

Controlled scale: pure SSMs still lose on copy, ICL, and long-context retrieval

The strongest same-data evidence in this pass is Waleffe et al. 2024: 8B Mamba, Mamba-2, Transformer, and a Mamba-2-Hybrid (43% Mamba-2 / 7% attention / 50% MLP), all trained on the same data up to 3.5T tokens. Pure SSMs match or exceed the Transformer on many tasks but lag on strong copying, in-context learning (e.g. 5-shot MMLU, Phonebook), and long-context reasoning. The 8B hybrid exceeded the 8B Transformer on all 12 standard tasks (+2.65 points average) and was predicted up to 8× faster at token generation; on 23 additional long-context tasks at 16K–128K it continued to match or exceed the Transformer on average (Waleffe et al. 2024).

That ICL/retrieval gap is not only a large-LM artifact. Park et al. 2024 compare Mamba and Transformers on controlled ICL tasks: SSMs are comparable on standard regression ICL and better on sparse parity, but fall short on non-standard retrieval; a MambaFormer hybrid beats either family on the tasks each fails alone. The abstract’s implication is architectural complementarity, not SSM dominance.

One 7B counter-claim exists. Zuo et al. 2024 report Falcon Mamba 7B (pure Mamba, 5.8T tokens) surpassing Mistral 7B, Llama 3.1 8B, and Falcon2 11B, matching Gemma 7B, and beating RecurrentGemma 9B and RWKV-v6 Finch 7B/14B on the Open LLM Leaderboard, while also being faster and leaner at long-sequence generation. The authors explicitly frame this as evidence that pure Mamba can match or beat Transformer and hybrid designs. This is a single technical report, not a same-data control; Waleffe’s design is the one that holds architecture constant and varies only the stack. Treat Falcon Mamba as existence proof that a well-trained pure SSM can look strong on standard leaderboards, not as a refutation of the copy/ICL gap.

Hybrids are the current viability path

Across independent groups, the architecture that consistently claims Transformer-level quality and sub-quadratic serving is a hybrid:

• Sparse global attention inside an SSM stack. Waleffe’s 7% attention hybrid is the cleanest controlled win (Waleffe et al. 2024).
• Interleaved Transformer + Mamba + MoE (Jamba). Lieber et al. 2024 report a large-scale hybrid MoE that fits on a single 80GB GPU, with high throughput, a small memory footprint versus vanilla Transformers, “state-of-the-art” standard-benchmark and long-context results, and strong quality out to 256K tokens. How Transformer and Mamba layers are combined, and how experts are mixed, is described as crucial at large scale — i.e. hybrid design is not a free lunch.
• Mamba + sliding-window attention (Samba). Ren et al. 2024 scale Samba to 3.8B parameters / 3.2T tokens and report it significantly outperforming “state-of-the-art” models on a variety of benchmarks; pretrained at 4K, zero-shot perplexity improves out to 1M tokens; after 4K finetuning it extrapolates to 256K with perfect Passkey Retrieval and better Phonebook extrapolation than full attention. Throughput claims: 3.73× versus grouped-query-attention Transformers on 128K user prompts, and 3.64× when generating 64K tokens with unlimited streaming. These speedups are author-reported on specific serving setups.
• Gated linear recurrences + local attention (Griffin / RecurrentGemma). De et al. 2024 report that Hawk (gated linear RNN) exceeds published Mamba downstream numbers, while Griffin matches Llama-2 despite being trained on over 6× fewer tokens, extrapolates beyond training length, matches Transformer training efficiency, and has lower latency / higher inference throughput; they scale Griffin to 14B. Botev et al. 2024 release RecurrentGemma 2B and 9B (Griffin architecture) as comparable to similarly sized Gemma baselines despite fewer training tokens, with a fixed-size state for long-sequence inference. Falcon Mamba’s abstract later claims to beat RecurrentGemma 9B (Zuo et al. 2024) — another reminder that cross-paper leaderboard comparisons are not controlled.
• Distill a pretrained Transformer down to a hybrid. Wang et al. 2024 reuse attention projection weights to distill large Transformers into linear RNNs on academic GPUs. A hybrid that keeps a quarter of the attention layers is reported comparable to the original Transformer on chat benchmarks and better than open hybrid Mamba models trained from scratch on trillions of tokens. Distilled from Llama3-8B-Instruct, their top model reports a 29.61 length-controlled AlpacaEval 2 win rate versus GPT-4 and 7.35 on MT-Bench; it also shows almost-perfect needle-in-a-haystack accuracy at 20× the distillation length. This is one distillation study, but it is direct evidence that much of a Transformer’s quality can survive after most attention layers are removed.

Taken together, the abstracts support a narrower viability claim than “SSMs replace Transformers”: a small attention budget plus a linear-time backbone is enough to match or beat a dense Transformer at 8B–14B class, with large decode and long-context memory wins.

Linear attention and RetNet: same efficiency pitch, weaker quality record

Linear attention can be trained in parallel and decoded as an RNN with matrix-valued state, but “generally underperforms ordinary softmax attention,” and naive implementations are slower than I/O-aware softmax (FlashAttention) (Yang et al. 2023). Yang et al. 2023 introduce hardware-efficient FlashLinearAttention — claimed faster than FlashAttention-2 as a standalone layer even at 1K sequence length — and gated linear attention (GLA). GLA Transformers are reported competitive with a LLaMA-style Transformer and with RetNet and Mamba on moderate-scale language modeling, with length generalization from 2K training to >20K, and higher training throughput than a similarly sized Mamba (Yang et al. 2023).

Sun et al. 2023 present RetNet as a “successor to Transformer”: retention has parallel, recurrent, and chunkwise-recurrent modes, theoretically connecting recurrence and attention, with (O(1)) inference, linear long-sequence cost, favorable scaling, and “without sacrificing performance.” That successor claim is from the introducing paper; later controlled and hybrid work (Waleffe, Park, Yang, De) treats linear/recurrent layers as components, not as a settled replacement.

The 2025 survey’s practical takeaway matches this: efficient attention is being adopted both as all-efficient backbones and as hybrids that keep some global / local softmax (Sun et al. 2025).

Hardware viability is about KV cache and decode, not just training FLOPs

Even when FlashAttention makes training quadratic attention affordable at moderate context, autoregressive decode still pays a growing KV cache (Waleffe et al. 2024). Linear-time models with a fixed-size state (SSM / Hawk / Griffin / RecurrentGemma) attack that memory wall directly (De et al. 2024; Botev et al. 2024). Reported serving deltas in this corpus:

• Mamba: 5× inference throughput vs Transformers (Gu et al. 2023).
• Mamba-2 layer: 2–8× faster than Mamba’s selective SSM (Dao et al. 2024).
• 8B Mamba-2-Hybrid: up to 8× faster decode (predicted) (Waleffe et al. 2024).
• Samba: ~3.6–3.7× vs GQA Transformers at 64K–128K (Ren et al. 2024).
• Jamba: high throughput and small memory versus vanilla Transformers; one configuration fits 80GB (Lieber et al. 2024).
• GLA: FlashLinearAttention faster than FlashAttention-2 even at 1K; GLA trains faster than same-size Mamba (Yang et al. 2023).

These numbers are not commensurate (different hardware, batching, and whether they include MLP/MoE). The directional claim that survives is: the inference-memory advantage is the most consistently advertised, and it grows with context length.

Expressivity: SSMs do not escape Transformer limits

A common hope is that recurrence gives SSMs extra state-tracking power that Transformers lack. Merrill et al. 2024 argue the opposite: SSMs, like Transformers, cannot express computation outside (\mathsf{TC}^0), so they cannot solve permutation composition and are “provably unable” (under standard complexity separations) to accurately track certain chess notations, evaluate code, or track entities in a long narrative. Experiments in that paper find Mamba-style SSMs indeed struggle at state tracking. The abstract’s punchline — the “state” in an SSM is an illusion — undercuts any viability story that treats SSMs as strictly more expressive than attention. Both families need depth, tools, or other scaffolding for genuine state tracking; the hybrid win on retrieval/copy is a different capability than this formal state-tracking class.

Contradictions and open questions

• Pure SSM vs hybrid, after controlling data. Waleffe’s same-data 8B study says pure SSMs lose on copy/ICL/long-context reasoning and hybrids win (Waleffe et al. 2024; Park et al. 2024). Falcon Mamba 7B claims a pure SSM can beat strong Transformer and hybrid baselines on the Open LLM Leaderboard (Zuo et al. 2024). Resolving this needs a same-data, same-eval rerun that includes Falcon’s recipe and copy/Phonebook/needle tests, not just the Open LLM Leaderboard.
• Author “successor” claims vs later consensus. RetNet is introduced as a Transformer successor with no performance sacrifice (Sun et al. 2023); GLA still describes vanilla linear attention as generally weaker than softmax (Yang et al. 2023); subsequent large-scale work converges on hybrids (Jamba, Samba, Griffin, Waleffe, Mamba-in-Llama). The successor claim has not been independently established at frontier scale in this corpus.
• Expressivity: recurrence helps ICL-parity but not (\mathsf{TC}^0) state tracking. Park finds SSMs better than Transformers on sparse parity ICL (Park et al. 2024); Merrill finds they still cannot do permutation-composition state tracking and share Transformer limits (Merrill et al. 2024). “State” in the architecture sense ≠ state tracking in the complexity sense.
• How little attention is enough? Abstracts give 7% (Waleffe), “a quarter of the attention layers” after distillation (Wang), sliding-window / local attention (Samba, Griffin), and interleaved Transformer blocks (Jamba). There is no agreed minimal attention budget, nor a published scaling law for attention fraction versus model size.
• Frontier scale is thin. The largest explicit scales in this pass are Jamba (large MoE, 256K context, single 80GB GPU fit), Griffin 14B, RecurrentGemma 9B, and 8B controlled Mamba/Transformer runs. This corpus does not contain a same-data 70B+ pure-SSM vs Transformer study. Abstracts also do not settle whether 2025–2026 frontier models that “linearize attention” are closer to GLA/Mamba-2 or still softmax Transformers with systems tricks.
• Training-time vs decode-time viability. FlashAttention and GQA shrink the training quadratic penalty; several papers still find linear methods faster at long context or even at 1K for a well-implemented GLA layer (Yang et al. 2023; Kacham et al. 2023). The regime where architecture change beats better softmax kernels is not pinned down as a function of (n), batch, and hardware.
• SETH-hardness vs practical approximations. Exact softmax is theoretically expensive to approximate (Kacham et al. 2023), yet production systems routinely use kernel, sparse, or recurrent stand-ins. Whether those stand-ins preserve the “global competition” that softmax provides is an open expressivity question the 2025 survey flags without settling (Sun et al. 2025).
• Full text needed. Abstracts do not give Waleffe’s per-task tables, Jamba’s exact parameter/active-parameter counts, Falcon Mamba’s data mix, or Merrill’s experimental details. Promote-and-ingest should treat numerical speedups as author-reported until the PDFs are clipped.

Provenance

Rounds run: 3 (full)

Sub-questions by round:

Round 1 (broad survey):

1. Do state-space models (Mamba, S4, S6) match Transformer language-modeling quality at comparable parameter and compute scale?
2. How do linear attention and kernelized attention compare to softmax attention on quality and long-context performance?
3. What do empirical scaling studies and surveys find about the quality–compute tradeoff of sub-quadratic sequence models versus Transformers?
4. In which sequence-length and hardware regimes do sub-quadratic models outperform Transformers on wall-clock time and memory?
5. Do hybrid Transformer–SSM or linear-attention hybrids outperform both pure Transformers and pure sub-quadratic models?

Round 2 (drill-down):

1. Why do pure SSMs fail at copying and in-context retrieval — expressivity, state size, or training dynamics? — targeted the Waleffe/Park gap
2. How little attention is enough in hybrids to close the ICL/copying gap? — targeted the 7% hybrid result
3. How do RetNet, RWKV-class, GLA, and DeltaNet-style linear models compare to softmax Transformers on quality? — targeted the second sub-quadratic family

Round 3 (resolve remaining uncertainty):

1. Do SSMs have a formal expressivity advantage over Transformers (state tracking / circuit complexity)? — targeted the “recurrence is more powerful” claim
2. Do large-scale (7B–50B-class) hybrid or pure-SSM LLMs match Transformer baselines on standard benchmarks? — targeted the frontier-viability gap
3. How do Griffin / Hawk / RecurrentGemma / RWKV compare to Transformers at multi-billion scale? — targeted the Google recurrent line and other non-Mamba stacks

Papers reviewed (15 total; R1: 6, R2: 5, R3: 4):

Round 1:

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu et al. (2023). ArXiv — 8469 citations. Foundational selective-SSM result; 5× decode; Mamba-3B vs same-size and 2× Transformers.
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao et al. (2024). ArXiv — 1775 citations. SSD theory; Mamba-2 2–8× faster, competitive with Transformers at small/medium scale.
• An Empirical Study of Mamba-based Language Models — R. Waleffe et al. (2024). ArXiv — 200 citations. Controlled 8B / 3.5T comparison; pure SSM ICL/copy gap; 7% attention hybrid wins.
• Jamba: A Hybrid Transformer-Mamba Language Model — Opher Lieber et al. (2024). ArXiv — 485 citations. Large-scale hybrid MoE; 256K context; 80GB-GPU fit.
• Efficient Attention Mechanisms for Large Language Models: A Survey — Yutao Sun et al. (2025). ArXiv — 25 citations. Survey of linear vs sparse efficient attention and hybrid adoption.
• PolySketchFormer: Fast Transformers via Sketches for Polynomial Kernels — Praneeth Kacham et al. (2023). ArXiv — 30 citations. SETH-hardness of sub-quadratic softmax; polynomial-sketch workaround; 2.5–4× vs FlashAttention at 32k (GPT-2-class).

Round 2:

• Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks — Jongho Park et al. (2024). ArXiv — 132 citations. ICL: SSMs match regression, win sparse parity, lose non-standard retrieval; MambaFormer hybrid.
• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling — Liliang Ren et al. (2024). ArXiv — 177 citations. Mamba + SWA; 3.8B / 3.2T; 1M PPL; 256K passkey; ~3.7× throughput.
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models — Junxiong Wang et al. (2024). ArXiv — 133 citations. Distill Llama3-8B to hybrid keeping 1/4 attention; chat parity; 20× needle extrapolation.
• Gated Linear Attention Transformers with Hardware-Efficient Training — Songlin Yang et al. (2023). ArXiv — 498 citations. Linear attention weaker than softmax; FlashLinearAttention; GLA competitive with LLaMA/RetNet/Mamba.
• Retentive Network: A Successor to Transformer for Large Language Models — Yutao Sun et al. (2023). ArXiv — 740 citations. Retention as parallel/recurrent/chunkwise mechanism; O(1) inference claim.

Round 3:

• The Illusion of State in State-Space Models — William Merrill et al. (2024). ArXiv — 188 citations. SSMs in (\mathsf{TC}^0) like Transformers; no extra state-tracking power; Mamba struggles on those tasks.
• Falcon Mamba: The First Competitive Attention-free 7B Language Model — Jingwei Zuo et al. (2024). ArXiv — 52 citations. Pure Mamba 7B / 5.8T claims to beat Llama 3.1 8B and Mistral 7B on Open LLM Leaderboard (not same-data).
• Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — Soham De et al. (2024). ArXiv — 265 citations. Hawk > published Mamba; Griffin matches Llama-2 on >6× fewer tokens; scaled to 14B.
• RecurrentGemma: Moving Past Transformers for Efficient Open Language Models — Aleksandar Botev et al. (2024). ArXiv — 53 citations. Open Griffin 2B/9B comparable to Gemma on fewer tokens; fixed-size state.

Not retained (seen in search, deprioritized): Xu et al. 2026 Softmax Linear Attention (1 citation); Hartl et al. 2026 xLSTM/Mamba-2/Gated DeltaNet comparison (0 citations); Fu et al. 2023 Monarch Mixer; Trockman et al. 2024 mimetic init for SSM recall; Chen et al. 2024 circuit-complexity corroboration of Merrill; Lenz et al. 2025 Jamba journal version (no abstract returned); ALISA / KV-cache systems papers that optimize Transformers rather than replace them.

Tools used: mcp__consensus__search (Consensus — covers Semantic Scholar, PubMed, Scopus, ArXiv). Each query returned 19–20 hits but the Cursor integration surfaced the top 3 abstracts per query; retained papers are from those surfaced results after URL-dedup. Filters applied: none. Generated: 2026-08-18 15:17 UTC (11:17 America/New_York)