questionopenartificial-intelligence
How little attention is enough in hybrid sequence models?
Notes
How little attention is enough in hybrid sequence models?
The question
What is the minimal softmax / local-attention budget that closes the copy / ICL / long-context-retrieval gap of pure SSMs, and does that fraction stay constant as models scale?
Why it matters
If a few percent attention is enough at 8B–14B, the "replace the transformer" debate is the wrong debate — the live question is how to spend a small attention budget on a linear-time backbone. That is the viability claim of sub-quadratic-vs-transformer-viability.
What we currently believe
Abstracts (2026-08-18 academic synthesis) give several non-commensurate answers and no scaling law. Belief is: unknown; hybrids work at several different budgets; no agreed minimum.
Evidence we have
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "How little attention is enough? Abstracts give 7% (Waleffe), “a quarter of the attention layers” after distillation (Wang), sliding-window / local attention (Samba, Griffin), and interleaved Transformer blocks (Jamba). There is no agreed minimal attention budget, nor a published scaling law for attention fraction versus model size."
- From 2026-08-18-academic-research-sub-quadratic-model-viability-vs: "This corpus does not contain a same-data 70B+ pure-SSM vs Transformer study."
Evidence we need
- Same-data, same-eval sweeps of attention fraction at ≥8B that include copy / Phonebook / needle tests, not just Open LLM Leaderboard.
- A 70B+ controlled comparison.
How to resolve
Clip the Waleffe / Wang / Jamba PDFs for per-task tables; do not invent a budget from abstracts.
Related
Referenced by