Post-transformer architectures (and why the transformer keeps winning anyway)
Post-transformer architectures (and why the transformer keeps winning anyway)
Vintage: 2026-07. Two Moonshots episodes recorded 2026-07-16 and 2026-07-18. Architecture-capability claims age fast — re-validate before leaning on them.
One-line summary: A live tension in frontier AI — a well-funded effort (liquid-ai's Liquid Neural Networks, plus state-space / linear-attention lines) tries to replace the transformer with something more efficient and nature-inspired, yet the strongest 2026 open model (Kimi K3) turns out to be "still essentially a transformer," and even Liquid's own architecture-search keeps re-discovering transformer-adjacent components.
The insight
Two claims sit in productive tension.
(1) The transformer is bloated and someone should replace it. ramin-hasani in 2026-07-17-podcast-moonshots-mira-murati-s-975b-open-model-ramin-hasani-on built liquid-ai on exactly this bet: Liquid Neural Networks derive from the continuous-time, analog dynamics of the C. elegans worm's 302-neuron nervous system — "these are based on recurrent neural networks... based on continuous time processes." dave-blundin frames the motive: "everybody knows the transformer architecture and the whole attention mechanism is bloated. And if you really go back to founding principles... you might be able to build something dramatically, like massively better."
(2) But the transformer keeps winning. alexander-wissner-gross in 2026-07-19-podcast-moonshots-urgent-update-ai-sputnik-moment-kimi-k3-released, inspecting Kimi K3's published architecture: "there's no magic in it and that's pretty striking... it's still essentially a transformer. They've made obviously a number of innovations... how they linearize attention... but it's still basically a recognizable transformer." He lands on the punchline: "attention is still all you need." Even inside Liquid, ramin-hasani concedes the convergence: when their STAR / AFMD architecture-search runs "without a human bias," the winning design comes back "double gated convolution kind of mechanisms as 80% of the network" — expressive, but a hybrid that still carries "a little bit of transformers in them."
The reconciliation is that "post-transformer" is really a spectrum, not a replacement. State-space models (Mamba-class), gated/linear attention, and Liquid's continuous-time recurrences are all linearizations of a more general dynamical-system family that you adopt to scale — and the pieces that survive at scale look transformer-like. The gains that actually shipped Kimi K3 were data-mix, MoE, and the Muon optimizer, not a new base architecture (see chinese-open-weight-frontier-parity).
Where post-transformer wins today: the edge
The one place the non-transformer premise pays off cleanly is efficiency for on-device deployment. ramin-hasani: the mission is "efficient general purpose AI at every scale... that brings the same level of intelligence than a frontier model into, let's say on a CPU." Liquid's sub-1GB multimodal model runs on a ~$60 automotive chip (Mercedes-Benz). That is the edge-inference-shift story — architecture chosen per hardware constraint, not per leaderboard.
Contradictions / tensions
- alexander-wissner-gross presses ramin-hasani directly: Liquid "looks more and more like either transformer or transformer plus MoE... a good business selling customized transformer derivatives to Mercedes." Ramin's defense is that the search framework (STAR/AFMD) is the real product, not any single architecture — but that concedes the base is convergent.
- RSI-via-architecture-design is asserted by Ramin as Liquid's edge, but he also argues real recursive self-improvement is computationally intractable — see autoresearch-recursive-self-improvement.
Open questions
- Does any non-transformer base ever lead the frontier, or does it only win at the efficiency/edge margin?
- Is the Muon-optimizer + data-mix route (Kimi) more of the near-term gain than any architecture change?
Sources
- 2026-07-17-podcast-moonshots-mira-murati-s-975b-open-model-ramin-hasani-on (multi-context, vault/sources/) — Ramin Hasani on Liquid AI / LNN / STAR / AFMD.
- 2026-07-19-podcast-moonshots-urgent-update-ai-sputnik-moment-kimi-k3-released (multi-context, vault/sources/) — AWG/Emad on Kimi K3 being "still a transformer," Muon optimizer.