The capability gap that doesn't go away
Transformers maintain an explicit KV cache where every past token is directly accessible to every later position. SSMs compress history into a fixed state. Hyena's filters are content-independent. RWKV's state decays. RetNet's decay is per-head fixed. Only Transformers preserve the original tokens at lookup time, and that's the property that makes in-context learning genuinely robust.
The 2025 NeurIPS spotlight "Achilles' Heel of Mamba" made this rigorous: pure SSMs systematically fail on copy and recall tasks where Transformers succeed, and the failure can't be patched with more data — it's structural. Five-shot MMLU is the canonical practical demonstration: pure SSMs lag substantially because in-context learning requires looking back at the few-shot examples and pattern-matching, and compressed state blurs that lookup.
This is why Transformers won't be replaced wholesale
The recall gap is the single most important fact for understanding why production frontier models in 2026 are still Transformers (or hybrids with significant attention components). Tasks like multi-step tool use, agentic workflows that accumulate observations, code generation that must respect distant function signatures, and RAG that retrieves and reasons over 10–100 chunks at once — all of these depend on exactly the kind of precise per-token recall that Transformers do natively and that compressed-state alternatives cannot.
You can build hybrids that recover most of this capability with much less attention. You cannot build a pure-replacement architecture that recovers it. That asymmetry is why every successful production deployment we've discussed in this quest is either a Transformer, a hybrid with attention layers, or a domain-specific model where exact recall doesn't matter.
The frontier reasoning angle
Frontier reasoning workloads — complex math, multi-step proof, sophisticated tool use, code-wide refactors — are particularly recall-heavy. The model needs to maintain coherent reference to many specific facts, intermediate results, and constraint definitions over thousands of tokens. This is the hardest workload in modern AI, and it's the one where Transformer's advantage is largest. Don't expect the frontier to leave the Transformer family until something seriously addresses this gap, which no current alternative does at production scale.