Five reasons the Transformer hasn't been displaced
1. Ecosystem maturity
vLLM, TensorRT-LLM, llama.cpp, MLX, SGLang — the entire serving stack is optimized for attention-based Transformers. Switching to a non-Transformer architecture means re-discovering an inference-optimization story from scratch. That cost is enormous.
2. Scaling-law confidence
We know how Transformers scale. Chinchilla, the Llama 3 scaling work, the DeepSeek scaling notes — there is decade-long empirical confidence in how Transformer parameters and compute trade off. Alternative architectures lack this characterization above 100B parameters; nobody wants to bet a multi-million-dollar training run on a less-charted scaling curve.
3. Benchmark design bias
Most evaluation benchmarks were designed around tasks where Transformers excel — exact recall over long passages, precise multi-step reasoning, structured QA. Alternative architectures with bounded recurrent state may be better-or-worse depending on the benchmark, but the benchmark distribution itself favors attention.
4. Industry risk tolerance
When a training run costs $50M, "promising new architecture" is not what teams pick. Proven architectures win. This is rational — first-mover penalties on architectural changes can mean re-doing the entire training pipeline.
5. Reasoning workloads need attention's strengths
Extended chain-of-thought reasoning relies on exact recall over the thinking-token sequence. SSMs and recurrent variants compress past context into bounded state, making this kind of reasoning harder. The reasoning paradigm has, somewhat unexpectedly, made attention's strengths more valuable, not less.
How to stay current without drowning
- Primary sources. Model cards, system cards, technical reports. Most reliable.
- Survey papers. Annual/quarterly surveys aggregate the noisy paper firehose. Worth reading 2–3 per year.
- Curated newsletters. Sebastian Raschka's Ahead of AI, Nathan Lambert's Interconnects, the Hugging Face blog. Selective and high-signal.
- Communities. r/LocalLLaMA for open-weight tinkering, Papers with Code for code-linked papers.
The 3-question filter
For any architectural claim: (1) Is this a backbone change, a training change, or an inference change? (2) Is there a peer-reviewed or technical-report writeup? (3) Can someone reproduce it on the open frontier? "Yes" to fewer than two means treat with skepticism.
What to avoid
Twitter/X hype threads, YouTube clickbait, marketing blog posts from labs launching products. They are designed to spread, not to inform. Default skepticism.