How to read new architecture papers without getting fooled
The post-Transformer literature is full of impressive-sounding numbers that don't always reproduce. Here's the framework that separates credible claims from paper-flavored hype:
Scope: Does this architecture beat Transformers only in a niche, or across the board? "Beats Transformer at long-context summarization on 7B scale" is a real and useful claim. "Beats Transformer" without qualifiers usually has a hidden niche.
Baselines: Is the Transformer comparison against an optimized Transformer (FlashAttention-3, GQA, modern training recipe) or against a naive implementation that nobody would actually deploy? Many "alternative beats Transformer" results disappear when the Transformer baseline includes the optimizations a real production team would use.
Maturity: Is the hardware/software stack mature enough for production, or are you reading a paper whose code requires a specific CUDA version, a custom kernel, and three months of integration work? The architecture and the deployment readiness are separate axes.
Reproducibility: Are training recipes published and reproducible? Has any third party replicated the headline results? Published-but-not-reproduced results are common and worth less than they look.
Two myths to discard
Myth: "Mamba replaces attention everywhere." Reality: pure Mamba fails on ICL/recall — the most successful Mamba deployments (Jamba, Nemotron-H, Granite 4.0) all keep attention layers. Hybrids are the practical path.
Myth: "Efficient attention and attention-free models are the same thing." Reality: they're fundamentally different. Efficient attention (FlashAttention, sparse attention) preserves the attention mechanism but optimizes its implementation. Attention-free models (SSMs, RWKV) replace attention entirely. The quality-efficiency trade-offs are completely different. Don't lump them together.
The single most important habit
Theoretical complexity (O(n) vs. O(n²)) doesn't always translate to wall-clock speedup due to constant factors and hardware optimization. Always benchmark with your actual workload, not with paper numbers. Five-times speedup at 1M context on a hand-tuned synthetic doesn't help you if your P95 input is 4K and your serving hardware doesn't have the kernel.