The sweet spot for efficient attention
If your sequences are mostly under 64K tokens and you need strong recall, efficient Transformer variants are the pragmatic choice. Sliding window (Mistral-style), GQA-aggressive variants, NSA-style learned sparsity — these are the upgrades that compose with everything else in your stack and require no architectural retraining.
The "efficient enough" Transformer beats the "theoretically better" alternative that lacks ecosystem support. A 10% quality loss from an immature alternative architecture often costs more in production (bugs, support time, missed features) than the 3× inference cost savings would buy.
The crossover point — really
Below 20–30K tokens, optimized Transformers (FA3 + GQA + sliding window) are typically faster on real wall-clock measurements than SSMs and hybrids. The asymptotic complexity arguments don't kick in until you're past the constant-factor zone. If your P95 input is under 20K, you're entirely below the crossover and you should stop reading post-Transformer papers as if they're directly relevant to your stack.
The myth of "linear is always better"
Myth: "Linear-time always wins." Reality: below 20–30K tokens, optimized Transformers win on wall-clock latency. The crossover depends on hardware (faster GPUs push it later), batch size (larger batches push it later), and implementation maturity (more mature kernels push it later). A 2026 H100 with FlashAttention-3 might cross over later than a 2024 A100 with naive attention.
The takeaway: always benchmark with your actual workload. "Linear attention is faster" without a sequence-length and hardware specifier is a nearly-meaningless claim. "Linear attention is faster on H100s above 64K tokens with batch size 4" is a specific, useful claim.