The practical guideline
Based on 2025 research and production experience, here's when efficient attention is the right answer:
- General NLP under 32K tokens, quality priority: standard Transformer + FlashAttention-3 + GQA. The optimized baseline is so strong that switching to anything else costs you more than it saves.
- Enterprise long-context (legal, medical, code) at 32–128K: SSM-attention hybrid (Jamba, Nemotron-H, Granite 4.0) for best cost-quality balance. Sparse attention alone may not cover the cost gap.
- Edge / on-device serving at variable context: pure SSM or RWKV for constant-memory inference.
- Frontier reasoning workloads (complex math, multi-step logic, tool use): pure Transformer where recall quality is non-negotiable, even at the cost.
- Long-context summarization or retrieval over very long documents with quality near-parity acceptable: Kimi Linear-class linear attention, or sliding window if you want maturity.
The crossover point
The crossover where SSMs/hybrids start beating Transformers on real wall-clock latency (not just paper-FLOP) is around 20–30K tokens. Below that, optimized Transformers (FA3 + GQA + sliding window) are actually faster — better constant factors, more mature kernel implementations, more battle-tested tooling. Above 32K, the alternatives gain ground rapidly. By 128K the difference is decisive.
Know your sequence length distribution before you decide
The single most important measurement before architecture-shopping is your workload's sequence length distribution. If your P95 input length is 4K tokens, you're never hitting the regime where alternatives matter — pick Transformer + sparse attention if you want efficiency, or just default Transformer if you want simplicity. If your P95 is 64K, you're in the regime where the choice matters and you should benchmark.
Most teams choose architecture based on what's exciting in the literature instead of what their workload actually needs. Don't be that team. The boring "99% of our requests are under 8K" is a totally legitimate reason to ignore this entire quest's worth of alternatives and stay on FA3+GQA forever.