Three families inside the attention frame
If you stay inside the attention frame, three approaches dominate. Kernelized attention approximates the softmax with kernel functions, factorizing computation to be linear in sequence length. Sparse attention attends only to a subset of tokens — sliding windows, global tokens, learned sparsity patterns. Structured attention rearranges the computation order or the attention shape (multi-head latent attention, grouped attention, multi-query attention) for efficiency.
The common thread: all three preserve the attention abstraction at the API level. A model trained with FlashAttention-3 can have a layer swapped for sliding-window attention without changing the rest of the stack. This drop-in compatibility is the killer feature — you keep the entire Transformer ecosystem (vLLM, SGLang, TRT-LLM, HF Transformers) and just lower the cost.
Why this matters strategically
For most teams in 2026, the realistic upgrade path from a vanilla Transformer is not "replace with Mamba" but "replace some attention layers with sparse attention". The cost of architectural risk is low — you're still in the attention paradigm — and the upside (1.5–4× efficiency at long context) is meaningful. The pure-replacement architectures (Mamba, RWKV) require more substantial commitment.
This is also the lane where most Transformer improvements from 2023–2026 actually shipped. GQA, MQA, sliding window, NSA, MoBA, Kimi Linear — these all crossed the line from research to production faster than any pure-replacement architecture. The ecosystem effect compounds: improvements in this lane stack on top of FlashAttention-3, GQA, speculative decoding, and quantization, while improvements in the pure-replacement lane usually have to rebuild each integration from scratch.