Not every sequence architecture is a Transformer. State-space models (SSMs) and a few RNN-Transformer hybrids offer O(n) complexity alternatives that are getting close on language quality while being meaningfully cheaper at long context.
Mamba (Gu & Dao, 2023)
Mamba replaces self-attention with selective state-space models: input-dependent state transition matrices that decide what information to keep or forget. O(n) compute and O(1) state, vs O(n²) for attention. Mamba 1 demonstrated 5× throughput and 3× wall-clock speedup at long sequences vs comparable Transformers.
Mamba 2 (2024) reframes SSMs as structured linear attention, allowing the same computation to be expressed as matrix multiplications. Result: 2-8× faster training than Mamba 1 with simpler implementation.
RWKV
An RNN-Transformer hybrid: trains in parallel like a Transformer, runs inference in O(1) memory like an RNN. Through versions V4-V8, it has accumulated tricks: multi-headed matrix states, dynamic LoRA-based token shifting, attention-free mixing layers. Comparable quality on many benchmarks; dramatically lower inference memory.
| Architecture | Training compute | Inference memory | Long context behavior |
|---|---|---|---|
| Transformer (dense attention) | O(n²) | O(n) | 1× (baseline) |
| Mamba 1 | O(n) | O(1) | 5× throughput |
| Mamba 2 | O(n) | O(1) | 2-8× faster training than Mamba 1 |
| RWKV (V6+) | O(n) train, O(1) inference | O(1) | Comparable quality, smaller memory |
Despite their theoretical edges, SSMs have not displaced Transformers for production language tasks as of 2026. Attention's direct any-position connectivity remains powerful for complex reasoning, and the engineering ecosystem (Flash Attention, vLLM, training frameworks) has tuned Transformers to extract every drop of efficiency. Hybrids — alternating attention and SSM layers — are an active research direction (e.g., Jamba, Zamba).