The pragmatic frontier
Pure-SSM and pure-RWKV models are still niche. The most influential post-Transformer designs in 2025–2026 are hybrids — Transformers with some attention layers replaced by SSM or recurrent layers. These get most of the efficiency gains while keeping the strengths of attention where they matter most.
The 7:1 to 10:1 ratio
Multiple independent research groups (NVIDIA, IBM, AI21, Mistral) have converged on a similar finding: roughly 7:1 to 10:1 efficient layers per attention layer gives most of the speedup with minimal quality loss. This is one of the more striking convergences in recent architecture research.
Notable hybrid families
- Jamba (AI21). Mamba + Transformer hybrid, includes a 52B variant with MoE. One of the first frontier-scale hybrids.
- Nemotron-H (NVIDIA). 8B and 56B hybrid variants achieving roughly 3× speedup over pure Transformers at comparable quality.
- IBM Granite 4.0. Hybrid architecture targeting ~70% memory reduction for long contexts.
- Falcon H1R (TII). Hybrid with RWKV-style layers replacing some attention layers.
Why hybrids work
Most attention computation is "wasted" — many attention heads attend mostly locally, doing what a cheaper layer could do. Replacing those layers with SSM or recurrent variants saves compute. The few attention layers that genuinely benefit from full global attention can be preserved. The result is asymmetric: keep what helps, replace what doesn't.
What hybrids still cannot deliver
- Truly massive frontier scale. The largest hybrid models (Jamba 52B, Nemotron-H 56B) are still well below the trillion-parameter MoE frontier.
- Drop-in replacement for fine-tuning. Most fine-tuning tooling is dense/MoE-Transformer-shaped. Hybrid fine-tuning is possible but the ecosystem is thinner.
The reading rule for hybrids
Hybrids are usually clearly marked in announcements ("Mamba-Transformer hybrid", "SSM-attention hybrid", explicit ratios). If a model is a hybrid, the announcement will tell you. If you cannot tell, it is probably not a hybrid.