The arithmetic of "both"
Pure Transformers have perfect recall but quadratic cost. Pure SSMs have linear cost but lossy memory. The natural question is whether you can get both — and the empirical answer the field has converged on is: yes, by mixing them strategically. Use SSM (or RWKV-class) layers for the bulk of your stack — say, 80–90% — and sparse attention layers for the rest, placed at strategic depths where the model needs precise recall.
The arithmetic explains why this works. If 90% of your layers are O(n) cost and 10% are O(n²), the dominant cost is still the O(n²), but the constant factor is 10× smaller. At long context (where attention's quadratic cost matters most), this constant-factor reduction is roughly an order of magnitude in practice — and you keep the recall capability that pure SSMs lose.
How few attention layers is enough?
The empirically-found answer keeps coming up around 1 attention layer per 7–9 SSM layers. NVIDIA's Mamba-2-Hybrid study found that even adding 4 attention layers among 28 total substantially recovered in-context learning capability. Below that ratio, ICL collapses; much above it, you've given up most of the efficiency. The 1:7 to 1:9 range is a stable empirical sweet spot found by independent teams.
This is the kind of empirical convergence that's worth taking seriously. Multiple labs that weren't talking to each other — AI21, NVIDIA, IBM, Zyphra — independently arrived at similar attention:SSM ratios. That's a strong signal that the ratio reflects something real about the recall-vs-efficiency tradeoff at production scale.
The strategic-depth question
It's not just "how many" attention layers — it's where. Most production hybrids place attention layers in the middle-to-late depths, where the model is doing higher-level integration. The early layers (which are mostly token-level processing) and the very last layers (which are output projection) tend to work fine as pure SSM. The middle layers are where recall matters most, so attention earns its place there.
This is a heuristic, not a theorem. Different production designs put attention in slightly different places. If you're building a hybrid from scratch, study Jamba's, Nemotron-H's, and Granite 4.0's depth distributions; you'll find they cluster but don't agree exactly.