The largest hybrid SSM-attention model
NVIDIA Nemotron-H (April 2025) is the canonical enterprise-scale hybrid: 8B and 56B / 47B versions, with ~92% of attention layers replaced by Mamba-2. About 3× faster than Llama-3.1 70B at 65K context. The 56B model was pretrained in FP8 on 6,144 H100 GPUs using 20T tokens — production pretraining at frontier scale.
This is the data point that ends the "is hybrid SSM-attention production-ready" question. NVIDIA, the most hardware-aware company in the field, bet 6,144 H100s on a hybrid architecture and shipped it as their commercial offering. That's not a research bet; it's a serving bet.
MiniPuzzle compression — 56B → 47B
NVIDIA also showed that hybrid architectures are amenable to aggressive post-training compression. The 56B model was distilled / pruned to 47B parameters with a 20% speed gain on top of the architectural advantage. The compression worked partly because hybrid stacks have natural redundancy across SSM layers that pure Transformer stacks don't have to the same degree.
Cosmos-Reason 1 — vision-language application
Nemotron-H serves as the backbone for Cosmos-Reason 1, NVIDIA's physical AI vision-language model. The hybrid backbone is what makes Cosmos-Reason's long-context multi-modal reasoning practical — high-res video frames at scale require exactly the long-context efficiency that hybrids provide.
Nemotron 3 — the next generation
Nemotron 3 (December 2025 / March 2026) is NVIDIA's next-generation hybrid: MoE architecture, 1M context, and NVFP4 precision (a new low-precision format optimized for Blackwell GPUs). The combination of MoE sparsity, hybrid SSM-attention, and aggressive low-precision training represents NVIDIA's bet on what 2026–2027 production stacks look like.
The clear pattern: NVIDIA is committing more to hybrid SSM-attention with each generation, not less. That commitment from the hardware vendor itself is a signal worth taking seriously when planning longer-term architecture decisions.