Transformers are predictable at scale
Transformers have been scaled to hundreds of billions of parameters with dynamics that are now well-understood. Every failure mode is documented and has a known mitigation: loss spikes, attention entropy collapse, attention sinks, learning-rate cliffs, gradient norm divergence. The recipes for training a stable 70B Transformer are mature. The recipes for training a stable 7B Transformer are extremely mature.
This predictability has real economic value. Frontier labs spend tens of millions of dollars on a single training run. The cost of a failed training run isn't the FLOPs wasted; it's the calendar time you can't get back and the team time you redirected away from other work. Transformers are predictable enough that you can budget for a training run with high confidence it will produce something.
SSM training is narrower
SSMs show narrower optimal learning rate ranges. A 2025 study found Mamba's performance on recall tasks jumps from near-zero to near-optimal within a tight LR window — choose poorly and the model is unusable, with no graceful degradation. The published recipes from NVIDIA (Nemotron) and AI21 (Jamba) help significantly, but "Mamba is finicky" remains true in a way that "Transformer is finicky" no longer is.
For research labs and well-funded teams, this fragility is a tax — they can run more sweeps to find the right LR. For startups and product teams running small training budgets, it's a real risk. A single LR mistake on a $1M Mamba run is more painful than a single LR mistake on a $1M Transformer run because you have less institutional fallback.
Initialization, schedule, regularization
The same fragility applies across the board: initialization choices matter more, schedule shapes matter more, regularization gains and losses are more variable. The grace period for hyperparameter mistakes is shorter. None of this is fatal — Mamba and other alternatives obviously do train successfully — but the Transformer's predictability is itself a competitive advantage that's hard to fully appreciate until you've felt the difference.