C.W.K.
Stream
Lesson 04 of 05 · published

Scaling and Training Stability

~12 min · scaling, training, stability

Level 0Observer
0 XP0/50 lessons0/14 achievements
0/100 XP to next level100 XP to go0% complete

Transformers are predictable at scale

Transformers have been scaled to hundreds of billions of parameters with dynamics that are now well-understood. Every failure mode is documented and has a known mitigation: loss spikes, attention entropy collapse, attention sinks, learning-rate cliffs, gradient norm divergence. The recipes for training a stable 70B Transformer are mature. The recipes for training a stable 7B Transformer are extremely mature.

This predictability has real economic value. Frontier labs spend tens of millions of dollars on a single training run. The cost of a failed training run isn't the FLOPs wasted; it's the calendar time you can't get back and the team time you redirected away from other work. Transformers are predictable enough that you can budget for a training run with high confidence it will produce something.

SSM training is narrower

SSMs show narrower optimal learning rate ranges. A 2025 study found Mamba's performance on recall tasks jumps from near-zero to near-optimal within a tight LR window — choose poorly and the model is unusable, with no graceful degradation. The published recipes from NVIDIA (Nemotron) and AI21 (Jamba) help significantly, but "Mamba is finicky" remains true in a way that "Transformer is finicky" no longer is.

For research labs and well-funded teams, this fragility is a tax — they can run more sweeps to find the right LR. For startups and product teams running small training budgets, it's a real risk. A single LR mistake on a $1M Mamba run is more painful than a single LR mistake on a $1M Transformer run because you have less institutional fallback.

Initialization, schedule, regularization

The same fragility applies across the board: initialization choices matter more, schedule shapes matter more, regularization gains and losses are more variable. The grace period for hyperparameter mistakes is shorter. None of this is fatal — Mamba and other alternatives obviously do train successfully — but the Transformer's predictability is itself a competitive advantage that's hard to fully appreciate until you've felt the difference.

External links

Exercise

Train a small Mamba (≤30M params) on a small dataset, sweeping learning rate across log-spaced values from 1e-5 to 1e-2. Then do the same with a small Transformer at similar parameter count. Plot final validation loss vs. learning rate for both. You should see the Mamba's curve has a sharper minimum (wider range of LRs that produce divergence or convergence to bad local optima); the Transformer's curve is gentler. That sharpness is the training-stability gap.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.