What the optimizer decides
The loss tells you how wrong you are; the optimizer decides what to do about it. Given a gradient, it chooses the size and direction of each weight's update. That single choice is why the same model with the same data can converge cleanly or thrash forever — the optimizer is the policy, the gradient is just the raw signal.
Keras ships every optimizer that matters. Start here:
| Optimizer | Key Feature | When to Use |
|---|---|---|
Adam | Adaptive learning rates | Default starting point |
AdamW | Adam + weight decay | Fine-tuning pretrained models |
SGD | Simple + momentum | Still competitive, good with schedules |
RMSprop | Adaptive per-parameter | RNNs, non-stationary objectives |
Muon | Newton-style updates | New in Keras 3.10, experimental |
If you're unsure, reach for Adam — it's the default for a reason, and it forgives a sloppy learning rate better than anything else. Switch to AdamW the moment you fine-tune a pretrained model, because decoupled weight decay is what keeps those large pretrained weights from drifting.
The learning rate is a curve, not a number
A fixed learning rate is a compromise: too high to finish cleanly, too low to start fast. A schedule resolves the tension by changing the LR over training. The Code section sets up cosine decay with a warmup — ramp up gently for the first few hundred steps so early noisy gradients don't blow up the weights, then decay smoothly toward a floor so the final epochs settle into a sharp minimum. Pass the schedule object straight into the optimizer's learning_rate and Keras advances it every step for you.