The failure is almost always one of five
When training goes wrong it rarely fails loudly with a useful message — it fails quietly, with a loss that won't move or a gap between train and validation that won't close. The skill is reading the symptom back to its cause. Most of what you'll hit fits in one small table:
| Problem | Symptom | Solution |
|---|---|---|
| Shape mismatch | ValueError about incompatible shapes | Print shapes at each layer; check model.summary() |
| NaN loss | Loss becomes NaN during training | Lower LR, check data for NaN/inf, add gradient clipping |
| Overfitting | Train acc high, val acc low | More data, add dropout, L2 regularization, augmentation |
| Underfitting | Both train and val acc low | Bigger model, more epochs, lower regularization |
| Slow training | Epochs take too long | Check GPU utilization, prefetch data, mixed precision |
NaN loss: clip before you despair
NaN loss is the one that looks scariest and is usually the easiest. It means a gradient blew up — often a learning rate too high, or a single bad sample with an inf in it. Beyond lowering the LR and scrubbing the data, gradient clipping is a cheap guardrail: clipnorm rescales the whole gradient vector if its norm exceeds a threshold, so one exploding step can't poison the weights.