Reading the Loss Curves

Loss curves are diagnostic, not decorative

A good loss curve plot tells you what's wrong with your training in the first 30 seconds of looking. A junior engineer plots loss as decoration; a senior engineer reads it like a stethoscope. Spend the time learning what each shape means.

Eight common shapes

Both flat at the initial value — model isn't learning at all. Check loss function, data normalization, learning rate.
Both falling smoothly together — healthy training. Keep going.
Train falling, val rising after a point — classic overfitting. Regularize or stop early.
Train and val both fall then plateau — converged. Try larger model or different architecture.
Loss explodes to NaN — exploding gradients or numerical issue. Add clipping, lower lr, check for div-by-zero.
Loss zig-zags wildly — learning rate too high or batch size too small. Lower one or both.
Val loss spikes periodically — usually evaluation noise, sometimes data corruption in val set.
Train loss higher than val loss — usually means dropout is on in train but off in eval (correct), or your loss has different reduction in each.

Tip: Print a sentence about what you expect the loss curve to look like before you start training. After training, compare. The mismatch is often more informative than the metric itself.

Smoothing matters

Per-step training loss is noisy. Plot a smoothed running average (EMA with α=0.99 is fine) overlaid on the raw values. Validation loss is once per epoch, plot raw.

Principle: Loss curves are the cheapest debugging tool you have. If you can't read the curve, you can't tune the model. Get fluent in the shapes early.

Code

EMA smoothing for noisy training loss·python

def ema_smooth(values, alpha=0.99):
    out, ema = [], None
    for v in values:
        ema = v if ema is None else alpha * ema + (1 - alpha) * v
        out.append(ema)
    return out

import matplotlib.pyplot as plt
plt.plot(train_losses, alpha=0.3, label="train (raw)")
plt.plot(ema_smooth(train_losses, 0.99), label="train (EMA 0.99)")
plt.plot(val_steps, val_losses, label="val", marker="o")
plt.legend(); plt.show()

Loss curves are diagnostic, not decorative

Eight common shapes

Smoothing matters

Code

External links

Exercise

Progress

Comments 0