Overfitting in Neural Networks

What overfitting actually looks like

Overfitting is when your training loss keeps going down while your validation loss starts going up. The model is memorizing patterns specific to the training set that don't transfer to held-out data. Big neural networks can fit pure noise (Zhang et al., 2017) — capacity is rarely the bottleneck. The art is keeping the model honest.

Three signals you should learn to read on a loss-curve plot: (1) the gap between train and val loss; (2) the slope of val loss after the gap opens; (3) whether the gap shows up in epoch 1 (data leak / no regularization) or epoch 30 (capacity catching up to dataset size).

Tip: Plot train and val loss on the same axes from step 0. The shape of the gap tells you what to fix: huge gap from epoch 1 = leak or too much capacity, gap that grows after epoch 20 = needs more regularization, no gap and high loss = underfitting / wrong architecture.

The standard regularization toolbox

Weight decay (L2 / AdamW) — penalizes large weights, the cheapest default.
Dropout — randomly zero units during training, smooth the gradient noise.
Data augmentation — synthetically expand the training set (the highest-leverage move when you have it).
Early stopping — stop training when val loss stops improving.
Smaller model — sometimes the right answer.

The 'just give it more data' answer

The single most reliable cure for overfitting is more (good) data. Augmentation is the cheap version. Pretraining on a related domain is the medium version. Active learning to label hard examples is the expensive version. All three beat any clever regularizer.

Principle: Overfitting is mostly a data problem dressed up as a modeling problem. Reach for more (or better) data first; reach for regularization knobs second; shrink the model only when both fail.

Code

Diagnose overfitting from a loss curve·python

import matplotlib.pyplot as plt

train_losses = []   # one per step
val_losses   = []   # one per epoch
val_steps    = []   # the step at which each val_loss was measured

# ... after training ...
plt.plot(range(len(train_losses)), train_losses, label="train", alpha=0.5)
plt.plot(val_steps, val_losses, label="val", marker="o")
plt.xlabel("step"); plt.ylabel("loss"); plt.legend(); plt.show()
# Read the shape:
# - val rising while train falling -> overfitting, regularize or stop
# - both flat and high -> underfitting, more capacity / longer schedule
# - both falling, modest gap -> healthy

What overfitting actually looks like

The standard regularization toolbox

The 'just give it more data' answer

Code

External links

Exercise

Progress

Comments 0