Overfitting: When the Model Memorizes Instead of Learning

The Trap That Eats Beginners

You train a model. Training loss goes to zero. You're proud. Then you evaluate on test data — performance is awful. You overfit. The model memorized the training set instead of learning the underlying pattern.

The signature: training loss low, test loss high. The gap is overfitting.

Why It Happens

Models with too many parameters can memorize the training data exactly — every quirk, every noise sample. They lose the ability to generalize because they didn't have to. The only signal during training is "make training loss small," and a big enough model can do that by brute force.

The Antidotes

More data — the cheapest fix when available. Memorizing 10M examples is harder than memorizing 100.
Regularization — penalize complexity. L1/L2 weight penalties. Dropout (randomly turning off neurons). Early stopping.
Train/Validation/Test split — hold out data the model never sees during training. Track val loss to detect overfitting in flight.
Data augmentation — slightly perturb training samples (image rotations, text paraphrases) to increase effective dataset size.
Smaller model — fewer parameters means less capacity to memorize. Sometimes the best fix.

Training accuracy is a vanity metric. Validation accuracy is the truth. If they're far apart, your model is memorizing, not learning. Always hold out data the model never sees.

Code

Polynomial overfit demo·python

import numpy as np
from numpy.polynomial import polynomial as P

# Fit polynomials of increasing degree — watch overfitting
np.random.seed(0)
x = np.linspace(0, 1, 10)
y = np.sin(2 * np.pi * x) + np.random.normal(0, 0.1, 10)

x_test = np.linspace(0, 1, 100)
y_test = np.sin(2 * np.pi * x_test)        # the truth

for degree in [1, 3, 9]:
    coeffs = np.polyfit(x, y, degree)
    train_pred = np.polyval(coeffs, x)
    test_pred  = np.polyval(coeffs, x_test)
    train_mse = np.mean((y - train_pred) ** 2)
    test_mse  = np.mean((y_test - test_pred) ** 2)
    print(f"degree {degree}: train MSE = {train_mse:.4f}, test MSE = {test_mse:.4f}")
# Degree 9 fits training near perfectly but explodes on test — classic overfit.

Exercise

Generate 20 points along y = sin(2πx) + small noise. Fit polynomials of degree 1, 3, 9, 15. For each, plot the fit on a fine grid and compute test MSE. Notice how test MSE shoots up at high degrees — that's overfitting visually.

Hint

Higher degree = more flexibility = better training fit but worse generalization. The 'right' degree depends on the noise level and how much data you have.

Overfitting: When the Model Memorizes Instead of Learning

The Trap That Eats Beginners

Why It Happens

The Antidotes

Code

External links

Exercise

Progress

Comments 0