Overfitting: When the Model Memorizes Instead of Learning

Excellent Fit, Poor Generalization

Overfitting occurs when a fitted system performs much better on its training sample than on relevant new data. Memorization can be one cause, but the operational signal is a generalization gap under a trustworthy evaluation protocol—not a claim about what the model internally “understood.”

Why It Happens

Capacity, noisy labels, weak inductive bias, distribution mismatch, repeated model selection, and a small or unrepresentative dataset can all contribute. Parameter count alone does not determine the outcome; modern overparameterized systems can still generalize well when data, optimization, and regularization interact favorably.

Ways to Improve Generalization

Better or more representative data, with attention to label quality and coverage.
Regularization, including weight penalties or decoupled weight decay, dropout, augmentation, and early stopping.
Appropriate capacity and architecture for the task and dataset.
Leakage-resistant validation that respects time, groups, and repeated tuning.
Distribution-aware evaluation on the environments that matter at deployment.

Training metrics measure fit; validation metrics estimate behavior under a chosen validation distribution. A large gap can indicate overfitting, but validation data is not “truth”: it can be biased, shifted, noisy, or indirectly overfit through repeated model selection.

Code

Polynomial overfit demo·python

import numpy as np
from numpy.polynomial import polynomial as P

# Fit polynomials of increasing degree — watch overfitting
np.random.seed(0)
x = np.linspace(0, 1, 10)
y = np.sin(2 * np.pi * x) + np.random.normal(0, 0.1, 10)

x_test = np.linspace(0, 1, 100)
y_test = np.sin(2 * np.pi * x_test)        # the truth

for degree in [1, 3, 9]:
    coeffs = np.polyfit(x, y, degree)
    train_pred = np.polyval(coeffs, x)
    test_pred  = np.polyval(coeffs, x_test)
    train_mse = np.mean((y - train_pred) ** 2)
    test_mse  = np.mean((y_test - test_pred) ** 2)
    print(f"degree {degree}: train MSE = {train_mse:.4f}, test MSE = {test_mse:.4f}")
# Degree 9 fits training near perfectly but explodes on test — classic overfit.

Exercise

Generate 20 points along y = sin(2πx) + small noise. Fit polynomials of degree 1, 3, 9, 15. For each, plot the fit on a fine grid and compute test MSE. Notice how test MSE shoots up at high degrees — that's overfitting visually.

Hint

Higher degree = more flexibility = better training fit but worse generalization. The 'right' degree depends on the noise level and how much data you have.

Excellent Fit, Poor Generalization

Why It Happens

Ways to Improve Generalization

Code

External links

Exercise

Progress

Comments 0