Overfitting and Underfitting

~26 min · overfitting, underfitting, diagnostics

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The two failure shapes

Overfitting: training loss is small, validation loss is much larger. The model memorized noise. Underfitting: training loss and validation loss are both poor. The model lacks capacity or the wrong representation. Diagnose by plotting both losses on the same chart.

Knobs that fight each failure

Failure	Knob
Overfitting	more data, more regularization, simpler model, early stopping, dropout, fewer features
Underfitting	more capacity, better features, more interactions, less regularization

Learning curves

Plot training and validation loss as data size grows. Both still falling means you need more data. Validation flat while training drops means more capacity will not help; you need regularization or features. The curves are a 30-second diagnosis.

Code

Learning curve diagnostic·python

from sklearn.model_selection import learning_curve
import numpy as np

sizes, train_scores, val_scores = learning_curve(
    pipe, X, y, cv=5, scoring="average_precision",
    train_sizes=np.linspace(0.1, 1.0, 8), n_jobs=-1
)
print("train mean:", train_scores.mean(axis=1))
print("val mean:  ", val_scores.mean(axis=1))

Early stopping for boosting models·python

import lightgbm as lgb

model = lgb.LGBMClassifier(n_estimators=2000, learning_rate=0.03)
model.fit(
    X_tr, y_tr, eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(50)]
)

External links

Exercise

Plot the learning curve of your best model. From the shape, decide which is the bigger payoff: more data, more capacity, or more regularization. Do that one thing next, not all three.