Overfitting and Underfitting

The two failure shapes

Overfitting means training loss is small while validation loss is much larger: the model learned sample noise. Underfitting means both losses are poor and similar: the model or representation cannot capture enough structure. Diagnose by plotting both on the same scale rather than guessing from one validation score.

Check the data boundary first

A large gap can also come from a time or group distribution mismatch, and excellent validation with bad production may signal leakage. Confirm the split, metric, and preprocessing path before prescribing a model change. Otherwise you optimize the symptom of a broken experiment.

Knobs that fight each failure

Failure	Useful knobs
Overfitting	more representative data, more regularization, a simpler model, early stopping, dropout, fewer features
Underfitting	more capacity, better representations, justified interactions, less regularization

Learning curves

Plot training and validation loss as data size grows. If validation keeps improving, more data can help. A persistent large gap suggests variance and regularization. Two high curves converging to a plateau suggest bias and a need for better features or capacity.

Complexity curves

Vary depth, regularization, or training iterations and plot both scores. The point where training continues to improve while validation reverses marks overfitting. Prefer a stable region around the optimum over a fragile peak.

Early stopping is a cheap regularizer

For boosting, monitor an independent validation set and stop when it has not improved for a fixed patience. Reusing the same validation set for many searches still adapts to it, so preserve a final test set and record the chosen iteration.

Change one cause at a time

Do not add data, features, regularization, and a new model in one experiment. Choose the dominant failure hypothesis, change one major lever, and rerun on the same split. A wrong prediction from the hypothesis is useful evidence for the next diagnosis.

Code

Learning curve diagnostic·python

from sklearn.model_selection import learning_curve
import numpy as np

sizes, train_scores, val_scores = learning_curve(
    pipe, X, y, cv=5, scoring="average_precision",
    train_sizes=np.linspace(0.1, 1.0, 8), n_jobs=-1
)
print("train mean:", train_scores.mean(axis=1))
print("val mean:  ", val_scores.mean(axis=1))

Early stopping for boosting models·python

import lightgbm as lgb

model = lgb.LGBMClassifier(n_estimators=2000, learning_rate=0.03)
model.fit(
    X_tr, y_tr, eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(50)]
)