Bias-Variance Tradeoff

The decomposition

Imagine repeatedly drawing a new training sample from the same world. Bias measures how far the average model prediction sits from the true relationship. Variance measures how much predictions change across samples. Irreducible noise is the randomness no available features and model can remove. Under squared loss, these components organize generalization error.

High bias misses structure

A shallow tree or an untransformed linear model on a curved relationship can make the same systematic mistake across samples. Training and validation performance are both poor and close together. Better features, more capacity, interactions, or weaker regularization can lower bias.

High variance follows sample accidents

A deep tree may fit training almost perfectly but change completely when a few rows move. Training performance is high, validation is lower, and results wobble across folds and seeds. More representative data, stronger regularization, simplification, and bagging can lower variance.

How tools change the tradeoff

Regularization accepts some bias to reduce variance.
More data mainly lowers variance for a fixed model family.
Bagging averages high-variance learners such as trees.
Boosting sequentially corrects residual structure to reduce bias, but can overfit if unchecked.

Why random forests work

Bootstrap samples and random feature subsets make individual trees different. Averaging cancels some of their sample-specific errors. Feature randomness matters because averaging nearly identical trees would provide little variance reduction.

Why boosting works

Boosting does not average independent trees. Each new weak learner attacks the loss gradient left by the ensemble, increasing representational power. A small learning rate and early stopping keep that bias reduction from continuing into noise.

Use the frame to pick one experiment

High training performance with a validation gap suggests variance; poor performance on both suggests bias. Distribution shift and leakage can imitate these shapes, so validate the data boundary first. Then choose one lever matched to the dominant error instead of tuning blindly.

Code

Empirically split bias from variance with bootstrap·python

import numpy as np
from sklearn.utils import resample

predictions = []
for _ in range(50):
    Xb, yb = resample(X_train, y_train, random_state=None)
    fit = pipe.fit(Xb, yb)
    predictions.append(fit.predict_proba(X_test)[:, 1])

preds = np.array(predictions)
mean_pred = preds.mean(axis=0)
var_pred = preds.var(axis=0)
print("mean variance across bootstrap models:", var_pred.mean())

Bagging to reduce a high-variance model·python

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=None),
    n_estimators=200, random_state=7, n_jobs=-1
).fit(X_train, y_train)