Bias-Variance Tradeoff

~26 min · bias, variance, theory

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The decomposition

Generalization error decomposes into bias squared (how far the average prediction is from truth), variance (how much predictions wobble across training sets), and irreducible noise. Models with too little capacity have high bias. Models with too much capacity have high variance. The sweet spot minimizes the sum.

How tools change the tradeoff

Regularization raises bias to lower variance.
More data lowers variance without changing bias.
Bagging (random forests) averages high-variance trees to reduce variance.
Boosting reduces bias by sequentially correcting errors; can overfit if unchecked.

Why it matters

You cannot eliminate both at once. Knowing which one dominates your current model tells you which knob to turn next. High train accuracy and low validation accuracy = variance problem. Both low = bias problem. The framework saves weeks of guessing.

Code

Empirically split bias from variance with bootstrap·python

import numpy as np
from sklearn.utils import resample

predictions = []
for _ in range(50):
    Xb, yb = resample(X_train, y_train, random_state=None)
    fit = pipe.fit(Xb, yb)
    predictions.append(fit.predict_proba(X_test)[:, 1])

preds = np.array(predictions)
mean_pred = preds.mean(axis=0)
var_pred = preds.var(axis=0)
print("mean variance across bootstrap models:", var_pred.mean())

Bagging to reduce a high-variance model·python

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=None),
    n_estimators=200, random_state=7, n_jobs=-1
).fit(X_train, y_train)

External links

Exercise

For your current best model, plot training score vs validation score across 5 random seeds. If the gap is large with low variance across seeds, you have bias. If the gap is small but validation score wobbles between seeds, you have variance. Pick one knob to fix it next.