Calibration

~28 min · calibration, probabilities

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

What calibration means

A classifier is calibrated when, among examples it scored at probability p, the empirical positive rate is also p. Logistic regression is approximately calibrated by construction. Boosted trees and random forests are usually overconfident at the extremes and need calibration. Calibration matters whenever a downstream system uses the probability (cost-weighted thresholds, cascades, expected value).

Diagnostic: the reliability curve

Bin predictions into deciles, compute the empirical positive rate per bin, and plot it against the predicted probability. The closer the line is to y = x, the better the calibration. The Brier score is the single-number summary; lower is better.

Two practical fixes

Platt scaling — fit a logistic regression on raw scores; works well with sigmoid-shaped miscalibration.
Isotonic regression — fits a non-parametric monotone curve; needs more data but more flexible.

Both are wrapped by sklearn's CalibratedClassifierCV.

Code

Reliability curve and Brier score·python

from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss
import matplotlib.pyplot as plt

frac_pos, mean_pred = calibration_curve(y_val, probs, n_bins=10)
print("Brier:", brier_score_loss(y_val, probs))
plt.plot([0, 1], [0, 1], "k--")
plt.plot(mean_pred, frac_pos, marker="o")
plt.xlabel("predicted probability")
plt.ylabel("empirical positive rate")
plt.show()

Calibrate with isotonic regression on a held-out fold·python

from sklearn.calibration import CalibratedClassifierCV

calibrated = CalibratedClassifierCV(model, method="isotonic", cv="prefit")
calibrated.fit(X_calib, y_calib)
probs_calibrated = calibrated.predict_proba(X_val)[:, 1]

External links

Exercise

Plot the reliability curve for your classifier and report the Brier score. If the curve sags away from the diagonal, fit CalibratedClassifierCV(method='isotonic') on a held-out fold and re-plot. Decide whether calibration changes any downstream decision.