Calibration

A score of 0.8 should mean eight in ten

A probabilistic classifier is calibrated when cases scored near 0.8 are positive about 80% of the time in the evaluated population. Calibration differs from ranking: two models can have similar ROC-AUC while one produces far more trustworthy probabilities.

Calibration matters when a decision uses probability

Expected value, risk tiers, resource allocation, and cost-based thresholds interpret score magnitude. No algorithm is guaranteed to stay calibrated: specification errors, regularization, class rebalancing, and population shift can distort logistic regression, trees, or boosting. Measure rather than assume.

Reliability curves show honesty by score range

Group predictions and compare each bin's mean score with its observed positive rate. Bin choice affects the picture, especially for rare outcomes, so show counts and consider quantile bins. Inspect critical slices instead of relying only on the overall diagonal.

Brier score summarizes squared probability error

Lower Brier score is better, but the score combines calibration and resolution. Log loss offers another view and punishes confident mistakes strongly. Use diagrams and at least one proper scoring rule rather than reducing probability quality to one number.

Platt scaling fits a smooth sigmoid

Sigmoid calibration learns a logistic mapping from raw scores to probabilities and is useful when miscalibration has an approximately S-shaped pattern. Fit it on data not used to train the base estimator.

Isotonic regression is flexible but data-hungry

Isotonic calibration learns a monotone piecewise mapping without assuming a sigmoid. It can match complex distortions but may overfit when calibration samples are few, so compare it on held-out evidence.

Keep calibration inside the validation boundary

CalibratedClassifierCV can create out-of-fold scores for fitting a calibrator. The final test set must remain outside model, calibrator, and threshold selection. Choose the operating threshold after calibration using explicit cost and capacity.

Recheck calibration when the environment changes

Calibration depends on population and time. A prevalence shift can invalidate probabilities even when rank ordering stays stable. Monitor reliability when delayed labels arrive, report uncertainty for small bins, and recalibrate only with a new protected evaluation.

Code

Reliability curve and Brier score·python

from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss
import matplotlib.pyplot as plt

frac_pos, mean_pred = calibration_curve(y_val, probs, n_bins=10)
print("Brier:", brier_score_loss(y_val, probs))
plt.plot([0, 1], [0, 1], "k--")
plt.plot(mean_pred, frac_pos, marker="o")
plt.xlabel("predicted probability")
plt.ylabel("empirical positive rate")
plt.show()

Calibrate with isotonic regression on a held-out fold·python

from sklearn.calibration import CalibratedClassifierCV

calibrated = CalibratedClassifierCV(model, method="isotonic", cv="prefit")
calibrated.fit(X_calib, y_calib)
probs_calibrated = calibrated.predict_proba(X_val)[:, 1]