Drift Monitoring

Data starts moving as soon as a model is deployed

User populations, policies, devices, seasons, and upstream systems change. Drift is a family of signals, not one diagnosis. A marketing campaign may change inputs harmlessly, while a broken currency conversion can mimic drift but is an incident.

Input drift

Feature distributions change relative to a reference window. Monitor schema, missingness, category coverage, ranges, and units before statistical distances. Kolmogorov-Smirnov statistics, population stability index, or divergence measures summarize change but do not measure business impact by themselves.

Prediction drift

Score, probability, decision, or alert-volume distributions change. This can reveal upstream or policy changes before labels arrive, but does not prove accuracy deteriorated. Track these outputs with the active model and feature versions.

Concept drift

The relationship between features and the true outcome changes. Confirming it requires delayed labels joined by stable prediction IDs. Report the deployed metric, calibration, and errors across important slices and time rather than inferring concept drift from inputs alone.

Version the reference window

Choose a baseline that represents the intended population and a recent window with enough observations. Record dates, filters, feature definitions, model version, and expected seasonality. Thresholds depend on sample size and behavior; a p-value is not an operational severity level.

Every alert threshold needs an action

Define owner, severity, supporting evidence, and runbook before enabling an alarm. Route schema breaks differently from modest population shifts. A dashboard that cannot tell anyone what to inspect or decide is only persistent visual noise.

Retraining is not an automatic cure

Investigate pipeline failures, label delays, and policy changes first. Retrain on a cadence only when change rate, label arrival, and validation evidence justify it; weekly is not a universal default. Without lineage, retraining can reproduce the same defect.

Validate and release the replacement gradually

Compare a candidate with the current model on a recent deployment-shaped holdout. Verify rollback and use shadow or canary release when consequences warrant it. Monitor both immediate operating signals and delayed outcomes before promotion.

Code

PSI to compare two windows·python

import numpy as np

def psi(reference, current, bins=10):
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    p_ref, _ = np.histogram(reference, bins=edges)
    p_cur, _ = np.histogram(current, bins=edges)
    p_ref = np.clip(p_ref / p_ref.sum(), 1e-6, None)
    p_cur = np.clip(p_cur / p_cur.sum(), 1e-6, None)
    return ((p_cur - p_ref) * np.log(p_cur / p_ref)).sum()

for col in numeric_cols:
    score = psi(reference[col], current[col])
    if score > 0.20:  # >0.10 = caution, >0.25 = serious
        print(f"⚠️  PSI {col} = {score:.3f}")

Output drift via predicted-probability shift·python

from scipy.stats import ks_2samp

stat, pval = ks_2samp(reference_probs, current_probs)
if pval < 0.01:
    alert("output distribution drifted", stat=stat, pval=pval)