Drift Monitoring

~28 min · drift, monitoring, production

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Three things drift

Input drift — the feature distribution shifts (e.g., a marketing campaign changes the user mix).
Output drift — the prediction distribution shifts even when inputs look stable.
Concept drift — the relationship between features and target changes (the world changed).

How to monitor each

Input drift: per-feature distance metrics (Kolmogorov-Smirnov, Population Stability Index) compared to a reference window. Output drift: the moving distribution of predicted probabilities. Concept drift: rolling window of evaluation metrics on labeled production data.

Triggering retraining

Set thresholds on each metric and route them through your alerting system. Retrain on a cadence (weekly is a sane default) and ad-hoc when a drift alert fires. Always shadow-deploy a new model before promoting it.

Code

PSI to compare two windows·python

import numpy as np

def psi(reference, current, bins=10):
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    p_ref, _ = np.histogram(reference, bins=edges)
    p_cur, _ = np.histogram(current, bins=edges)
    p_ref = np.clip(p_ref / p_ref.sum(), 1e-6, None)
    p_cur = np.clip(p_cur / p_cur.sum(), 1e-6, None)
    return ((p_cur - p_ref) * np.log(p_cur / p_ref)).sum()

for col in numeric_cols:
    score = psi(reference[col], current[col])
    if score > 0.20:  # >0.10 = caution, >0.25 = serious
        print(f"⚠️  PSI {col} = {score:.3f}")

Output drift via predicted-probability shift·python

from scipy.stats import ks_2samp

stat, pval = ks_2samp(reference_probs, current_probs)
if pval < 0.01:
    alert("output distribution drifted", stat=stat, pval=pval)

External links

Exercise

Pick the five most important features for your model. Compute PSI between your training window and the most recent two weeks of production data. For each feature with PSI > 0.10, write a one-sentence hypothesis for the cause.