Target Leakage

~32 min · leakage, splits, evaluation

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

What leakage is

Target leakage is when the feature set contains information that would not exist when the prediction is made. It is the single most common reason ML models look great in the notebook and disappoint in production. Leakage produces beautiful validation numbers and a model that cannot be deployed.

Three flavors

Time leakage — using future aggregates (last-30-days revenue computed across the entire dataset).
Post-event leakage — keeping columns that only exist after the outcome (cancel reason, refund amount, support call about the very issue).
Preprocessing leakage — fitting a scaler, imputer, or encoder on the full dataset before splitting. Train statistics now know the test set.

Defense

Build features inside the pipeline, not before it. Split first, fit transformers on training only, and audit any column whose absence drops validation score by an unrealistic amount. If a single feature carries the model, it is leakage until proven otherwise.

Code

Wrong: scaler sees the test set·python

# DO NOT DO THIS
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit(X)  # fit on full dataset → leakage
X_scaled = scaler.transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y)

Right: scaler lives inside the pipeline·python

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=500)),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="average_precision")

External links

Exercise

Audit the top five features by importance in your current model. For each, write a one-sentence proof that the feature exists at prediction time. Throw out anything you cannot prove and retrain.