Target Leakage

Leakage steals information from the future

Target leakage occurs when features contain information that would not exist when a prediction is made. It creates beautiful validation numbers and a model that cannot be deployed. The danger is that the defect improves the metric instead of breaking the program, so the team may celebrate after the experiment has already become invalid.

Time leakage

A “last 30 days” aggregate can include future events if it is computed across the full dataset instead of being cut off at each row's prediction time. Define an as-of timestamp for every example and aggregate only earlier events. The same time rule must be reproducible during retraining and serving.

Post-event leakage

Cancellation reason, refund amount, or a support call caused by the cancellation may be valid columns that appear only after the target event. Ask when each value is first written in the source workflow; its name alone cannot prove that it was available to the decision.

Preprocessing leakage

Fitting an imputer, scaler, category vocabulary, or feature selector before splitting lets validation and test statistics influence training even if the target column is never read directly. Split first, fit transformations on training rows, and repeat that fitting independently inside every cross-validation fold.

Group leakage

Rows from the same customer, device, document, or video can leak identity across a random split. The model then recognizes an entity instead of generalizing to a new one. Define the independent unit and keep every row from that unit on one side of the split.

Defend with timelines and pipelines

Record creation and availability times for every feature, keep learned preprocessing with the model, and audit any column whose removal causes an implausibly large score drop. Reproduce a real scoring request from only the values available at that moment. If leakage is found, do not discount or adjust the old score: repair the feature and split boundaries and rerun from the beginning. Leakage invalidates the experiment rather than merely reducing its performance.

Code

Wrong: scaler sees the test set·python

# DO NOT DO THIS
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit(X)  # fit on full dataset → leakage
X_scaled = scaler.transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y)

Right: scaler lives inside the pipeline·python

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=500)),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="average_precision")

Leakage steals information from the future

Time leakage

Post-event leakage

Preprocessing leakage

Group leakage

Defend with timelines and pipelines

Code

External links

Exercise

Progress

Comments 0