Cross-Validation

One split contains too much luck

A single validation set is a noisy estimate. With small or imbalanced data, which examples land in validation can matter more than the gap between candidates. K-fold cross-validation rotates the held-out fold, yielding a mean performance estimate and a view of variability.

Put the entire pipeline inside each fold

Imputation, scaling, encoding, feature selection, and resampling must be fitted on the training portion of every fold. Preprocessing the full dataset first leaks validation statistics. Cross-validation protects the boundary only when it receives the raw-data pipeline.

Use stratification for classification

StratifiedKFold keeps class proportions similar across folds and reduces folds with too few rare positives to evaluate. It does not solve repeated-entity leakage or time travel. If the number of positives is smaller than K, reduce the fold count or obtain more data.

Keep related rows in one group

Rows from one user, patient, session, or document must not cross training and validation boundaries. Use GroupKFold to measure generalization to unseen groups, or consider StratifiedGroupKFold when both group integrity and class balance matter.

Move from past to future for temporal problems

TimeSeriesSplit trains on earlier periods and validates on later ones. Observation and target windows may require a gap so overlapping events cannot leak across the boundary. Scores by time fold also reveal whether the environment is changing.

Read mean and variability together

Report every fold score, mean, and standard deviation. If variability is similar to the model gap, the comparison is unresolved. Repeat where sample counts allow, inspect the worst fold, or declare a tie and prefer the simpler model.

Keep a final test set

Model and hyperparameter choices adapt to cross-validation results. After all choices are frozen, evaluate once on an independent test set. Cross-validation reduces split luck; it does not make evidence immune to repeated selection.

Code

Stratified 5-fold cross-validation with std·python

from sklearn.model_selection import StratifiedKFold, cross_val_score
import numpy as np

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
scores = cross_val_score(pipe, X, y, cv=cv, scoring="average_precision")
print(f"PR-AUC = {scores.mean():.3f} ± {scores.std():.3f}")

Group-aware CV when rows belong to users·python

from sklearn.model_selection import GroupKFold

gkf = GroupKFold(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=gkf, groups=df["user_id"], scoring="roc_auc")

Time-respecting CV for forecasting·python

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5, gap=14)  # 14-day gap to avoid leakage
scores = cross_val_score(pipe, X_sorted, y_sorted, cv=tscv, scoring="neg_root_mean_squared_error")