Cross-Validation

~28 min · cv, validation, model-selection

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Why a single split is not enough

One held-out validation set is a noisy estimator of model quality. With small or imbalanced data, the noise can be larger than the difference between models you are trying to choose between. K-fold cross-validation averages over multiple splits so the comparison is honest.

Three flavors that matter

StratifiedKFold — preserves class proportions per fold for classification.
GroupKFold — keeps all rows of one group (user, session, patient) inside the same fold.
TimeSeriesSplit — train on the past, validate on the future, never the other way.

Reading CV scores honestly

Always report the mean and the standard deviation across folds. If the std is half the gap between two candidate models, the comparison is noise. Increase folds, average over multiple seeds, or admit you cannot tell them apart yet.

Code

Stratified 5-fold cross-validation with std·python

from sklearn.model_selection import StratifiedKFold, cross_val_score
import numpy as np

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
scores = cross_val_score(pipe, X, y, cv=cv, scoring="average_precision")
print(f"PR-AUC = {scores.mean():.3f} ± {scores.std():.3f}")

Group-aware CV when rows belong to users·python

from sklearn.model_selection import GroupKFold

gkf = GroupKFold(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=gkf, groups=df["user_id"], scoring="roc_auc")

Time-respecting CV for forecasting·python

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5, gap=14)  # 14-day gap to avoid leakage
scores = cross_val_score(pipe, X_sorted, y_sorted, cv=tscv, scoring="neg_root_mean_squared_error")

External links

Exercise

Run StratifiedKFold (n_splits=5, shuffle=True) on your problem. Report mean and std. Then run with n_splits=10 and compare the std. Decide which fold count gives a stable enough estimate to compare two models that differ by 0.5%.