Train, Validation, and Test Split

~30 min · splits, validation, evaluation

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Three roles, three sets

Training data fits the model. Validation data tunes hyperparameters and picks between models. Test data answers one question: did the choice you made on validation generalize? If you ever look at the test score before the final model is frozen, you have just turned the test set into another validation set and lost the only honest estimate you had.

Stratify, group, time-respect

For classification with class imbalance, stratify the split by the label. For grouped data (multiple rows per user), split by user. For time-series, split by time and never let future rows enter training. Mixing strategies is how leaderboards get gamed and production systems get embarrassed.

The frozen test set ritual

Pickle the indices of the test set on day one and never recompute them. If the data changes, build a new dataset, do not slide the test set. The frozen-test-set ritual is the difference between a calibrated team and a team that lies to itself.

Code

Stratified split for binary classification·python

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.30, stratify=y, random_state=7
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, stratify=y_temp, random_state=7
)

Group-aware split when multiple rows belong to one user·python

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=7)
train_idx, test_idx = next(splitter.split(X, y, groups=df["user_id"]))

Time-respecting split for forecasting·python

cutoff = "2026-04-01"
train = df[df["event_time"] < cutoff]
test  = df[df["event_time"] >= cutoff]

External links

Exercise

For your dataset, decide which split strategy is correct: random stratified, grouped, or time-respecting. Write the code that produces train/val/test indices. Save the test indices to disk and treat them as immutable.