Train, Validation, and Test Split

Three roles, three sets

Training data fits model parameters. Validation data tunes hyperparameters and selects candidates. Test data asks one final question: did the choices made on validation generalize? If a test score causes another feature or model change, the test set has become validation data and its estimate is no longer independent.

Random is not always the deployment reality

Random splitting is reasonable only when rows are independent and time is irrelevant. Stratify imbalanced classification labels when each set needs comparable prevalence. When several rows belong to one user, patient, or document, split by that entity so the model cannot recognize the same object on both sides.

When time matters, put the future later

For demand, churn, or fraud, train on earlier periods and validate on more recent periods. Random shuffling can teach the model future policy, prices, seasonality, or behavior and then ask it to predict the past. Rolling windows can show whether performance depends on one fortunate interval and can mirror the planned retraining cadence.

Validation wears out too

Comparing dozens of models on the same validation rows adapts the team's decisions to that set's noise. Record experiment count and decision logic. Use cross-validation when data is small, but still preserve an independent final test set after feature, model, and hyperparameter selection.

Freeze the test set

Store stable row or entity identifiers and the dataset version on day one. A random seed alone is insufficient because changed source data produces different rows with the same seed. If the data definition changes, create a new versioned dataset and test set. At final evaluation, report baseline difference, uncertainty or fold variability, and important subgroup errors—not one score alone.

The split is part of the deployment contract

Version the grouping unit, time boundary, generation query, identifiers, and data hash. These details state which unseen future the score is meant to represent and allow the same evaluation question to be reconstructed. The split is the history of the experiment, not disposable setup code.

Code

Stratified split for binary classification·python

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.30, stratify=y, random_state=7
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, stratify=y_temp, random_state=7
)

Group-aware split when multiple rows belong to one user·python

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=7)
train_idx, test_idx = next(splitter.split(X, y, groups=df["user_id"]))

Time-respecting split for forecasting·python

cutoff = "2026-04-01"
train = df[df["event_time"] < cutoff]
test  = df[df["event_time"] >= cutoff]

Three roles, three sets

Random is not always the deployment reality

When time matters, put the future later

Validation wears out too

Freeze the test set

The split is part of the deployment contract

Code

External links

Exercise

Progress

Comments 0