Train/Val/Test Split: The Discipline That Prevents Self-Deception

Three Datasets, Three Roles

Training set — the model sees this. Updates its weights.
Validation set — the model never trains on this. You use it to tune hyperparameters (model size, learning rate, regularization strength).
Test set — the model never trains on AND you never tune on. Used once, at the end, to estimate real-world performance.

Why All Three

If you only have train + test, you'll be tempted to tune hyperparameters by checking test performance — and the test set becomes secretly training data. Your final reported number will be optimistic. The validation set absorbs the tuning, leaving the test set genuinely held out.

Common Disasters

Data leakage — test data sneaking into training (often via preprocessing). Standardize using only training statistics, not the whole dataset.
Test set peeking — running the test set repeatedly during development. After enough peeks, your test set is contaminated.
Time-series leakage — for time data, the future leaking into the past. Always split chronologically for time series.

The Kaggle leaderboard effect. Top teams sometimes overfit the public leaderboard (essentially a validation set), then drop dramatically on the private (test) leaderboard. The math punishes peeking, even when peeks are aggregated.

The test set is sacred. Look at it once. Twice means you've corrupted your evaluation.

Code

Three-way split with scikit-learn·python

import numpy as np
from sklearn.model_selection import train_test_split

X = np.random.randn(1000, 10)
y = np.random.randint(0, 2, 1000)

# Two-step split: 60% train, 20% val, 20% test
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=42)

print(f"train: {len(X_train)}, val: {len(X_val)}, test: {len(X_test)}")

Exercise

With a tabular dataset (random or real), split into 60/20/20 train/val/test. Standardize using ONLY training-set statistics (mean, std). Apply the same transformation to val and test. Why is this important?

Hint

If you compute mean/std on the whole dataset, your validation/test sees information about its own distribution during preprocessing — that's leakage. Compute once on train, reuse the same numbers for val/test.

Train/Val/Test Split: The Discipline That Prevents Self-Deception

Three Datasets, Three Roles

Why All Three

Common Disasters

Code

External links

Exercise

Progress

Comments 0