Normalize vs Standardize: Two Ways to Tame Numbers

Two Ways to Align Different Scales

Pixels may range from 0 to 255, ages span tens, and incomes can reach millions. For methods driven by distances or gradients, a large numerical scale can give one feature disproportionate influence. Two common remedies are min-max normalization and z-score standardization.

Min-max normalization: $x^{'} = (x - x_{min}) / (x_{ma x} - x_{min})$ . It maps the training data's minimum and maximum to 0 and 1.
Z-score standardization: $z = (x - μ) / σ$ . It gives the training data mean 0 and standard deviation 1.

Neither Operation Changes the Distribution's Shape

Both are affine transformations: they shift values and divide by a positive constant. A right-skewed distribution remains right-skewed, and a bimodal distribution remains bimodal. Z-score standardization does not turn arbitrary data into a standard normal distribution; it only matches that distribution's mean and standard deviation.

Min-max values can fall below 0 or above 1 when new observations lie outside the training range. Z-scores are unbounded from the start, and both methods are sensitive to outliers. For severe outliers, a robust scaler based on the median and interquartile range may be a better fit.

Prevent Data Leakage

Compute minima, maxima, means, and standard deviations on the training set only, then reuse those values for validation and test data. Fitting the scaler on the full dataset leaks evaluation-distribution information into the training pipeline.

Scaling Is Model-Dependent

Neural networks, k-nearest neighbors, SVMs, and linear models often benefit from aligned feature scales. Decision-tree families are comparatively insensitive to monotonic rescaling because they split on per-feature thresholds. Choose the transformation for the model and data-generating process.

Normalization aligns a range; standardization aligns a center and scale. Neither magically makes non-normal data normal.

Code

Both flavors in 4 lines·python

import numpy as np

img = np.random.randint(0, 256, size=(64, 64, 3)).astype(np.float32)

# Min-max normalization → [0, 1]
img_norm = img / 255.0
print(img_norm.min(), img_norm.max())   # 0.0  1.0 (or close)

# Z-score standardization → mean 0, std 1
mean, std = img.mean(), img.std()
img_std = (img - mean) / std
print(img_std.mean().round(3), img_std.std().round(3))  # ~0.0  ~1.0

PyTorch flavor·python

import torch

# Same operations as PyTorch tensors — same vibe, GPU-ready
x = torch.randn(1000) * 50 + 100   # mean 100, std 50, very un-normalized

x_norm = (x - x.min()) / (x.max() - x.min())  # [0, 1]
x_std  = (x - x.mean()) / x.std()             # mean 0, std 1

Exercise

Take a 1-D NumPy array of 100 random integers in [0, 1000]. Apply (a) min-max normalization, (b) z-score standardization. Print mean & std for each. Which one has a guaranteed range? Which one has a guaranteed mean?

Hint

np.random.randint(0, 1001, size=100). Then divide and (subtract-then-divide). The 'guaranteed' answers are 'normalization' and 'standardization' respectively — but stating it isn't the lesson; *seeing* it on numbers is.