Population vs Sample: The Glimpse Problem

The Whole Group and the Part You Observed

A population is the complete set addressed by a question: voters in an election, Seoul adults whose commute is being studied, or requests a deployed service will receive. A sample is the subset actually observed. The population should be defined by the question, not inflated to “all data that could ever exist.”

If a question concerns all Seoul adults but the sampling frame contains only users of one app, the mismatch matters more than raw sample size.

A Large Sample Is Not Enough

With appropriate random sampling, a larger sample reduces chance variation in estimates such as a mean. It does not automatically remove selection bias, nonresponse bias, dependence, or measurement error. A survey can question a million systematically unrepresentative people and become precisely wrong.

Representation: are important subgroups missing?
Selection: who had what probability of entering the sample?
Independence: are many observations effectively duplicates, such as one household or a bot network?
Distribution shift: will deployment conditions differ from collection conditions?

In Machine Learning

Training, validation, and test data are samples from the environment in which the model will be used. Generalization depends on collection pathways, label quality, temporal change, and coverage—not only row count. Data quality includes both sample size and the process that generated the sample.

A sample is the result of a sampling process, not automatically a miniature population. Ask who entered and who was excluded before counting rows.

Code

Estimation as sample size grows·python

import numpy as np

# True population mean (which you don't know in real life)
population = np.random.normal(loc=100, scale=15, size=1_000_000)
true_mean = population.mean()
print(f"true mean: {true_mean:.3f}")

# Take a sample of 30 — what do we estimate?
sample = np.random.choice(population, size=30)
print(f"sample mean (n=30): {sample.mean():.3f}")

# Take a bigger sample
sample = np.random.choice(population, size=10_000)
print(f"sample mean (n=10000): {sample.mean():.3f}")
# Bigger samples → estimates converge to the truth

Exercise

Generate a population of 1,000,000 random numbers. Draw random samples of size 10, 100, 1,000, and 10,000, then compute each mean. Compare the wobble of small samples with the stability of large ones.

Hint

np.random.choice(population, size=n) draws a sample. As n grows, the estimate should stabilize near the population mean—the law of large numbers at work. A biased sampling process is not repaired by size alone.

The Whole Group and the Part You Observed

A Large Sample Is Not Enough

In Machine Learning

Code

External links

Exercise

Progress

Comments 2