Selection Bias: Your Sample Was Lying to You

"The data you analyzed is not a random sample of the population you cared about. Almost no data is. Knowing the filter is the citizen-statistician's primary defense."

What Selection Bias Is

Selection bias occurs when the process that produced your sample is correlated with the outcome you are trying to measure. The classic forms:

Self-selection: subjects volunteer to participate. Volunteers differ from non-volunteers in ways that often correlate with the outcome.
Non-response bias: subjects refuse to answer. The refusers differ from the responders.
Survivorship: only some subjects are reachable for measurement (already covered in lesson 3).
Geographic / demographic filtering: the sample only includes people from places or groups that are accessible to the researcher.
Convenience sampling: the researcher uses 'whoever is around,' which is rarely a representative slice.

Each of these produces a sample whose composition is no longer the population's, and inferences drawn from the sample do not generalize.

The 1948 Election Polling Disaster

In the 1948 US presidential election, every major pre-election poll predicted Dewey would defeat Truman. The polls were spectacularly wrong. The cause was selection bias: polls were conducted by telephone, and in 1948, telephones were disproportionately held by wealthier, more Republican voters. The poll sample was systematically skewed toward Republican-leaning respondents, and the projection from the sample to the population was therefore biased. The 'Dewey Defeats Truman' headline became one of the most famous photographs in journalism history, and the methodology of political polling changed substantially as a result.

The Modern Form: Online Polls and 'Engagement'

Modern online polls and social-media 'sentiment' metrics suffer the same selection bias. People who fill out online polls are not a random slice of the population; they are more engaged, more opinionated, more online. Polls that draw from one platform (Twitter/X versus Reddit versus Instagram) draw from different demographic and ideological distributions. Aggregating across platforms or weighting carefully can partially correct, but the underlying filter never fully disappears. 'Internet sentiment' is the sentiment of the people who chose to express it, which is not the same as the sentiment of the population.

The Operating Principle

Before drawing inferences from any sample, ask: what filtering process produced this sample? Who is in it, and who is excluded? Does the filter correlate with the outcome being measured? If yes, the sample is biased and inferences from it do not generalize to the broader population without correction. Most public-facing data — polls, surveys, online sentiment, customer feedback, app reviews — suffers from substantial selection bias, often unnamed in the headline. Naming the filter is the citizen's first sanity check.

Code

Selection bias: response correlates with outcome·python

import numpy as np
rng = np.random.default_rng(270)

# Simulate a population with a true 50/50 split on some question.
# But people who feel strongly about it (either way) are 3x more likely
# to respond to a poll. The poll sample is biased toward strong opinions.

N = 100_000
true_opinion = rng.choice(['yes', 'no'], size=N, p=[0.5, 0.5])
strength = rng.choice(['mild', 'strong'], size=N, p=[0.7, 0.3])

# Response probability depends on strength.
response_prob = np.where(strength == 'strong', 0.6, 0.2)
responded = rng.random(N) < response_prob

sample = true_opinion[responded]
yes_pct_sample = (sample == 'yes').mean() * 100
yes_pct_true = (true_opinion == 'yes').mean() * 100

print(f"True population yes-rate:    {yes_pct_true:.2f}%")
print(f"Poll sample yes-rate:        {yes_pct_sample:.2f}%")
# In this sim the yes rates happen to be similar because the filter is on
# strength rather than on direction. Now change strength to correlate with
# direction (people who say 'yes' are more passionate)...
response_prob2 = np.where(
    (true_opinion == 'yes') & (strength == 'strong'), 0.7,
    np.where((true_opinion == 'no') & (strength == 'strong'), 0.5,
             0.2)
)
responded2 = rng.random(N) < response_prob2
sample2 = true_opinion[responded2]
yes_pct_sample2 = (sample2 == 'yes').mean() * 100
print(f"\nWhen 'yes' supporters respond more (selection bias):")
print(f"Poll sample yes-rate:        {yes_pct_sample2:.2f}%")
print(f"True yes-rate:               {yes_pct_true:.2f}%")
# The poll now over-estimates 'yes' simply because 'yes' voters are more
# willing to respond. This is selection bias in plain numbers.

Exercise

Pick a recent dataset you've encountered (a poll, a survey, online sentiment, an app rating, a customer feedback report). Identify the selection filter: who is in the sample, and who is excluded? Does the filter correlate with the outcome being measured? If yes, how would the headline change if the missing groups were represented?

Hint

Almost all online data is selection-biased toward the engaged, opinionated, and willing-to-click. The unrepresented are the silent majority, and their absence systematically skews most data you read.

What Selection Bias Is

The 1948 Election Polling Disaster

The Modern Form: Online Polls and 'Engagement'

The Operating Principle

Code

External links

Exercise

Progress

Comments 0