P-Hacking and Multiple Comparisons

"With enough comparisons, the bell curve will hand you significance by accident. The replication crisis is what happens when this is mistaken for discovery."

The Mechanism

Under a true null hypothesis, ~5% of statistical tests come up p < 0.05 by random chance alone (the Type I error rate at α = 0.05). If you run 20 independent tests against the null, expectation is one of them will be 'significant' even though nothing real is happening.

'P-hacking' is exploiting this — explicitly or accidentally — by running many tests, choosing the one that hit p < 0.05, and reporting only that one. Once that selection process is hidden, the reported p-value no longer has its advertised evidential interpretation for the overall procedure.

The Common Forms

Trying many outcome measures: 'we measured 15 different things; one was statistically significant.' Unsurprising.
Trying many subgroups: 'overall no effect, but in left-handed women aged 30-40 there was a p < 0.05 result.'
Trying many model specifications: 'using these covariates and excluding these outliers, the effect becomes significant.'
Stopping data collection when results favor you: 'we ran the study until p crossed 0.05 and then stopped.'
The 'garden of forking paths': researchers make many small analytical decisions that, together, amount to running many implicit tests — even without intending to p-hack.

The Multiple-Comparison Correction

When you must run many tests, choose a procedure that controls the error criterion relevant to the study. Bonferroni controls family-wise error by using α/M for M planned tests—0.0025 per test for 20 tests at family-wise α = 0.05. False-discovery-rate procedures such as Benjamini–Hochberg answer a different error-control question; they are not simply “better” in every setting.

Pre-registration is the social fix: announce in advance what you will measure, how you will measure it, and what test you will run. Then any 'exploratory' analysis is labeled as such and cannot masquerade as confirmatory.

The Citizen's Sniff Test

When a result is presented as 'one significant finding,' ask how many outcomes, subgroups, stopping rules, and model choices were available. Selective reporting is one contributor to replication problems, alongside low power, measurement error, publication bias, model misspecification, and other incentives. Pre-registration and multiplicity-aware analysis help make the analysis path visible.

Code

False-positive inflation under multiple tests·python

import numpy as np
from math import erf, sqrt
rng = np.random.default_rng(140)

# Show that with enough tests under a true null, 'significance' becomes
# almost guaranteed somewhere — pure chance, no real effect.

N_per_test = 200
M_tests_per_experiment = 20
M_experiments = 5_000
p_true = 0.5     # null is true everywhere

# Track: in each experiment of 20 tests, how often does AT LEAST ONE hit p<0.05?
hit_rate_at_alpha_05 = 0
for _ in range(M_experiments):
    significant = False
    for _ in range(M_tests_per_experiment):
        flips = rng.binomial(n=1, p=p_true, size=N_per_test)
        p_hat = flips.mean()
        z = (p_hat - 0.5) / np.sqrt(0.5 * 0.5 / N_per_test)
        p_val = 2 * 0.5 * (1 - erf(abs(z) / sqrt(2)))
        if p_val < 0.05:
            significant = True
            break
    if significant:
        hit_rate_at_alpha_05 += 1

print(f"P(at least one of 20 tests hits p<0.05 under true null): "
      f"{hit_rate_at_alpha_05 / M_experiments:.4f}")
print(f"Expected: 1 - 0.95^20 = {1 - 0.95**20:.4f}")
print(f"\nBonferroni-corrected threshold for 20 tests: alpha = 0.05/20 = 0.0025")
# About 64% of 'experiments' that ran 20 null tests will have at least one
# 'significant' result — pure noise, no real effect. That is the multiple-
# comparison hazard in plain numbers.

Exercise

Identify a recent surprising scientific finding you remember from news (a coffee study, a wine study, a nutrition study, a behavioral finding). Ask: was this a pre-registered confirmatory study, or an exploratory analysis where the researcher looked at many variables? If exploratory, the reported p-value is almost certainly overstating the strength of evidence. Most non-replicating findings come from this category.

Hint

Pre-registered studies tend to be much more replicable. Exploratory analyses are useful for hypothesis generation but should not be reported as if they were definitive.

The Mechanism

The Common Forms

The Multiple-Comparison Correction

The Citizen's Sniff Test

Code

External links

Exercise

Progress

Comments 0