P-Hacking and Multiple Comparisons

"With enough comparisons, the bell curve will hand you significance by accident. The replication crisis is what happens when this is mistaken for discovery."

The Mechanism

Under a true null hypothesis, ~5% of statistical tests come up p < 0.05 by random chance alone (the Type I error rate at α = 0.05). If you run 20 independent tests against the null, expectation is one of them will be 'significant' even though nothing real is happening.

'P-hacking' is exploiting this — explicitly or accidentally — by running many tests, choosing the one that hit p < 0.05, and reporting only that one. The reported p-value is technically correct for the single test it describes. The selection process — looking at many and reporting one — is the problem. The published p-value omits the fact that the researcher rolled the dice many times before reading the result.

The Common Forms

Trying many outcome measures: 'we measured 15 different things; one was statistically significant.' Unsurprising.
Trying many subgroups: 'overall no effect, but in left-handed women aged 30-40 there was a p < 0.05 result.'
Trying many model specifications: 'using these covariates and excluding these outliers, the effect becomes significant.'
Stopping data collection when results favor you: 'we ran the study until p crossed 0.05 and then stopped.'
The 'garden of forking paths': researchers make many small analytical decisions that, together, amount to running many implicit tests — even without intending to p-hack.

The Multiple-Comparison Correction

The fix, when you must run many tests, is to correct the threshold for the number of comparisons. The Bonferroni correction is the simplest: if you run M tests, lower your per-test threshold to α/M. So 20 tests, individually at α = 0.05/20 = 0.0025 each. This is conservative; better procedures (False Discovery Rate, Benjamini-Hochberg) exist for high-dimensional settings.

Pre-registration is the social fix: announce in advance what you will measure, how you will measure it, and what test you will run. Then any 'exploratory' analysis is labeled as such and cannot masquerade as confirmatory.

The Citizen's Sniff Test

When a result is presented as 'one significant finding,' ask: 'how many things did the researcher look at to find this one?' The single reported p-value tells you nothing about the selection process that produced it. The replication crisis is what happens when an entire field forgets this question. Pre-registration, p-curve analysis, and multiple-comparison correction are the tools the field has invented to bring honesty back.

Code

False-positive inflation under multiple tests·python

import numpy as np
from math import erf, sqrt
rng = np.random.default_rng(140)

# Show that with enough tests under a true null, 'significance' becomes
# almost guaranteed somewhere — pure chance, no real effect.

N_per_test = 200
M_tests_per_experiment = 20
M_experiments = 5_000
p_true = 0.5     # null is true everywhere

# Track: in each experiment of 20 tests, how often does AT LEAST ONE hit p<0.05?
hit_rate_at_alpha_05 = 0
for _ in range(M_experiments):
    significant = False
    for _ in range(M_tests_per_experiment):
        flips = rng.binomial(n=1, p=p_true, size=N_per_test)
        p_hat = flips.mean()
        z = (p_hat - 0.5) / np.sqrt(0.5 * 0.5 / N_per_test)
        p_val = 2 * 0.5 * (1 - erf(abs(z) / sqrt(2)))
        if p_val < 0.05:
            significant = True
            break
    if significant:
        hit_rate_at_alpha_05 += 1

print(f"P(at least one of 20 tests hits p<0.05 under true null): "
      f"{hit_rate_at_alpha_05 / M_experiments:.4f}")
print(f"Expected: 1 - 0.95^20 = {1 - 0.95**20:.4f}")
print(f"\nBonferroni-corrected threshold for 20 tests: alpha = 0.05/20 = 0.0025")
# About 64% of 'experiments' that ran 20 null tests will have at least one
# 'significant' result — pure noise, no real effect. That is the multiple-
# comparison hazard in plain numbers.

Exercise

Identify a recent surprising scientific finding you remember from news (a coffee study, a wine study, a nutrition study, a behavioral finding). Ask: was this a pre-registered confirmatory study, or an exploratory analysis where the researcher looked at many variables? If exploratory, the reported p-value is almost certainly overstating the strength of evidence. Most non-replicating findings come from this category.

Hint

Pre-registered studies tend to be much more replicable. Exploratory analyses are useful for hypothesis generation but should not be reported as if they were definitive.

The Mechanism

The Common Forms

The Multiple-Comparison Correction

The Citizen's Sniff Test

Code

External links

Exercise

Progress

Comments 0