The P-Value: What It Actually Says

"The p-value is one of the most misread numbers in modern life. Naming what it is not is half the lesson."

The Definition

The p-value is: assuming the null hypothesis is true, the probability of observing data as extreme as (or more extreme than) the actual data. Formally:

p-value = P(data this extreme or more | H₀ is true)

A small p-value means the observed data would be unlikely if the null were true. A small p-value gives you reason to reject the null. It does NOT, by itself, tell you the probability that the null is true, the probability that the alternative is true, the size of the effect, the practical importance of the effect, or whether the result will replicate.

The Misread Citizens Make

The most common misreading: 'p = 0.03 means there's a 3% chance the null hypothesis is true.' Wrong. That sentence reverses the conditional. The p-value is P(data | null), not P(null | data). Inverting the two is, once again, the prosecutor's fallacy.

To go from P(data | null) to P(null | data), you need Bayes' rule and a prior — exactly the move Track 08 will make. The frequentist p-value alone cannot give you that posterior. Most published 'statistically significant' findings are silently treated as if they did, and that silent slip is one of the major engines of the replication crisis in modern science.

What 'Statistical Significance' Really Buys You

'Significant at α = 0.05' means: if the null is true, this kind of evidence shows up no more than 5% of the time by random noise. It does NOT mean the effect is real, large, important, or reproducible. It means the data is incompatible with the null at the 5% noise tolerance. That is a much smaller claim than the words 'statistically significant' suggest in everyday speech.

The Three Things to Carry Forward

(1) p-value is a conditional on the null, not a probability about the null. (2) Statistical significance is a tolerance for false alarms, not a claim of truth. (3) To convert a p-value into 'how likely is the hypothesis?', you need a prior — and that takes you out of the pure frequentist toolkit into the Bayesian one. Memorize these three; you will then immediately read most p-values in news and papers more accurately than the headline that quoted them.

Code

p-values under a true null: the noise floor·python

import numpy as np
from math import erf, sqrt
rng = np.random.default_rng(130)

# Show that under the null hypothesis, p-values are uniformly distributed.
# We simulate 100,000 'experiments' where there is NO real effect (p_true = 0.5).
# For each, we compute a one-sample z-test p-value.
# Under the null, the p-values should be roughly uniform on [0, 1].

N = 200
M = 100_000
p_true = 0.5     # null is exactly true

flips = rng.binomial(n=1, p=p_true, size=(M, N))
p_hat = flips.mean(axis=1)
z = (p_hat - 0.5) / np.sqrt(0.5 * 0.5 / N)

# Two-tailed p-value from |z|.
p_values = np.array([2 * 0.5 * (1 - erf(abs(zi) / sqrt(2))) for zi in z])

for threshold in (0.05, 0.01, 0.001):
    fraction_significant = (p_values < threshold).mean()
    print(f"P(p < {threshold:.3f}) under TRUE null: {fraction_significant:.4f}  "
          f"(expected: {threshold:.4f})")

# Under a TRUE null, ~5% of experiments still get p < 0.05 by chance.
# That 5% is the Type I error rate, baked into the alpha threshold.
# The lesson: a single p < 0.05 is not strong evidence — it's the noise floor.

Exercise

Find a paper or news article that quotes a p-value (anything from medicine, psychology, finance, or social science). For each, restate the p-value precisely: 'assuming the null hypothesis [name it explicitly] is true, the probability of observing data this extreme is [p].' Then notice whether the article's wording silently slips into 'there's a [p] chance the effect isn't real' — that's the misread we're trying to fix.

Hint

Most popular science writing makes this slip. Spotting it once is enough to recalibrate how you read every future result.

The Definition

The Misread Citizens Make

What 'Statistical Significance' Really Buys You

The Three Things to Carry Forward

Code

External links

Exercise

Progress

Comments 0