The P-Value: What It Actually Says

"The p-value is one of the most misread numbers in modern life. Naming what it is not is half the lesson."

The Definition

The p-value is: under the null hypothesis and the test's assumptions, the probability of obtaining a test statistic at least as incompatible with H₀ as the observed one. What counts as “at least as extreme” depends on the statistic and whether the test is one- or two-sided. Informally:

p-value = P(test statistic this extreme or more | H₀ and model assumptions)

A small p-value means the observed data would be unlikely if the null were true. A small p-value gives you reason to reject the null. It does NOT, by itself, tell you the probability that the null is true, the probability that the alternative is true, the size of the effect, the practical importance of the effect, or whether the result will replicate.

The Misread Citizens Make

The most common misreading: 'p = 0.03 means there's a 3% chance the null hypothesis is true.' Wrong. That sentence reverses the conditional. The p-value is P(data | null), not P(null | data). Inverting the two is, once again, the prosecutor's fallacy.

To go from P(data | null) to P(null | data), you need Bayes' rule and a prior — exactly the move Track 08 will make. The frequentist p-value alone cannot give you that posterior. Most published 'statistically significant' findings are silently treated as if they did, and that silent slip is one of the major engines of the replication crisis in modern science.

What 'Statistical Significance' Really Buys You

'Significant at α = 0.05' means the pre-specified decision rule rejected H₀ because the p-value was at or below 0.05. Under the procedure's assumptions, the rule is designed to limit its long-run Type I error rate to 5%. It does NOT mean the effect is real, large, important, or reproducible.

The Three Things to Carry Forward

(1) A p-value is computed under H₀ and model assumptions, not a probability that H₀ is true. (2) Statistical significance is a decision-rule result, not a claim of truth or importance. (3) A posterior probability requires an explicit model, likelihood, and prior; it cannot be obtained by simply inverting a p-value.

Code

p-values under a true null: the noise floor·python

import numpy as np
from math import erf, sqrt
rng = np.random.default_rng(130)

# Show that under the null hypothesis, p-values are uniformly distributed.
# We simulate 100,000 'experiments' where there is NO real effect (p_true = 0.5).
# For each, we compute a one-sample z-test p-value.
# Under the null, the p-values should be roughly uniform on [0, 1].

N = 200
M = 100_000
p_true = 0.5     # null is exactly true

flips = rng.binomial(n=1, p=p_true, size=(M, N))
p_hat = flips.mean(axis=1)
z = (p_hat - 0.5) / np.sqrt(0.5 * 0.5 / N)

# Two-tailed p-value from |z|.
p_values = np.array([2 * 0.5 * (1 - erf(abs(zi) / sqrt(2))) for zi in z])

for threshold in (0.05, 0.01, 0.001):
    fraction_significant = (p_values < threshold).mean()
    print(f"P(p < {threshold:.3f}) under TRUE null: {fraction_significant:.4f}  "
          f"(expected: {threshold:.4f})")

# Under a TRUE null, ~5% of experiments still get p < 0.05 by chance.
# That 5% is the Type I error rate, baked into the alpha threshold.
# The lesson: a single p < 0.05 is not strong evidence — it's the noise floor.

Exercise

Find a paper or news article that quotes a p-value (anything from medicine, psychology, finance, or social science). For each, restate the p-value precisely: 'assuming the null hypothesis [name it explicitly] is true, the probability of observing data this extreme is [p].' Then notice whether the article's wording silently slips into 'there's a [p] chance the effect isn't real' — that's the misread we're trying to fix.

Hint

Most popular science writing makes this slip. Spotting it once is enough to recalibrate how you read every future result.

The Definition

The Misread Citizens Make

What 'Statistical Significance' Really Buys You

The Three Things to Carry Forward

Code

External links

Exercise

Progress

Comments 0