The Hypothesis-Test Frame

"A hypothesis test is a courtroom in miniature: a presumption of one thing, an attempt to find evidence against it, and a decision rule for when the evidence has crossed a threshold."

The Structure

A frequentist hypothesis test has four moving parts:

Null hypothesis (H₀): the default state of the world; usually 'no effect,' 'no difference,' 'the coin is fair.' This is what is presumed true unless the evidence overturns it.
Alternative hypothesis (H₁): the claim you'd accept if the evidence is strong enough.
Test statistic: a number computed from the data that captures the strength of evidence against H₀.
Decision rule: a threshold (α, usually 0.05) above which the evidence is deemed strong enough to reject H₀.

The frame is asymmetric. H₀ is the default; H₁ is the challenger. We do not 'accept' H₀ when the evidence is weak; we 'fail to reject' it. This wording is not academic fussiness — it reflects the fact that failing to find evidence is not the same as confirming the negative.

The Courtroom Analogy

Track 06 will make this explicit, but it's worth previewing here. The legal system is a hypothesis test:

H₀ = innocent (the presumption of innocence).
H₁ = guilty.
Test statistic = strength of the prosecution's evidence.
Decision rule = 'beyond reasonable doubt' = a very small α.

The legal system's choice to set α very small reflects its judgment that Type I errors (convicting the innocent) are far worse than Type II errors (acquitting the guilty). The frame is the same as a statistical test; the calibration of α is the policy decision.

What Hypothesis Tests Cannot Do

A hypothesis test does not give you P(H₀ | data). It gives you P(data | H₀) — the probability of the data given the null. These two are different (the prosecutor's fallacy again). A hypothesis test also does not tell you the size of the effect — just whether the evidence is strong enough to reject H₀ at your chosen α. A statistically significant tiny effect and a statistically insignificant huge effect are both possible, and both are common.

The Frame to Keep

A hypothesis test is a structured procedure for asking 'is the evidence strong enough to overturn the default assumption?' It is not a procedure for asking 'is the alternative true?' The two questions look similar; they are not. Track 06 (Courtroom) is where this distinction does the most citizen-relevant work; Track 08 (Bayesian Frame) is where it gets fully inverted.

Code

A coin-fairness hypothesis test, run repeatedly·python

import numpy as np
rng = np.random.default_rng(120)

# A classic hypothesis test: is this coin fair?
# H0: P(heads) = 0.5
# H1: P(heads) != 0.5
# We collect N=200 flips and observe k heads.
# Test statistic: z = (k/N - 0.5) / sqrt(0.5*0.5/N).

N = 200
p_true = 0.55     # secretly, the coin is slightly unfair
for trial in range(3):
    flips = rng.binomial(n=1, p=p_true, size=N)
    k = flips.sum()
    p_hat = k / N
    se = np.sqrt(0.5 * 0.5 / N)     # SE under the null
    z = (p_hat - 0.5) / se
    print(f"Trial {trial+1}: k={k}/{N}  p_hat={p_hat:.3f}  z={z:+.2f}  "
          f"(|z| > 1.96 → reject H0 at alpha=0.05)")

# Each trial may or may not reject H0 depending on noise.
# The TEST is well-defined; the decision can swing trial to trial.
# That is exactly the meaning of 'significance at alpha=0.05':
# we accept a 5% rate of falsely rejecting a true H0.

Exercise

Set up a hypothesis test you might run in everyday life. Example: 'is my morning latte at this café actually larger than the menu says?' Define H0 (the latte is exactly the advertised volume), H1 (it differs), how you would collect data (weigh ten lattes over two weeks), and your decision threshold. The exercise is not to actually run it — it's to feel the discipline of separating 'what I'd expect under the default' from 'what would convince me to change my mind.'

Hint

The default is usually 'no special effect.' The discipline is to require evidence calibrated to your tolerance for false alarms, not to update on every observation.

The Structure

The Courtroom Analogy

What Hypothesis Tests Cannot Do

The Frame to Keep

Code

External links

Exercise

Progress

Comments 0