Confidence Intervals: What They Really Say

"A confidence interval is a statement about the procedure, not about the parameter."

The Mechanic of a CI

Take a sample, compute the sample mean, compute the standard error (σ/√N), and build an interval like:

CI(95%) = sample mean ± 1.96 × SE

That ±1.96 comes from the bell — about 95% of a standard normal's mass sits between −1.96 and +1.96. The result is the 95% confidence interval for the mean. The arithmetic is simple. The interpretation is what trips citizens up.

The Citizen Misreading and the Correct Reading

Citizens (and many scientists, casually) read '95% CI [42, 58]' as 'there's a 95% probability that the true mean is between 42 and 58.' That is wrong in the frequentist frame. The true mean is a fixed but unknown number; it is either in the interval or it isn't. Probability does not apply to it directly.

The correct frequentist reading: 'the procedure used to construct this interval, if repeated many times across many studies, would produce intervals that contain the true mean 95% of the time.' The probability is about the procedure (the long-run hit rate of the CI-construction process), not about this particular interval.

For the citizen, the practical difference is small: in most reasonable cases, treating the CI as a plausible range for the true value is fine. The mathematicians' insistence on precise wording matters because the citizen reading silently elides preconditions (independence, normality, large N) that the procedure-based reading keeps visible.

What Determines the CI's Width

Sample size N: bigger N → smaller SE → narrower CI. The dependency is √N — to halve the CI width, you need four times the data.
Population variability σ: more variable data → larger SE → wider CI. You can't change σ; you can only acquire more data.
Confidence level: 99% CI is wider than 95%, which is wider than 90%. Higher confidence buys you a wider net but doesn't improve precision.

The Preconditions That Bite

The CI's coverage guarantee rests on the same preconditions as the CLT: independence and finite variance. Correlated samples produce CIs that are too narrow (the procedure's true hit rate is below the nominal 95%). Fat-tailed data produces CIs that are arithmetic-only with no meaningful coverage guarantee. When you read '±X' in a published paper, the implicit invitation is to trust that the authors verified the preconditions. They often did not.

Bayesian Footnote

The 'probability the parameter is in the interval' reading IS valid in the Bayesian frame, but only when you compute a Bayesian credible interval — a different procedure with a different interpretation, requiring a prior. Track 08 will cover this. The frequentist CI and the Bayesian credible interval often agree numerically for clean data; they diverge in interpretation always, and in numbers when priors are influential.

Code

What '95% confidence' actually means, verified·python

import numpy as np
rng = np.random.default_rng(110)

# Demonstrate the 'long-run hit rate' interpretation of a 95% CI.
# Take 10,000 samples of N=100 from a normal with mean 50, std 15.
# For each sample, compute its 95% CI for the mean. Then ask: what fraction
# of those CIs actually cover the true mean of 50?

true_mean = 50
N = 100
sigma = 15
M = 10_000

coverage = 0
for _ in range(M):
    sample = rng.normal(loc=true_mean, scale=sigma, size=N)
    sample_mean = sample.mean()
    se = sigma / np.sqrt(N)              # use known sigma for clarity
    low, high = sample_mean - 1.96 * se, sample_mean + 1.96 * se
    if low <= true_mean <= high:
        coverage += 1

print(f"95% CI coverage rate across {M:,} simulated studies: {coverage / M * 100:.2f}%")
# Should be close to 95%. That number IS what '95% confidence' means.
# It's a property of the procedure across many repeats, not a probability
# statement about any single interval.

Exercise

Find a recent poll or scientific study that reports a confidence interval (e.g., '52% support, 95% CI [49%, 55%]'). Restate the CI in the procedure-based language: 'if this study were repeated many times under the same conditions, 95% of constructed intervals would contain the true value.' Then ask: under what conditions might the procedure's hit rate fall below 95%? Almost always: correlated respondents or fat-tailed outcomes.

Hint

Most public-facing reports use the citizen wording. The procedure wording is what protects you from over-trusting an interval whose preconditions weren't checked.

The Mechanic of a CI

The Citizen Misreading and the Correct Reading

What Determines the CI's Width

The Preconditions That Bite

Bayesian Footnote

Code

External links

Exercise

Progress

Comments 0