Confidence Intervals: What They Really Say

"A confidence interval is a statement about the procedure, not about the parameter."

The Mechanic of a CI

For a simple mean with independent observations and known population standard deviation σ, compute the sample mean and standard error σ/√N, then build an interval like:

CI(95%) = sample mean ± 1.96 × SE

That ±1.96 comes from the bell — about 95% of a standard normal's mass sits between −1.96 and +1.96. The result is the 95% confidence interval for the mean. The arithmetic is simple. The interpretation is what trips citizens up.

The Citizen Misreading and the Correct Reading

Citizens (and many scientists, casually) read '95% CI [42, 58]' as 'there's a 95% probability that the true mean is between 42 and 58.' That is wrong in the frequentist frame. The true mean is a fixed but unknown number; it is either in the interval or it isn't. Probability does not apply to it directly.

The correct frequentist reading: 'the procedure used to construct this interval, if repeated many times across many studies, would produce intervals that contain the true mean 95% of the time.' The probability is about the procedure (the long-run hit rate of the CI-construction process), not about this particular interval.

A confidence interval can be used as a range of values compatible with the chosen model and procedure, but it is not a posterior probability distribution. The precise wording keeps the sampling design, model, estimator, and coverage procedure visible.

What Determines the CI's Width

Sample size N: bigger N → smaller SE → narrower CI. The dependency is √N — to halve the CI width, you need four times the data.
Population variability σ: more variable data → larger SE → wider CI. You can't change σ; you can only acquire more data.
Confidence level: 99% CI is wider than 95%, which is wider than 90%. Higher confidence buys you a wider net but doesn't improve precision.

The Preconditions That Bite

A CI's coverage depends on the assumptions of the particular construction method. Dependence, heavy tails, small samples, biased sampling, or a misspecified standard-error formula can spoil nominal coverage. Other methods—such as design-based, robust, bootstrap, or exact intervals—use different assumptions. When you read '±X,' ask what the interval is, how it was constructed, and which sources of uncertainty it includes.

Bayesian Footnote

The 'probability the parameter is in the interval' reading IS valid in the Bayesian frame, but only when you compute a Bayesian credible interval — a different procedure with a different interpretation, requiring a prior. Track 08 will cover this. The frequentist CI and the Bayesian credible interval often agree numerically for clean data; they diverge in interpretation always, and in numbers when priors are influential.

Code

What '95% confidence' actually means, verified·python

import numpy as np
rng = np.random.default_rng(110)

# Demonstrate the 'long-run hit rate' interpretation of a 95% CI.
# Take 10,000 samples of N=100 from a normal with mean 50, std 15.
# For each sample, compute its 95% CI for the mean. Then ask: what fraction
# of those CIs actually cover the true mean of 50?

true_mean = 50
N = 100
sigma = 15
M = 10_000

coverage = 0
for _ in range(M):
    sample = rng.normal(loc=true_mean, scale=sigma, size=N)
    sample_mean = sample.mean()
    se = sigma / np.sqrt(N)              # use known sigma for clarity
    low, high = sample_mean - 1.96 * se, sample_mean + 1.96 * se
    if low <= true_mean <= high:
        coverage += 1

print(f"95% CI coverage rate across {M:,} simulated studies: {coverage / M * 100:.2f}%")
# Should be close to 95%. That number IS what '95% confidence' means.
# It's a property of the procedure across many repeats, not a probability
# statement about any single interval.

Exercise

Find a recent poll or scientific study that reports a confidence interval (e.g., '52% support, 95% CI [49%, 55%]'). Restate the CI in the procedure-based language: 'if this study were repeated many times under the same conditions, 95% of constructed intervals would contain the true value.' Then ask: under what conditions might the procedure's hit rate fall below 95%? Almost always: correlated respondents or fat-tailed outcomes.

Hint

Most public-facing reports use the citizen wording. The procedure wording is what protects you from over-trusting an interval whose preconditions weren't checked.

The Mechanic of a CI

The Citizen Misreading and the Correct Reading

What Determines the CI's Width

The Preconditions That Bite

Bayesian Footnote

Code

External links

Exercise

Progress

Comments 0