Base Rate Neglect: The Citizen's Built-In Bias

"Daniel Kahneman called it 'base rate neglect': humans reliably ignore prior probabilities and overweight vivid case-specific information. Bayes' rule is the math; base-rate neglect is the cognitive failure that makes the math feel counterintuitive."

The Pattern, Named

Base-rate neglect is the systematic human tendency to ignore prior probabilities (base rates) in favor of vivid, specific information about an individual case. It is one of the most reliable cognitive biases in the literature; Kahneman and Tversky documented it in dozens of experiments and it forms a central pillar of Thinking, Fast and Slow.

The Classic Illustration

'Linda is 31, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice. Which is more probable: (a) Linda is a bank teller, or (b) Linda is a bank teller AND active in the feminist movement?' Many respondents pick (b). But (b) is a subset of (a), so it cannot be more probable. This is the conjunction fallacy. It is related to representativeness and is often discussed near base-rate neglect, but it is not itself a direct base-rate calculation.

The medical false positive of the previous lesson is the same shape: the vivid '99% accurate test' overwhelms the boring fact that the disease is rare. The prosecutor's fallacy is the same shape: the vivid '1 in a million DNA match' overwhelms the boring question of base rate of guilt. People can underweight base rates in some tasks, but the size and direction of the effect depend on wording, information format, and context. Explicit natural frequencies or a Bayes calculation can help.

Why the Bias Exists

Evolutionary speculation aside, the bias is consistent with how human reasoning under uncertainty actually works: we substitute easy questions for hard ones. The hard question is 'given the evidence and the base rate, what is the posterior probability?' The easy question is 'how strongly does the evidence remind me of the hypothesis?' Substituting the easy question for the hard one (Kahneman's 'attribute substitution') gets us reliable answers very fast, often correct enough for everyday life, but systematically biased when the base rate matters.

The Corrective

The corrective for base-rate neglect is the explicit Bayes calculation, written out with the prior, likelihood, and posterior named. When you find yourself reaching for a posterior conclusion based on vivid case-specific evidence, write the prior down explicitly. Compare your gut posterior to the Bayesian posterior. The two often differ by an order of magnitude or more. The discipline of writing the prior down is what turns base-rate neglect from a reliable bias into a noticed one.

Code

The cab problem: gut vs Bayes·python

# Demonstrate base-rate neglect with the cab problem.
#
# In a city, 85% of cabs are Green and 15% are Blue. An accident occurred at night.
# A witness identifies the cab as Blue. Witnesses tested under similar conditions
# are correct 80% of the time and wrong 20% of the time.
# Question: what is the probability the cab was actually Blue?
#
# Gut answer: 80% (the witness's accuracy).
# Bayes answer: ~41%.

def bayes(prior, P_E_given_H, P_E_given_not_H):
    num = P_E_given_H * prior
    den = num + P_E_given_not_H * (1 - prior)
    return num / den

prior_blue = 0.15           # base rate
P_says_blue_given_blue = 0.80
P_says_blue_given_green = 0.20
posterior = bayes(prior_blue, P_says_blue_given_blue, P_says_blue_given_green)

print(f"Base rate P(Blue): {prior_blue:.2f}")
print(f"Witness accuracy: 80% correct identifications")
print(f"Posterior P(Blue | 'witness says Blue'): {posterior:.4f}")
print()
print("Gut intuition says ~80%. Bayesian math says ~41%.")
print("The base rate (only 15% of cabs are Blue) drags the posterior down")
print("to below 50%, despite the witness being 80% accurate.")
print("That gap is base-rate neglect made numerical.")

Exercise

Pick a vivid story you have read recently in news or social media that prompted you to update a belief. Identify: (1) the vivid case detail that did the work; (2) the implicit base rate that the story did not mention; (3) the posterior you would compute if you took the base rate seriously. Does your initial reaction survive the comparison? Most do not.

Hint

'A friend's cousin got X side effect from Y vaccine' is a vivid case detail; the relevant base rate is the overall side-effect rate in the population that received the vaccine. The vivid story is dramatic; the base rate is what should drive policy.

The Pattern, Named

The Classic Illustration

Why the Bias Exists

The Corrective

Code

External links

Exercise

Progress

Comments 0