Survivorship Bias: We Only Hear from Those Who Made It

"The successful tell us their strategies. The failed are not around to tell us they tried the same strategies. The advice is filtered by survival, not by validity."

The Canonical Story: Wald and the Bombers

In World War II, the US military analyzed the bullet-hole patterns on bombers returning from missions. The patterns showed dense damage on the wings and fuselage and sparse damage on the engines and cockpit. The natural conclusion seemed to be: add armor to the wings and fuselage, where the planes are getting hit.

The returning-aircraft example is widely associated with Abraham Wald and captures the logic of conditioning on survival, though popular retellings often present a cleaner dialogue and decision than the surviving archival record establishes. The bombers being analyzed were the ones that returned. The sparse damage on engines and cockpit did not mean those areas were getting hit less; it meant that bombers hit there did not return to be analyzed. The missing damage suggests that vulnerable regions on aircraft that failed to return must be considered; the actual engineering decision would also require aircraft geometry, hit exposure, mission data, and weight constraints.

This is the foundational story of survivorship bias. The sample of survivors does not represent the population of all who tried; it represents the population conditional on having survived. Inferences from the survivor sample to 'what works' are systematically biased by the survival filter.

Why Successful-People Advice Is Often Wrong

Books and podcasts featuring 'successful entrepreneurs' or 'top traders' or 'world-class athletes' are by construction sampling on success. The advice extracted from them is conditional on the same kinds of success having been achieved. The many people who used the same strategies and failed are not in the sample. The 'successful advice' may be little more than the survival rate of a strategy times the rate at which it's heard about, with no actual causal relationship to success at all.

The useful test is whether the strategy predicts outcomes among everyone exposed to it, including failures and censored cases. Survivor testimony may still contain useful evidence, but it cannot estimate success rates or causal effects by itself.

The Operating Principle

Before taking 'what successful X did' as causal advice, ask whether you could have run the same survey on the people who tried X and failed. If you could not, the advice is conditional on survival rather than on validity, and the failure rate is invisible. The right question is not 'what did successful people do?' but 'what is the ratio of success to failure among everyone who did this thing?' That denominator is what survival filtering hides.

Code

Survivor sample distorts strategy attribution·python

import numpy as np
rng = np.random.default_rng(260)

# Simulate 100,000 startups trying one of two strategies.
# Strategy A: 'work harder' — success rate 0.05.
# Strategy B: 'be lucky' — success rate 0.02.
# Both strategies are noisy; we cannot tell them apart from outcomes alone.
#
# We INTERVIEW only successful founders and ask 'what strategy did you use?'

N = 100_000
strategy = rng.choice(['A', 'B'], size=N, p=[0.5, 0.5])
success_rate = np.where(strategy == 'A', 0.05, 0.02)
success = rng.random(N) < success_rate

survivors = strategy[success]
print(f"Total founders:       {N:,}")
print(f"Surviving founders:   {success.sum():,}")
print(f"\nSurvivor breakdown:")
for s in ['A', 'B']:
    count = (survivors == s).sum()
    pct = count / len(survivors) * 100
    print(f"  Strategy {s}: {count:,} survivors ({pct:.1f}% of survivors)")

print(f"\nFrom this survivor sample alone, you would conclude that Strategy A")
print(f"is dominant. That conclusion is correct in this sim, BUT the conclusion")
print(f"is not visible without knowing the denominator. Without the failures,")
print(f"you would have no way to tell A's success rate from B's.")
print("This is what survivor advice looks like in the absence of failure data.")

Exercise

Pick a piece of 'how to succeed at X' advice you've found compelling. Identify the survivor sample it is built on. Estimate: how many people followed the same strategy and did NOT succeed, and how would the advice change if their experiences were included? Most advice books cannot survive this question seriously asked.

Hint

If the advice was extracted from people who succeeded, you are inferring the strategy's effectiveness from a survivor-only sample. The failure rate, the universe of strategies that also failed, and the role of luck are all invisible from this vantage.

The Canonical Story: Wald and the Bombers

Why Successful-People Advice Is Often Wrong

The Operating Principle

Code

External links

Exercise

Progress

Comments 0