How do I avoid Survivorship Bias?

Ask yourself: 'What data is missing?' Look for the failures.

Survivorship Bias

Category: Probability & Belief

The logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility.

How it works

We analyze the things that made it through a selection process and never see the things that didn't, so our sample is silently, catastrophically skewed. The winners are visible, loud, and available for study; the losers are gone, quiet, and invisible. Drawing lessons from the survivors alone is like studying lottery winners to learn how to win the lottery.

The trap is that the missing data doesn't announce itself. Nothing in the survivors says 'remember the hundreds who tried this and vanished.' So we confidently extract the 'secrets of success' from the few who succeeded, their habits, their bold bets, their college-dropout origin stories, without checking whether the failures had all the same traits. Often they did, which means those traits explain nothing.

This is what makes survivorship bias so seductive and so wrong: every example you can find supports the conclusion, because the counterexamples have been deleted from view. The pattern looks airtight precisely because the disconfirming cases were filtered out before you ever started looking.

Where you'll see it

'Drop out and follow your passion!' draws on Gates, Jobs, and Zuckerberg, the handful who made it, while the millions of dropouts whose ventures quietly failed never get a TED talk, biasing the whole lesson.
Mutual-fund advertising touts 'top performers over ten years,' but funds that did badly get quietly closed or merged away, so the surviving list *looks* like investing is easy, the losers were deleted from the chart.
A new manager copies the 'work 90-hour weeks and never compromise' habits of one celebrated founder, not realizing countless founders with the identical habits burned out and went bankrupt, leaving only the lucky survivor to be studied.

Where it comes from

The most famous illustration comes from World War II and statistician Abraham Wald at the Statistical Research Group. The military examined bombers returning from missions and wanted to add armor where the planes showed the most bullet holes, the wings and fuselage. Wald's insight inverted the logic: those were the planes that survived being hit there. The places with no holes on returning aircraft, engines and cockpit, were precisely where a hit was fatal, so those planes never came back to be counted. Armor belonged where the survivors showed no damage. The story is the canonical demonstration that the missing data, not the visible data, holds the answer.

How to counter it

Always ask the haunting question: 'Where are the ones that didn't make it?' Before you copy a success, deliberately hunt for the failures, the dead startups, the closed funds, the dropouts who didn't become billionaires, and check whether they shared the very traits you're about to credit for success.

Look for the selection filter. Whenever you're handed a sample, ask what process determined who got into it. If only winners are visible (top performers, returning planes, published studies), you're looking at survivors, and the conclusion needs the invisible group to be valid.

Weigh base rates over highlight reels. 'How many people tried this, and what fraction succeeded?' is the question that converts an inspiring anecdote back into honest odds. The visible success story is one draw from a distribution whose failures you'll have to go dig up on purpose.

The tell

You're doing it when you're studying only the winners to learn what works and never asking what the losers had in common.