Survivorship bias in engineering advice

What it is

Survivorship bias is drawing conclusions from the cases that remained visible (successful companies, popular libraries, teams that shipped) while the cases that failed or left are missing from the sample. The standard illustration is Abraham Wald's wartime work on aircraft damage at the Statistical Research Group: returning aircraft showed hits on the fuselage and few around the engines, and the inference was that the engine hits were the ones that kept aircraft from returning. The American Mathematical Society column that retells the story also warns that the popular version is a plausible reconstruction with little source material beyond Wald's memoranda and a colleague's memoir; the anecdote used to teach source criticism is itself lightly sourced.

Why it matters

Most engineering advice is written by survivors: architecture talks come from companies that grew, postmortems from services still running, "how we scaled" from the subset that scaled. Practices that were common among abandoned projects are rarely written up, because nobody is paid to describe them. Advice therefore tends to over-credit whatever the visible winners happened to do, including things that were neutral or harmful.

How to apply

• For any "winners do X" claim, ask what fraction of non-winners also did X, and where their accounts would be found.
• Look for the denominator: a list of projects that adopted a tool is not evidence without the list of those that abandoned it.
• Weight sources that report failures and abandoned attempts (postmortems, "what we would do differently", deprecation notices, retracted recommendations) at least as highly as success stories.
• When writing advice, state the population it was drawn from ("three services that reached this size") and what is unknown about those that did not.
• Treat "no one complained" as absence of surviving complainers: users who left silently do not file issues, and requests that timed out do not appear in the latency histogram.

Pitfalls

The mirror error: assuming that what failed projects did must be harmful. Confusing survivorship with selection on the outcome in your own data, such as dropping errored requests before computing latency. Retelling the Wald story as if it were documented in detail; cite it as an illustration, not as evidence.