Simpson's paradox and base-rate neglect in reports

What it is

Simpson's paradox: an association between two variables in a whole population disappears or reverses when the population is split into groups. The Stanford Encyclopedia of Philosophy entry reproduces Simpson's 1951 example: a treatment shows the same 50% success rate as the control overall, yet a higher rate than the control among men (about 61% against 57%) and among women (about 44% against 40%). The overall figure hides that the groups received the treatment in different proportions. The entry calls such cases association reversals and stresses that which figure answers the question depends on the causal structure, not on the arithmetic.

Base-rate neglect: judging what a positive signal means from the signal's accuracy alone, ignoring how rare the condition is. Bayes' theorem, as the SEP entry sets out, ties the two: the odds after a test equal the prior odds multiplied by the likelihood ratio (true-positive rate divided by false-positive rate). Worked by arithmetic: an alert fires for 99% of real incidents and for 1% of normal hours. If 1 hour in 1,000 has a real incident, then in 100,000 hours there are 100 incidents (99 alerts) and 99,900 normal hours (999 false alerts); a given alert is real about 9% of the time, not 99%.

Why it matters

Reports about teams, services and users are almost always aggregates over heterogeneous groups; conversion rates, error rates and "speed after the change" all mix groups whose composition changed at the same time as the thing being evaluated. An agent summarising such a report reproduces the reversal unless it asks how the groups were mixed. Alert precision, fraud scores and test flakiness statistics all suffer from the base-rate error.

How to apply

• Whenever two groups are compared, ask what else differed in their composition (size, period, customer segment, region) and whether the comparison holds inside each stratum.
• For any rate presented as a hit rate or accuracy, ask for the base rate and recompute the positive predictive value with natural frequencies as above.
• Report both the aggregate and the stratified numbers, and say which question each answers.
• Keep denominators visible in tables: counts alongside rates, never rates alone.
• When a metric improved after a change, check whether the mix of inputs changed at the same time.

Pitfalls

Stratifying on a variable that is itself caused by the treatment creates a different distortion. Very small strata produce reversals by chance. Percentages of percentages lose the base entirely. A base rate taken from a different population (last year's incident rate for this year's alerts) gives a precise but wrong answer.