Statistical Paradoxes and Surprises

Numbers feel solid, yet the same honest figures can point in opposite directions depending on how you slice them. A hospital can look deadlier than its rival while treating every type of patient better; a fund can look like a safe bet only because its failures have quietly vanished from the record. These are not tricks or lies. They are structural surprises baked into how data is grouped and collected.

Learning to see them changes how you read almost any statistic. Once you know a pooled average can hide a reversed pattern, and that the data in front of you may be missing its most important cases, you stop taking summaries at face value and start asking where the rest of the story went.

Simpson's paradox

Simpson's paradox occurs when a relationship holds within every subgroup but flips once the subgroups are combined. It happens because a lurking variable is spread unevenly across the groups. If one group is both larger and facing tougher conditions, its results can drag the pooled average in a direction that contradicts what each group experienced on its own. The paradox is a warning that the level at which you aggregate can invert the conclusion.

Survivorship bias

Survivorship bias is the error of drawing conclusions from the cases that made it through while ignoring those that did not. The survivors are visible and the losses are silent, so the sample you can see is not the sample you need. Fund records that quietly drop closed funds, or lists of the habits of successful founders drawn only from companies still trading, both overstate success because the failures were never counted.

The missing-cases question

Both surprises share one cure: ask what you cannot see. For Simpson's paradox, ask whether a hidden grouping variable could be steering the pooled figure. For survivorship bias, ask which cases dropped out before the data reached you. Naming the missing cases usually dissolves the surprise.

Consider two illustrative treatments given to 1,000 patients each. Treatment A is used mostly on hard cases, Treatment B mostly on easy ones. On easy cases, A saves 95% (95 of 100) and B saves 90% (810 of 900). On hard cases, A saves 50% (450 of 900) and B saves 45% (45 of 100). Treatment A is better in both subgroups. Yet overall A saves 545 of 1,000 (54.5%) while B saves 855 of 1,000 (85.5%). B wins the pooled comparison purely because it treated the easier patients. The lurking variable is case difficulty, and pooling hides it.

The paradox is not inevitable. Suppose both treatments had faced the same mix of easy and hard cases, in the same proportions. Then the group that wins each subgroup also wins overall, and the pooled average tells the true story. Simpson's paradox needs an imbalance in the lurking variable to appear. When groups are comparable, aggregation is safe. The reversal is a symptom of an uneven mix, not of arithmetic gone wrong.

In 1975, Bickel, Hammel and O'Connell published an analysis in the journal Science of graduate admissions at the University of California, Berkeley, for the autumn of 1973. Overall, about 44% of male applicants were admitted against about 35% of women, which looked like bias against women. But examining departments one at a time, the pattern largely disappeared; the authors reported a small bias, if anything, in favour of women. The explanation was a lurking variable: women applied disproportionately to competitive departments with low admission rates for everyone, while men applied more to departments that admitted a high share of applicants.

Survivorship bias has an equally famous case. During the Second World War, the statistician Abraham Wald, working with the Statistical Research Group, was asked where to add armour on bombers, based on where returning planes showed the most bullet holes. Wald reasoned the opposite way: holes on survivors marked places a plane could be hit and still come home. The areas with few holes on returning planes, such as the engines, were where stricken aircraft went down and never returned. He recommended armouring the parts that were not hit. (The popular bullet-hole diagram is a modern illustration, but Wald's wartime analysis is documented in his reports.)

• "If a treatment wins in every group, it must win overall." Not when the groups differ in size and difficulty.
• "More data always clarifies." Pooling can hide the truth; splitting the data can reveal it.
• "The successful examples show what works." Survivors alone cannot tell you how often the same approach failed.
• "Aggregated statistics are neutral." The choice of how to group is itself an analytical decision that can flip the story.

Find a headline that compares two groups by a single average, such as pass rates, salaries, or recovery rates. Ask one question: could a hidden variable be split unevenly between the groups? Then look for the same claim broken down by that variable. Note whether the comparison holds, weakens, or reverses once you can see the subgroups.

Think Like a Maester: Before trusting any average, ask who is missing from the count and what hidden group might be steering the total.

1. What condition must hold for Simpson's paradox to appear when subgroups are pooled?
2. A fund advert shows only the funds still open today. What bias is at work, and what is missing?
3. In the Berkeley case, which lurking variable explained the apparent bias against women?
4. Why did Abraham Wald recommend armouring the parts of bombers with the fewest bullet holes?
5. You read that one treatment beats another overall. What single question best guards against being misled?

Simpson's paradox and survivorship bias are two ways honest data can mislead. In Simpson's paradox, an uneven lurking variable lets a trend reverse when subgroups are combined, as the Berkeley admissions figures showed. In survivorship bias, the failures are invisible, so the survivors flatter the record, as Wald saw in the returning bombers. The shared defence is to disaggregate the numbers and to ask which cases never reached the data. When a statistic surprises you, treat it as an invitation to look for the missing part of the picture.

• Misleading Statistics
• Statistics in Sports and Games