Statistics ยท medium
What is Simpson's paradox?
Asked in Data Analyst interviews, in the Statistics round.
Short answer
A trend within groups reverses when groups are combined due to a confounder.
How to answer it
Definition
A trend that holds in every subgroup reverses when the groups are pooled. It is not a statistical curiosity: it happens because a lurking variable is distributed unevenly across the groups.
Example
Treatment A Treatment B
small stones 81/87 = 93% small stones 234/270 = 87%
large stones 192/263 = 73% large stones 55/80 = 69%
A wins on small stones. A wins on large stones.
Pooled: A 273/350 = 78% Pooled: B 289/350 = 83%B wins overall while losing both subgroups. Nothing is miscalculated. A was given to far more of the hard cases, so its overall rate is dragged down by the mix rather than by its performance.
Why it happens
The subgroup, here stone size, affects both the treatment assigned and the outcome. Pooling averages over a mix that differs between groups, so the overall numbers compare different populations rather than different treatments.
Which number to trust
Not automatically the disaggregated one. It depends on the causal structure:
- The subgroup is a CONFOUNDER, decided before treatment, like stone size. Trust the subgroups.
- The subgroup is a MEDIATOR, caused by the treatment, like whether a user reached checkout. Conditioning on it removes part of the effect you are measuring. Trust the pooled number.
Only a causal claim distinguishes these. The data alone cannot.
Key points
- Every subgroup can point one way and the total the other. It is arithmetic, not a mistake.
- The cause is an unequal mix combined with a variable that affects the outcome.
- Whether to disaggregate is a causal question, not a statistical one.
- Randomization prevents it by making the mix equal in expectation.
- In observational data, always look for a variable that drives both the grouping and the outcome.
FAQ
- How do I spot it? Compute the metric overall and by the obvious segments. If they disagree in direction, look for the confounder.
- Does it affect A/B tests? Randomization protects the overall result, but slicing after the fact can reintroduce it, especially with unequal traffic across segments.
- Is it common? Very, in observational data. Any comparison across a changing mix is exposed to it.
The 30-second answer
Simpson's paradox is when a trend that holds inside every subgroup reverses once you pool them. The classic case is a treatment that wins on small stones and wins on large stones, yet loses overall, because it was given to far more of the hard cases and the mix drags its overall rate down.
The cause is a lurking variable that affects both which group you land in and the outcome. Nothing is calculated wrongly.
The part I would add is that the disaggregated number is not automatically the right one. If the subgroup was decided before treatment it is a confounder and I trust the subgroups. If the treatment caused the subgroup it is a mediator, and conditioning on it removes part of the effect I am trying to measure. Deciding that needs a causal argument, not more data.
In the interview
- What they are checking: whether you can explain the mechanism (unequal group sizes plus a confounder), not just recite the name.
- Common mistake: presenting the pooled number without checking whether the mix changed between arms.