Statistics · hard
Asked in Data Scientist interviews, in the Statistics round.
Extreme measurements are partly luck, so the next measurement of the same thing tends to be closer to average. Pick the worst-performing stores, intervene, and they improve on their own. The fix is a control group selected the same way.
Any measurement is signal plus noise. When you select the extremes, you are selecting for lucky noise as much as for real signal, so the next measurement, with fresh noise, sits closer to the average. Nothing has to change for the extreme group to look better next time.
The trap it sets: a team picks the ten worst-converting landing pages, rewrites them, and conversion improves. Some of that is the rewrite and some is regression to the mean, and without a control you cannot separate them. The same thing happens with "our best customers spent less this quarter" and "the users who complained most are happier after the fix".
How to defend against it: select the group, then randomly split it into treated and control. Both regress, so the difference between them is the treatment effect. If you cannot randomize, compare to a similar group selected the same way in a period with no intervention. And be suspicious of any before-and-after story that starts with "we picked the worst".
It also explains why a metric that spiked last week "returns to normal" without anyone doing anything, which is worth saying out loud before a stakeholder credits the wrong fix.