Statistics · hard
Explain regression to the mean, and how it fools an analysis.
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Extreme measurements are partly luck, so the next measurement of the same thing tends to be closer to average. Pick the worst-performing stores, intervene, and they improve on their own. The fix is a control group selected the same way.
How to answer it
Definition
Extreme measurements are extreme partly because of luck, and luck does not repeat. So a group selected for being extreme tends to be closer to average next time, with no intervention at all.
Why it happens
observed = true ability + noiseSelecting the highest observed values selects for high true ability AND for favorable noise. The ability persists; the noise resamples. The second measurement therefore sits closer to the mean, and the more noise there is relative to real differences, the stronger the pull.
Example
Pick the 50 worst-performing stores this quarter.
Run a training programme.
Next quarter they improve by 8%.
The programme looks like it worked. But the 50 stores were selected partly
because they had a bad quarter, and a bad quarter is partly bad luck. Some
of that 8% was always going to happen.How it fools people
- Any intervention aimed at the worst performers looks effective.
- Any intervention aimed at the best performers looks harmful, which is where the myth that criticism works better than praise comes from.
- Sports and finance are full of it: rookie of the year underperforming next season, or a top-decile fund reverting.
The fix
A control group selected the SAME way. Take the 100 worst stores, treat 50 at random, leave 50 alone. Both groups regress; the difference between them is the effect. There is no way to correct for this afterwards by cleverness in the analysis, which is the point worth making.
Key points
- It is a property of measurement with noise, not a real force pulling things back.
- Any selection on an extreme value exposes you to it.
- The noisier the measure, the stronger the apparent regression.
- The only fix is a control group selected by the same rule.
- Before-and-after comparisons on a selected group are the classic way to get fooled.
FAQ
- Is it the same as the gambler's fallacy? No. That expects compensating outcomes; this is about noise not repeating in a re-measurement.
- Does randomization solve it? Yes, provided the control group is selected the same way as the treatment group.
- Where does it bite in product analytics? Targeting churn-risk users, or any campaign aimed at a segment picked for a bad recent metric.
The 30-second answer
Regression to the mean is that extreme measurements are extreme partly because of luck, and luck does not repeat, so a group picked for being extreme moves toward average on its own.
It fools analyses whenever you select on a bad period and then intervene. Take the fifty worst stores, run a training programme, and they improve, because part of why they were worst was a bad quarter rather than something durable. The programme gets the credit.
The only real fix is a control group chosen by exactly the same rule. Take the hundred worst, treat half at random, and compare. Both halves regress, so the difference between them is the actual effect. You cannot correct for it afterwards with a cleverer analysis.
In the interview
- What they are checking: the selection-on-noise mechanism and the control group fix.
- Common mistake: reporting a before-and-after improvement on a group chosen for being extreme.