Statistics · medium
Asked in Data Analyst interviews, in the Statistics round.
A trend within groups reverses when groups are combined due to a confounder.
A trend that holds inside every group reverses when the groups are pooled, because the groups are of very different sizes and the grouping variable is related to both the treatment and the outcome.
The example to have ready: two hospitals. Hospital A has a higher survival rate than B for mild cases and for severe cases. Pooled, B looks better, because A takes most of the severe cases. Severity is the lurking variable; it is correlated with which hospital you go to and with whether you survive.
In product data the same shape appears constantly: a new design wins on mobile and wins on desktop but loses overall, because the test ran while a marketing push shifted the mix towards mobile, where conversion is lower for everyone.
# same numbers, opposite conclusions
by_segment = df.groupby(["variant", "device"]).agg(conv=("converted", "mean"), n=("converted", "size"))
pooled = df.groupby("variant")["converted"].mean()What to do about it: decide the comparison unit before looking, randomise so that the mix is balanced between arms, and when reading observational data, always ask what else changed when the groups did. If the segments disagree with the total, the segments are usually right and the total is telling you about the mix.