Statistics · easy
What is the difference between statistical and practical significance?
Asked in Data Analyst interviews, in the Statistics round.
Short answer
Statistical significance says an effect is unlikely to be noise. Practical significance says it is big enough to matter. With enough data a 0.01% lift is significant, and it is still not worth shipping. Report the effect size with its interval, not just the p-value.
| Statistical significance | Practical significance | |
|---|---|---|
| Answers | Is this likely to be noise | Is this big enough to act on |
| Measured by | P-value, confidence interval | Effect size against a threshold you set |
| As n grows | Any tiny effect becomes significant | Unchanged, the effect is the effect |
| Decided by | The data | The business |
| Report | The interval, not just the p-value | The effect size and what it is worth |
How to answer it
Definition
Statistical significance says an effect is unlikely to be noise. Practical significance says it is large enough to be worth acting on. The table above compares them; they are independent, and all four combinations occur.
The four cases
significant + practical ship it
significant + not practical real, and too small to care about
not significant + would be
practical if real inconclusive, and worth more data
not significant + not practical stop looking hereThe second row is the one that wastes teams' time, and the third is the one that gets misreported as "no effect".
Why large samples force the distinction
n per arm lift p-value worth shipping?
1,000 2.0% 0.21 maybe, but unproven
100,000 0.05% 0.03 significant, and worthlessAny non-zero effect becomes statistically significant with enough data, because significance is about distinguishing from zero and the standard error keeps shrinking. Practical significance does not move: the effect is the size it is.
How to judge practical significance
Set the threshold before the test, in business terms. What lift would repay the engineering cost, the added complexity, the risk? That number is the MDE you should have powered the test for, which is why the two questions are really one question asked at different times.
Key points
- Significance is about noise; practical significance is about value.
- With a large enough sample every real effect becomes significant.
- Decide the threshold that matters before running the test, not after.
- Report the effect size and its interval, since a p-value carries neither.
- Not significant is not the same as no effect. The interval shows what you could rule out.
FAQ
- Can something be practically significant but not statistically? Yes, and that usually means underpowered. The answer is more data, not a different conclusion.
- Who sets the practical threshold? The business, not the analyst. Your job is to insist it exists.
- How should I report it? The effect size with a confidence interval, next to the threshold that would justify acting.
The 30-second answer
Statistical significance says the effect is unlikely to be noise. Practical significance says it is big enough to be worth doing something about. They are independent, and the case that causes trouble is significant but tiny.
The reason it comes up is that with a large enough sample any non-zero effect becomes statistically significant, because the standard error keeps shrinking while the effect stays the size it is. At a hundred thousand users per arm a twentieth of a percentage point can be significant and completely worthless.
So I would agree the threshold that would justify shipping before running the test, in business terms, and report the effect size with its confidence interval rather than a p-value, because the interval is what tells you whether you cleared that bar.
In the interview
- What they are checking: that you separate "is it real" from "does it matter", and that you report effect sizes.
- Common mistake: declaring a test a success because p was below 0.05, without saying how large the effect was.