Statistics · easy
Asked in Data Analyst interviews, in the Statistics round.
Statistical significance says an effect is unlikely to be noise. Practical significance says it is big enough to matter. With enough data a 0.01% lift is significant, and it is still not worth shipping. Report the effect size with its interval, not just the p-value.
A p-value answers one question: could an effect this large be noise. It says nothing about whether the effect is worth anything.
With a million users per arm, a 0.02% change in conversion is statistically significant. Whether it is practically significant depends on the business: at that traffic it might be real money, and for a change that costs engineering time it might not. The number that answers this is the effect size with its confidence interval, read against the minimum effect the team said would matter before the test.
The reverse case is just as common. A 5% lift with a p-value of 0.12 in a small test is not "no effect", it is "not enough data". The interval probably spans zero and a large gain. The honest answer is to say what the interval covers and, if the decision is worth it, run longer.
Present a result as: effect, interval, p-value, and the decision threshold, in that order. Most product stakeholders only need the first and the last.