Statistics · medium · Reported at Meta
How do you determine sample size for an A/B test?
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Use power analysis: baseline rate, MDE, alpha, power (0.8) to compute n per arm.
How to answer it
Definition
Sample size is fixed by four quantities, and choosing three of them determines the fourth: the baseline rate, the minimum effect worth detecting, the significance level, and the power.
Formula
n per arm ~= 16 * p * (1 - p) / MDE^2 for a proportion,
alpha = 0.05, power = 0.80
p baseline conversion rate
MDE minimum detectable effect, in ABSOLUTE percentage pointsExample
Baseline p = 0.04 (4% conversion)
MDE = 0.004 (a 10% relative lift, so 0.4 points)
alpha = 0.05
power = 0.80
n = 16 * 0.04 * 0.96 / 0.004^2
= 16 * 0.0384 / 0.000016
= 38,400 per armOutput
~38,400 per arm, so ~76,800 users total.
At 5,000 users a day that is about 16 days.The MDE is a business decision
This is the part interviewers listen for. The MDE is not a statistical quantity you look up; it is the smallest lift that would change what you do. If a 0.1 point lift would not justify shipping the change, do not power the test to find it, because the cost is quadratic: halving the MDE quadruples the sample.
Key points
- Fix the sample size BEFORE the test starts. Deciding when to stop after watching the results inflates the false-positive rate badly.
- The relationship is quadratic. A twice-smaller effect needs four times the data.
- Baseline rate matters: rarer events need much more traffic for the same relative lift.
- Power of 0.80 is convention. It means a real effect is missed one time in five, which is worth saying out loud.
- Convert the answer into days at current traffic. That is the number the room actually cares about.
FAQ
- What if we cannot wait that long? Then raise the MDE, accept lower power, or reduce variance with CUPED or a better metric. Say which you traded.
- Do more than two arms change it? Yes: more comparisons need a multiple-comparison correction, which raises the required n per arm.
- What about continuous metrics like revenue? Same logic, but driven by the variance rather than p(1-p), and revenue's variance is large.
The 30-second answer
Four things determine it: the baseline rate, the minimum effect worth detecting, the significance level and the power. Fix three and the sample size falls out. For a proportion I use roughly sixteen times p times one minus p over the MDE squared, which gives eighty percent power at the usual five percent significance.
The part I would push on is the MDE, because it is a business decision rather than a statistical one: it is the smallest lift that would actually change what we ship. It matters because the relationship is quadratic, so halving the effect you want to detect quadruples the traffic you need.
I would also fix the sample size before starting and convert it to days at current traffic, since peeking and stopping early is what quietly ruins the false-positive rate.
In the interview
- What they are checking: that you know the MDE is a product decision, and that you can turn n into days.
- Common mistake: running until significant. That is the peeking problem, and it inflates the false positive rate several times over.