Statistics · medium · Asked at Meta

How do you determine sample size for an A/B test?

Asked in Data Scientist interviews, in the Statistics round.

Short answer

Use power analysis: baseline rate, MDE, alpha, power (0.8) to compute n per arm.

How to answer it

Four inputs and one formula. Name the inputs before touching the formula.

• Baseline rate: the metric today, say 3.0% conversion.
• Minimum detectable effect: the smallest lift worth acting on, say a 10% relative lift, so 3.3%. This is a business decision, not a statistical one.
• Significance level, alpha: usually 0.05.
• Power: usually 0.8, meaning an 80% chance of detecting the effect if it exists.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
effect = proportion_effectsize(0.033, 0.030)          # Cohen's h
n_per_arm = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8, ratio=1)
# ~ 27,000 per arm for these numbers

Then turn the number into a duration: per-arm n divided by daily eligible traffic per arm, rounded up to whole weeks so weekday and weekend behaviour are both covered. If that is 9 weeks and the team wanted 1, the honest conversation is about a larger MDE or a more sensitive metric, not a shorter test.

The follow-ups worth pre-empting: the MDE drives everything (halving it roughly quadruples n); fixing n in advance is what makes the p-value valid, so no peeking and stopping early; and for a continuous metric the same calculation uses the metric's standard deviation instead of a rate.

• What they are checking: that you know the MDE is a product decision, and that you can turn n into days.
• Common mistake: running until significant. That is the peeking problem, and it inflates the false positive rate several times over.

Related questions

• When would you use a t-test vs a chi-square test?
• What is Simpson's paradox?
• Explain p-value in plain language.
• What is the difference between Type I and Type II error?
• Explain the Central Limit Theorem and why it matters.
• What is p-hacking and how do you avoid it?

Practice this for real

• Check your resume against a Data Scientist job description — free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews — two free, no card