Statistics · medium

Explain statistical power and what increases it.

Asked in Data Scientist interviews, in the Statistics round.

Short answer

Power is the probability of detecting an effect that is really there, one minus the Type II error rate. It rises with sample size, effect size and a higher alpha, and falls with variance. 80% is the usual target.

How to answer it

Definition

Power is the probability of detecting a real effect: the chance the test comes back significant given that the effect genuinely exists. It equals 1 minus beta, the false-negative rate.

Formula

power = P(reject the null | the null is false)
      = 1 - beta

convention: 0.80, which accepts missing 1 real effect in 5

What increases it

larger sample                power up      the usual lever, and quadratic
larger true effect           power up      not yours to choose
lower variance               power up      better metric, CUPED, blocking
higher alpha                 power up      at the cost of more false positives
one-tailed test              power up      only when genuinely defensible
paired design                power up      removes between-subject variance

Variance reduction deserves more attention than it usually gets. Trimming the metric's noise, or using pre-experiment data as a covariate, buys power without buying traffic, which is often the only lever available.

The underpowered trap

An underpowered test that comes back significant is not reassuring. Because only a large observed effect can clear the threshold on a small sample, the effects that do get reported from underpowered studies are systematically overstated. This is the winner's curse, and it is why replications so often disappoint.

Key points

• Power is about false negatives, the mirror of alpha's false positives.
• 0.80 is convention: a real effect is missed one time in five.
• Sample size is quadratic. Half the effect needs four times the data.
• Reducing variance raises power without more traffic and is usually underused.
• A significant result from an underpowered test likely overstates the effect.

FAQ

• Should I compute power after the test? Observed power adds nothing beyond the p-value. Report the confidence interval instead.
• What if the test is underpowered and null? It is inconclusive, not evidence of no effect. The interval shows what you could and could not rule out.
• Is 0.80 always right? Raise it when missing a real effect is expensive; a safety-critical decision might want 0.95.

The 30-second answer

Power is the probability of detecting an effect that is genuinely there, so it is one minus the false-negative rate. Eighty percent is the usual target, which means accepting that one real effect in five gets missed.

Four things raise it: a larger sample, a larger true effect, lower variance, and a looser significance threshold. Sample size is the obvious lever and it is quadratic, so halving the effect I want to detect quadruples the traffic. The underrated one is variance reduction, using a less noisy metric or pre-experiment data as a covariate, because it buys power without buying traffic.

I would also flag that a significant result from an underpowered test tends to overstate the effect, since only a large observed effect could have cleared the bar at that sample size.

In the interview

• What they are checking: the four levers and the fact that power is planned up front.
• Common mistake: running until significance appears. That inflates the false positive rate, it is not power.

Practice this for real

• Check your resume against a Data Scientist job description, free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews, two free, no card