Statistics · hard

How do you correct for multiple comparisons?

Asked in Data Scientist interviews, in the Statistics round.

Short answer

Testing twenty metrics at alpha 0.05 gives a 64% chance of at least one false positive. Bonferroni divides alpha by the number of tests and is safe but conservative. Benjamini Hochberg controls the false discovery rate and keeps more real findings.

How to answer it

Definition

Every test carries its own false-positive risk, so testing many hypotheses at once multiplies the chance that at least one looks significant by luck. A correction tightens the threshold to hold some error rate across the family.

The problem

tests    P(at least one false positive) at alpha 0.05
    1                                    5%
    5                                   23%
   20                                   64%
  100                                   99.4%

The two families

Bonferroni       alpha / m
                 controls the family-wise error rate: P(any false positive)
                 simple, conservative, and loses real effects when m is large

Benjamini-Hochberg
                 sort the p-values, compare p(i) against (i/m) * alpha
                 controls the false discovery rate: the expected SHARE of
                 discoveries that are false
                 far more power, and the right default when m is large

Example

m = 10 tests, alpha = 0.05

Bonferroni:  each test must beat 0.005

Benjamini-Hochberg, sorted p-values:
  rank  p        (i/m)*0.05     verdict
     1  0.001    0.005          pass
     2  0.008    0.010          pass
     3  0.039    0.015          fail -> stop; ranks 1 and 2 are the discoveries

Which to use

Family-wise control when a single false positive is expensive: a drug trial, a launch you cannot reverse. False discovery rate control when you are screening many candidates and can tolerate a known share of false leads, which describes most analytics work.

Key points

• The correct family is the set of tests you actually ran, including the ones you did not report.
• Bonferroni controls any false positive; BH controls the proportion of your discoveries that are false.
• BH has substantially more power once m is more than a handful.
• The best correction is often fewer tests: pick one primary metric in advance.
• Peeking at a running experiment is multiple comparisons over time, and needs a sequential method rather than these.

FAQ

• Do I correct for metrics I looked at but did not report? Yes. The family is what you tested, not what you published.
• Is Bonferroni too strict? With many tests, yes, which is what BH exists to fix.
• What about correlated tests? Both are conservative under positive correlation. Bonferroni-Holm or permutation approaches do better.

The 30-second answer

Testing many hypotheses multiplies the false-positive risk. At five percent, twenty independent tests give roughly a sixty-four percent chance of at least one significant result even when nothing is real.

There are two things you might want to control. Bonferroni divides alpha by the number of tests and controls the chance of any false positive at all, which is right when one false positive is expensive, but it gets very conservative as the count grows. Benjamini-Hochberg controls the false discovery rate, the expected share of my discoveries that are wrong, and keeps far more power when there are many tests.

For most analytics work I would use Benjamini-Hochberg, and I would count every test I ran in the family, not just the ones I reported.

In the interview

• What they are checking: the arithmetic of inflation, the difference between family-wise error and false discovery rate, and pre-registering a primary metric.
• Common mistake: correcting the primary metric along with fifty exploratory ones and calling the real effect noise.

Practice this for real

• Check your resume against a Data Scientist job description, free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews, two free, no card