Statistics · hard
Asked in Data Scientist interviews, in the Statistics round.
Testing twenty metrics at alpha 0.05 gives a 64% chance of at least one false positive. Bonferroni divides alpha by the number of tests and is safe but conservative. Benjamini Hochberg controls the false discovery rate and keeps more real findings.
Every test you run at alpha 0.05 has a 5% false positive rate. Run twenty on the same experiment and the chance of at least one false positive is 1 minus 0.95 to the power 20, about 64%. That is the multiple comparisons problem, and it shows up as "the test moved metric fourteen".
Bonferroni: divide alpha by the number of tests. Twenty tests, alpha 0.0025 each. It controls the family-wise error rate, the chance of any false positive, and it is conservative, so it throws away real effects when the tests are correlated.
Benjamini Hochberg: sort the p-values, compare the k-th smallest to k over m times alpha, and accept everything up to the largest one that passes. It controls the false discovery rate, the expected share of your "wins" that are false, which is usually what a product team actually cares about.
The better answer is design, not correction: declare one primary metric before the test, a few guardrails, and treat everything else as exploratory. A correction is what you apply when you did not.
from statsmodels.stats.multitest import multipletests
reject, p_adj, *_ = multipletests(pvals, alpha=0.05, method="fdr_bh")