Statistics · hard

What is p-hacking and how do you avoid it?

Asked in Data Scientist interviews, in the Statistics round.

Short answer

Pre-register hypotheses, correct for multiple comparisons, fix sample size in advance.

How to answer it

P-hacking is any practice that lets you keep testing until something clears 0.05: trying many metrics and reporting the one that worked, slicing by segment until one segment is significant, stopping the test the day it crosses the line, or adjusting the model until p drops. Each is a way of running many tests while reporting one, and at 20 tests you expect one false positive for free.

Avoiding it is mostly about deciding things before seeing data:

• Pre-register: one primary metric, the hypothesis, the sample size and the stopping rule, written down before the test starts.
• Fix the sample size and do not stop early. If you need to look mid-way, use a sequential method that budgets for it.
• Correct for multiple comparisons when you do test several metrics or segments: Bonferroni if there are a few, Benjamini-Hochberg to control the false discovery rate if there are many.
• Treat segment findings as hypotheses for the next test, not as results.

from statsmodels.stats.multitest import multipletests
reject, p_adj, *_ = multipletests(p_values, alpha=0.05, method="fdr_bh")

And say the uncomfortable part: the pressure to p-hack is organisational. A team whose tests "have to win" will find wins. The fix is reporting the negatives as routinely as the positives.

• What they are checking: whether you can name concrete practices, and whether you understand why multiple looks inflate the error rate.
• Common mistake: describing p-hacking only as fraud. Most of it is well-meaning people peeking.

Related questions

• When would you use a t-test vs a chi-square test?
• What is Simpson's paradox?
• Explain p-value in plain language.
• How do you determine sample size for an A/B test?
• What is the difference between Type I and Type II error?
• Explain the Central Limit Theorem and why it matters.

Practice this for real

• Check your resume against a Data Scientist job description — free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews — two free, no card