Statistics ยท hard
What is p-hacking and how do you avoid it?
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Pre-register hypotheses, correct for multiple comparisons, fix sample size in advance.
How to answer it
Definition
Searching, consciously or not, until something crosses p < 0.05, then reporting that result as if it had been the plan all along. The individual steps all look reasonable; the problem is that the search itself is never declared.
What it looks like
- Peeking at a running test and stopping the moment it turns significant.
- Testing twenty metrics and reporting the one that moved.
- Slicing by country, device and plan until a segment shows an effect.
- Dropping "outliers" after seeing that it helps the p-value.
- Trying several models and reporting only the one that worked.
Why it works so well
1 test at alpha 0.05 5% chance of a false positive
20 independent tests 1 - 0.95^20 = 64%Twenty honest tests on twenty useless metrics gives about a coin flip's chance of at least one significant result. Nobody lied; the search space did the work.
Peeking is the same problem
Checking a test daily and stopping at significance is multiple comparisons spread over time. A fixed-horizon test checked ten times can carry a real false-positive rate several times the nominal 5%.
How to avoid it
- Pre-register: the metric, the sample size and the stopping rule, written down before the data arrives.
- One primary metric. Everything else is secondary and labeled exploratory.
- Correct for multiplicity when genuinely testing many things: Bonferroni or Benjamini-Hochberg.
- Use a sequential method if you must monitor: group sequential boundaries or always-valid inference are designed for it.
- Report the whole search. "We looked at twelve segments and this one moved" is honest and readable.
- Confirm a discovery on fresh data. An exploratory finding is a hypothesis, not a result.
Key points
- It is usually not fraud. It is a search that went undeclared.
- Twenty tests at 5% give a 64% chance of at least one false positive.
- Peeking converts a fixed-horizon test into many tests.
- Pre-registration is the single highest-value habit here.
- Exploratory analysis is fine and valuable. Calling it confirmatory is the error.
FAQ
- Is exploratory analysis wrong? No. It generates hypotheses. It just cannot also confirm them.
- Which correction? Bonferroni when false positives are costly and the tests are few; Benjamini-Hochberg when there are many and a known false-discovery rate is acceptable.
- What if a secondary metric moves? Treat it as a hypothesis and design a test for it.
The 30-second answer
P-hacking is searching until something crosses the significance threshold and then presenting it as though it had been the plan. Usually nobody is cheating: they peek at a running test and stop when it turns green, or check twenty metrics and report the one that moved.
The arithmetic is what makes it dangerous. Twenty independent tests at five percent give about a sixty-four percent chance of at least one false positive, so the search itself manufactures the result.
I avoid it by writing down the primary metric, the sample size and the stopping rule before the test runs, correcting for multiplicity when I really am testing many things, and treating anything found by slicing as a hypothesis to confirm on fresh data rather than as a finding.
In the interview
- What they are checking: whether you can name concrete practices, and whether you understand why multiple looks inflate the error rate.
- Common mistake: describing p-hacking only as fraud. Most of it is well-meaning people peeking.