Statistics · easy
What is the difference between Type I and Type II error?
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Type I is a false positive (rejecting a true null), Type II is a false negative (failing to reject a false null).
| Type I error | Type II error | |
|---|---|---|
| The truth | No real effect | There is a real effect |
| Your conclusion | There is an effect | No effect |
| Also called | False positive | False negative |
| Rate | Alpha, the significance level you pick | Beta. Power is 1 minus beta |
| What it costs | You ship a change that does nothing | You bin a change that worked |
| To reduce it | Lower alpha, correct for multiple tests | More sample, bigger effect, less noise |
How to answer it
Definition
Two ways to be wrong, and which one you made depends on what was true and what you concluded. The table above sets them side by side.
The tradeoff
They trade against each other at a fixed sample size. Lowering alpha to make false positives rarer moves the threshold further out, so real effects clear it less often and beta rises. The only way to reduce both at once is more information: a bigger sample, a larger effect, or less noise.
Example
alpha = 0.05 1 in 20 tests on a useless change will look like a winner
power = 0.80 so beta = 0.20: 1 in 5 real wins is missedRun 100 A/B tests on changes that genuinely do nothing and about 5 come back significant. That is not a bug in the method; it is what alpha means.
Which error is worse
It depends entirely on the cost, and saying so is the answer:
- A medical screening test: a false negative sends someone home sick. Favor power.
- A spam filter: a false positive deletes a real email. Favor precision.
- An A/B test on a cheap, reversible UI change: a false positive costs little, so a relaxed alpha may be rational.
Key points
- Type I is a false positive, controlled directly by alpha.
- Type II is a false negative, with rate beta; power is 1 minus beta.
- At fixed n they trade off. Only more data, a bigger effect or less variance improves both.
- Running many tests multiplies Type I errors, which is why corrections exist.
- The right balance is a cost question, not a statistics question.
FAQ
- Why is 0.05 standard? Convention from Fisher, not a derived value. Fields with expensive false positives use much smaller thresholds.
- What raises power? A larger sample, a larger true effect, lower variance, and a one-tailed test where it is defensible.
- Is there a Type III error? Informally, answering the wrong question correctly, or getting the direction of a real effect backwards.
The 30-second answer
A Type I error is a false positive: I conclude there is an effect when there is not, and its rate is alpha, the significance level I choose. A Type II error is a false negative: there is a real effect and I miss it, at rate beta, and power is one minus beta.
The thing worth adding is that they trade against each other at a fixed sample size, so tightening alpha to avoid false positives means missing more real effects. The only way to improve both is more data, a bigger effect, or less noise.
Which one I would rather make depends on the cost. For a medical screen a false negative is far worse, so I would protect power. For an expensive irreversible launch I would protect alpha.
In the interview
- What they are checking: whether you can say which error matters more in a given setting, not just the definitions.
- Common mistake: getting them backwards under pressure. Anchor on "Type I = reject a true null = false alarm" and derive the other.