Statistics · medium
Explain the bias-variance tradeoff.
Asked in ML Engineer interviews, in the Statistics round.
Short answer
Bias is underfitting error, variance is overfitting error, and the goal is the lowest total generalization error.
How to answer it
Definition
Prediction error splits into three pieces: bias, variance and irreducible noise. Bias is error from a model too simple to capture the pattern; variance is error from a model so flexible it fits the noise in the training sample. The chart above shows the sum.
Formula
Expected error = bias^2 + variance + irreducible noiseOnly the first two are yours to trade. The third is a property of the problem.
What each looks like
High bias, low variance underfitting
train error high, test error high, and the two are close
example: a straight line through a curve
Low bias, high variance overfitting
train error very low, test error much higher
example: a deep tree that memorised the training setThe gap between training and test error is the practical read on variance. A model that is excellent in training and mediocre in test has a variance problem, not a bias one.
Moving along the curve
- Reduce variance: more data, regularisation, fewer features, a shallower tree, bagging, early stopping.
- Reduce bias: a richer model, more features, interactions, boosting, less regularisation.
Ensembles are worth naming because they attack different terms. Bagging and random forests average many high-variance models to cut variance. Boosting builds sequentially on the errors of the last model to cut bias.
Key points
- Total error is U-shaped in complexity. The best model is in the middle, not at either end.
- More data reduces variance but does nothing for bias.
- The train-test gap is your variance diagnostic.
- Regularisation deliberately adds bias to remove more variance than it adds.
- Irreducible noise sets a floor. If you are near it, more modeling will not help.
FAQ
- Does deep learning break this? Very large models can show double descent, where test error falls again past the interpolation point. The classic curve still describes most tabular work.
- How do I measure them separately? Rarely directly. Learning curves and the train-test gap are the practical proxies.
- Is high bias always bad? No. A simple, slightly biased model that is stable and explainable often beats a twitchy one in production.
The 30-second answer
Prediction error breaks into bias, variance and irreducible noise. Bias is error from a model too simple to capture the real pattern, so it underfits. Variance is error from a model so flexible it fits the noise in the particular sample it trained on, so it overfits. Total error is the sum, which makes it U-shaped in complexity, and the best model sits in the middle.
In practice I read it off the gap between training and test error. A large gap means variance, so I would add data, regularise, or simplify. Both errors high and close together means bias, so I would use a richer model or better features.
It is also why regularisation works: it deliberately adds a little bias to remove more variance than it adds.
In the interview
- What they are checking: that you can diagnose which one you have from the training and validation curves.
- Common mistake: defining the terms correctly and then having no answer to "so what do you do about it?".