Statistics · medium

Explain the bias-variance tradeoff.

Asked in ML Engineer interviews, in the Statistics round.

Short answer

Bias = underfitting error; variance = overfitting error; minimize total generalization error.

How to answer it

A model's error on new data has two controllable parts. Bias is the error from the model being too simple to represent the pattern: it is wrong in the same way on every dataset. Variance is the error from the model being too sensitive to the particular training set: retrain on a different sample and it changes a lot. Plus irreducible noise you cannot do anything about.

Simple models have high bias and low variance; flexible models have low bias and high variance. As you add capacity, bias falls and variance rises, and total error is U-shaped. The goal is the bottom of the U, not either end.

How you see it in practice is the useful part:

• High bias: training error and validation error are both high and close together. The model is underfitting. Add features, add capacity, reduce regularisation.
• High variance: training error is low and validation error is much higher. The model is overfitting. More data, regularisation, early stopping, ensembling, fewer features.

from sklearn.model_selection import validation_curve
train_scores, val_scores = validation_curve(
    model, X, y, param_name="max_depth", param_range=range(1, 20), cv=5)
# the gap between train and val opening up is variance; both low is bias

Mention that modern deep networks complicate the classic curve (double descent), but the diagnostic, comparing training error with validation error, still works.

• What they are checking: that you can diagnose which one you have from the training and validation curves.
• Common mistake: defining the terms correctly and then having no answer to "so what do you do about it?".

Related questions

• When would you use a t-test vs a chi-square test?
• What is Simpson's paradox?
• Explain p-value in plain language.
• How do you determine sample size for an A/B test?
• What is the difference between Type I and Type II error?
• Explain the Central Limit Theorem and why it matters.

Practice this for real

• Check your resume against a ML Engineer job description — free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews — two free, no card