Machine Learning · hard

What is gradient boosting and how does it differ from bagging?

Asked in ML Engineer interviews, in the Machine Learning round.

Short answer

Boosting fits trees sequentially on residuals (reduce bias); bagging trains independently (reduce variance).

How to answer it

Bagging trains many models independently on bootstrap samples and averages them; it reduces variance. Boosting trains models one after another, each one fitting what the previous ones got wrong; it reduces bias.

Gradient boosting makes "what they got wrong" precise: at each step, fit a small tree to the negative gradient of the loss with respect to the current predictions. For squared error that gradient is the residual, so the next tree literally predicts the errors of the ensemble so far. Add it in, scaled by a learning rate, and repeat. The model is a sum of small trees, each correcting the last.

from sklearn.ensemble import GradientBoostingRegressor
gb = GradientBoostingRegressor(n_estimators=500, learning_rate=0.05, max_depth=3, subsample=0.8)
# XGBoost / LightGBM / CatBoost: same idea, faster, with regularisation and better handling of missing values

The consequences of sequential training are the interview:

• Trees are shallow (depth 3 to 6) because each only needs to correct a little; a forest uses deep trees.
• It can overfit by adding trees, unlike a forest, so use early stopping on a validation set.
• Learning rate and number of trees trade off; lower rate, more trees, better generalisation, more time.
• It cannot be parallelised across trees the way bagging can, only within a tree.

Practical guidance: boosting usually wins on tabular data when tuned; a forest is the robust default when you cannot tune.

• What they are checking: sequential versus independent, bias versus variance, and the residual-fitting mechanism.
• Common mistake: describing boosting as "weighting the misclassified rows", which is AdaBoost, and not the gradient view.

Related questions

• What metrics would you use for an imbalanced fraud model?
• How do you handle class imbalance in classification?
• Explain L1 vs L2 regularization.
• Explain how a random forest works.
• How would you detect data leakage in a pipeline?
• What is cross-validation and why use k-fold?

Practice this for real

• Check your resume against a ML Engineer job description — free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews — two free, no card