Machine Learning · hard
What is gradient boosting and how does it differ from bagging?
Asked in ML Engineer interviews, in the Machine Learning round.
Short answer
Boosting fits trees sequentially on residuals to reduce bias, bagging trains them independently to reduce variance.
| Bagging (random forest) | Boosting (GBM, XGBoost) | |
|---|---|---|
| Trees are built | In parallel, independently | In sequence, each on the residuals |
| Each tree sees | A bootstrap sample | The errors still left over |
| Mainly reduces | Variance | Bias |
| Base learner | Deep trees, low bias | Shallow trees, weak learners |
| Main risk | Little, hard to overfit | Overfits if you boost too long |
| Tuning that matters | Number of trees, features per split | Learning rate, depth, early stopping |
How to answer it
Definition
Boosting builds trees in sequence, each one fitted to the errors the previous ones are still making. Bagging builds them independently and averages. The table above compares them; the difference in what they fix is the point.
How boosting works
1. Start with a constant prediction, usually the mean
2. Compute the residuals: what the current model gets wrong
3. Fit a small tree to those residuals
4. Add it to the model, scaled by the learning rate
5. RepeatEach tree is a weak learner, typically only 3 to 6 levels deep. The strength comes from the sequence, not from any individual tree.
Why the learning rate matters
learning rate 0.3, 100 trees fast, and prone to overfitting
learning rate 0.03, 1000 trees slower, and usually generalises betterThe learning rate shrinks each tree's contribution, so more trees are needed and no single one dominates. Low rate with many trees and early stopping is the standard recipe.
The key difference
Bagging parallel, independent trees, each deep
attacks VARIANCE by averaging
more trees converges and never overfits
Boosting sequential, each tree fits what is left over, each shallow
attacks BIAS by refining
more trees DOES eventually overfit, so early stopping mattersKey points
- Boosting reduces bias; bagging reduces variance. That is the sentence to say.
- Boosting trees are shallow on purpose. Deep trees would fit the residual noise.
- Learning rate and number of trees trade off directly. Tune them together.
- Boosting can overfit with too many rounds, so use a validation set and early stopping.
- It is sequential, so it parallelises less well than a random forest.
FAQ
- XGBoost, LightGBM, CatBoost? All gradient boosting with different split-finding and categorical handling. LightGBM is fast on large data; CatBoost handles categoricals well.
- Why is it so strong on tabular data? It builds capacity gradually and directly targets the remaining error, which suits heterogeneous features.
- What is AdaBoost? An earlier variant that reweights misclassified examples rather than fitting residuals.
The 30-second answer
Gradient boosting builds trees in sequence, where each new tree is fitted to the errors the model so far still makes, and is added scaled by a learning rate. The trees are deliberately shallow, so the strength comes from the sequence rather than from any one of them.
Bagging, like a random forest, does the opposite: it builds deep trees independently on bootstrap samples and averages them. So bagging attacks variance by averaging, and boosting attacks bias by refining.
The practical consequence is that adding trees to a forest converges harmlessly, while adding rounds to a boosted model eventually overfits, so I would use a low learning rate with many trees and early stopping on a validation set.
In the interview
- What they are checking: sequential versus independent, bias versus variance, and the residual-fitting mechanism.
- Common mistake: describing boosting as "weighting the misclassified rows", which is AdaBoost, and not the gradient view.