Machine Learning ยท easy
Explain how a random forest works.
Asked in ML Engineer interviews, in the Machine Learning round.
Short answer
Bagged decision trees with feature subsampling, averaged or voted to reduce variance.
How to answer it
Definition
Many decision trees trained on different random views of the data, whose predictions are averaged for regression or voted for classification. The randomness is the point: it makes the trees' errors partly independent, so averaging cancels them.
The two sources of randomness
1. Bagging each tree trains on a bootstrap sample of the ROWS
(sampled with replacement, so about 63% of rows appear)
2. Feature subsampling
at each split, only a random subset of COLUMNS is
considered (commonly sqrt(p) for classification)The second one is what distinguishes a random forest from plain bagged trees. Without it, one dominant feature would sit at the top of nearly every tree and the trees would be highly correlated, so averaging would gain little.
Why averaging works
variance of the mean of n correlated variables
= rho * var + (1 - rho) * var / nThe second term shrinks with more trees. The first does not, and it is set by rho, how correlated the trees are. That formula is the whole design: bagging and feature subsampling exist to push rho down so averaging can work.
Key points
- Individual trees are deep and overfit deliberately: low bias, high variance. Averaging removes the variance.
- More trees never overfits. It converges. Depth and leaf size are the parameters that control overfitting.
- Out-of-bag error is nearly free validation, since each row is unseen by about 37% of the trees.
- Feature importances are biased toward high-cardinality features. Prefer permutation importance or SHAP.
- It handles mixed types and needs no scaling, which is why it is such a strong tabular baseline.
FAQ
- Random forest or gradient boosting? Boosting usually wins on accuracy with tuning; forests are more forgiving and harder to get wrong.
- How many trees? A few hundred is usually plenty. Watch the out-of-bag error flatten.
- Does it work for regression? Yes, averaging the leaf means instead of voting.
The 30-second answer
It trains many decision trees, each on a bootstrap sample of the rows, and at every split each tree only considers a random subset of the features. Then it averages the predictions for regression, or votes for classification.
Each individual tree is deliberately deep and overfits, so it has low bias and high variance. Averaging many of them cancels the variance, provided their errors are not too correlated, and that is exactly what the two sources of randomness are for. The feature subsampling in particular stops one strong predictor sitting at the top of every tree and making them all alike.
Practically it is a strong tabular baseline: no scaling needed, handles mixed types, and out-of-bag error gives validation almost for free. Adding more trees never overfits; depth and leaf size are what control that.
In the interview
- What they are checking: that you name both sources of randomness and connect them to variance reduction.
- Common mistake: describing it as boosting, or claiming it reduces bias. It reduces variance.