Machine Learning · easy

Explain how a random forest works.

Asked in ML Engineer interviews, in the Machine Learning round.

Short answer

Bagged decision trees with feature subsampling; average/vote to reduce variance.

How to answer it

Many decision trees, each trained on a different random view of the data, whose predictions are averaged.

Two sources of randomness, and each has a job:

• Bootstrap sampling: each tree trains on a random sample of rows drawn with replacement. Different trees see different data, so their errors are less correlated.
• Feature subsampling: at each split, a tree considers only a random subset of features (root of the feature count for classification, by default). This stops every tree from splitting on the same strong feature first, which would make them near copies of each other.

Individual deep trees overfit; they have low bias and high variance. Averaging many trees whose errors are only weakly correlated keeps the low bias and cuts the variance. That is the whole idea, and it is why more trees never hurt (only cost time) and why the forest is hard to overfit by adding trees.

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=500, max_features="sqrt", oob_score=True, n_jobs=-1).fit(X, y)
rf.oob_score_           # free validation estimate from the left-out bootstrap rows
rf.feature_importances_

Out-of-bag error is a nice detail: each tree left out about a third of the rows, so those rows give a validation estimate without a holdout set. On feature importance, say that impurity-based importance inflates high-cardinality features and permutation importance is more honest.

• What they are checking: that you name both sources of randomness and connect them to variance reduction.
• Common mistake: describing it as boosting, or claiming it reduces bias. It reduces variance.

Related questions

• What metrics would you use for an imbalanced fraud model?
• How do you handle class imbalance in classification?
• Explain L1 vs L2 regularization.
• How would you detect data leakage in a pipeline?
• What is gradient boosting and how does it differ from bagging?
• What is cross-validation and why use k-fold?

Practice this for real

• Check your resume against a ML Engineer job description — free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews — two free, no card