Statistics ยท medium
How do you detect and handle multicollinearity?
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Check VIF and correlations, then drop or combine features, or use regularization or PCA.
How to answer it
Definition
Two or more predictors carrying much the same information. The model can still predict well; what breaks is the interpretation of individual coefficients, because the data cannot separate their contributions.
How to detect it
# 1. correlation matrix: catches pairwise collinearity only
X.corr()
# 2. VIF: catches a variable explained by a COMBINATION of others
from statsmodels.stats.outliers_influence import variance_inflation_factor
[variance_inflation_factor(X.values, i) for i in range(X.shape[1])]VIF = 1 / (1 - R2_i) R2_i from regressing predictor i on all the rest
VIF 1 uncorrelated with the others
VIF 5 often flagged
VIF 10+ usually treated as a problemThe symptom
A high R-squared with individually insignificant coefficients, or a coefficient whose sign flips when you add another variable. That combination is close to diagnostic.
Handling it
- Drop one of the pair. Cheapest, and fine when they are near-duplicates.
- Combine them: an index, or a principal component, at the cost of interpretability.
- Regularise. Ridge in particular shares weight between correlated predictors and stabilises them.
- Get more data, where it is the sample rather than the structure creating the correlation.
- Do nothing, when prediction is the goal. Collinearity does not harm predictive accuracy on similar data.
Key points
- It inflates standard errors, so coefficients become unstable and insignificant.
- It does not bias the predictions or reduce overall fit.
- A correlation matrix misses multi-way collinearity; VIF catches it.
- Perfect collinearity, such as including all dummy levels plus an intercept, makes the model unidentifiable outright.
- If you only need predictions, this may be a non-issue. Say so rather than fixing reflexively.
FAQ
- Where does the VIF threshold come from? Convention. 5 and 10 are rules of thumb, not tests.
- Does it affect a random forest? Not the accuracy, but it does split and dilute feature-importance scores between correlated features.
- What is the dummy variable trap? Including every category level plus an intercept gives perfect collinearity. Drop one level.
The 30-second answer
Multicollinearity is when predictors carry overlapping information, so the model cannot attribute effect between them. I detect it with a correlation matrix for pairs and VIF for the cases where one variable is explained by a combination of several, which a pairwise correlation will not show. Above about five is worth a look and above ten usually needs action.
The symptom I watch for is a good overall R-squared with individually insignificant coefficients, or a sign that flips when another variable is added.
How I handle it depends on the goal. If I only need predictions I might do nothing, because it does not hurt accuracy. If I need to interpret coefficients I would drop one, combine them into an index, or use ridge, which shares weight between correlated predictors rather than letting them fight.
In the interview
- What they are checking: VIF over a correlation matrix, and the distinction between prediction and inference.
- Common mistake: dropping features on correlation alone and losing signal the model needed.