Machine Learning · medium
Explain L1 vs L2 regularization.
Asked in ML Engineer interviews, in the Machine Learning round.
Short answer
L1 (Lasso) drives weights to zero for sparsity and selection, L2 (Ridge) shrinks weights smoothly.
| L1 (lasso) | L2 (ridge) | |
|---|---|---|
| Penalty | Sum of absolute weights | Sum of squared weights |
| Weights go to | Exactly zero | Small but non-zero |
| Selects features | Yes | No |
| Correlated features | Keeps one, drops the rest | Shares weight between them |
| Reach for it when | You want a sparse, explainable model | Everything is somewhat predictive |
How to answer it
Definition
Both add a penalty on coefficient size to the loss, trading a little bias for less variance. They differ in the shape of the penalty, and that shape decides whether coefficients reach exactly zero. The table above compares them.
Formula
L1 (lasso): loss + lambda * sum(|w|)
L2 (ridge): loss + lambda * sum(w^2)
elastic net: loss + lambda * (a * sum(|w|) + (1 - a) * sum(w^2))Why L1 gives exact zeros
The absolute-value penalty has a constant gradient right up to zero, so it keeps pushing a small coefficient all the way there. The squared penalty's gradient shrinks as the coefficient does, so it approaches zero and never arrives. Geometrically the L1 constraint region is a diamond with corners on the axes, and the solution tends to land on a corner, which is a zero.
Example
from sklearn.linear_model import Lasso, Ridge
Lasso(alpha=0.1).fit(X, y).coef_
# array([ 0. , 1.42, 0. , 0. , -0.87]) <- 3 features removed
Ridge(alpha=0.1).fit(X, y).coef_
# array([ 0.03, 1.31, -0.02, 0.01, -0.79]) <- all kept, all shrunkWhich to use
- L1 when you want a sparse, explainable model, or suspect most features are irrelevant.
- L2 when most features carry a little signal, and especially with correlated predictors, since ridge shares weight between them rather than arbitrarily picking one.
- Elastic net when both apply: many features, and groups of correlated ones.
Key points
- Both fight overfitting by penalising large coefficients.
- L1 performs feature selection; L2 does not.
- With correlated features L1 keeps one arbitrarily, which makes it unstable across resamples. L2 spreads the weight.
- Scale your features first. The penalty is on coefficient magnitude, so unscaled features are penalised unequally.
- Lambda controls the strength and is chosen by cross-validation, not by taste.
FAQ
- Does regularisation add bias? Yes, deliberately. It buys a larger reduction in variance.
- Should the intercept be penalised? No. Standard implementations exclude it, since shrinking it just biases the predictions.
- Is dropout related? Conceptually yes: it is a regularisation technique for neural networks, achieving a similar variance reduction differently.
The 30-second answer
Both add a penalty on the size of the coefficients to fight overfitting, and the difference is the shape of that penalty. L1, lasso, penalises absolute values and drives some coefficients to exactly zero, so it does feature selection. L2, ridge, penalises the squares and shrinks everything smoothly without eliminating anything.
The reason L1 reaches zero is that the gradient of the absolute value stays constant all the way down, while the squared penalty's gradient shrinks as the coefficient does, so it approaches zero without arriving.
I use L1 when I want a sparse explainable model, and L2 when most features carry some signal or when they are correlated, because ridge shares weight between correlated predictors instead of picking one arbitrarily. Either way I scale the features first, since the penalty is on magnitude.
In the interview
- What they are checking: "L1 gives zeros, L2 shrinks", and why.
- Common mistake: skipping feature scaling, so the regularisation strength means something different for every column.