Machine Learning ยท easy
What is cross-validation and why use k-fold?
Asked in Data Scientist interviews, in the Machine Learning round.
Short answer
Split data into k folds to estimate generalization robustly and use all data for validation.
How to answer it
Definition
Split the data into k parts, train on k minus one of them and validate on the held-out part, rotate through all k, and average the scores. Every row is used for validation exactly once and for training k minus one times.
Why not a single split
A single train-test split gives one estimate from one arbitrary partition. On a small dataset that estimate has high variance: a lucky split flatters the model and an unlucky one condemns it. K-fold averages over k splits, which both reduces that variance and uses all the data.
Example
from sklearn.model_selection import cross_val_score, StratifiedKFold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc")
print(scores.mean().round(3), scores.std().round(3))Output
0.842 0.011 the mean is the estimate,
the spread tells you how much to trust itThe standard deviation matters as much as the mean. A model scoring 0.84 plus or minus 0.01 is a different proposition from 0.84 plus or minus 0.09.
Choosing the variant
StratifiedKFold classification: keeps the class balance in every fold
TimeSeriesSplit temporal data: always train on the past, test on the future
GroupKFold repeated units: keep one user entirely in one foldPlain KFold on time series or on grouped data is a leak, not a choice.
Key points
- Every fold is used for validation exactly once, so the estimate uses all the data.
- 5 or 10 folds is the usual tradeoff. More folds means less bias and more compute.
- Fit every preprocessing step INSIDE the fold. Scaling or imputing on the full dataset leaks the validation set.
- Stratify for classification, especially with imbalance.
- Never shuffle time series. Training on the future to predict the past inflates the score enormously.
- Keep a final untouched test set if you also used CV to tune, since tuning on CV overfits it.
FAQ
- What is leave-one-out? k equal to n. Nearly unbiased, high variance, and expensive.
- Nested CV? An inner loop for tuning and an outer loop for the honest estimate. Correct, and costly.
- Does CV pick the model? It estimates performance. Refit on all the data once you have chosen.
The 30-second answer
Split the data into k folds, train on all but one and validate on the one left out, rotate through all of them, then average. Every row gets used for validation exactly once.
The reason to prefer it over a single split is variance. One split gives one estimate from one arbitrary partition, and on a smallish dataset that number moves a lot with the luck of the split. Averaging over five or ten folds gives a more stable estimate and uses all the data.
The mistake I would watch for is fitting preprocessing outside the loop. Scaling or imputing on the whole dataset before splitting leaks the validation set into training. And for time series I would use TimeSeriesSplit rather than shuffling, because training on the future to predict the past makes any model look excellent.
In the interview
- What they are checking: the variance argument, and whether you know when random folds are wrong.
- Common mistake: fitting the scaler on all the data and then cross-validating, which leaks validation statistics into training.