Machine Learning · easy
Asked in Data Scientist interviews, in the Machine Learning round.
Split data into k folds to estimate generalization robustly and use all data for validation.
Cross-validation estimates how a model will do on data it has not seen, using the data you have. Split the data into k folds; train on k minus 1, validate on the held-out one; rotate so every fold is the validation set once; average the k scores.
Why not a single train/validation split: one split gives one number, and that number depends on which rows landed where. On a small dataset the variance of that estimate is large enough to pick the wrong model. K-fold uses every row for validation exactly once, so the estimate is steadier, and it gives k scores whose spread tells you how much to trust the mean.
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipeline, X, y, cv=cv, scoring="average_precision")
scores.mean(), scores.std()The parts that show judgement: stratify so each fold has the same class balance; use grouped folds when rows share an entity (all of one user's rows in the same fold, or the model memorises users); use time-based splits when the data is a time series, because random folds let the model train on the future. And put preprocessing inside the pipeline so scaling and imputation are fit on the training folds only.
Five or ten folds is the convention. Leave-one-out is the extreme and rarely worth its cost.