Machine Learning · hard
How would you detect data leakage in a pipeline?
Asked in ML Engineer interviews, in the Machine Learning round.
Short answer
Check for features that use future or target information, fit transforms only on the training set, and audit unrealistically high scores.
How to answer it
Definition
Leakage is when information that would not exist at prediction time reaches the model during training. It produces excellent validation scores and a model that fails in production, which is why it is worth catching early.
The tell
A model that is too good. An AUC of 0.99 on a genuinely hard problem is far more likely to be leakage than a breakthrough, and treating it as suspicious rather than as success is most of the skill here.
The common sources
Target leakage a feature computed from the outcome
"days_until_churn", or refund_amount in a fraud model
Temporal leakage a feature whose value is only known after the prediction
time, such as an account status updated at resolution
Preprocessing leak scaler, imputer or SMOTE fitted on the full dataset
before splitting
Group leakage the same user in both train and test, so the model
recognizes the person rather than the pattern
Duplicate rows near-duplicates split across the two sets
Tuning leakage the test set used repeatedly for model selection until
it no longer measures anythingHow to detect it
- Rank feature importances and interrogate the top one. Leaky features usually dominate.
- Ask of every feature: would this value have existed, with this value, at prediction time? A timestamp per feature makes that answerable rather than a memory exercise.
- Split by time even when the problem is not time series. A model that scores far worse on a temporal split than a random one is telling you something.
- Drop the suspicious feature and see how much the score falls. A large drop from one feature is a strong signal.
- Compare validation to early production performance. A big gap is leakage until proven otherwise.
Key points
- The symptom is implausibly good performance, so treat that as a bug report.
- Fit every transform inside the CV fold, using a scikit-learn Pipeline so it cannot be forgotten.
- Use GroupKFold when rows repeat per entity.
- Give every feature a timestamp and enforce that it precedes the prediction time.
- A feature store with point-in-time correctness is the structural fix at scale.
FAQ
- Is it always obvious in hindsight? Usually, which is why a checklist beats intuition.
- Does leakage always inflate the score? In validation, yes. In production it shows up as a collapse.
- What is the single best guard? Wrap preprocessing in a Pipeline and split by time. Those two catch most of it.
The 30-second answer
Leakage is information reaching the model that would not be available at prediction time, and the symptom is a model that looks too good. So the first thing I do with a suspiciously high score is treat it as a bug rather than a result.
I would look at feature importances and interrogate whatever dominates, then ask of each feature whether that value would have existed at prediction time, with that value. The classic cases are a feature derived from the target, a field updated after the event, and preprocessing fitted before the split.
Structurally I prevent it by putting every transform inside a scikit-learn Pipeline so it is fitted per fold, using GroupKFold when the same user appears many times, and splitting by time even when the problem is not time series, because a large gap between a random split and a temporal one is usually leakage.
In the interview
- What they are checking: that you name time as the usual culprit and that you can propose a test, not just a definition.
- Common mistake: celebrating an AUC of 0.99 on a problem where the previous best was 0.75.