Machine Learning · medium · Reported at Stripe
What metrics would you use for an imbalanced fraud model?
Asked in Data Scientist interviews, in the Machine Learning round.
Short answer
Precision, recall, PR-AUC and F-beta. Avoid raw accuracy, and consider cost-weighted metrics.
How to answer it
Definition
Metrics that ignore the enormous true-negative class and expose the tradeoff between catching fraud and annoying legitimate customers. Accuracy and ROC-AUC both fail that test here.
What to use
Recall of the real fraud, how much did we catch
Precision of what we flagged, how much was actually fraud
PR-AUC precision-recall across all thresholds; the headline number
F-beta one number, weighted toward recall when a miss costs more
than a review
Cost-weighted expected money lost, using real costs per error type
Recall at fixed
review capacity the operational version: how much fraud can we catch given
that the team reviews 500 cases a dayWhy not accuracy or ROC-AUC
1% fraud rate, model predicts "legitimate" always:
accuracy 99% meaningless
ROC-AUC uses the false positive rate = FP / (FP + TN).
With 99% legitimate, TN is enormous, so FP/(FP+TN) stays tiny even when
most of what you FLAG is wrong. ROC-AUC looks great while precision is awful.
PR-AUC ignores true negatives entirely, which is why it is the honest curve.The cost matrix is the real answer
false negative: fraud goes through, average loss $220
false positive: a good customer is blocked, cost $8 review
plus churn risk that is harder to priceWith those numbers a false negative is worth about 27 false positives, which sets the threshold directly. Volunteering this is what separates a strong answer from a list of metric names.
Key points
- PR-AUC over ROC-AUC whenever the positive class is rare.
- Accuracy is unusable at these ratios and should be said out loud.
- Convert errors into money. The threshold follows from the cost ratio.
- Constrain by capacity: precision at the top N cases is often what operations actually needs.
- Monitor after deployment. Fraud patterns shift, so today's threshold decays.
FAQ
- What about F1? Fine as a summary, but it weighs precision and recall equally, which the business rarely does. F-beta lets you say which matters more.
- Should I report a single number? Report PR-AUC for the model and precision at the operating recall for the decision.
- How do I set the threshold? From the cost ratio and the review capacity, not from 0.5.
The 30-second answer
Not accuracy: at a one percent fraud rate, predicting legitimate every time scores ninety-nine percent. I would also avoid ROC-AUC, because the false positive rate divides by an enormous true-negative count, so it looks strong while most of what you flag is wrong.
I would use precision and recall, with PR-AUC as the headline since it ignores true negatives, and F-beta weighted toward recall if a missed fraud costs more than a review.
The answer I would actually push for is a cost matrix. If a missed fraud is two hundred dollars and a false positive costs about eight in review time, one miss is worth roughly twenty-seven false alarms, and that ratio sets the threshold. I would also report precision at whatever recall the review team can actually handle, because that is the real operational constraint.
In the interview
- What they are checking: PR over ROC, precision at capacity, and whether you convert the tradeoff into cost.
- Common mistake: reporting ROC-AUC of 0.98 and calling it done.