Machine Learning · medium · Asked at Stripe

What metrics would you use for an imbalanced fraud model?

Asked in Data Scientist interviews, in the Machine Learning round.

Short answer

Precision, recall, PR-AUC, F-beta; avoid raw accuracy; consider cost-weighted metrics.

How to answer it

Not accuracy. With 0.2% fraud, a model that flags nothing is 99.8% accurate.

The metrics that describe a fraud model's actual job:

• Precision at the operating point: of the transactions you block or review, how many were fraud. This is the false-positive cost, and false positives are blocked customers.
• Recall at the operating point: of the fraud that happened, how much you caught. This is the loss you prevented.
• PR-AUC to compare models across thresholds. ROC-AUC looks flattering on rare positives because the huge negative class dominates the false positive rate; PR-AUC does not.
• Precision at a fixed review capacity: if the ops team can review 500 cases a day, what is precision in the top 500? That is the number the team lives with.

Then turn it into money. A missed fraud costs the transaction; a false block costs a customer, some of whom leave. Put those two costs on the confusion matrix and choose the threshold that minimises expected cost. That is the answer to "which is more important, precision or recall": it depends on the two prices, and you can compute it.

from sklearn.metrics import average_precision_score, precision_recall_curve
ap = average_precision_score(y_val, scores)          # PR-AUC
prec, rec, thr = precision_recall_curve(y_val, scores)

Mention that fraud drifts, so the metric must be reported on recent data, and that evaluation must respect time: train on the past, test on the future.

• What they are checking: PR over ROC, precision at capacity, and whether you convert the tradeoff into cost.
• Common mistake: reporting ROC-AUC of 0.98 and calling it done.

Related questions

• How do you handle class imbalance in classification?
• Explain L1 vs L2 regularization.
• Explain how a random forest works.
• How would you detect data leakage in a pipeline?
• What is gradient boosting and how does it differ from bagging?
• What is cross-validation and why use k-fold?

Practice this for real

• Check your resume against a Data Scientist job description — free, no account needed.
• Every data interview question, by role and topic
• Scored AI mock interviews — two free, no card