Product Sense · hard · Reported at Spotify
Design an experiment to test a new recommendation algorithm.
Asked in ML Engineer interviews.
Short answer
Randomize at user level, define engagement metric, guardrails, run long enough for novelty decay.
How to answer it
Define success first
Before any design, agree what the algorithm is for. Recommendations optimized for clicks and recommendations optimized for satisfaction produce different systems, and the metric decides which one you build.
Primary one metric. Perhaps conversion, or long-session rate
Secondary engagement depth, items discovered, return within 7 days
Guardrails latency, error rate, revenue, unsubscribes, support ticketsGuardrails matter here specifically because a recommender can win on engagement while quietly harming the experience: clickbait raises clicks and lowers return visits.
The design
Unit the user, not the session or the request. A user seeing both
algorithms across sessions contaminates the comparison
Assignment deterministic hash of user id with experiment id
Split 50/50, or a 5% ramp first if there is any risk to revenue
Duration the pre-computed sample size, and at least two full weeks
to cover the weekly cycle and outlast novelty
Exposure log when a user actually SAW recommendations, not everyone
who was eligibleThe recommender-specific traps
- Novelty: a new algorithm lifts engagement simply because the items are different. Split by new against returning users and watch the effect over the weeks. If it decays and lives only in returning users, it is novelty.
- Feedback loops: the model trains on data its own recommendations generated, so a bad early recommendation suppresses an item that then never gets shown. A small random exposure slice keeps collecting unbiased data.
- Interference: on a marketplace, promoting an item to one group can starve inventory for another, which breaks the independence the test assumes. Cluster or geo-randomize where that is real.
- Cold start: new users and new items behave differently. Segment the analysis, or the average hides both.
Key points
- Agree the primary metric before the design, because it determines the system.
- Randomize on the user, so nobody sees both algorithms.
- Guardrails are essential here: engagement can rise while satisfaction falls.
- Run at least two weeks and check whether the effect decays.
- Watch for interference on a marketplace, where one group's gain is another's loss.
FAQ
- Why not per session? A user seeing both versions gets an inconsistent experience and contaminates the measurement.
- How do I measure satisfaction? Return rate, session length after the click, explicit feedback, complaint rate. No single proxy is sufficient.
- What about an interleaving experiment? Excellent for ranking sensitivity with far less traffic, and it does not measure downstream business outcomes.
The 30-second answer
I would agree the primary metric first, because a recommender optimized for clicks and one optimized for satisfaction are different systems. So one primary metric, a few secondary ones, and guardrails on latency, revenue and return rate, since engagement can go up while the experience gets worse.
Randomization on the user rather than the session, using a deterministic hash, so nobody sees both algorithms. Fifty-fifty, or a small ramp first if revenue is exposed. At least two full weeks to cover the weekly cycle and outlast novelty, and I would log exposure rather than eligibility.
The traps specific to recommenders are novelty, which I would check by splitting new against returning users and watching the effect decay, and interference on a marketplace, where promoting an item to one group can starve another and break the independence the test assumes.
In the interview
- What they are checking: user-level assignment, an outcome metric rather than clicks, and the novelty and feedback-loop issues.
- Common mistake: a two-day test on click-through rate.