Randomize at user level, define engagement metric, guardrails, run long enough for novelty decay.
How to answer it
The experiment has to isolate the algorithm's effect on the outcome the company cares about, over a period long enough for novelty to wear off.
Unit and assignment: randomise at the user level, so a user sees one algorithm consistently. Session-level assignment lets the same person see both and contaminates the measurement.
Primary metric: the downstream outcome, not the click. For a streaming service, hours consumed from recommendations or 30-day retention; for commerce, purchases attributed to the recommendation slot. Click-through rate is a secondary metric because a clickbait recommender wins on clicks and loses on satisfaction.
Guardrails: total engagement (the new algorithm must not just shift attention within the page), diversity of what is consumed, latency of the recommendation call, and complaint or skip rate.
Duration: long enough to see the novelty spike decay, typically three to four weeks, and sized on the primary metric with power analysis before starting.
Interference: recommendations feed back into training data. If both arms train on pooled data during the test, the arms leak into each other; freeze the models for the test, or train each on its own arm.
Worked line: "Two arms, user-level, 4 weeks, primary metric is 28-day retention, guardrails are total hours and skip rate. Week 1 showed a 6% click lift that decayed to 1% by week 3, but retention was up 0.8% and held. Ship on the retention result; the click lift was novelty."
What they are checking: user-level assignment, an outcome metric rather than clicks, and the novelty and feedback-loop issues.
Common mistake: a two-day test on click-through rate.