System Design · hard · Reported at Airbnb
Design a metrics/experimentation platform.
Asked in Data Scientist interviews, in the System Design round.
Short answer
Assignment service, event logging, metric computation, stats engine, guardrails, dashboards.
How to answer it
Requirements first
What it has to do: define metrics once, assign users to experiments consistently, compute results correctly, and stop people shipping on noise. The hard parts are organizational as much as technical.
Architecture
Metric layer one definition per metric, in version control,
with owner and description. Everything reads from here.
Assignment deterministic hashing of (user_id, experiment_id) to a
bucket, so the same user always lands in the same arm
without storing an assignment table
Exposure logging log WHEN a user actually saw the variant, not when they
were merely eligible. This is the one people get wrong.
Computation scheduled jobs producing effect size, interval and
p-value per metric per experiment
Guardrails a fixed set of metrics checked on every experiment:
latency, errors, revenue, unsubscribes
Interface results with intervals, not just a green tickDeterministic assignment
Hash the user id with the experiment id and take a modulus. It needs no storage, survives restarts, and is consistent across services. Including the experiment id in the hash is what stops the same users landing in the treatment arm of every experiment, which would otherwise correlate all your results.
Exposure, not eligibility
Analyzing everyone who was eligible dilutes the effect toward zero, because most of them never saw the change. Analyzing only those who actually reached the surface is what gives a measurable effect. Getting this wrong is the most common bug in a home-grown platform, and it is invisible: the numbers look plausible and are simply smaller than they should be.
Guardrails and sanity checks
- Sample ratio mismatch: if a 50/50 split arrives 52/48, the assignment or the logging is broken and the result is void. Check it automatically on every experiment.
- A/A tests: run them regularly. About 5% should come back significant. If far more do, the pipeline is wrong.
- Guardrail metrics on every experiment, so a conversion win that doubles latency is caught.
Key points
- One metric definition, in version control, read by everything.
- Deterministic hashed assignment including the experiment id.
- Log exposure, not eligibility.
- Sample ratio mismatch checks are the cheapest correctness guard there is.
- Show intervals, not just significance, or people ship noise.
FAQ
- Build or buy? Buy early. The value is in the metric definitions and the discipline, not the plumbing.
- How do I handle many simultaneous experiments? Independent hashing makes overlaps random, which is usually fine; use mutually exclusive layers when two changes genuinely interact.
- Frequentist or Bayesian? Either, consistently. Bayesian output tends to be read more correctly by non-specialists.
The 30-second answer
The core pieces are a metric layer where each metric is defined once in version control, deterministic assignment, exposure logging, a computation job, and guardrail metrics.
For assignment I would hash the user id together with the experiment id and take a modulus, so it needs no storage and the same user always lands in the same arm. Including the experiment id matters, otherwise the same users end up in treatment for everything and the results correlate.
The detail I would emphasise is logging exposure rather than eligibility. Analyzing everyone who could have seen the change dilutes the effect toward zero, because most of them never reached it. And I would run automatic sample ratio mismatch checks, because a fifty-fifty split arriving fifty-two forty-eight means the result is void, and that is the cheapest correctness guard available.
In the interview
- What they are checking: deterministic assignment, exposure versus assignment, one metric definition, and SRM.
- Common mistake: designing the UI first. The platform is the metric registry and the stats engine, the UI is the last week.