Five components, and the hard problems live in the seams between them.
Assignment: a service that, given a user and an experiment, returns an arm deterministically (hash of user id and experiment salt), so the same user always sees the same arm and assignment can be recomputed. Supports mutual exclusion between experiments that would interfere, and traffic ramps.
Exposure logging: an event when the user actually sees the treatment. Analysing on assignment rather than exposure dilutes the effect with users who were assigned but never reached the feature.
Metric definitions: a registry where each metric is defined once (numerator, denominator, unit, window, filters) and computed by one pipeline. This is the part that decides whether teams trust the platform; two definitions of "conversion" and the platform is over.
Stats engine: computes per-arm metrics, confidence intervals, and p-values, with the corrections that matter at scale: CUPED or regression adjustment for variance reduction, delta method for ratio metrics, sequential testing if peeking is allowed, and multiple-comparison control across metrics.
Results and guardrails: a UI that shows primary and guardrail metrics with the pre-registered hypothesis, flags underpowered tests and sample ratio mismatch, and requires a decision to be recorded.
The seams: the assignment log and the metrics pipeline must join on the same user id and the same timestamps; exposure must be logged by the client that renders the treatment; and the metrics pipeline must respect the experiment window per user, not calendar time.
Operational: sample ratio mismatch checks run automatically, because a 50/50 split that comes out 52/48 means assignment or logging is broken and every result is invalid.
What they are checking: deterministic assignment, exposure versus assignment, one metric definition, and SRM.
Common mistake: designing the UI first. The platform is the metric registry and the stats engine; the UI is the last week.