Statistics · medium
Asked in Data Scientist interviews, in the Statistics round.
Resample the data with replacement many times, recompute the statistic each time, and read the spread off those results. Use it when there is no clean formula for the standard error: medians, ratios, percentiles, or a metric per user with heavy tails.
The bootstrap treats your sample as the population. Draw n rows with replacement, compute the statistic, repeat a few thousand times, and the spread of those statistics estimates the sampling distribution. A 95% interval is the 2.5th and 97.5th percentiles of the bootstrap results.
rng = np.random.default_rng(0)
stats = [np.median(rng.choice(x, size=len(x), replace=True)) for _ in range(5000)]
lo, hi = np.percentile(stats, [2.5, 97.5])When it beats a formula: the statistic has no textbook standard error (a median, a 90th percentile latency, a ratio of two means like revenue per session), the data is heavy-tailed so the normal approximation is poor, or you want an interval on something you computed through a pipeline rather than a closed-form estimator.
When it does not help: the sample is tiny (resampling ten rows tells you about those ten rows), the rows are not independent (bootstrap by user or by cluster instead of by row), or the statistic depends on the extremes, like a maximum.
For A/B tests the natural version is to bootstrap the difference between arms and check whether the interval excludes zero. It handles the whales in revenue metrics that break a t-test on small samples.