Statistics · medium
What is the bootstrap, and when would you use it instead of a formula?
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Resample the data with replacement many times, recompute the statistic each time, and read the spread off those results. Use it when there is no clean formula for the standard error: medians, ratios, percentiles, or a metric per user with heavy tails.
How to answer it
Definition
Resample your data with replacement many times, recompute the statistic on each resample, and use the spread of those values as its sampling distribution. It replaces a formula for the standard error with simulation.
Approach
- Draw a sample of the same size as the original, with replacement.
- Compute the statistic on it.
- Repeat several thousand times.
- Take the 2.5th and 97.5th percentiles of the results for a 95% interval.
Example
import numpy as np
rng = np.random.default_rng(0)
def boot_ci(x, stat=np.median, n=10_000, alpha=0.05):
x = np.asarray(x)
stats = [stat(rng.choice(x, size=len(x), replace=True)) for _ in range(n)]
lo, hi = np.percentile(stats, [100 * alpha / 2, 100 * (1 - alpha / 2)])
return lo, hi
print(boot_ci(revenue))Output
(41.20, 47.85) a 95% interval for the MEDIAN,
which has no simple closed-form standard errorWhen it earns its place
- The statistic has no convenient formula: a median, a 90th percentile, a ratio of two metrics, a Gini coefficient.
- The parametric assumptions are doubtful and the sample is not large enough for the CLT to rescue them.
- The quantity is a complicated function of several estimates, where propagating error by hand is painful and error-prone.
When not to
- The sample is very small. The bootstrap resamples what you have, and if that is unrepresentative it faithfully reproduces the problem.
- The statistic depends on extremes, such as the maximum. Resampling cannot invent values beyond the observed range.
- The observations are dependent. Plain resampling destroys the dependence structure; a block bootstrap is needed for time series.
Key points
- It estimates uncertainty from the data instead of from an assumed distribution.
- It works for statistics with no tractable standard error, which is its main appeal.
- It does not create information. A biased sample bootstraps to a confidently biased answer.
- A few thousand resamples is usually plenty; more reduces simulation noise, not real uncertainty.
- For dependent data, resample blocks rather than individual points.
FAQ
- How many resamples? 1,000 for a rough interval, 10,000 when you want stability in the tails.
- Percentile or BCa interval? Percentile is fine for symmetric cases; BCa corrects for bias and skew and is safer for a skewed statistic.
- Can I bootstrap a p-value? Yes, usually by permutation, which is the closely related tool for testing.
The 30-second answer
The bootstrap estimates the sampling distribution of a statistic by resampling the data with replacement, computing the statistic each time, and using the spread of those values. A ninety-five percent interval is just the two-point-five and ninety-seven-point-five percentiles of the results.
I reach for it when there is no clean formula for the standard error, which covers medians, percentiles and ratios of two metrics, or when the parametric assumptions look shaky and the sample is not big enough for the central limit theorem to cover me.
What I would be careful about is that it does not manufacture information. With a small or biased sample it reproduces that bias confidently, and for time series I would resample blocks rather than individual points, because plain resampling destroys the dependence.
In the interview
- What they are checking: the resampling idea in one sentence, a case where it is the right tool, and the independence caveat.
- Common mistake: bootstrapping rows when the unit of randomization was the user.