Statistics · easy
What is a confidence interval, and what does 95% actually mean?
Asked in Data Analyst interviews, in the Statistics round.
Short answer
A range built from the sample by a procedure that would contain the true value in 95% of repeated samples. It is a statement about the procedure, not a 95% probability that this particular interval holds the truth.
How to answer it
Definition
A range of plausible values for a parameter, built from the data by a procedure with a known long-run success rate. The 95% describes the procedure, not the particular interval in front of you.
What 95% means
Repeat the experiment many times, building an interval the same way each time, and about 95% of those intervals contain the true value. Any single interval either contains it or does not; the probability lives in the method.
Formula
estimate +/- 1.96 * standard error for a 95% interval on a mean
standard error = sigma / sqrt(n)Example
Conversion lift: 0.8 percentage points
95% CI: [0.2, 1.4]
Readable as: the data is consistent with a lift anywhere from 0.2 to 1.4
points. It excludes zero, so it is significant at the 5% level, and it is
also telling you the effect might be small enough not to matter.Why it beats a p-value
An interval carries three things at once: the direction, the size, and the precision. A p-value carries only distinguishable from noise, or not. An interval of [0.2, 1.4] and one of [0.79, 0.81] can share a p-value while supporting completely different decisions.
Key points
- The 95% is a property of the procedure across repetitions, not of one interval.
- Width is uncertainty. It shrinks with the square root of n, so precision is expensive.
- If the interval excludes zero the result is significant at that level, which is why the p-value is often redundant.
- An interval that excludes zero but includes only trivial values is significant and not worth acting on.
- To say "95% probability the value is in here" you want a credible interval, which needs a prior.
FAQ
- Why 1.96? It is the z-score cutting off 2.5% in each tail of the normal. With small samples use the t-distribution instead.
- Can I compare two intervals by eye? Roughly. Non-overlapping means a significant difference, but overlapping intervals can still be significantly different, so test properly.
- Does it assume normality? It assumes the ESTIMATE is roughly normal, which the CLT usually provides even for skewed data.
The 30-second answer
A confidence interval is a range of plausible values for a parameter, and the ninety-five percent describes the procedure rather than the interval in front of you. If I repeated the experiment many times and built an interval each time the same way, about ninety-five percent of those intervals would contain the true value.
It is tempting to say there is a ninety-five percent probability the truth is in this one, but in the frequentist frame the truth is fixed and the interval is what varies. The statement people want belongs to a Bayesian credible interval.
Practically I prefer reporting the interval over the p-value, because it carries the size and the precision as well as the significance. An interval that excludes zero but only contains trivial values tells you to move on, which a p-value never would.
In the interview
- What they are checking: that you know the 95% is about repeated sampling, and that you can say what changes the width.
- Common mistake: reading two overlapping intervals as "no significant difference".