Statistics · easy

What is a confidence interval, and what does 95% actually mean?

Asked in Data Analyst interviews, in the Statistics round.

Short answer

A range built from the sample by a procedure that would contain the true value in 95% of repeated samples. It is a statement about the procedure, not a 95% probability that this particular interval holds the truth.

How to answer it

A 95% confidence interval is a range computed from your sample by a procedure that, over many repeated samples, captures the true parameter 95% of the time.

The subtle part: once you have a specific interval, say 4.1% to 5.3% conversion, the truth is either in it or not. The 95% belongs to the procedure. Saying "there is a 95% probability the true rate is between 4.1% and 5.3%" is the Bayesian credible-interval reading, and most interviewers will let it pass in conversation but will probe if you claim it is the definition.

What makes it narrower: a bigger sample (the width shrinks with root n), less variance, or a lower confidence level. A 90% interval is narrower than a 95% one, which is a trade you should name rather than hide.

For a proportion, the quick form is p plus or minus 1.96 times the square root of p(1 minus p) over n. For an A/B test, report the interval on the difference between arms, not two separate intervals. Two overlapping intervals can still be a significant difference.

Related questions

Practice this for real