Statistics · hard
Explain confidence intervals vs credible intervals.
Asked in Data Scientist interviews, in the Statistics round.
Short answer
A confidence interval is frequentist coverage over repeated samples, a credible interval is Bayesian probability given the data.
| Confidence interval | Credible interval | |
|---|---|---|
| Framework | Frequentist | Bayesian |
| What is random | The interval | The parameter |
| 95% means | 95% of such intervals cover the truth | 95% probability the value is in this one |
| Needs a prior | No | Yes |
| The tempting wrong reading | Treating it as the credible one | None, it reads as people expect |
How to answer it
Definition
Both give a range for a parameter. They differ in what is treated as random, which is why only one of them supports the sentence everybody wants to say. The table above compares them.
The awkward part
A 95% confidence interval does NOT mean there is a 95% probability the true value is inside it. In the frequentist frame the true value is fixed; the interval is what varies from sample to sample. The 95% describes the procedure: repeat the experiment many times and 95% of the intervals it produces contain the truth.
A credible interval does support that sentence. Given the prior and the data, there is a 95% probability the parameter lies in the interval. That is what most people mean, which is why the confidence interval is so persistently misread.
Example
Frequentist: 95% CI for the lift is [0.2%, 1.4%]
"95% of intervals built this way contain the true lift."
Bayesian: 95% credible interval is [0.3%, 1.3%]
"Given the prior and this data, 95% probability the lift is in here."
and also: "P(lift > 0) = 0.98", which a CI cannot give you.Key points
- Confidence interval: the interval is random, the parameter is fixed.
- Credible interval: the parameter is random, the interval is fixed.
- Only the credible interval licenses a probability statement about the parameter.
- The credible interval needs a prior, which is an input you must defend.
- With a lot of data and a weak prior the two are usually close, so the practical difference shrinks as evidence grows.
FAQ
- Which should I use for an A/B test? Either works. Bayesian output is easier for stakeholders because "94% probability B is better" is what they already assume they are reading.
- Does the prior make it subjective? It makes an assumption explicit that the frequentist version leaves implicit in the design. Report a sensitivity check.
- Is a wider interval worse? It is more honest. Width is uncertainty, and narrowing it needs more data, not a different method.
The 30-second answer
Both are ranges for a parameter, and the difference is what is treated as random. In the frequentist view the parameter is fixed and the interval varies from sample to sample, so ninety-five percent confidence describes the procedure: repeat it many times and that share of the intervals cover the truth.
A credible interval is Bayesian, where the parameter has a distribution, so you genuinely can say there is a ninety-five percent probability the value is in this range, given the prior and the data.
That matters because the sentence everybody wants to say about a confidence interval is only true of the credible one. In practice, with plenty of data and a weak prior the two nearly coincide, and I would often report the Bayesian version to stakeholders because it says what they already think it says.
In the interview
- What they are checking: that you can state both definitions precisely and know which one supports "probability the effect is in this range".
- Common mistake: describing the confidence interval as "95% probability the true value is inside". That is the credible interval's sentence.