Statistics ยท easy
Explain p-value in plain language.
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Probability of observing data as extreme as this, assuming the null hypothesis is true.
How to answer it
Definition
The probability of seeing a result at least this extreme if the null hypothesis were true. It is a statement about the DATA under an assumption, not a statement about the assumption.
What it is not
- Not the probability the null hypothesis is true.
- Not the probability the result happened by chance.
- Not a measure of how big or important the effect is.
All three are the same error: reading a conditional backwards. P(data given no effect) is not P(no effect given data). A test that almost never fires on healthy people still returns mostly false alarms if the disease is rare enough.
Example
Control: 4.0% conversion, 20,000 users
Variant: 4.4% conversion, 20,000 users
p = 0.03Read it as: if the two versions were genuinely identical, a gap this large or larger would show up about 3 times in 100 experiments. That is uncommon enough to act on, and it is not a 97% chance the variant is better.
Key points
- Small p means the data would be surprising under the null. That is all it means.
- The threshold, usually 0.05, is a convention chosen by Fisher, not a fact about the world.
- A p-value says nothing about effect size. With enough traffic a 0.01% lift is significant and worthless.
- p = 0.06 is not "no effect". It is weaker evidence than 0.04, and the cliff between them is arbitrary.
- Report the confidence interval alongside it. The interval carries the size and the uncertainty; the p-value carries neither.
FAQ
- What does p = 1 mean? The data is entirely unsurprising under the null. It is not proof the null is true.
- Does a big sample make p smaller? Yes, for any real non-zero effect. That is why practical significance has to be judged separately.
- Should I ever use 0.01? Yes, when a false positive is expensive, or when running many tests at once.
The 30-second answer
A p-value is the probability of seeing a result at least this extreme if there were genuinely no effect. So it is a statement about the data under an assumption, not about the probability that the assumption is true.
The misreading I would flag is treating it as the chance the null is true, or as the chance the result was a fluke. Those invert the conditional, and it is the most common mistake in the room.
In practice I would not report it alone. The p-value tells you whether something is distinguishable from noise, not whether it is big enough to matter, so I would give the effect size and its confidence interval next to it, because that is what a decision actually needs.
In the interview
- What they are checking: that you can state the conditional correctly ("assuming no effect") and separate significance from size.
- Common mistake: "there is a 95% chance the effect is real". Say that and the follow-up questions get harder.