Statistics · easy

What is a one-tailed vs two-tailed test, and when is one-tailed defensible?

Asked in Data Analyst interviews, in the Statistics round.

Short answer

Two-tailed asks whether there is any difference in either direction, one-tailed asks about one direction only and puts all of alpha there. One-tailed is defensible only when the other direction is impossible or irrelevant by design, decided before the data.

How to answer it

A two-tailed test splits alpha across both tails: the effect could be positive or negative and you want to detect either. A one-tailed test puts all of alpha in one tail, so it reaches significance with a smaller effect in that direction and can never call the other direction.

That is the whole appeal and the whole danger. One-tailed tests are easier to pass, so switching to one after seeing the data is p-hacking with extra steps.

When it is defensible: the decision only depends on one direction and was framed that way before the test. "We will ship the new checkout only if it is better, and if it is worse we do not need to know by how much" is a real one-tailed decision. Even then, many teams run two-tailed because a significant decrease is worth knowing about.

Say which you used, why, and that you chose it before looking. That sentence is most of the mark.

Related questions

Practice this for real