Statistics · medium
When would you use a t-test vs a chi-square test?
Asked in Data Analyst interviews, in the Statistics round.
Short answer
A t-test compares means of a continuous variable, a chi-square test checks association between categorical variables.
| T-test | Chi-square | |
|---|---|---|
| Outcome is | A number, like revenue or time | A count in categories, like converted or not |
| Compares | Two means | Observed against expected counts |
| Typical A/B use | Average order value per group | Conversion rate per group |
| Assumes | Roughly normal means, similar variance | Expected count of at least 5 per cell |
How to answer it
Definition
The outcome you measured decides the test. A number per unit points to a t-test; a category per unit points to a chi-square. The table above compares them directly.
Example
Question: did the new checkout page help?
Metric: conversion rate -> each user converted or not
-> counts in categories
-> chi-square on the 2x2 table
Metric: average order value -> each user has a number
-> comparing means
-> t-test, Welch's versionThe 2x2 table
converted not converted
control 800 19,200
variant 880 19,120Chi-square asks whether the split of converted against not converted depends on which group you were in.
Always Welch's
For two means, use Welch's t-test rather than Student's. It does not assume equal variances, it costs nothing when the variances happen to be equal, and the equal-variance assumption is rarely true in product data. Most libraries default to it; scipy.stats.ttest_ind needs equal_var=False.
Key points
- Numeric outcome, comparing means, so a t-test.
- Categorical outcome, comparing counts, so a chi-square.
- The t-test relies on the sample MEAN being roughly normal, not the raw data, which the CLT usually provides.
- Chi-square wants an expected count of at least about 5 per cell. Below that, Fisher's exact test.
- A conversion rate is a proportion, so a two-proportion z-test is equivalent to the chi-square here.
FAQ
- What about more than two groups? ANOVA for means, or a larger chi-square table for categories, then post-hoc tests with a correction.
- Paired data? A paired t-test when the same unit is measured twice, or McNemar's test for paired categorical data.
- Revenue is skewed. Is a t-test still valid? Usually yes at a large n thanks to the CLT, but consider a log transform or a bootstrap when the tail is extreme.
The 30-second answer
It comes down to the type of the outcome, not the question. If each unit gives me a number and I am comparing means, that is a t-test, and I would use Welch's version because it does not assume equal variances and costs nothing when they happen to be equal.
If each unit gives me a category and I am comparing counts, that is a chi-square on the contingency table. So for a conversion rate I would use chi-square, and for average order value on the same experiment a t-test.
The check I would mention for chi-square is the expected count per cell, around five or more; below that I would use Fisher's exact test instead.
In the interview
- What they are checking: whether you classify by outcome type first, and whether you know Welch's is the default.
- Common mistake: running a t-test on a 0/1 conversion column. It roughly works at large n, but it signals you did not think about the data type.