Statistics · medium
How long should an A/B test run, and what are novelty and seasonality effects?
Asked in Data Scientist interviews, in the Statistics round.
Short answer
Run for the sample size the power calculation requires and for whole weeks, at least one and usually two, so weekday and weekend behavior are both covered. Novelty effects fade as users get used to a change, so check whether the lift holds for later cohorts before you trust it.
How to answer it
Definition
Long enough to reach the pre-computed sample size, AND long enough to cover the business cycle and outlast novelty. Whichever is longer governs, and the second condition is the one people forget.
The three constraints
1. Statistical reach the sample size you computed in advance
2. Seasonality at least one full week, ideally two
weekday and weekend users behave differently, and a test
run Monday to Thursday measures a biased slice
3. Novelty long enough for the reaction to the CHANGE to fade
from the effect of the change itselfNovelty and primacy
- Novelty effect: existing users click the new thing because it is new. The lift is real in week one and gone by week three.
- Primacy effect: the opposite. Users are slower at first because they knew the old flow, and the change looks worse than it is.
Both are strongest for existing users and mostly absent for new ones, which gives you the diagnostic: split the result by new against returning users. If the effect lives only in returning users and decays week over week, you are looking at novelty rather than a durable improvement.
Key points
- Fix the horizon in advance and do not stop the moment it turns significant.
- Run in whole weeks. Stopping mid-week weights the days unevenly between arms.
- Watch the effect over time, not just at the end. A decaying lift is a warning.
- Segment by new and returning users to separate novelty from a real effect.
- Avoid holidays and launches unless the test is specifically about them.
FAQ
- Can I stop early if it is clearly winning? Not with a fixed-horizon test. Use a sequential design if early stopping matters.
- What if it needs six weeks? Reconsider the MDE, reduce variance, or accept that the effect may be too small to be worth detecting.
- Two weeks always? A good default, not a rule. What matters is covering the cycle and outlasting novelty.
The 30-second answer
Long enough to hit the sample size I computed beforehand, and long enough to cover a full business cycle, which usually means at least one and preferably two complete weeks. Whichever is longer wins.
The reason for whole weeks is that weekday and weekend users behave differently, so a test running Monday to Thursday measures a biased slice.
Novelty is when existing users engage with something simply because it is new, so the lift is real in week one and gone by week three. Primacy is the reverse, where people are slower because they knew the old flow. I separate either from a genuine effect by splitting on new against returning users and watching the effect over time. If it only exists for returning users and decays, it is novelty rather than something worth shipping.
In the interview
- What they are checking: power-driven duration, whole weeks, and that you would check whether the lift decays.
- Common mistake: peeking daily and stopping on the first significant day.