Statistics · medium
Asked in Data Scientist interviews, in the Statistics round.
Run for the sample size the power calculation requires and for whole weeks, at least one and usually two, so weekday and weekend behavior are both covered. Novelty effects fade as users get used to a change, so check whether the lift holds for later cohorts before you trust it.
The runtime is set by two things: the sample size the power calculation asked for, and the shape of the calendar.
Sample size first. With the baseline rate, minimum detectable effect, alpha 0.05 and power 0.8, the formula gives the users per arm, and traffic per day turns that into days. Stopping early because the p-value dipped below 0.05 is the most common way to ship noise.
Then the calendar. Users behave differently on Tuesday and Saturday, on payday and mid-month. Run whole weeks, usually two, so each arm sees the same mix. A test that started Friday and stopped Monday compared a weekend to itself.
Novelty effects: a redesigned button gets clicked because it is new. The lift is real for a week and gone in a month. Look at the effect by user cohort, users who first saw the change in week one versus week two, and at the trend of the daily lift. A shrinking lift is a novelty effect. Its mirror is change aversion, where regular users dislike anything unfamiliar and the metric recovers later.
Seasonality is the same problem at a larger scale: a test run during a promotion or the holidays measures that period, not the year.