Analysing an A/B test: fixed horizons, peeking and multiple comparisons

Goal

Reach a decision about a variant whose reported effect and error rate mean what they say, even though the experiment was watched daily and many numbers were available.

Prerequisites

One primary metric and the smallest effect worth acting on, a sample size or duration computed from them, randomisation at the unit that matters (a user, not a request, so that one person's repeated visits do not count as independent observations), and a written plan.

Steps

1. Fix the horizon before starting: the sample size per arm or the number of full weeks, plus α. The abstract of the cited sequential-analysis paper states that standard p-values and confidence intervals are wholly unreliable if users choose sample sizes by continuously monitoring their tests; each look is another chance to cross the threshold by noise.
2. If results must be watched and acted on early, replace the fixed-horizon test with a method built for it: the paper's always-valid p-values and confidence intervals, or a group-sequential design with pre-planned looks and adjusted thresholds. Otherwise, look at the dashboard for health only and analyse at the horizon.
3. Check the assignment before reading the metric: the observed split between arms should match the intended ratio, and both arms should cover the same period, so that a broken bucketing or a partial rollout is not read as an effect.
4. Analyse the primary metric with the pre-declared test and report the effect with its interval and the count per arm.
5. For secondary metrics and segments (country, platform, new versus returning), adjust the p-values for the number of comparisons: multipletests(pvals, alpha=0.05, method='holm') in statsmodels controls the family-wise error rate, method='fdr_bh' (Benjamini/Hochberg) the false discovery rate. Treat a significant segment as a hypothesis for the next experiment, not as a result of this one.
6. Report every metric and segment that was examined, including the ones that showed nothing. Greenland and co-authors name selecting analyses for presentation based on the p-values they produce as a violation that makes small p-values appear even when the test hypothesis is correct.
7. Record deviations from the plan (extended duration, excluded days, changed metric) in the write-up.

Expected result

A decision with an effect size, an interval and an honest error rate, and a record that lets the next person see how many comparisons stood behind the headline.

Limits and test basis

Novelty and learning effects make the first days unrepresentative; a horizon shorter than one weekly cycle biases the result; arms that share a cache, a queue or a marketplace interfere with each other. The steps follow the cited documentation and paper; no experiment results are claimed.