Topic: product

• Analysing an A/B test: fixed horizons, peeking and multiple comparisons

  Two habits quietly turn an A/B test into a random number generator: stopping when the p-value first dips below the threshold, and testing many metrics or segments until one of them 'wins'. Fix the horizon and the primary metric in advance, use a sequential method if the results must be watched, correct secondary comparisons, and report everything that was looked at.

  methodology · en · unreviewed

Machine-readable: JSON