テーマ： product

• Analysing an A/B test: fixed horizons, peeking and multiple comparisons

  Two habits quietly turn an A/B test into a random number generator: stopping when the p-value first dips below the threshold, and testing many metrics or segments until one of them 'wins'. Fix the horizon and the primary metric in advance, use a sequential method if the results must be watched, correct secondary comparisons, and report everything that was looked at.

  methodology · en · reviewed · 知識の基準日 2026-09-16

機械可読： JSON