Estimating how many samples a comparison needs before collecting them

Goal

Know, before running a benchmark, canary or A/B test, how many observations per group are needed so that a difference of the size that matters would show up, and a difference that does show up is not an artefact of a handful of samples.

Prerequisites

A single primary metric; a pilot or historical data from which its spread (standard deviation σ) can be estimated; the smallest difference δ worth detecting; and the two error rates: α, the risk of declaring a difference that is not there, and β, the risk of missing one that is (power is 1 − β). The NIST/SEMATECH handbook states that there is no correct answer to "how many measurements" without such assumptions.

Steps

1. Write down δ in the metric's units ("20 ms at p95", "0.5 percentage points"), σ from the pilot, α and the power, and where σ came from.
2. For the mean of a roughly normal metric with known σ, the handbook gives the two-sided sample size as N = (z₁₋α/₂ + z₁₋β)² (σ/δ)², where δ is the difference or shift to be detected. For an estimate of the mean alone with a 95% interval half-width of δ it gives N ≥ (1.96/δ)² σ²; with σ twice δ that is 1.96² × 4 ≈ 15.4, so 16 observations (arithmetic).
3. Or let a library solve it: TTestIndPower().solve_power(effect_size=delta/sigma, alpha=0.05, power=0.8, ratio=1.0) in statsmodels returns the observations per group for a two-sample t-test; exactly one of its parameters is left as None and solved for.
4. Read the sensitivity: N grows with the square of σ/δ, so halving the detectable difference quadruples the sample; a noisier metric costs the same way.
5. If N is infeasible, change the design rather than the error rates: a less noisy metric (median instead of mean), a paired design (same inputs through both variants), or a larger δ, stated openly.
6. For metrics far from normal (latency tails, rates near zero), replace the formula with a simulation: generate data from the pilot's distribution with the hypothesised shift and count how often the planned test detects it.
7. Write N, the assumptions and the stopping rule into the experiment plan before collecting data.

Expected result

A number per group with its reasons, so that "no difference" can be read as "no difference of at least δ" and a small pilot is not mistaken for evidence either way.

Limits and test basis

The formulas assume independent observations and a known σ; a σ from a small pilot is itself uncertain, and autocorrelated measurements (consecutive runs on one machine) need more samples than the formula says. Based on the cited documentation; no measurements are claimed.