Effect size versus statistical significance: which one decides

What it is

An effect size is the magnitude of a difference in the units of the question: 30 ms of p95 latency, 0.4 percentage points of conversion, 12% fewer retries. A standardized effect size divides the difference by the spread so that effects on different metrics can be compared; the statsmodels power documentation defines its effect_size as the difference between the two means divided by the standard deviation (Cohen's d). Statistical significance is a different quantity: it says whether the observed data would be unusual if there were no effect, and it depends on the sample size as much as on the effect. Greenland and co-authors state the two failure modes directly: especially when a study is large, very minor effects or small assumption violations can lead to statistically significant tests; especially when a study is small, even large effects may be drowned in noise and fail to reach significance. Their remedy is the same in both cases: look at the confidence interval to see which effect sizes of practical importance are compatible with the data.

Why it matters

Big systems produce big samples. With millions of requests, a 0.1 ms difference is "significant" and irrelevant. Small pilots produce small samples: a 20% improvement across ten runs can be "not significant" and very relevant. Deciding on significance alone means shipping irrelevant changes and abandoning promising ones, depending only on how much data happened to be available.

How to apply

• Before the experiment, write down the smallest effect worth acting on: "ship if p95 drops by at least 20 ms", "roll back if error rate rises by more than 0.1 percentage points".
• Report the effect in natural units with its interval; add the standardized effect only when comparing across metrics or sizing the next experiment.
• Compare the interval to the threshold: entirely beyond it, act; entirely short of it, do not act even if p is small; straddling it, collect more data or accept the stated risk.
• Give absolute and relative effects together: "50% fewer errors" can mean two per day instead of four.
• Use the standardized effect and the observed spread to size the next experiment rather than to judge the current one.

Pitfalls

Growing the sample until something becomes significant. Relative effects on tiny baselines. Post-hoc power computed from the observed data, which Greenland and co-authors describe as a direct transformation of the p-value that provides no test of the alternatives. Thresholds chosen after seeing the result.

When the interval straddles the threshold

Extending an experiment because its interval straddles the smallest effect worth acting on is a data-dependent stopping rule: the extension happens only in the ambiguous cases, and the error rate fixed in advance no longer holds. Decide before the data arrive what a straddling interval will mean. Either plan the extension as a group-sequential design with pre-set looks and adjusted thresholds, or treat straddling as 'not established' and act on the stated risk, or run a second pre-registered experiment sized from the first one's spread. For the 'entirely short of the threshold' case there is a formal test: two one-sided tests for equivalence (ttost_ind in statsmodels) ask whether the effect lies inside the interval of practical irrelevance. Collecting more is acceptable only in a pilot whose purpose is sizing the real experiment.