Confidence intervals in outline: what the interval says and what it does not

What it is

The NIST/SEMATECH handbook describes a confidence interval as a lower and upper limit for a quantity such as the mean, computed from the sample; the narrower the interval, the more precise the estimate. For the mean of roughly normal data the limits are the sample mean plus and minus a t-distribution quantile times the sample standard deviation divided by the square root of N, so a larger sample narrows the interval and a larger spread widens it. The handbook is explicit that a 95% interval does not mean there is a 95% probability that the interval contains the true mean: the interval computed from a given sample either contains the true value or it does not, and the level of confidence is associated with the method of calculating the interval, that is, with the proportion of samples whose intervals would contain the truth. Greenland and co-authors list "the specific 95% confidence interval presented by a study has a 95% chance of containing the true effect size" among 25 common misinterpretations, and recommend shifting attention from the null hypothesis to the full range of effect sizes compatible with the data under the model used.

Why it matters

A point estimate invites false precision. "Conversion rose by 2.1%" and "conversion changed by 2.1 percentage points, 95% interval −1.5 to +5.7" are different messages: the second shows that a loss is compatible with the data. Intervals let a reader see whether a difference is distinguishable from zero and whether it could be large.

How to apply

• Report estimate, interval, level and count together: "p95 latency 410 ms (95% CI 380–450, n = 2,000)".
• Choose a method that fits the statistic: t-based intervals for means of roughly symmetric data, bootstrap intervals for medians, percentiles, ratios and differences.
• When comparing two variants, compute the interval of the difference. Greenland and co-authors list "if two confidence intervals overlap, the difference is not significant" as a misinterpretation: overlapping intervals can still belong to a clearly non-zero difference.
• Use the width to plan: halving the width takes roughly four times the data (arithmetic from the square-root-of-N term).
• State the assumptions: independent observations, a sample size fixed in advance, no selection of the best of many variants.

Pitfalls

Treating an interval that excludes zero as proof and one that includes zero as "no effect". Intervals from consecutive time-series samples that are correlated with each other come out too narrow. An interval computed after picking the best of twenty variants is not a 95% interval for that variant.