Benchmarking a change: warm-up, repetitions, variance and what to report

Goal

Produce a timing comparison between two versions of a function, query or command that another person can reproduce and that does not report noise as an improvement.

Prerequisites

A machine with nothing else running (a laptop on mains power with a fixed power profile, or a dedicated host), the exact inputs, the interpreter or compiler versions, and a decision on what is measured: wall-clock time of a whole command, or a microbenchmark of one function.

Steps

1. Fix the workload before running anything: same input data, size and configuration, written down in the benchmark script.
2. Warm up. Discard the first executions so that file caches, JIT compilers and lazy initialisation do not count against one variant. hyperfine has --warmup N for whole commands and, for the opposite question, --prepare to run a cache-clearing command before each timing run.
3. Repeat, interleaved. Run each variant many times in the order A, B, A, B rather than all A then all B, so that drift of the machine (thermal throttling, a background job) affects both equally.
4. Choose the statistic before looking at data. The Python timeit documentation states that the lowest value gives a lower bound for how fast the machine can run the snippet, that higher values typically come from other processes interfering, and that one should look at the whole result vector rather than report mean and standard deviation. That fits CPU-bound microbenchmarks; for whole-system throughput with I/O, medians and percentiles describe what users see.
5. Record the spread: the number of runs and min, median and max (or a percentile spread) per variant. hyperfine performs statistical outlier detection to flag interference from other programs and caching effects; a flagged run is a reason to rerun, not data to delete silently.
6. Note the environment: CPU model, frequency scaling, container CPU limits, language version, and whether garbage collection was disabled (timeit turns it off during timing by default).
7. Report the numbers, the exact command, the environment and the comparison rule. A difference smaller than the run-to-run spread is not a result.

Expected result

A table where each variant has N runs, min/median/max and a stated environment, plus the script that produced it; a reader can rerun it and land inside the reported spread.

Limits and test basis

A microbenchmark measures a function in isolation; its effect on the real program can be smaller (the function is not hot) or larger (cache and allocation effects). Shared CI runners add noise that no statistic removes; whether the minimum is the most robust statistic there is a separate hypothesis on this wiki. Based on the cited documentation; no measurements are claimed.