Benchmark methodology: warm up, interleave repetitions, report the spread

Goal

Produce a timing comparison between two versions of a function, a query or a command that another person can reproduce and that does not present noise as an improvement.

Prerequisites

A machine with nothing else running on it (a laptop on mains power with a fixed power profile, or a dedicated host); the exact inputs; the versions of the interpreter or compiler; and a decision on what is being measured: the wall-clock time of a whole command, or a microbenchmark of a single function.

Steps

1. Fix the workload before anything runs: the same data, sizes and settings, hard-coded in the benchmark script.
2. Warm up and discard: the first executions do not count, so that file caches, JIT compilers and lazy initialization are not charged to one variant. hyperfine offers --warmup N for this; for the opposite question (cold cache), use --prepare with a command that clears the cache before each run.
3. Repeat in interleaved order: A, B, A, B rather than all A runs followed by all B runs, so that machine drift (thermal throttling, a background job) affects both equally.
4. Choose the metric beforehand. The documentation of Python's timeit states that, in the typical case, the lowest value is a lower bound on how fast the machine can run the snippet, and that higher values are usually caused by other processes. For this reason it advises against reporting the mean and standard deviation. For the throughput of whole systems with input and output, the median and percentiles describe what users experience.
5. Record the spread: the number of runs, plus the minimum, median and maximum for each variant. hyperfine performs statistical outlier detection that points to interference from other programs and to cache effects; a flagged run is a reason to repeat the measurement, not a data point to be silently dropped.
6. Note the environment: CPU model, frequency scaling, CPU limits in the container, language version, and whether garbage collection was disabled (timeit disables it during measurement by default).
7. Report: the numbers, the exact command, the environment and the comparison rule. A difference smaller than the run-to-run spread is not a result; a difference of a few percent needs a very large number of runs to become visible at all.

Expected result

For each variant, a table with N runs, minimum, median and maximum, plus the environment, together with the script. A reader can run it and land within the reported spread.

Limits and test basis

A microbenchmark measures a function in isolation; the effect in the real program can be smaller (the function is not hot) or larger (cache and allocation effects). Shared CI runners add noise that no statistics can remove; whether the minimum is the most robust metric there is a separate hypothesis in the wiki. The procedure rests on the cited documentation; no measured values are claimed.