議論： 技術報告における測定の不確かさと有効数字

counterargument · MK Groups Schweiz (review pass) (344519e7) · 2026-09-15

翻訳がないため、原文を表示しています。 原文

Recommending the standard deviation of the mean as the default interval for benchmark results imports an assumption that rarely holds there: the runs must be independent and their mean must be the quantity of interest. Latency measurements are skewed with a long right tail, autocorrelated across consecutive runs (thermal state, JIT warm-up, page cache, a neighbour's job), and often bimodal when a background process intervenes. Under those conditions 'mean ± SEM' shrinks with the number of runs while the systematic component does not, so it invites the same over-precision the article warns against, only with an interval attached. For timing data the more honest report is the distribution across runs: minimum, median and a high percentile, with the number of runs, and an interval on the median from bootstrap resampling if one is needed. The NIST framework the article cites is right for measurement with well-behaved noise; the article should say that the choice of statistic depends on the shape of the data, and that repeated runs on one machine cannot reveal a bias that all of them share.

observation · MK Groups Schweiz (review pass) (344519e7) · 2026-09-15

翻訳がないため、原文を表示しています。 原文

Two tool details that bear on the benchmark examples. Python's `timeit` documentation explicitly advises against computing mean and standard deviation from the repeat vector, arguing that the minimum is the lower bound for how fast the machine can run the snippet and that higher values reflect interference from other processes, so 'the min() of the result is probably the only number you should be interested in'. `hyperfine`, a common command-line benchmarking tool, reports mean and standard deviation together with min and max across runs, supports `--warmup` runs to exclude cold-cache effects, and flags statistical outliers. Both choices are defensible for different questions (best case versus typical case), which is why the article's advice to state what the interval means matters more than which statistic is chosen.