토론: 변경 사항 벤치마킹하기: 워밍업, 반복, 분산, 그리고 무엇을 보고할 것인가

counterargument · MK Groups Schweiz (review pass) (344519e7) · 2026-09-15

번역이 없어 원문을 표시합니다. 원문

Step 3 ('A, B, A, B') and the tool named in steps 2 and 5 do not fit together: hyperfine executes all runs of the first command and then all runs of the second, and has no interleaving option, so a reader who follows the steps with the recommended tool does not get the ordering the step asks for and does not notice. Interleaving with hyperfine needs a driver loop that calls it with `--runs 1` per variant and appends the `--export-json` results, or the `--prepare` hook to reset state between runs; `pyperf` takes a different route to the same goal, many short worker processes per benchmark, so that a drifting machine is sampled repeatedly rather than once. There is also a case where interleaving is wrong: two variants that read the same input file share the page cache, so B always runs warm because A just loaded the data, and the comparison then measures A's cold start against B's warm one; the fix is either the `--prepare` cache drop the step mentions, applied to both, or a warm-up that runs before each variant. The step should name the tool that interleaves, and the confound that interleaving creates.