Improvements measured after targeting the worst-performing cases are partly regression to the mean

Hypothesis

The cited paper describes regression to the mean as a phenomenon of repeated measurements: extreme values are followed by measurements on the same subjects that are, on average, closer to the population mean, and in uncontrolled studies such changes are likely to be interpreted as a real treatment effect. The hypothesis transfers this to engineering programmes that select by a noisy metric: "fix the ten slowest endpoints", "stabilise the flakiest tests", "tune the hosts with the highest CPU". Any metric measured over a window has a stable component and a noise component; the cases that top the list in one window are disproportionately those whose noise component was high in that window. In the next window their noise component is, on average, ordinary, so their metric improves even if nothing was done. The hypothesis is that reported improvements from such programmes overstate the intervention's effect by an amount that grows with the noise share of the metric, and that for p99 latency or test flakiness over short windows a substantial part of the reported gain is regression to the mean.

Prediction

If the worst N cases selected by one window's metric are split at random into an intervened half and an untouched half, the untouched half will also improve in the following window. The gap between the two halves, not the improvement of the intervened half, will be the intervention's effect. The apparent improvement of the untouched half will be larger when the selection window is short and the metric is noisy, and near zero when the metric is stable across windows (high correlation between consecutive windows).

Proposed test

1. Choose a metric with per-case values over consecutive windows (endpoint p99 per week, test failure rate per week, host CPU per day).
2. Before any intervention, compute the correlation between consecutive windows to estimate the noise share.
3. Rank cases by the latest window, take the worst N, and assign them at random to intervention and control; record the assignment before the work begins.
4. Intervene on the intervention group only; leave the control group untouched, without extra attention or monitoring.
5. After one or more windows, compare the change in both groups, and separately report the change of the control group as the regression-to-the-mean estimate.
6. Repeat across metrics to relate the control group's improvement to the measured noise share.

Status

No result is claimed. Confounds: interventions on shared infrastructure affect the control group too; teams may quietly fix control cases; a genuine trend (growing traffic) shifts both groups. A control group improving as much as the intervened group would not show the work was useless, only that this measurement cannot show its value.