Replacing servers from images instead of patching them in place reduces configuration drift findings

Hypothesis

Immutable infrastructure means that a running instance is never modified after launch; a change is a new image, built by a tool such as Packer (cited: identical machine images from a single source configuration) or a container build, followed by replacement of the instance. The Terraform apply documentation (cited) advises, for automated applies with -auto-approve, making sure no one can change the infrastructure outside the Terraform workflow, because that minimises unpredictable changes and configuration drift. The hypothesis is that the replace-only discipline reduces drift not only in the resources the infrastructure code manages but also inside the instances, because the operations that cause drift (hotfixes over SSH, partial configuration runs, package upgrades that succeed on some hosts) have no place to land.

Prediction

Across comparable services, a scheduled terraform plan -refresh-only (or the equivalent for another tool) will report drift on fewer runs and touching fewer resources for fleets operated replace-only than for fleets patched in place. Within an in-place fleet, the number of drifted attributes will correlate with instance age; within a replace-only fleet, it will not, because no instance is old. Incidents whose postmortem names "this host was different" will be rarer in replace-only fleets.

Proposed test

1. Select services in the same organisation that differ in operating style (replace-only versus in-place) but use the same infrastructure code tool and cloud.
2. Run a refresh-only plan daily for at least one quarter per service and record: runs with any drift, resources drifted, attributes drifted, and the age of each instance involved.
3. For instances, add a periodic in-instance audit (package list, configuration file hashes) diffed against the image's manifest; count differing files per instance and instance age.
4. Compare distributions between styles; report per-service counts, not only totals, since one noisy service can dominate.
5. Record confounders: team size, on-call practices, change volume, and whether the in-place fleets run configuration management on a schedule or only on demand.

Status

No result claimed. Replace-only operation has costs the hypothesis does not weigh: longer lead time for small fixes, image build pipelines to maintain, and state that must live outside the instance. Drift found by the cloud API and drift inside the operating system are different measurements and should be reported separately.