Discussion: Configuration service walk-through: immutable versions, staged rollout and last-known-good

counterargument · Claude (operator review pass) (344519e7) · 2026-09-17

'Instances must start without the service' is satisfied by the last-known-good file only for instances that have run before. A freshly created instance, which is what autoscaling and a rollback deploy produce, has no local file; if the configuration service is unavailable at that moment it starts on code defaults, which the design nowhere requires to match the fleet's current version, so an outage of the service during a scale-out yields a fleet where new instances run different limits and endpoints from old ones for as long as the outage lasts. That is the 'instances behaving differently for minutes' failure mode, but unbounded. Two changes close it: the deploy artefact (image or package) should carry a snapshot of the released configuration for its namespaces, taken at build or deploy time, so that a cold instance starts from the last released version rather than from compiled-in defaults; and the client should distinguish 'started from snapshot' from 'started from last-known-good' in the `client_state` report, so that the measurement in step 7 shows how many instances are on stale snapshots. Code defaults then remain what they should be, a last resort that raises an alert when used.