Configuration service walk-through: immutable versions, staged rollout and last-known-good

Goal

Distribute runtime configuration (limits, endpoints, tuning values) to many instances without redeploying, roll changes out in stages with an automatic halt, and guarantee that every instance always has a usable configuration.

Prerequisites

A boundary between configuration and secrets (secrets stay in the secrets manager), a schema per namespace, and a per-instance health signal the rollout can read.

Steps

1. Constraints: versions are immutable and attributable; instances must start without the service; a bad change stops before reaching everyone; rollback is a pointer move, not a new edit.
2. Components: a version store; a validator checking candidates against the namespace schema; a rollout controller assigning versions to cohorts; a distribution API that clients poll with their version, or watch (the etcd documentation, for example, describes a Watch API that streams key changes by continuously watching from a given revision); a client library keeping a last-known-good file.
3. Data model: config_version(namespace, version, content, checksum, author, created_at); rollout(namespace, version, stages[], current_stage, state: running|held|halted|complete); stage(cohort_selector, hold_duration, halt_condition); client_state(instance, namespace, version_applied, reported_at, healthy). Cohorts come from a stable hash of the instance id, so membership holds mid-rollout.
4. Rollout: a few instances, then a share, then all; the controller advances only while the halt condition (error rate or health of the cohort against the rest) stays clear for the hold time, otherwise it halts and reverts the cohort to the previous version.
5. Client behaviour: load the local file on start, then fetch; apply a new version atomically by swapping a pointer; write it to disk; report version_applied and health; keep the old version if the new one fails local validation.
6. Failure modes: a change valid by schema but wrong in effect (the staged halt is the defence, so its signal must be a real health metric); instances behaving differently for minutes (document it; keep interdependent values in one namespace); many clients polling a large namespace (revision- or ETag-based fetches, jittered intervals); secrets creeping into configuration (the validator rejects known patterns); urgent changes bypassing the rollout (a logged emergency path, not a back door).
7. Measure: propagation time to full coverage, share of instances on the target version, halts and their causes, validation rejections, clients on last-known-good.
8. Not first: a web editor, per-request evaluation (that is a feature-flag service), dependencies between keys, cross-environment promotion workflows.

Expected result

A change is a versioned object that reaches a small cohort first, and an instance restarted while the service is down comes up on the last version it applied.

Limits and test basis

Proposed design, no measurements. The halt condition needs enough traffic in the first cohort to carry information; on a low-traffic service it may hold without learning anything.