Configuration rollouts are distributed writes — Rhys

A configuration change can alter production behaviour as decisively as a code deployment. When a fleet receives that change at different times, the rollout is a distributed state transition. I would give it explicit versions, compatibility rules and acknowledgement of applied state instead of assuming that saving a central document changes every process at once.

Distinguish desired state from applied state

A control plane can successfully store a new configuration while some workers continue using an older one. They may be disconnected, overloaded, unable to parse the new format or intentionally held back by a canary policy. A green write response therefore proves only that the desired state changed at the control plane's boundary. I would track which version each relevant process has validated and activated, along with failures and last successful contact. The operational question is whether the intended population is running the intended behaviour, not merely whether the configuration file was saved.

Google's configuration-design guidance treats configuration as a human-facing interface whose complexity and validation affect reliability. My interpretation extends that concern to rollout semantics. The configuration format should make dangerous combinations difficult to express, and the deployment system should make partial activation visible. A field with an innocent name can still redirect traffic, disable a protection or multiply resource use. Review should therefore focus on the behavioural effect and the population that receives it, rather than classifying every configuration edit as low risk because no application binary changed.

References: [1] Google SRE Workbook: Configuration Design and Best Practices

Publish a coherent bundle rather than unrelated keys

Suppose a request needs a backend address, credential reference and protocol version that must agree. Updating three independent keys lets a worker observe a mixture that never represented a valid intended configuration. A versioned immutable bundle gives the worker one object to validate and activate. Etcd's API offers transactions over key-value operations and revision-based comparisons, which can help a control plane publish coherent state. That does not make activation across every application process simultaneous. It establishes a storage-side boundary from which clients still need a loading and application protocol.

Within a process, build and validate the candidate before replacing the active reference. Each request can capture one immutable snapshot so it does not read a different configuration halfway through execution. The example below models this rule with a frozen Python dataclass in a single-threaded program. It is not a complete concurrent configuration service, and nested mutable structures would need additional care. The useful invariant is that invalid candidates never become active and in-flight work retains a coherent view while later work receives the newly activated version.

Single-process immutable snapshot model. Real publication needs concurrency-safe reference management and validation of external dependencies. python

from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    revision: int
    timeout_ms: int
    max_attempts: int

def validate(candidate):
    if candidate.timeout_ms <= 0 or not 1 <= candidate.max_attempts <= 3:
        raise ValueError('invalid request policy')

active = Config(10, 800, 2)
in_flight = active
candidate = Config(11, 600, 2)
validate(candidate)
assert candidate.revision > active.revision
active = candidate
assert in_flight.revision == 10
assert active.revision == 11
try:
    validate(Config(12, -1, 2))
except ValueError:
    pass
else:
    raise AssertionError('invalid configuration accepted')
print(in_flight, active)

References: [2] etcd API [1] Google SRE Workbook: Configuration Design and Best Practices

Use revisions to order activation and recovery

A monotonic activation revision helps a worker reject an old notification arriving after a newer one. The revision identifies the rollout event, while a content digest can identify the bundle itself. Those are different concepts. A rollback can activate the contents of revision 10 under a new revision 12, preserving the fact that the decision occurred after revision 11. Reusing the old revision number can confuse stale-update checks and acknowledgements. I would keep a history connecting activation events, content identities and the reason for the change.

Watch streams need a gap-recovery path. Etcd documents revision-based watches and compaction of old history; a client that can no longer resume from its saved revision must obtain a fresh consistent view and restart from an appropriate point. More generally, reconnecting is not evidence that every missed update was received. The worker should either reconstruct the current desired state or use a supported resumable protocol. If configuration is a complete immutable bundle, skipping intermediate versions may be safe. If updates are imperative commands with side effects, that assumption fails and the system needs a different delivery contract.

References: [2] etcd API

Compatibility is a fleet-level property

Even if each process activates a bundle atomically, the fleet can run several versions at once. A change to request routing or message format must therefore tolerate communication between old and new participants during the rollout. I would define the supported version combinations before choosing the rollout speed. A new producer cannot safely emit a format that old consumers reject merely because the control plane has stored both changes. Coordination by wall-clock activation time also requires explicit assumptions about clock error, delayed delivery and processes that were offline at the scheduled instant.

Some changes need a staged protocol: first teach all readers to accept a new form, then change writers, then remove the old form after the compatibility window closes. Other changes can be local and independent, such as lowering a nonshared cache size within safe bounds. Treating both categories identically wastes effort in one case and creates risk in the other. The key question is whether mixed versions preserve the system's invariants. If they do not, the rollout must introduce a compatibility phase or a stronger coordination mechanism rather than hoping the fleet converges quickly enough.

References: [1] Google SRE Workbook: Configuration Design and Best Practices

A canary needs stable exposure and relevant signals

A configuration canary should receive a known subset of traffic or workers long enough to reveal the expected failure modes. Randomly reassigning every request can make comparison noisy and expose nearly every user to a supposedly small experiment. Stable assignment by an appropriate identity can improve interpretation, although it must respect the feature's semantics and privacy requirements. I would compare the behaviours the configuration changes: latency for a timeout adjustment, destination errors for a routing change or resource usage for a concurrency increase.

The rollback trigger should be stated before rollout, along with the limits of the observation window. A setting that affects a nightly job cannot be validated by ten quiet minutes of daytime traffic. Nor does a clean canary guarantee safety at full scale when the change creates shared load. Increasing a per-worker concurrency limit from ten to twenty may look harmless on one worker and overwhelm a destination when applied to a thousand. The plan should include aggregate capacity checks and progressive exposure so the shared-resource consequence becomes visible before the entire fleet receives it.

References: [1] Google SRE Workbook: Configuration Design and Best Practices

Last known good has conditions

Keeping the previous valid configuration during control-plane failure often preserves availability. It is not automatically safe for every setting. A revoked credential, emergency traffic block or expiring authority may require a bounded lifetime or a different failure policy. I would classify configuration by how long stale values remain acceptable and what the process should do after that interval. A single global rule to keep running forever or fail closed immediately is unlikely to fit every dependency and product promise.

The counterargument is that versioned bundles, activation tracking and staged rollout make a simple settings file feel like a deployment platform. For a single process with infrequent low-impact changes, that complexity may be unnecessary. The machinery becomes justified as the number of independently updating participants and the consequence of disagreement grow. Start with the smallest contract that captures the real failure modes: atomic local loading, validation and visible version reporting may be enough initially. Add coordinated rollout when mixed versions or shared resource effects become material, rather than building generic infrastructure without a concrete need.

References: [1] Google SRE Workbook: Configuration Design and Best Practices [2] etcd API

Audit the transition as well as the final value

A useful change record includes the previous and new content identities, activation revision, validation result, rollout population and applied-state evidence. Sensitive values should remain protected; auditability does not require copying secrets into logs. For metrics, avoid attaching an unbounded history of configuration digests to every business measurement. A bounded active-version view or a separate information metric can expose rollout state without multiplying all historical series. Detailed change identities can live in controlled event records linked from the operational view.

I would test an invalid bundle, a delayed older update, a disconnected worker, a watch history gap and a rollback while requests are in flight. Assert that requests use coherent snapshots and that the control plane reports the remaining mixed population accurately. Configuration reliability then becomes a set of observable state transitions rather than a belief that text files are harmless. The central write expresses intent; validated activation expresses behaviour. Keeping those concepts separate is what makes a distributed rollout understandable when the fleet does not all move together.

References: [2] etcd API [1] Google SRE Workbook: Configuration Design and Best Practices

Sources and further reading

Google SRE Workbook: Configuration Design and Best Practices
Primary operational guidance discusses configuration interfaces and reliability. The rollout protocol, examples and acceptance criteria are the essay's analysis.
etcd API
Primary documentation defines transactions, revisions and watches. The essay distinguishes storage atomicity from fleet-wide activation and presents independent client-side design examples.

Distinguish desired state from applied state

Publish a coherent bundle rather than unrelated keys

Use revisions to order activation and recovery

Compatibility is a fleet-level property

A canary needs stable exposure and relevant signals

Last known good has conditions

Audit the transition as well as the final value

Sources and further reading

RELATED NOTES

Cache invalidation is a versioning problem

Schema evolution without flag days

WebSocket reconnection is state reconciliation