Schema evolution without flag days

Name the compatibility direction

A new reader consuming an old message and an old reader consuming a new message are different compatibility questions. A rollout may require both while versions overlap. A replay job introduces a third question: can the current reader consume every still-retained historical version? I would write those combinations down before changing a field. Otherwise a schema check that compares only adjacent versions can pass while an old mobile client or a delayed worker fails as soon as the new writer becomes active.

Avro's specification makes the distinction concrete by defining resolution between a writer schema and a reader schema, including handling for fields and defaults. The exact rules depend on the format; there is no universal interpretation of an optional field. My practical consequence is to test the actual serializer, generated code and validation path used by each consumer. A compatible binary encoding does not automatically imply compatibility through a JSON gateway, a database mapper or an application validator that rejects an unfamiliar enum value.

References: [1] Apache Avro 1.12.0 specification: Schema Resolution

Semantic compatibility is stricter than wire compatibility

Consider a hypothetical product message with a field named price_cents. The system wants to support multiple currencies and introduces a Money value containing amount_minor and currency_code. Simply renaming the old field is insufficient. The old contract may have implicitly assumed US dollars and two decimal places, while the new representation must identify both the currency and the unit. A reader can parse the number 1499 correctly and still display the wrong amount if it applies the wrong unit convention. The schema cannot infer the historical business assumption.

I would document the old invariant before writing an adapter. If every old value is contractually USD cents, mapping 1499 to amount_minor 1499 with currency_code USD preserves that meaning. If the old data mixes currencies without recording which one applies, the migration needs another authoritative source or an unresolved state. Guessing a default would manufacture information. A successful migration can legitimately surface records that need correction. Hiding those records behind a convenient default makes the rollout look cleaner while weakening the meaning of the data that downstream systems rely on.

Add the new representation before removing the old one

An expand-and-contract rollout first makes readers understand both representations, then changes writers, then removes the old path after evidence shows it is no longer needed. In the illustrative money migration, the transitional reader prefers the new Money field when present and falls back to the old cents field only under the documented USD rule. The writer may emit both fields temporarily, with a consistency check between them. This overlap costs code and bytes, but it avoids requiring every service and client to deploy at one instant.

Protocol Buffers documents wire-compatibility rules and warns against reusing deleted field numbers; reserving removed numbers and names protects future changes. I would apply those rules separately from the rollout policy. Deprecating a field does not prove nobody uses it, and retaining its number does not preserve its old meaning if a developer repurposes the value. The example below is a transitional schema, not the final contracted state. Removing the legacy field later requires evidence about readers, retained messages and rollback, followed by reserving the retired identity.

syntax = "proto3";

message Money {
  int64 amount_minor = 1;
  string currency_code = 2;
}

message Product {
  string id = 1;
  optional int64 price_cents = 2 [deprecated = true];
  Money price = 7;
}
// After removing price_cents, reserve its number and name.

References: [2] Protocol Buffers: Proto3 Language Guide

Defaults are interpretations of missing information

A default can make an old payload readable, but the chosen value becomes an interpretation of history. If a new field records whether a customer consented to a feature, absence in an old event cannot automatically mean consent. If it records a numeric threshold, zero may be a meaningful configured value rather than a safe representation of unknown. I would distinguish missing, explicitly empty and explicitly set values wherever those states affect behaviour. The correct fallback belongs to the domain contract, not to whichever zero value the generated language happens to expose.

This also affects enum changes. An old client that receives a new status needs a defined behaviour: preserve an unknown value, display a generic state, decline the operation or fetch a compatible representation. Crashing or mapping every unrecognised value to the first enum member is usually an accidental policy. I would test unknown values through the entire path, including logging, persistence and re-emission. A gateway that drops information it does not understand can break an otherwise compatible round trip, so forward compatibility is a property of the processing chain rather than one parser.

Retained events keep old contracts alive

Removing the last old service instance does not necessarily remove the last old message. A queue can contain delayed work, an archive can feed a replay, and a restored backup can reintroduce an earlier representation. The compatibility horizon therefore follows data retention and recovery policy as well as deployment inventory. I would maintain a small corpus of representative historical payloads with their writer schemas and expected interpretations. Each new reader should process that corpus before it is allowed to become the only supported recovery implementation.

For a worked horizon calculation, assume online consumers may lag three days, event retention is ninety days and disaster recovery can restore a snapshot from seven days ago before catching up from the log. If arbitrary ninety-day replay is promised, reader compatibility must cover the retained history regardless of the shorter normal lag. The exact required set depends on which schema versions actually appear in that interval, not on ninety being a special number. A compatibility ledger should record those versions explicitly so deleting an old decoder becomes a deliberate contract change.

Rollback needs a data plan as well as a binary

A deployment rollback can restore old code without restoring the old world. New writers may already have emitted messages or stored values that the previous reader cannot interpret. In the money example, once a writer accepts a non-USD price, an old reader that only understands USD cents cannot represent the value faithfully. Continuing to dual-write a misleading legacy field would not make rollback safe. I would define the irreversible transition point before enabling such data, and keep the fallback deployment capable of understanding the expanded representation.

The strongest counterargument is that compatibility layers become permanent because nobody is willing to remove them. That is a real maintenance failure. Expand-and-contract needs an owner, measurable exit criteria and a removal plan. Count legacy reads and writes, inventory supported clients, verify the archive horizon and rehearse the fallback. If a breaking versioned endpoint or a scheduled maintenance window is simpler and acceptable to users, choose it honestly. Avoiding a flag day is valuable when independent deployment is a requirement; it is not a reason to preserve every historical interface forever.

Validate meaning across the rollout matrix

I would build migration fixtures around the combinations that matter: old writer to new reader, new writer to old reader during overlap, current reader to archived payload, and rollback reader to data produced before rollback. Include missing fields, explicit zero values, unknown enums, maximum supported integers and contradictory dual-written values. Assert the interpreted business object, not merely that decoding returns without an exception. In the price example, a useful assertion includes the currency and unit as well as the numeric amount, because those fields jointly define the value.

The final contract should explain which representation is authoritative when both appear, how missing information is handled and when the legacy path can disappear. That makes the migration reviewable by people who operate consumers as well as people who edit the schema. A format's compatibility rules provide essential mechanical constraints, but they cannot decide whether a new default lies about old data or whether rollback can express a newly accepted business case. Schema evolution works without a coordinated deployment when the whole transition preserves those meanings deliberately.