Replay is an architectural feature

Decide which question replay must answer

There are several different operations commonly called replay. One reconstructs the state that an earlier software version produced. Another recomputes history using a corrected algorithm. A third rebuilds a current projection after losing its storage. These operations can require different inputs and produce deliberately different answers. I would name them separately in the operator interface. Otherwise a person trying to repair a search index might unknowingly execute a historical business workflow, or a person investigating an old decision might apply today's rules and believe they reproduced yesterday's result.

Fowler's Event Sourcing description connects retained state-changing events with reconstruction and temporal queries. My narrower recommendation does not require every application to adopt event sourcing. A service can retain enough immutable input to rebuild one derived view while keeping an ordinary database as its authority. The important question is whether the replay's target is derivable from the retained information. A log of notification summaries may be useful for debugging but insufficient to reconstruct all the state that existed when the original computation ran.

References: [1] Martin Fowler: Event Sourcing

A deterministic function has more inputs than its payload

Suppose a hypothetical classification job reads an event and then looks up the customer's current plan. Replaying the same event next month may produce a different answer because the plan changed. The event bytes are identical, yet the effective input is not. Wall-clock reads, random numbers, exchange rates, feature flags and model versions create similar hidden dependencies. I would either retain the decision-relevant values or retain immutable references that can recover them. Recording only a code version cannot reproduce a calculation whose external context has disappeared.

A useful conceptual model is output equals a transformation of event, reference snapshot and configuration version. For historical reproduction, all three are fixed. For a corrected rebuild, the transformation changes intentionally while the original evidence remains available. This distinction also helps explain a diff: an output changed because the code changed, because a reference dataset changed, or because the input population changed. Without that separation, a large replay can finish successfully while leaving no reliable explanation for why its results differ from the old projection.

replay_run = {
  input_range: [start_offset, end_offset],
  transform_version: 'classifier-v3',
  reference_snapshot: 'plans-2026-09-01',
  configuration_digest: 'sha256:...',
  output_namespace: 'shadow-run-42'
}

Rebuild state without reissuing commands

The most dangerous replay bug is an old event triggering a new external action. Rebuilding an invoice projection should not charge the customer again. Reconstructing a notification history should not resend every message. I would separate pure state transitions from effect dispatch so a replay can compute the intended state while suppressing or redirecting effects. That separation should be enforced by the execution environment and interfaces, not a boolean that each individual handler must remember to inspect correctly.

An effect ledger can make the distinction explicit. The computation produces a desired effect with a business identity; a separate dispatcher decides whether the effect is new and authorised for execution. During a shadow replay, desired effects are recorded in an isolated namespace for comparison rather than sent. If the purpose of the run really is to repair missing effects, review the difference between desired and completed identities and dispatch only the approved remainder. That workflow is slower than blindly running handlers, but it makes the irreversible boundary observable and recoverable.

Retained logs have different reconstruction power

Kafka permits consumers to revisit retained offsets, but its compaction documentation also makes clear that compaction preserves useful keyed state rather than every historical update. That distinction changes what can be rebuilt. A compacted topic can be appropriate for restoring the latest known value of each key. It is not automatically an audit trail of every transition. I would choose retention and compaction based on the exact reconstruction question, then test that question against data old enough for the cleanup policies to have actually run.

Deletion deserves particular attention. If a projection receives a delete marker during normal operation but a rebuild starts after that marker is no longer available, an older snapshot can resurrect data unless the recovery protocol accounts for the missing deletion. A snapshot and a log position must describe one coherent starting point. The snapshot cannot merely be the newest file with a plausible timestamp. Store its source boundary, transformation version and validation record, and ensure the remaining retained history begins early enough to bridge from that boundary to the desired target.

References: [2] Apache Kafka 4.1 design: Consumer position and log compaction

Parallel replay still needs an order contract

If events for independent accounts commute, replay can process those accounts in parallel. Events within one account may still require order. A balance set to 10 and then incremented by 3 does not mean the same thing as incrementing first and setting later. I would state the partitioning key and any cross-key dependencies before increasing worker count. A global timestamp sort may be both expensive and semantically wrong if timestamps do not encode the original dependency relation. The correct unit of parallelism follows the state transition model.

For a hypothetical backlog of 900 million events, an isolated replay rate of 50,000 events per second suggests five hours of processing. If live input continues at 10,000 per second and shares that capacity, catching the moving frontier takes 900 million divided by 40,000, or 6.25 hours under constant-rate assumptions. This estimate ignores skew and pauses deliberately. One hot key can dominate completion even when aggregate throughput looks healthy. I would measure the oldest unfinished partition and the slowest required dependency, rather than declare success from an average consumption rate.

Compare outputs before moving the read path

A shadow projection provides a safer place to discover semantic mistakes. Build the new result beside the current one, then compare records at a shared source boundary. A whole-table row count is a useful first check but a weak correctness argument: one missing record and one extra record can cancel out. Compare keys, selected field distributions, domain invariants and partitioned hashes over a canonical representation. Investigate differences by category, especially whether they reflect the intended correction or an unintended change in input handling.

The counterargument is that retaining all this context can cost more than rebuilding from a current database snapshot. That is often true when historical behaviour does not matter. I would avoid promising arbitrary historical reproduction unless the product needs it and can support its storage and privacy obligations. Replayability has degrees. A daily snapshot plus a short retained change log may provide exactly the recovery objective required. The mistake is presenting a limited recovery mechanism as unlimited replay, then discovering its missing history during a migration or incident.

A replayable system is one that has rehearsed replay

The practical proof is a rehearsal from an empty destination. Choose a bounded input interval, rebuild it using the documented command and compare the result with a known reference. Interrupt the run after output writes but before checkpoint persistence, restart it, and verify that duplicate execution does not create duplicate state. Include a record encoded with an older schema and a record that references an unavailable external dependency. These tests reveal whether the archive, decoder and environment actually support the advertised recovery path.

I would make the final replay report an artifact of the run: input boundary, code and configuration versions, output namespace, exception count, validation results and the eventual cutover decision. That record turns replay from an improvised script into an operation someone else can repeat. The architectural benefit is not merely disaster recovery. Once state can be rebuilt deliberately, changing projections, testing corrections and investigating past decisions become more tractable. The benefit depends on preserving meaning across time, not simply preserving a large quantity of old messages.