DISTRIBUTED SYSTEMSENGINEERING ESSAY · 7 MIN READ

WebSocket reconnection is state reconciliation

Notebook dates are an editorial chronology, separate from publication dates.

Opening a new WebSocket restores a transport connection. It does not establish that the client and server agree about application state. I would design reconnection around a reconciliation protocol with an explicit last-applied position, a replay horizon and a snapshot fallback, then put the socket retry loop around that protocol.

A live transport is not a shared history

RFC 6455 defines the WebSocket protocol, including connection establishment, framing and closure. It does not define how an application recovers missed business events after a connection disappears. That responsibility belongs to the application protocol. A client can show a green connected indicator while its order list remains several changes behind the server. I would therefore expose transport connectivity separately from synchronisation state. A useful client state machine distinguishes disconnected, connecting, reconciling and current, rather than treating an open socket as evidence that the interface is up to date.

Consider a hypothetical client that has applied updates through position 120. While it is offline, the server produces positions 121 through 137. On reconnect, receiving position 138 does not repair the missing changes. If those changes include a deletion, the client may retain a record indefinitely even though new traffic flows normally. The protocol needs either the missing interval or a replacement state that includes its effects. Retrying the TCP connection more aggressively can reduce the frequency of this situation, but it cannot make the missing interval cease to exist.

References: [1] RFC 6455: The WebSocket Protocol

A cursor must describe application progress

A resume cursor should identify the last event whose effect is safely incorporated into client state, not merely the last frame received. Parsing, rendering and durable local storage may complete at different times. If the client persists a cursor before its corresponding state, a crash can restart from a position that skips unapplied work. I would commit the local state and cursor together when offline persistence matters. For an in-memory client, losing both on reload can instead trigger a fresh snapshot, provided that behaviour is part of the contract.

The cursor also needs a scope and an epoch. A position in one user's subscription cannot be reused for another user, and position 120 after a stream reset may refer to entirely different history. Include the stream identity or an opaque server-issued token that encodes it. If updates are independently partitioned, a single scalar cursor may be insufficient unless a server layer provides a coherent sequence. I would avoid forcing a global order merely for convenient client code when per-resource revisions and a scoped subscription position can express the actual guarantees.

Join the snapshot and stream without a gap

A naive reconnect fetches a snapshot, then subscribes to updates. A change between those steps can be missed. Reversing the steps creates another problem if buffered updates are applied to a snapshot that already includes them. The solution needs a shared boundary: obtain a snapshot identified by position Q and guarantee access to every required change after Q. The server might construct both from a durable log and a versioned projection, or implement an explicit subscription barrier. The necessary invariant matters more than the particular sequence of HTTP and WebSocket requests.

For a worked example, suppose the server returns a snapshot complete through position 137. The client replaces its local state with that snapshot, records 137, then applies buffered updates 138, 139 and 140 in order. Any buffered event at or below 137 is already represented and should be discarded under this replacement protocol. If position 139 is missing, the client must not silently claim to be current after applying 140 when every intermediate event matters. It should request the missing interval or restart reconciliation according to a bounded, observable policy.

Illustrative client protocol for a contiguous, server-defined stream. text
on reconnect:
  send RESUME(stream_id, epoch, last_applied)

on SNAPSHOT(state, cursor):
  atomically replace local_state and last_applied

on UPDATE(event, cursor):
  if cursor <= last_applied: ignore duplicate
  else if cursor != last_applied + 1: request reconciliation
  else: atomically apply event and advance last_applied

The replay horizon is a resource decision

Retaining every update forever is rarely the right reconnect strategy. Assume a hypothetical stream averages 300 updates per second and a client is disconnected for ninety seconds. It misses 27,000 updates. At an assumed 240 bytes per update, replay carries about 6.48 megabytes before protocol overhead. If a current snapshot is two megabytes and the application only needs current state, replacement may be cheaper. If each missed event represents a notification the user must see, a snapshot alone cannot satisfy the requirement regardless of its smaller size.

A useful server response can therefore distinguish resume accepted from cursor expired. Expiration is not an exceptional parser error; it is a normal outcome of finite retention. The client should discard incompatible pending assumptions, obtain an authorised snapshot and communicate any user-relevant loss of historical detail. I would also limit replay work per connection and per account. A fleet reconnecting after an outage can overwhelm the server if every client requests a large historical interval simultaneously. Jittered reconnect timing and controlled replay admission address load without weakening the state contract.

Sending and applying are different acknowledgements

The browser WebSocket standard exposes bufferedAmount to describe data queued by send operations. That is useful for observing local transmission pressure, but it is not evidence that a remote application has committed a command. If a client sends a create-order command and disconnects before receiving a response, the outcome is ambiguous. I would give the command a stable operation identifier and use an application-level acknowledgement that names the accepted result. Reconnection then queries or retries that identity rather than inventing a new command each time the socket changes.

Server-to-client backpressure deserves the same care. A slow client can accumulate updates faster than it applies them, even on a healthy connection. Bound the queue and decide which messages can be coalesced. Replacing several cursor-position updates with the newest value may be harmless; dropping one inventory decrement from a transition stream may corrupt state. When the safe queue limit is exceeded, an explicit resynchronisation response is better than an unbounded buffer or silent loss. The policy depends on message semantics, so it should be attached to the subscription type.

References: [2] WHATWG WebSockets Standard

The distribution layer must support the recovery promise

A durable client cursor is useless if the server's fan-out layer cannot recover the referenced events. Redis documents at-most-once delivery for Pub/Sub and notes that messages missed during a disconnect are lost to that subscriber. Pub/Sub can still be a sensible low-latency notification path, but I would not make it the only source behind a promise of resumable history. A durable log, authoritative versioned state or snapshot mechanism must supply the recovery information. The live path and recovery path can differ as long as their boundary is coherent.

Authorisation is another state boundary. Reconnecting is a new opportunity to verify the user's current access, not permission to replay everything an old connection could see. A role change may require removing local records as well as changing future subscriptions. Cursors should not allow a client to select another tenant's history or bypass a new filter. I would bind resume tokens to their scope and explicitly invalidate the local projection when that scope changes. State reconciliation includes removing information that is no longer part of the authorised view.

References: [3] Redis: Pub/Sub delivery semantics

Sometimes refreshing the page is the right protocol

The strongest counterargument is that an elaborate resume protocol is unnecessary for a small dashboard. That can be correct. If snapshots are cheap, changes are infrequent and the interface has no unsaved local edits, reconnecting by replacing the whole view may be the simplest reliable design. Polling may even meet the freshness requirement without a persistent connection. I would choose incremental replay only when its savings or user experience justify the extra identities, retained history and failure states. Complexity should purchase a specific recovery capability.

Whatever design is chosen, test more than whether the socket opens again. Disconnect between receipt and local application, expire the cursor, restart the server with a new epoch, duplicate an update and omit one sequence position. Change permissions while the client is offline. Verify that the interface either reaches a demonstrably current state or clearly remains in reconciliation. The useful invariant is that the client can explain which authoritative state it represents. A reconnect loop is one mechanism for reaching that invariant, not the invariant itself.

Sources and further reading

  1. RFC 6455: The WebSocket Protocol

    Primary protocol specification for transport behaviour. Application cursors, snapshots and reconciliation are designs proposed by this essay, not guarantees provided by RFC 6455.

  2. WHATWG WebSockets Standard

    Defines browser API behaviour including send and bufferedAmount. The distinction between transport progress and business acknowledgement is application-level analysis.

  3. Redis: Pub/Sub delivery semantics

    Documents at-most-once Pub/Sub delivery. The recommended recovery path and hypothetical bandwidth calculation are original examples.

FROM THE NOTEBOOK.

Back to all notes