A lease is a time-bounded permission, not a mechanism for stopping a process. A worker can pause, lose its lease and later continue exactly where it left off. Correctness depends on whether the resource receiving its next write can recognise that the worker's authority has been superseded.
Put an ordering token on every protected operation
A fencing token is an increasing number associated with a grant of authority. If A received token 41 and B later received token 42, requests carry those values to the protected resource. That resource records the greatest accepted token and rejects requests with smaller values. Hazelcast's FencedLock documents this pattern and supplies increasing tokens through its coordination mechanism. The essential extra component is still the receiving service. A token returned by a lock library does nothing unless every mutation path enforces the token when it changes the protected state.
There is a subtle limit: token 41 is not automatically rejected at the instant a coordinator issues token 42. A resource that has not yet learned about 42 may still accept 41. The simple rule guarantees that after the resource accepts the newer authority, an older token cannot overwrite it. If the application requires revocation to become effective before B performs any ordinary write, acquiring ownership must include an appropriate fencing operation at the resource or a stronger shared authority check. I would state which of these guarantees the product actually needs.
References: [2] Hazelcast FencedLock
Make comparison and mutation atomic
The following in-memory example illustrates the receiver's decision, assuming calls are executed serially. It deliberately accepts repeated uses of the current token because one owner may perform several operations. A production receiver needs an atomic transaction or compare-and-swap around reading the highest token, comparing it and updating both token and value. Otherwise two concurrent requests can both pass an earlier comparison and then write in the wrong order. A database transaction is useful only if its locking or conditional update actually covers the contested record.
The example also separates fencing from request deduplication. Two different requests from the same owner both carry 42, so the token cannot tell whether a retried debit has already executed. Add a stable operation identifier and a recorded result when the operation requires retry safety. Similarly, fencing cannot repair an incorrect command created by the current legitimate owner. It establishes an ordering of authority at one resource boundary. Business validation, transaction isolation and idempotency remain distinct requirements that must be checked where their corresponding effects occur.
class Resource:
def __init__(self):
self.highest = -1
self.value = None
def write(self, token, value):
if token < self.highest:
return False
self.highest, self.value = token, value
return True
resource = Resource()
assert resource.write(41, 'A first write')
assert resource.write(42, 'B replacement')
assert not resource.write(41, 'A stale replacement')
assert resource.value == 'B replacement'
assert resource.write(42, 'B next write')
print(resource.highest, resource.value)References: [1] How to do distributed locking[2] Hazelcast FencedLock
The token issuer becomes part of the trust boundary
Generating tokens from local wall clocks is tempting because timestamps already look ordered. It also imports clock skew, backward adjustments and duplicated timestamps into an authority protocol. What matters is a monotonically increasing sequence across grants for the same protected domain, including coordinator failover. A counter stored in a system with the required consistency can provide that property; a set of independent counters cannot without additional coordination. The token does not have to increase by exactly one. It does have to avoid reusing an old authority value after a newer one has become visible.
Restoring a coordinator from backup deserves particular care. If the restored counter moves backwards while resources retain higher accepted tokens, legitimate new owners may be rejected indefinitely. If resources are restored too, old requests still in flight can become unexpectedly valid again. I would include an epoch or a carefully managed recovery procedure that establishes an ordering across restored identities. The procedure should describe what happens to outstanding work and how every receiver learns the new authority. Calling the incident a restore does not exempt it from the normal rules for ownership changes.
References: [2] Hazelcast FencedLock
Choose the scope that can actually be fenced
A single global token is simple but can create unnecessary coupling. Independent objects may use independent authority domains, provided a token for one object cannot be used to mutate another. The request should bind the token to the resource identity, and authorisation should check that binding rather than treating a large integer as a universal permission. Multi-object operations complicate the picture. Accepting a new token on object X does not establish the same fence on object Y. An application claiming atomic ownership of both needs a transaction or another protocol covering the entire invariant.
Some destinations cannot enforce a fencing token at all. An ordinary email recipient does not compare ownership epochs before displaying a message. A third-party API may expose only a request-level idempotency key. In those cases I would stop describing the lease as an exactly-once guarantee and design for the destination's actual semantics. A transactional outbox can control which intentions are durable; provider idempotency can suppress repeat requests; reconciliation can investigate unknown outcomes. None of those mechanisms should be replaced by confidence that the old worker will probably stop quickly.
References: [1] How to do distributed locking
Longer leases change costs and detection delay
One counterargument is that a sufficiently long lease makes stale owners irrelevant. Longer durations can reduce accidental expiration during pauses, but they also delay takeover after a real crash. Shorter durations improve detection while increasing renewal traffic and sensitivity to temporary stalls. These are availability and efficiency tradeoffs, not a proof that old requests cannot arrive. I would choose the duration using observed pause and network distributions, then keep the correctness boundary independent of ordinary timing where the resource supports it.
A second objection is that fencing imposes extra state on every resource. That is true, and it can be the deciding cost for a low-risk workload. Duplicate recomputation of a deterministic thumbnail may be harmless if publication uses an immutable content key. In that case the simpler contract can explicitly tolerate concurrent workers. The important distinction is between removing a correctness requirement and leaving it unsatisfied. I would document whether the lease is an optimisation, an admission rule or part of a safety protocol, because the same lock-shaped API can represent all three.
References: [1] How to do distributed locking
Test the delayed request, not only the crashed worker
A useful failure test pauses A after it has prepared its mutation, grants B a newer token, lets B write, and finally releases A's delayed request. The expected result is that A cannot replace B's state. Run a second test where the delayed old request reaches the resource before B's first request; this exposes the weaker boundary of a highest-seen-token design. Tests should also cover concurrent receiver requests, coordinator restart, resource restore and a code path that accidentally omits the token. The last case often matters more than arithmetic on the counter.
Operational evidence should distinguish lease acquisition, renewal failure, fencing rejection and business-operation failure. A rise in rejected stale tokens can indicate a healthy safety mechanism containing unhealthy pauses; it is not automatically a reason to disable the check. I would retain enough correlation information to identify the ownership generation and operation without putting unbounded resource identifiers into every metric. The design is complete when delayed execution has a defined outcome at the resource, not when the coordinator's ownership table looks orderly during the happy path.
References: [2] Hazelcast FencedLock
Sources and further reading
- How to do distributed locking
Kleppmann's primary analysis explains delayed clients and fencing. The receiver model, restore scenarios and design checklist are original illustrative reasoning.
- Hazelcast FencedLock
Primary product documentation describes fencing tokens and session-based ownership. This essay does not assume its API automatically fences arbitrary external services.