Cancellation is a protocol

Stopping observation is not stopping an action

Consider an illustrative client that asks a service to create an export, waits five seconds and then times out. The client knows it stopped receiving a result. It does not necessarily know whether the service accepted the job, wrote the file or sent a notification. A local timeout is evidence about the client's waiting policy, not a complete history of the remote operation. Retrying immediately can produce duplicate work unless the service offers a way to identify the original request and discover its eventual state.

I would distinguish three events in the vocabulary: cancellation requested, cancellation acknowledged and operation completed. They can race. An operation might complete between the request and its acknowledgement; a cancellation message might be lost; the initiating process might disappear entirely. The protocol needs a valid interpretation for each ordering. A boolean cancelled flag often records only the first event while readers assume it proves the second. That gap is where apparently careful timeout handling turns into contradictory user-visible outcomes.

Place the commitment point where it actually exists

An operation can often stop cheaply before it commits an externally visible effect. After commitment, cancellation may mean stop additional work, record a compensation request or return the completed result. Those are different actions. In an illustrative export workflow, cancelling before reserving storage differs from cancelling after a durable job record exists, and both differ from cancelling after a recipient has downloaded the result. The meaningful boundary comes from the operation's semantics, not from the position of the nearest await expression.

The illustrative state sketch below allows a cancellation request to compete with execution before commitment, while treating committed work as something that must be reconciled rather than wished away. A real implementation must make transitions atomic at the authoritative store. Merely checking a flag and then writing a side effect leaves a race between the check and the write. The design question is which system can decide the winner and what evidence it retains, especially when a caller cannot observe the final transition immediately.

queued -> running -> committed -> completed
queued -> cancelled
running -> cancelled, only before commitment
committed + cancel_request -> reconcile_or_compensate

client_timeout -> outcome_unknown
outcome_unknown + operation_id -> query_authoritative_status

Cooperative signals need cooperative code

Tokio's CancellationToken provides a way to signal cancellation to tasks, including child tokens whose cancellation does not cancel their parent. That structure can express ownership: a request can cancel its own work without shutting down the entire service. But a task must still observe the signal at a useful point and decide how to respond. A token cannot make a long CPU loop yield, roll back an external write or choose which cleanup operations remain necessary. It supplies communication, not the full application policy.

I would identify safe observation points explicitly. A chunked computation might check between chunks; an I/O workflow might race a wait against cancellation while retaining enough state to resume or clean up. Checks that are too sparse permit long cancellation latency. Checks inserted indiscriminately can interrupt an invariant halfway through an update. The right granularity depends on the maximum acceptable stop delay and the cost of leaving a partial operation. That is why cancellation behaviour deserves tests and documentation alongside the normal success path.

References: [1] Tokio util: CancellationToken

Dropping a future can discard progress

Tokio describes cancellation safety in terms of what happens if a future is dropped before completion and later recreated. Some operations can safely be retried that way; others can lose progress. A select loop therefore requires more than choosing whichever branch finishes first. If the losing future owned partial state, dropping it may erase information the next iteration needs. This is particularly easy to miss when a convenient helper internally performs several reads or writes while exposing only one asynchronous call.

An illustrative framing reader may have consumed a header and part of a body when another branch wins. Recreating the whole read as though it started at the next frame boundary can corrupt the protocol. One remedy is to keep the parser state outside the cancellable future so the next poll continues from the correct position. Another is to make cancellation close the connection and abandon that stream deliberately. Neither is universally better. The contract should state whether interruption preserves progress, abandons the resource or requires recovery.

References: [2] Tokio select!: cancellation safety

Cleanup has a budget and an owner

Cancellation usually creates work: release a reservation, return a buffer, delete temporary state, close a resource or record an unresolved outcome. That work cannot safely be owned by the request that has just disappeared unless its lifetime is extended deliberately. I would assign cleanup to a scope or supervisor that can outlive the caller, then bound how much cleanup remains in flight. Otherwise an overload event can generate an unbounded second workload of abandoned operations trying to clean up at the same time.

A shutdown sequence makes the ownership issue visible. Stop admitting new work, signal cancellation where appropriate, allow bounded graceful completion, then escalate according to documented semantics. The grace period should not be reset independently at every layer, or a nominal five-second shutdown can become many consecutive five-second waits. Conversely, forcing every cleanup into the original exhausted request deadline can prevent essential bookkeeping. Separate the caller's patience from the service's recovery budget, and ensure any work that survives the caller is observable and accountable.

Retries need identity, not optimism

An illustrative create-job API can accept an operation identifier chosen before the first attempt. Reusing that identifier lets a retry ask about the same logical operation instead of automatically creating another. The server still needs an atomic relationship between recording the identifier and performing the effect, a retention policy for deduplication records and a definition of what happens if the same identifier arrives with different input. Idempotency is a protocol property with storage and concurrency requirements; a field named requestId is only a starting point.

There are awkward edges. A status query may reach a replica that has not observed the commit yet. A deduplication record may expire before a delayed retry arrives. A compensation can fail even though the original operation succeeded. I would surface unresolved outcomes explicitly and preserve the identifiers needed for reconciliation. The client should not receive a confident cancelled result when the authoritative system only knows that cancellation was requested. That may make the interface less tidy, but it avoids turning uncertainty into a false account of history.

Test the races the happy path hides

A useful cancellation test matrix interrupts an illustrative operation before admission, while queued, during a partial read, immediately before commitment, immediately after commitment and during cleanup. It also repeats cancellation and simulates a lost acknowledgement. The assertions should concern resources and effects: no leaked permits, no buffer reuse before completion, no unintended duplicate operation, and no response that contradicts the authoritative state. Merely asserting that a future returned early misses most of the behaviours that make cancellation important to the user.

The counterargument is complexity: some operations are harmless to finish, and a full remote cancellation protocol would cost more than it saves. I agree that cancellation should be proportional. For a small read-only computation, stopping local waiting and letting bounded work complete may be a good policy. Say so, cap its resource use and measure abandonment. What I reject is an accidental policy produced by dropping a handle. A reliable system can choose best-effort cancellation, but it should know exactly which promise the best effort makes.

Observability should preserve the same distinctions as the protocol. Count requested cancellations, acknowledged stops, completed operations after cancellation and unresolved outcomes separately. One combined cancelled counter can improve while unwanted background work grows, because it records the caller's decision rather than the service's actual response to that decision.