A retry is another request to a system that may already be struggling. Backoff changes when that request arrives; a budget limits whether it should arrive at all. I would design retry policy around the extra work the whole dependency graph can tolerate, rather than a convenient constant in each client.
Count attempts across the dependency graph
Suppose a hypothetical request crosses five layers and each layer makes at most three attempts at the next layer. In the worst aligned failure case, one original request can produce 3 to the power of 5, or 243 attempts at the deepest dependency. This calculation uses three total attempts, not three retries after an initial attempt. It also assumes failures trigger every available attempt; healthy traffic does not automatically pay that cost. The point is that locally modest policies can compose into a globally large load multiplier during the exact period when capacity is least available.
Google's SRE guidance treats retries as part of overload control and discusses limiting the aggregate retry rate. My interpretation is that retry ownership should be a deliberate property of a call chain. The layer with enough context to judge the operation and its remaining deadline is often the best place to decide. Lower layers may still retry narrowly defined transport events, but those attempts must be included in the accounting. A service cannot budget work accurately while its SDK, proxy and application each independently believe they are the only component trying again.
References: [1] Google SRE: Handling Overload
Define what earns and spends the budget
One simple illustrative policy earns one retry token for every ten original requests and spends one token for each retry attempt. Retries do not earn more tokens. The bucket has a maximum balance so a long quiet period cannot finance an enormous future burst. With an initially empty bucket, one thousand original requests earn one hundred retry attempts. This bounds the extra attempt count at ten percent over that accounting interval, subject to any deliberately configured initial balance. It does not bound CPU usage if a retry can be much more expensive than an original request.
The example below uses integer credit units to avoid floating-point accumulation in the budget itself. It models a single serial policy owner and is not a distributed limiter. In a concurrent implementation, credit acquisition must be atomic; in a fleet, independent buckets multiply both the initial allowance and maximum burst. I would expose the policy's scope in configuration and dashboards. A per-process budget, a per-dependency budget and a fleet-wide budget can all be reasonable, but they answer different questions about how much excess work a recovering dependency will see.
class RetryBudget:
def __init__(self, burst_retries=100):
self.units = 0
self.capacity = burst_retries * 10
def original(self):
self.units = min(self.capacity, self.units + 1)
def retry(self):
if self.units < 10:
return False
self.units -= 10
return True
budget = RetryBudget()
for _ in range(1000):
budget.original()
assert sum(budget.retry() for _ in range(200)) == 100
assert not budget.retry()
print('100 retries admitted for 1000 original requests')References: [1] Google SRE: Handling Overload
Concurrency budgets protect a different resource
Envoy also uses the term retry budget for a policy that limits concurrent retries relative to active requests, with a configurable minimum allowance. That is not identical to the token accounting above. A concurrency limit prevents retries from occupying too many simultaneous upstream slots; a rate or attempt budget limits how much extra work occurs over time. A service with very fast failures can burn through many sequential retries while staying under a concurrency cap. A service with slow failures can exhaust connections with comparatively few attempts.
I would decide which resource needs protection before choosing the mechanism. Connection pools, outstanding memory and thread occupancy usually need concurrency bounds. A dependency billed per request or limited by total CPU needs work or rate accounting as well. The two controls can coexist: a retry must have budget credit, a free concurrency slot and enough remaining deadline. This sounds more complex than setting maxAttempts, but each gate has a measurable purpose. Combining them into one magic number hides which constraint is protecting the system and which one is responsible for a rejected retry.
References: [2] Envoy circuit breaking
Spend the caller's deadline, not a fresh timeout
A retry that starts after the user has abandoned the request is often pure waste. I would propagate an overall deadline and derive each attempt's timeout from the remaining time, reserving enough time to return a useful response. For an illustrative two-second deadline, an initial attempt that consumes 1.6 seconds cannot justify a second independent two-second timeout. The remaining four hundred milliseconds must cover backoff, connection acquisition, execution and response handling. If that is insufficient for a plausible successful attempt, the correct decision may be to stop early.
Exponential backoff with jitter helps spread retry arrivals, but it does not create capacity and should not extend the overall deadline. Nor should every kind of failure receive the same delay. A documented throttling response can carry information about when to try again; a validation error will not become valid through repetition. An overloaded upstream may recover more effectively when callers fail quickly and preserve resources for fresh work. I would record the final reason for stopping: deadline exhaustion, exhausted budget, nonretryable error or maximum attempt count. Those categories suggest very different remedies.
References: [1] Google SRE: Handling Overload
An ambiguous outcome changes the retry question
If a request times out after the server may have committed a mutation, retry safety depends on the operation contract. A stable idempotency key with recorded results can make a retry useful. A client-generated new identifier on every attempt can instead turn one intended purchase into several valid purchases. Even a read may be expensive or have ancillary effects such as audit logging. I would classify operations by semantics and cost, then make the allowed retry conditions explicit. A transport error is evidence about communication, not proof that application work did not happen.
Cancellation is another boundary. A caller can cancel its local future while the remote server continues processing. Starting a replacement request immediately can create overlapping work rather than sequential recovery. Where supported, propagate cancellation and still assume it may arrive too late. For costly operations, an asynchronous job identifier and status lookup may provide a better contract than repeated synchronous attempts. The budget remains useful in that design, but it protects polling and delivery traffic rather than pretending that every timeout should recreate the original computation.
References: [1] Google SRE: Handling Overload
Keep recovery traffic from defeating recovery
A counterargument is that strict budgets reduce availability when a dependency suffers a brief harmless glitch. That can happen. A small minimum allowance or carefully chosen initial balance lets low-volume clients recover without waiting for enough successful traffic to earn credit. The tradeoff should be visible: if ten thousand fresh processes each receive ten free retries, a deployment can create one hundred thousand attempts before ordinary rate accounting has any influence. I would consider coordinated startup, gradual traffic ramps and shared dependency limits when selecting those initial values.
Another objection is that a fleet-wide budget requires central coordination and may itself become a dependency. It does not always need to be exact. Conservative local allocations, periodically adjusted quotas or a shared concurrency boundary at the destination can provide useful protection with less coupling. The acceptable approximation depends on headroom and failure consequences. What I would avoid is calling a collection of unlimited local exceptions a global budget. During an incident, every exception tends to activate together, so the worst correlated case matters more than the comfortable average.
References: [1] Google SRE: Handling Overload[2] Envoy circuit breaking
Measure recovery efficiency rather than retry volume
The useful metric is how many original operations succeed because of a retry, alongside the additional work those retries consume. A high retry success rate can still conceal a harmful design if successful second attempts arrive after the client's deadline or compete with many more valuable original requests. I would separate original request rate, attempt rate, admitted retries, denied retries, remaining deadline and final outcomes. Labels should identify bounded dependency and operation classes rather than individual request identifiers, which belong in sampled traces or logs.
A meaningful load test introduces a finite period of latency or errors and observes whether the system returns to normal after the injected fault ends. Compare policies using the same original arrival stream, including a case where every layer is configured to retry. Check that queues drain, deadlines remain bounded and recovery does not produce a second load spike. The goal is not the largest possible number of attempts. It is the highest useful completion rate within a workload and resource budget the dependency can actually sustain during failure.
References: [1] Google SRE: Handling Overload
Sources and further reading
- Google SRE: Handling Overload
Primary operational guidance covers retry amplification and overload controls. The numeric examples, token model and proposed measurements are illustrative analysis.
- Envoy circuit breaking
Primary documentation defines Envoy's concurrent retry budget. This essay explicitly distinguishes that mechanism from its own attempt-credit example.