A rate limit answers how much traffic may pass a boundary. Fairness asks whose traffic should pass when demand exceeds capacity. Those questions require different information. I would define the identity, resource and allocation policy before choosing a counter algorithm, because a precise limiter can enforce a poorly chosen policy perfectly.
A token bucket controls bursts, not allocation
A token bucket with refill rate r and capacity b admits a sustained rate near r while allowing a bounded burst of up to b stored tokens, depending on initial state. If requests each cost one token, a bucket with r equal to 20 per second and b equal to 100 can admit an immediate burst of 100 after sufficient idle time. Over a later interval of t seconds, the total allowance is bounded by the starting balance plus 20t. That is a useful traffic envelope, but it says nothing about how several customers should share one bottleneck.
Suppose a destination can safely handle 100 units of work per second. Tenant A has a reservation of 60 and B a reservation of 40. If B is idle, a strict partition leaves 40 units unused; a work-conserving policy may let A borrow them. When B returns, the scheduler needs a rule for withdrawing borrowed capacity. The table below describes an illustrative desired allocation rather than an implemented scheduler. A token bucket can enforce each resulting allowance, but an allocation policy must decide those allowances and how quickly they change under contention.
| A demand | B demand | A admitted | B admitted |
|---|---|---|---|
| 100 | 0 | 100 | 0 |
| 100 | 40 | 60 | 40 |
| 30 | 100 | 30 | 70 |
References: [1] RFC 6585: Additional HTTP Status Codes
Requests are not interchangeable units of work
A lightweight metadata read and a large export can each count as one request while consuming radically different CPU, memory and downstream bandwidth. A request-count limit can therefore be fair according to its counter and unfair according to the shared resource. I would begin with a small number of bounded operation classes or explicit work units rather than a fragile attempt to predict every query's exact cost. An export might acquire a concurrent job slot and a byte budget, while ordinary reads use a faster request-rate allowance.
The Dominant Resource Fairness paper provides a deeper treatment of allocation when jobs consume several resource types. It reasons about each user's dominant resource share, rather than assuming one scalar request count describes all contention. I would not casually claim that a weighted HTTP limiter implements that algorithm. The relevant lesson for an application boundary is to identify which scarce resources are actually competing. If one tenant is CPU-heavy and another is memory-heavy, assigning the same request price may conceal both useful complementarities and serious overload risks.
References: [2] Dominant Resource Fairness
Concurrency is a separate dimension of fairness
A client sending only one request per second can still occupy many expensive slots if each request executes for a minute. Under steady conditions, completion rate multiplied by average execution time gives the average number actively executing; counting all requests in the system instead requires total time including queueing. In this illustrative case that is about sixty executing requests, assuming stable flow and no early cancellation. A rate limiter alone cannot prevent a slow workload from holding memory or connection slots. I would cap concurrent work separately and define whether queued requests count against that cap.
Queueing also determines who experiences the wait. A single first-in-first-out queue can let a burst of long tasks delay every small interactive request behind it. Separate queues, weighted scheduling or admission by workload class may give a better outcome, but each choice encodes priorities. Starvation protection matters when high-priority traffic remains continuously busy. I would bound waiting time and queue size, then reject work that can no longer meet its deadline. Keeping a request indefinitely in a fair-looking queue does not make the service useful to the person waiting for its result.
References: [2] Dominant Resource Fairness
Distributed counters need an error policy
A globally exact limiter can require coordination on the request path, which adds latency and creates another availability dependency. Independent local counters are faster but can overshoot a global allowance, particularly when traffic shifts between regions or a fleet scales out. Allocating quotas to workers makes the approximation explicit: each worker spends only its assigned share until renewal. The cost is temporary underutilisation when an idle worker holds tokens that a busy worker needs. I would quantify the maximum overshoot and stranded capacity rather than describing the implementation simply as distributed.
The failure policy should depend on what the limit protects. If it enforces a contractual spending ceiling, failing open may create an unacceptable bill. If it protects a healthy public information endpoint from occasional abuse, a conservative local fallback may preserve more useful availability than rejecting everyone when the central counter is unavailable. A bounded emergency allowance is different from unlimited fail-open behaviour. Configuration should state its duration, scope and observability, and recovery should avoid suddenly releasing every accumulated allowance at once when coordination returns.
References: [1] RFC 6585: Additional HTTP Status Codes
Explain rejection without creating a new attack surface
HTTP 429 gives clients a recognisable throttling signal, and RFC 6585 permits a Retry-After header. The response should help a well-behaved client slow down without promising that a particular future instant guarantees admission. Other traffic may consume capacity first. A useful error body can identify a bounded limit class and explain whether the client should reduce concurrency, batch work or wait. I would avoid exposing other customers' activity or precise internal capacity through overly detailed counters in an unauthenticated response.
There is a counterargument that all this nuance makes limits difficult for developers to understand. That is a real product concern. The public contract can remain simple: a sustained allowance, a burst allowance and a maximum number of concurrent expensive jobs. Internal scheduling can be richer without making every detail public. What should not remain hidden is a behaviour that changes how clients must act, such as a separate export quota or a shared organisation-wide ceiling. Predictability is part of fairness because clients need enough information to avoid repeatedly doing work that will be rejected.
References: [1] RFC 6585: Additional HTTP Status Codes
Test contested access rather than isolated throughput
A single-client benchmark verifies speed and perhaps the token arithmetic; it does not verify fairness. I would test two tenants with unequal demand, one tenant with many identities, mixed cheap and expensive operations, and a returning tenant whose reserved capacity has been borrowed. Measure admitted useful work, waiting time and rejection rate by bounded tenant class or plan, with detailed identities in controlled diagnostic data. Include process restarts and traffic relocation, because those events reveal whether per-process initial balances accidentally create new bursts.
The final policy should make its tradeoff legible. Equal request counts, equal weighted work, reserved minimums and proportional shares are all defensible in some settings, and they produce different outcomes. The choice should follow the service's promises and actual bottlenecks rather than whichever algorithm is easiest to copy. I would consider the limiter successful when overload stays bounded and the allocation matches that stated policy, including during partial failure. An accurate counter is one component of that result; the fairness contract is the part users actually experience.
References: [2] Dominant Resource Fairness[1] RFC 6585: Additional HTTP Status Codes
Sources and further reading
- RFC 6585: Additional HTTP Status Codes
The primary HTTP specification defines 429 and optional Retry-After without prescribing user identity or counting. Allocation examples and limiter policies are this essay's analysis.
- Dominant Resource Fairness
The original NSDI paper analyses multi-resource allocation. The essay uses its problem framing and explicitly does not equate a simple weighted request limiter with DRF.