Bounded queues are a product decision

Memory is only one of the limits

An unbounded queue turns a disagreement between arrival rate and processing rate into retained work. The first visible problem may be memory, but the earlier product failure can be time: requests are accepted long after there is any realistic chance of completing them usefully. A bounded queue makes that disagreement surface sooner. It forces the producer to wait, reject, drop, coalesce or persist elsewhere. Each action gives the caller a different promise. I would choose that promise before choosing a channel implementation or an integer capacity.

Consider an illustrative service receiving 1,500 jobs per second while it can finish 1,000. For every second those rates persist, unfinished work increases by 500 jobs. A capacity of 5,000 postpones saturation by roughly ten seconds if the queue starts empty and these simplified rates remain stable. It does not fix the deficit. Increasing the bound to 50,000 buys time at the cost of a much larger backlog. Whether that time is useful depends on whether demand will fall, capacity can rise and the jobs remain valuable.

Average queueing time is not a deadline guarantee

Little's relationship connects average population, throughput and time when the relevant long-run quantities exist and the boundary is consistent. An illustrative stable system completing 800 jobs per second with 160 jobs waiting on average has a mean queueing wait of 0.2 seconds, when those figures refer to the same queue population. It does not follow that a 160-slot queue guarantees a 200-millisecond wait. An average occupancy is not a maximum occupancy, and an average wait is not the worst case a particular job can experience.

I would use the relationship as an accounting check, not as a magic capacity calculator. Rejected arrivals are not completed throughput; jobs actively executing are not waiting jobs; retries may create additional attempts for one logical request. Mixing those quantities produces plausible arithmetic with the wrong meaning. For deadline-sensitive work, estimate service demand and examine the distribution of waiting under representative bursts. The final bound should reflect both resource limits and useful lifetime. A queue large enough to hold requests that will all expire is not providing meaningful resilience.

References: [1] MIT Urban Operations Research: Relationships in Queueing Theory

Backpressure can move the queue upstream

Tokio's bounded mpsc channel waits for capacity when sending asynchronously, while an unbounded channel does not impose that message-count bound. That behaviour is a useful mechanism, but it does not define a complete resource limit. If every incoming request spawns a task that waits to send, the channel can remain small while thousands of tasks retain request bodies and connection state outside it. The logical queue has moved. Measuring only the channel length would then report a reassuring number while total admitted work continues to grow.

My preferred admission boundary reserves the scarce resource before constructing expensive work or spawning an unlimited waiter. Sometimes that means a semaphore around total in-flight requests; sometimes a byte budget; sometimes both a per-tenant limit and a global limit. An illustrative 100-slot queue holding 1-megabyte bodies permits about 100 megabytes of payload, excluding overhead and work elsewhere. The same slot count with 100-megabyte bodies has entirely different consequences. Message count is a meaningful bound only when message cost is itself bounded or accounted for.

References: [2] Tokio: bounded and unbounded mpsc channels

Decide who loses when the queue fills

Rejecting the newest request preserves already accepted work, which is attractive when acceptance carries a durable promise. Dropping the oldest may be better for replaceable observations, such as a display that needs the newest state. Coalescing updates by key can preserve final state while discarding intermediate work, but it is invalid when every transition matters. These choices cannot be inferred from the data structure. A queue of account transfers, a queue of thumbnail requests and a queue of cursor positions have different semantics even if each holds a Rust struct.

The illustrative policy below distinguishes admission from execution. It refuses work that is already too old, enforces a resource reservation and treats successful insertion as the point at which responsibility transfers. A real implementation must make reservation and release exception-safe, and must state whether acceptance is durable across process failure. Returning accepted after inserting into volatile memory does not create persistence. If the product promises eventual execution, the acknowledgement must correspond to a storage boundary capable of supporting that promise, rather than to whatever operation happened to return first.

if request.deadline <= now:
    return expired
permit = try_reserve(request.estimated_cost)
if permit is absent:
    return overloaded
if queue.try_push(request, permit) fails:
    release permit
    return overloaded
return accepted_under_declared_durability_policy

Fairness needs a definition of the customer

A single FIFO queue seems fair because jobs are processed in arrival order. It can still let one producer occupy all capacity or let an expensive job delay many small ones. Per-tenant queues help isolate bursts, but choosing between them introduces another policy: equal jobs, equal bytes, equal compute time or paid priority. Those are not equivalent allocations. I would state which resource the scheduler intends to share and which customer boundary defines fairness before claiming that a queue discipline is fair in any useful sense.

An illustrative tenant submitting ten-second jobs can consume much more capacity than another submitting millisecond jobs at the same request rate. Weighted admission based on estimated cost may improve isolation, but estimates can be wrong or adversarial. Keep conservative caps, reconcile estimates against actual use and avoid letting one underestimated job monopolize an uninterruptible worker. If jobs cannot be preempted, separate classes or reserve capacity for short work. A policy that looks equitable at admission can become highly unequal once actual service times diverge.

Cancellation and retries are part of capacity

Expired work should release its reservation and leave the queue when doing so is semantically safe. Otherwise a timeout merely tells the caller to give up while the service continues spending resources on an abandoned request. A retry can then duplicate the same logical work, increasing the backlog that caused the original timeout. I would carry deadlines and stable request identifiers through the queue, and distinguish cancellation before execution from cancellation after an irreversible action has begun. The latter may require reconciliation rather than simple removal.

Retries also need an admission budget. If every rejected producer retries immediately, a small bounded queue can be surrounded by an enormous synchronized retry storm. Delayed retries, jitter and retry limits can reduce that pressure, but the protocol must remain understandable to callers. Some work should fail promptly rather than retry beyond its useful lifetime. A client-visible overload result is not necessarily worse than a long wait ending in failure; it can be the information needed to choose another action while there is still time to do so.

Operate the promise, not the container

I would monitor oldest admitted age, time waiting, service time, bytes retained, rejection reasons, expired-before-start jobs and completions by tenant or class. Occupancy alone cannot distinguish a healthy burst from a permanently stale backlog. Shutdown deserves the same accounting: stop admission, decide which accepted jobs will drain, impose a deadline and record what remains. Tokio documents a close-and-drain pattern for its receiver, but whether draining is appropriate depends on the promise made at admission and whether the work still has value.

The strongest case for a large queue is a finite burst of valuable durable work that can be completed later without violating its purpose. In that setting, buffering may be exactly the product. The case against it is not aesthetic minimalism; it is accepting an obligation the system cannot meet. I want a bound that can be explained in user terms: this much work, this much waiting, this recovery path. If changing a queue capacity changes those promises, it should be reviewed as a product behaviour change as well as a performance adjustment.