Backpressure for streaming AI

A stream is a chain of owners

A typical streaming response passes through model execution, server serialization, a network transport, a browser reader and a renderer. Voice adds synthesis, audio decoding and playback. Each stage can run at a different rate and retain its own buffered data. I would draw those stages before choosing a streaming API, because an await at one boundary does not imply that every upstream producer has stopped. The system needs a way to communicate demand, completion and cancellation across the full chain of ownership.

Reactive Streams defines a protocol for asynchronous streams with nonblocking backpressure, while the WHATWG Streams standard describes readable and writable streams, queueing strategies and propagation through stream operations. These are useful primary references for the control problem. My application-level argument is that the unit of demand must match the resource being constrained. A consumer requesting one chunk does not necessarily request one token, one byte or one millisecond of audio. Without a declared unit, a buffer limit can look precise while failing to bound the resource that actually causes trouble.

References: [1] Reactive Streams[2] WHATWG Streams Standard

Calculate how quickly a queue becomes old

Consider an illustrative producer emitting 40 tokens per second while a consumer processes 10. The backlog grows by 30 tokens per second. A queue capped at 200 tokens fills in about 6.67 seconds if it starts empty and both rates remain constant. At that point, draining the queued tokens alone takes 20 seconds at the consumer’s rate. Memory may still be modest, yet the response is already badly out of date. Queue length is therefore a latency budget as well as a storage budget.

Audio makes the distinction more obvious. A producer generating one and a half seconds of audio per wall-clock second while playback consumes one second creates half a second of additional lag every second. After one minute, the backlog is 30 seconds of audio. For illustrative 48 kHz mono 16-bit PCM, that is 2.88 million bytes before other overheads. Compressing the audio may reduce memory and bandwidth, but it does not remove the 30 seconds of conversational delay. The policy must address production and freshness, not merely representation size.

Choose a policy for each kind of data

When a queue reaches its limit, the system can slow production, reject new work, cancel obsolete work or discard some data. Those choices are not interchangeable. Dropping arbitrary tokens from generated prose corrupts the answer. Dropping intermediate progress counters may be harmless if a later counter supersedes them. Audio may require cancelling a whole response rather than removing random samples. I would classify messages by whether they are essential ordered content, replaceable state snapshots or optional diagnostics, then assign a queue policy to each class.

A stream carrying both answer text and status updates should not let verbose diagnostics consume the capacity needed for the answer. Separate queues or explicit priorities can help, but priority alone does not create capacity. Every class still needs a bound. The strongest counterargument is that these distinctions complicate a simple transport. That is true, yet the semantics already exist whether the implementation names them or not. Naming them early prevents accidental policies such as silently dropping the oldest bytes because a generic buffer happened to implement that behavior.

Propagate demand without hiding a second queue

A bounded writable stream can signal that it is not ready for more data. Respecting that signal is useful only if the producer does not continue accumulating output elsewhere. A wrapper that awaits the browser writer while an upstream callback pushes tokens into an unbounded array has moved the queue rather than bounded it. I would inspect every adapter between push and pull interfaces. Each adapter should either propagate demand upstream or document a fixed buffer and an explicit overflow action when the upstream API cannot pause.

The WHATWG stream model provides readiness and desired-size mechanisms, but application code must still connect them to its actual producer. The sketch below illustrates a cooperative source whose next chunk is requested only after the sink is ready. It deliberately omits transport-specific cancellation and error recovery, which need their own policy. If the model service supports only cancellation rather than pausing, the bounded response may need to stop generation instead of pretending it can exert precise token-level backpressure on an API that does not expose that control.

async function pump(
  source: AsyncIterator<Uint8Array>,
  sink: WritableStream<Uint8Array>,
) {
  const writer = sink.getWriter();
  let operationFailed = false;
  try {
    while (true) {
      await writer.ready;
      const chunk = await source.next();
      if (chunk.done) break;
      await writer.write(chunk.value);
    }
    await writer.close();
  } catch (error) {
    operationFailed = true;
    throw error;
  } finally {
    try {
      await source.return?.();
    } catch (cleanupError) {
      if (!operationFailed) throw cleanupError;
    } finally {
      writer.releaseLock();
    }
  }
}

References: [2] WHATWG Streams Standard

Cancellation needs a generation identity

A user can interrupt, navigate away or replace a request while old chunks remain in flight. I would assign a generation identifier to the entire response and carry it through text, synthesis and playback. Every consumer should reject chunks belonging to a superseded generation. Sending a cancellation request upstream is still useful because it saves work, but it is not sufficient for correctness: a late chunk may already have passed the cancellation point. The final output owner needs its own check before presenting content.

Completion also has several meanings. The model may have finished generating while the client still has a substantial queue. The network may have closed before the final chunk was rendered. A voice system may have synthesized an answer that was only partly played. I would track generated, delivered and consumed progress separately where those distinctions affect subsequent behavior. Otherwise the next turn may assume the user saw or heard information that remained buffered. A completion flag should identify the boundary it describes instead of implying that the entire chain finished simultaneously.

Bound the request before bounding its chunks

Per-stream buffers do not protect a server from admitting too many streams. If each request has a modest fixed queue, thousands of slow consumers can still consume large aggregate memory and retain expensive model state. I would combine per-request limits with global admission control and budgets for active generation. The scheduler should account for output that cannot be usefully delivered, especially when a stalled client continues holding a context allocation. A transport-level keepalive says the connection exists; it does not prove the user is consuming the response.

Fairness matters as well. A long-running stream should not monopolize a shared worker merely because it was admitted first, while repeatedly reconnecting clients should not bypass queue discipline. I would define how requests enter, yield, time out and release resources. Limits on output length and idle time should be part of the public behavior where relevant. These policies can be inconvenient at the margins, but an explicit refusal or cancellation is often more understandable than a service that accepts every request and gradually stops making useful progress for anyone.

Measure age, not just occupancy

A queue containing ten chunks can be healthy or disastrous depending on chunk size and age. I would record bytes, logical units and the age of the oldest undelivered content. For interactive generation, time from production to presentation can reveal problems that model-token latency misses. Track time to first useful output separately from later stalls. A system that emits an immediate placeholder and then freezes should not receive the same responsiveness score as one that begins delivering the requested content and continues at a usable pace.

Failure tests should deliberately slow the consumer, stop reading without closing, disconnect during a write and inject a late chunk after cancellation. Verify that memory remains bounded and that upstream resources are eventually released. Also inspect the user-visible result: no interleaving of two generations, no resumption of cancelled audio and no success marker after a truncated answer. These properties are easier to check with deterministic fake producers and consumers before adding a nondeterministic model. The stream protocol should be correct even when the generated content is just numbered test chunks.

Streaming should reduce waiting, not conceal it

There are cases where buffering a complete result is the better design. If the output is short, must pass a whole-document validation step or will be consumed atomically, streaming can add complexity without improving the experience. I would compare against that baseline rather than assuming every AI feature benefits from token-by-token delivery. Conversely, long explanations and voice interactions can benefit substantially from incremental output, provided the application accepts the responsibility to manage partial results, interruption and the uncertainty of unfinished work.

The useful design question is not whether the API supports a stream. It is whether each stage knows how much work the next stage can accept and what to do when that assumption changes. Backpressure makes that relationship explicit. Cancellation handles work that is no longer wanted. Admission control protects the aggregate system. Together they let streaming expose useful progress without turning the space between production and consumption into an unbounded hiding place for latency, memory and stale answers that will never help the user.