The KV cache is a systems problem

Start with the object whose lifetime keeps growing

During autoregressive decoding, previously computed keys and values can be retained so later tokens do not require recomputing them from the entire prefix. That retained state grows with the sequence. Model weights may be shared among requests, but the active context state usually scales with concurrent work. I find this distinction essential when reading a memory budget. A model fitting on the device says little about how many useful requests the server can sustain, especially when those requests have very different prompt and output lengths.

The PagedAttention paper identifies wasted KV-cache memory as a serving constraint and introduces fixed-size blocks with a mapping from logical sequence positions to physical storage. That is the paper’s central mechanism. The rest of this essay develops my own accounting and operational implications from that idea. I would not transfer its reported throughput improvements to a different model, hardware generation or traffic distribution without measurement. The relevant contribution is a way to manage growth, sharing and fragmentation, not a universal multiplier that follows from installing a library.

References: [1] Efficient Memory Management for Large Language Model Serving with PagedAttention

Calculate bytes per token before choosing a batch

An illustrative cache stores both keys and values for every layer, KV head and head dimension. Its approximate bytes per token are 2 × layers × KV heads × head dimension × bytes per element. For 32 layers, 8 KV heads, a head dimension of 128 and two-byte values, that is 131,072 bytes, or 128 KiB per token. A 4,096-token sequence therefore needs about 512 MiB of this cache. These are hypothetical architecture parameters; they are not measurements of a deployed model.

The expression also shows why the number of query heads is not automatically the number to use. Architectures can share keys and values among groups of query heads. The actual cache layout, precision and replication across devices determine the physical budget. Add alignment, allocator metadata and temporary execution buffers separately. I would put this calculation next to the admission controller, because a request count alone hides enormous variation. Ten short conversations and ten long document analyses can imply very different memory commitments even when both look like a batch of ten.

layers, kv_heads, head_dim, element_bytes = 32, 8, 128, 2
bytes_per_token = 2 * layers * kv_heads * head_dim * element_bytes
sequence_tokens = 4096
print(bytes_per_token // 1024, 'KiB/token')
print(bytes_per_token * sequence_tokens // (1024**2), 'MiB/sequence')

Use blocks to bound waste, then price the boundary

With blocks of 16 tokens, a 33-token sequence occupies three blocks, leaving 15 unused positions in the last block. At the illustrative 128 KiB per token above, that tail costs 1.875 MiB. The useful comparison is not zero waste; it is the alternative reservation and allocation policy under the same growth pattern. A contiguous allocation sized for a possible 4,096-token maximum would reserve vastly more for this particular short sequence. A resizing allocator might reduce reservation but pay copying or placement costs instead.

The block size controls a genuine trade-off. Small blocks reduce tail waste but increase the number of mappings and may make accesses less convenient for a kernel. Large blocks reduce mapping granularity while reserving more unused space at sequence ends. PagedAttention describes this trade-off directly. I would choose a candidate size using the distribution of actual sequence lengths and then benchmark the complete server. Optimizing a synthetic attention kernel while ignoring allocation behavior can select a setting that performs well in isolation and wastes capacity under real request churn.

References: [1] Efficient Memory Management for Large Language Model Serving with PagedAttention

Shared prefixes need ownership rules

Several continuations of the same prefix can initially read the same cached state. Once a continuation writes into a shared partially filled block, the implementation must preserve the other readers’ view. The PagedAttention paper uses reference-counted block sharing and copy-on-write for such cases. That mechanism makes the cache resemble a managed persistent data structure: sharing is safe while the shared region remains immutable, and divergence requires an ownership transition. The difficult part is keeping those transitions correct around completion, cancellation and reuse.

For an implementation review, I would draw the lifecycle of one physical block. Who creates it, who increments its references, what event releases each reference, and when can the allocator hand it to another sequence? A delayed device operation must not write into storage that has already been reassigned. Logical request cancellation does not necessarily mean all physical work has stopped. Testing should therefore include branching, cancellation and rapid allocation reuse together, rather than testing each operation only when the device queue happens to be empty.

References: [1] Efficient Memory Management for Large Language Model Serving with PagedAttention

Memory savings change the scheduling question

A larger feasible batch can improve utilization, but admitting every request that fits is not automatically a good latency policy. Prompt processing and incremental decoding impose different shapes of work. A long prompt entering at the wrong time may delay the next token for many existing streams. I would separate the memory feasibility decision from the scheduling decision: first determine which requests can be supported without violating reserves, then choose work according to explicit latency, fairness and throughput objectives. Capacity is a constraint, not the objective itself.

A hypothetical scheduler might reserve enough blocks for one further decoding step for every active sequence, then spend the remaining step budget on prompt chunks. That policy has failure cases: output lengths remain uncertain, arrivals can burst, and a long request can occupy memory while progressing slowly. I would measure time to first token and gaps between later tokens separately. A throughput number can improve while interactive service becomes noticeably worse, because the same amount of work is being arranged in a way that users experience differently.

Distinguish resident state from attention workspace

Paged allocation and memory-efficient attention kernels address related but different objects. The former organizes persistent per-sequence cache state; the latter can reduce data movement and avoid materializing large intermediates during an attention operation. FlashAttention’s IO-aware construction is a useful reference for that second concern. A server can benefit from both ideas because saving temporary workspace and reducing cache fragmentation free different portions of the memory budget. Calling either optimization simply a faster attention implementation obscures where its benefits actually originate.

I would track at least weights, live cache, reserved cache capacity, temporary workspace and other runtime allocations. Those counters should reconcile with device memory within known overheads. If the operating metric is merely free memory, a change in one category can conceal growth in another. The counters also make incidents more interpretable. A workload with longer outputs may expand resident cache without increasing prompt workspace, while a change in batch shape can alter temporary requirements even when the number of retained tokens is unchanged.

References: [2] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Eviction and reuse can move costs elsewhere

A cache is valuable only while its reuse saves more than preserving it costs. Holding a reusable prefix can reduce future computation, but it can also displace active requests or force admission delays. Recomputing evicted state spends compute; moving it to another memory tier spends transfer time and bandwidth. I would make those costs visible in the same request-level trace. Otherwise a cache-hit improvement can look impressive while the server quietly pays for more expensive misses, longer queues or repeated transfers elsewhere.

There is also an isolation boundary. A prefix cache key must identify the exact computation represented, including relevant model and tokenization choices, rather than merely a human-readable prompt string. Sharing across tenants can introduce additional policy questions even when the tensors happen to match. I would default to an explicit scope for reuse and expand it only with a clear contract. The paper’s memory technique does not relieve an application of deciding which requests are allowed to share state or how that decision is audited.

When the extra machinery is not worth it

For a single short request on an otherwise idle device, a simple contiguous cache may be entirely adequate. Block tables, sharing and scheduling introduce metadata and implementation complexity that must earn their place. The strongest case for paged management appears when variable-length concurrent requests would otherwise waste enough memory to constrain useful work. I would preserve a simple baseline and compare both implementations under identical token distributions, output policies and load, rather than assuming a sophisticated allocator must always win.

The experiment should report achieved throughput at a stated latency target, not just maximum throughput after the queue has become unacceptable. Include cancellation-heavy traffic, long-tail lengths and mixed prompt sizes. Check numerical equivalence where expected and memory reclamation after all requests finish. My reading of PagedAttention is that serving performance often depends on ordinary systems questions hiding inside a model-shaped application. What owns this state, how does it grow, and when can it be reused? Answering those questions can matter as much as making one matrix operation faster.