Prefix caching needs an identity

The cached object is computed state

A prefix cache commonly stores intermediate attention state for an already processed sequence. That state is not interchangeable with the visible text of the prompt. It depends on tokenization, model parameters, positions and other inputs to the computation. Two prompts that look identical to a person can produce different token sequences under different templates or tokenizers, while two requests with the same tokens can still use different adapters. I would define the cache identity from the actual model computation rather than build it from a convenient display string and hope the missing dependencies remain unchanged.

The vLLM prefix-caching documentation describes block identities that incorporate token content and additional distinguishing information, including a cache salt for isolating reuse. The important design lesson is broader than one implementation: reuse requires an equivalence relation. The service must be able to explain why two pieces of computed state are interchangeable. A cache hit is a claim that this equivalence holds, not merely that a hash lookup succeeded. Hashing is the mechanism used to represent the claim efficiently; the fields included in the identity determine whether the claim is actually justified.

References: [1] vLLM prefix caching design

Specify the dimensions that may change the result

I would begin with immutable model and tokenizer revisions, the exact token IDs, position semantics, adapter identity and any multimodal preprocessing identity. Runtime-specific state can add further requirements, including cache representation compatibility. A friendly model name is insufficient if it can resolve to different weights after an update. The same is true of an adapter alias that points to a mutable file. The cache namespace should use resolved revisions, and a deployment should change namespaces deliberately when a dependency changes. That policy is simpler to reason about than trying to repair stale entries after discovering that an alias moved.

The identity must also encode ordering and boundaries unambiguously. Naively concatenating strings can make different field combinations produce the same byte sequence before hashing. Structured serialization or a length-delimited format avoids that ambiguity. The toy example below uses canonical JSON for clarity, but a production implementation needs a specified encoding and version. It deliberately represents only a subset of possible dependencies. Its value is the testable invariant that changing the model or authorization namespace changes the key, even when the user-visible prefix remains identical. It is not a complete cache implementation or a cryptographic authorization mechanism.

import hashlib, json
def key(model_revision, token_ids, trusted_scope):
    identity = {
        'version': 1, 'model': model_revision,
        'tokens': token_ids, 'scope': trusted_scope,
    }
    data = json.dumps(identity, sort_keys=True, separators=(',', ':')).encode()
    return hashlib.sha256(data).hexdigest()
assert key('weights-a', [4, 9], 'tenant-a') == key('weights-a', [4, 9], 'tenant-a')
assert key('weights-a', [4, 9], 'tenant-a') != key('weights-b', [4, 9], 'tenant-a')
assert key('weights-a', [4, 9], 'tenant-a') != key('weights-a', [4, 9], 'tenant-b')

Sharing is an authorization decision

Cross-request reuse can be attractive when many users share a long system prefix. It also raises a question about which requests are permitted to share state and what a cache hit might reveal. A tenant-specific namespace or server-controlled salt can restrict reuse to an intended group. The service should derive that scope from authenticated policy, not accept a user-provided tenant label as proof of membership. If a caller can choose another tenant’s scope freely, the namespace is only a naming convention. It does not enforce the boundary it appears to represent.

Isolation can reduce hit rates, and that is a real cost rather than a reason to dismiss it. A public, fixed application prefix may be safely shared more broadly than user-specific retrieved material, but the classification should be explicit. I would separate those cases and review timing or other observable effects according to the threat model. The vLLM documentation’s salt mechanism provides one implementation tool; it does not determine the application’s trust policy. The application still decides which principals may share a namespace, how that decision is authenticated and when previously shared state should become unavailable.

References: [1] vLLM prefix caching design

Computation freshness differs from evidence freshness

Suppose a prompt contains a policy excerpt fetched yesterday. Reusing its cached attention state can be computationally correct for those exact tokens while producing an answer from outdated evidence today. The cache has not necessarily malfunctioned. The evidence assembly step supplied an old revision. I would keep source freshness checks before prompt construction and include source revisions in the trace. Clearing every prefix cache is an expensive and imprecise substitute for validating that the prompt contains the current material required by the request. Correct reuse and correct knowledge selection are related but distinct responsibilities.

HTTP caching, as specified in RFC 9111, offers a useful analogy through cache keys, freshness and validation, but a model’s intermediate state is not an HTTP response and the standard does not define its semantics. My proposed separation is inspired by that general discipline: identify the object, decide whether it is still applicable, and decide whether the requester may reuse it. A prefix cache usually answers the first question about computation. The surrounding application must answer the other questions about evidence and access. Treating every successful lookup as a universal freshness certificate collapses boundaries that need independent checks.

References: [2] RFC 9111: HTTP Caching

Only an actual common prefix can be reused

A shared paragraph somewhere in two prompts is not necessarily a shared prefix computation. Position and preceding tokens can affect the state that represents it. Reordering messages to improve cache reuse may therefore change the model input and its behavior, even if all visible words remain present. I would optimize prompt layout only within a tested semantic contract. Fixed instructions can often remain stable while request-specific data appears later, but that design must still preserve the intended hierarchy and evidence relationships. Cache efficiency is not a reason to move an exception away from the instruction it qualifies.

Partial reuse also interacts with block size. A cache may reuse complete matching blocks and recompute a trailing fragment, so the visible count of shared tokens need not equal the amount of work saved. Longer stable prefixes can improve the opportunity, while small edits near the beginning can invalidate later reuse. I would measure reused tokens and actual prefill time separately. A high hit rate on tiny prefixes may have little effect on latency, and a lower hit rate on expensive prefixes may be valuable. The performance claim should describe saved computation, not only successful key lookups.

Eviction is resource policy, not a correctness repair

Cached state competes with active requests for memory. Keeping every reusable prefix can reduce capacity for new work, so a service needs a bounded eviction policy and observability for memory pressure. Recency is one signal, but recomputation cost and expected future reuse may matter too. I would begin with a simple bounded policy and measure whether expensive, frequently reused entries are being displaced by one-off requests. Any smarter policy should justify its metadata and coordination cost. A cache that saves arithmetic while causing admission failures can make the overall service worse despite an impressive local hit rate.

Eviction should leave a request with the normal uncached computation, not a different answer contract. That makes it a useful correctness test: disabling the cache should preserve intended semantics, within the runtime’s documented numerical behavior. If a miss changes authorization, evidence selection or model identity, those responsibilities have been entangled with the cache. There are practical limits to demanding bitwise identity across every execution path, especially with different batching or kernels. The service should specify the relevant equivalence, test it and distinguish acceptable numerical variation from a genuinely stale or mismatched state reuse.

Make every hit explainable

For debugging, I want the trace to identify the cache namespace version, resolved model and adapter identities, matched prefix length and the reason for a miss. It should not log private prompt contents merely to make cache analysis convenient. Aggregate counters and opaque revision identifiers can answer many operational questions without copying sensitive data into another system. When a deployment changes a dependency, the expected hit-rate drop should be understandable. An unexplained improvement can be as concerning as a regression if it means requests are accidentally sharing state across a boundary that should have separated them.

The best prefix cache is boring about correctness and useful about performance. It has a precise notion of computational identity, an explicit sharing policy, bounded memory use and an ordinary recomputation path when reuse is unavailable. Those properties let the optimization remain local: the application can still reason separately about source freshness, user permissions and answer quality. I would accept a lower hit rate before weakening that separation. Saved prefill work is valuable only when the service can explain why the state it skipped computing was already the right state for this request.