Speculative decoding spends parallelism

What is being preserved

Leviathan and colleagues describe a draft model that proposes several tokens, followed by target-model verification that can evaluate the proposed positions together. Acceptance and correction are designed to preserve the target sampling distribution. Chen and colleagues independently describe a related speculative sampling approach. These papers concern distribution-preserving decoding under their stated algorithms, not a general guarantee that any mixture of small and large models produces equivalent answers. The probability correction is the part that makes the technique more than a heuristic shortcut.

Preserving a distribution also does not mean reproducing the same text for the same random seed. Different algorithms can consume random numbers in different orders while sampling from the same mathematical distribution. I would state the equivalence claim at that level and test implementations accordingly. Greedy decoding has its own deterministic comparison, while temperature and truncation policies define the distributions being sampled. If a verification path uses different logits processing from the target-only baseline, the system may preserve a distribution other than the one the application intended.

References: [1] Fast Inference from Transformers via Speculative Decoding[2] Accelerating Large Language Model Decoding with Speculative Sampling

A three-token acceptance calculation

Take an illustrative vocabulary containing A, B and C. Let the target probabilities p be [0.5, 0.3, 0.2], and the draft probabilities q be [0.6, 0.1, 0.3]. A token sampled from q is accepted with probability min(1, p divided by q) for that token. A is accepted with probability five sixths, B with probability one, and C with probability two thirds. Multiplying each draft probability by its acceptance probability gives accepted masses [0.5, 0.1, 0.2], totaling 0.8.

The remaining probability mass is 0.2. The positive part of p minus q is [0, 0.2, 0], so the normalized correction distribution chooses B whenever the draft proposal is rejected. Adding that correction produces [0.5, 0.3, 0.2], exactly the target distribution in this example. The calculation is original arithmetic applied to the paper’s rule. It shows why replacing a rejected token with an ordinary target sample would be wrong: accepted proposals have already contributed a nonuniform portion of the target mass.

p = [0.5, 0.3, 0.2]
q = [0.6, 0.1, 0.3]
accepted = [min(a, b) for a, b in zip(p, q)]
residual = [max(0.0, a - b) for a, b in zip(p, q)]
rejection_mass = 1.0 - sum(accepted)
correction = [x / sum(residual) for x in residual]
recovered = [a + rejection_mass*r for a, r in zip(accepted, correction)]
assert all(abs(a-b) < 1e-12 for a, b in zip(recovered, p))

References: [1] Fast Inference from Transformers via Speculative Decoding

A rejected prefix invalidates its descendants

A draft sequence is generated conditionally: its second token depends on its first, and so on. Verification cannot keep later draft tokens after rejecting an earlier one, because those later distributions were conditioned on a prefix that is no longer the accepted history. The algorithm keeps the accepted prefix and obtains the next token through correction; if all draft tokens survive, it can obtain an additional target token. This sequential acceptance boundary remains even though target evaluation of proposed positions can be performed in parallel.

I find this easiest to reason about as a branch of tentative state. Every proposal extends a branch that may become committed, but rejection cuts off the branch at the first invalid edge. That interpretation suggests concrete implementation tests. Force rejection at the first, middle and final proposed token, then check that returned tokens and cached state agree about the committed prefix. A token stream that looks correct while the cache retains an incompatible suffix can fail later in ways that are much harder to attribute to the original rejection.

References: [2] Accelerating Large Language Model Decoding with Speculative Sampling

Calculate useful work per verification round

Under a simplifying illustrative assumption that each successive proposal survives with the same independent probability alpha, a round with gamma draft tokens produces an expected 1 + alpha + alpha squared and so on through alpha to the gamma tokens. With alpha equal to 0.8 and gamma equal to four, that sum is 3.3616. This is an expectation under a model of acceptance, not a claim about any particular pair of language models. Real acceptance probabilities vary with the prefix and are not generally independent.

Now suppose a target-only token takes 10 milliseconds, one draft token takes 1 millisecond, and verification of four proposals plus the next distribution takes 12 milliseconds. A speculative round then costs roughly 16 milliseconds and produces 3.3616 tokens on average, or about 4.76 milliseconds per token. The corresponding illustrative speedup is about 2.1 times. Add synchronization, cache management or a more expensive verification batch and that result changes. The purpose of the calculation is to expose assumptions that a single acceptance-rate number leaves hidden.

High acceptance is not the only objective

A stronger draft model may agree with the target more often but cost enough to erase the gain. A weaker draft may be extremely cheap yet cause most proposed work to be discarded. I would optimize expected wall-clock cost per accepted output token under the serving workload, not draft accuracy in isolation. The best draft length can also change during a response. Predictable formatting and repetitive code may behave differently from a difficult transition in reasoning or a passage with many plausible next words.

A bounded adaptive policy could compare recent accepted-prefix lengths with measured drafting and verification costs, then choose among a small set of proposal lengths. That policy needs safeguards against oscillation and noisy estimates. A few easy tokens should not cause an enormous speculative batch just before the distribution changes. I would include a zero-speculation option so the server can fall back to target-only decoding when the expected benefit disappears. This makes speculation a resource allocation decision rather than a permanent declaration that one model pairing is always faster.

Parallelism can already be occupied

The appealing case is a target decode step that does not fully use available computation, so verifying several positions costs less than executing them as separate sequential steps. But an already saturated server may have little spare capacity. Additional speculative work can compete with other requests even while reducing latency for the request performing it. I would therefore distinguish single-request latency from throughput at a fixed service target. A method can improve one and worsen the other without either measurement being incorrect.

An illustrative counterexample is a server that could use its spare compute to advance another user’s request. Spending that compute verifying tokens that are later rejected has an opportunity cost. The right comparison depends on load, batching and fairness policy. I would run experiments across arrival rates and record total accelerator work, accepted tokens, wasted proposals, queue delay and inter-token latency. Reporting only tokens per second for an isolated request conceals the conditions under which the technique helps or burdens the rest of the system.

Exactness has implementation preconditions

The acceptance ratio needs the probability of the proposed token under the actual draft distribution used to sample it. Reconstructing that value with different truncation, normalization or precision can break the intended correction. The residual distribution also requires careful handling when its mass is tiny. Floating-point tolerances should be specified, and impossible branches should be treated as errors rather than silently replaced with a convenient fallback that changes behavior. Mathematical exactness describes an algorithm; numerical implementation still needs engineering discipline.

I would begin with exhaustive small-vocabulary tests like the example above, including disjoint supports, identical distributions and near-zero probabilities. Then sample many independent trials and compare empirical frequencies with the target within statistical uncertainty. Separately test cache truncation, end-of-sequence handling and maximum-output limits. These checks answer different questions: probability tests examine the sampler, while state tests examine the decoder integration. Passing a benchmark-quality comparison alone is too weak, because a biased sampler can still look similar on a small set of generated answers.

The useful lesson is broader than a speedup claim

The strongest objection to speculative decoding is operational complexity: two execution paths, more state, probability bookkeeping and workload-dependent gains. For a small target model, an expensive draft or a heavily loaded service, the simpler baseline may win. I would treat that as an expected outcome of the cost model, not as evidence that the underlying idea failed. The technique is valuable when it trades comparatively cheap parallel work for expensive sequential dependence under conditions that the actual hardware and scheduler can exploit.

What I take from the papers is a particularly clean example of speculation with a defined commit rule. The proposal mechanism can be approximate, but the acceptance mechanism preserves the desired distribution. That separation is stronger than hoping a fast component is usually right. It also tells us where to be skeptical: claims about equivalence need the full correction algorithm, and claims about speed need the full execution cost. Speculation is a way to spend resources differently, and its success depends on accounting for both the accepted work and the work thrown away.