Decoding is part of the product

The decoder makes choices the model leaves open

At an autoregressive generation step, a model produces a score for each possible next token. The system still has to decide how to transform those scores, which tokens remain eligible, how to select one and when to stop. A common pipeline applies temperature, filters the distribution, samples a token and checks termination conditions. Other pipelines change the order or add constraints. These are observable product decisions. A brainstorming assistant, a classification service and a code completion tool can share weights while needing very different policies for variety, repeatability and acceptable failure.

Holtzman and colleagues study degeneration in open-ended text generation and propose nucleus sampling, which selects from a changing set of tokens covering a target probability mass. I take the paper as evidence that generation strategy materially affects output, not as a universal instruction to sample every task. Its experimental setting does not establish that a particular top-p value produces factual answers or correct programs. The useful engineering question is narrower: which distribution of behaviours does this application need, and what decoder configuration produces it under the application's actual constraints and evaluation cases?

References: [1] Holtzman et al. — The Curious Case of Neural Text Degeneration

Temperature can change which tokens survive

Consider three illustrative token logits: two, one and zero. With temperature one, softmax gives probabilities of approximately 0.665, 0.245 and 0.090. At temperature one half they become 0.867, 0.117 and 0.016. At temperature two they are roughly 0.506, 0.307 and 0.186. Temperature changes relative concentration while preserving the ordering for positive temperatures. It does not insert new evidence into the model. A concentrated distribution can be confidently wrong, and a flatter distribution can expose plausible alternatives without providing any reliable estimate of whether those alternatives are true.

Now retain the smallest probability-ranked prefix whose mass reaches 0.8. The temperature-one distribution keeps two tokens; the colder distribution keeps only one; the warmer distribution also keeps two. After renormalization, the first case samples the surviving pair with probabilities about 0.731 and 0.269. The warmer case uses about 0.622 and 0.378. This interaction matters: changing temperature can alter both relative weights and the support of the subsequent nucleus filter. The example below specifies its own boundary convention explicitly. Production libraries may retain additional tokens or apply processors in another order, so test the actual implementation.

from math import exp

def distribution(logits, temperature):
    if temperature <= 0:
        raise ValueError('Use an explicit greedy policy for temperature zero')
    scores = [exp((x - max(logits)) / temperature) for x in logits]
    return [x / sum(scores) for x in scores]

def nucleus(probabilities, target):
    if not 0 < target <= 1:
        raise ValueError('target must lie in (0, 1]')
    chosen, mass = [], 0.0
    for index in sorted(range(len(probabilities)), key=probabilities.__getitem__, reverse=True):
        chosen.append(index)
        mass += probabilities[index]
        if mass >= target:
            break
    return {index: probabilities[index] / mass for index in chosen}

p = distribution([2, 1, 0], 1.0)
assert abs(p[0] - 0.665240956) < 1e-8
assert len(nucleus(distribution([2, 1, 0], 0.5), 0.8)) == 1
assert len(nucleus(p, 0.8)) == 2
assert len(nucleus(distribution([2, 1, 0], 2.0), 0.8)) == 2
assert abs(sum(nucleus(p, 0.8).values()) - 1) < 1e-12

Configuration names are not a complete specification

The Transformers generation documentation exposes controls including temperature, top-k, top-p and repetition penalties. Their presence in an API makes experimentation easy, but a configuration record needs more than a handful of familiar names. Top-k limits a count; top-p limits retained probability mass. Combining them can make one dominate the other. A constrained decoder may remove tokens that cannot complete the required grammar before sampling. Record the library version and processor sequence when those details affect behaviour, and include an integration fixture that inspects the allowed outputs at a deliberately constructed boundary.

Penalties also deserve semantic tests. Repeated text is sometimes a failure, but sometimes it is the answer: a repeated identifier in code, a recurring label in a table or a quotation that intentionally repeats a phrase. A global repetition penalty can discourage valid structure. Different penalty formulas are not interchangeable, and a control named frequency penalty should not be assumed equivalent to a multiplicative repetition penalty. I would start from the simplest configuration that meets the task, then add a control only when a measured failure justifies it and the corresponding regression cases are represented.

References: [2] Hugging Face Transformers — Generation configuration

Stopping is a result type

A response that ends because the model emitted its terminal token is different from one that hits a token budget, encounters a stop string or loses its connection. Collapsing these outcomes into a single successful string removes information the application needs. A budget-limited JSON document may look plausible until its missing suffix is parsed. A stop sequence can occur inside quoted content. A streamed fragment can end at an inconvenient display boundary. I would preserve a typed finish reason and let the consumer decide whether the result is complete, recoverable or unsuitable for the requested operation.

Continuation is not automatically repair. Asking the model to finish an interrupted object may duplicate a field or resume from an interpretation inconsistent with the prefix already shown. For structured actions, validate the complete assembled object before executing anything and retain an explicit incomplete state during streaming. For prose, a visible continuation affordance may be appropriate, but the application should not describe a budget cutoff as a finished answer. The decoder's maximum length is therefore part of the product's reliability policy, as well as a bound on generation cost and user waiting time.

Reproducibility has several layers

A seed identifies one input to a random process; it is not a complete description of the generation environment. Reproducing a result also requires the model revision, tokenizer, prompt assembly, decoding settings and relevant runtime behaviour. Even a deterministic selection rule can return a different answer if a new prompt template shifts the scores. I would store a compact generation manifest with each evaluation run and distinguish exact replay from a statistically comparable rerun. Both are useful, but they answer different questions about whether an observed change belongs to the decoder or another dependency.

For debugging, retain token identifiers and finish reasons where appropriate, rather than relying only on rendered text. Two token sequences can be displayed similarly while interacting differently with limits or stopping rules. For a production application, retention must also respect the sensitivity of prompts and responses; a reproducibility plan does not require indiscriminate permanent logging. A fixture with synthetic inputs can isolate the decoder without storing user content. The objective is to preserve enough evidence to reproduce a class of failure and attribute its cause, not to archive every interaction by default.

Evaluate the distribution of outcomes

A single attractive sample is a poor basis for choosing a sampling policy. For a creative task, compare several outputs per input and assess useful diversity alongside coherence and constraint satisfaction. For a task with an executable oracle, count successful completions under an equal generation budget and inspect repeated failures. Keep inputs paired across candidate configurations so difficult cases do not drift between experiments. Report the sample count and variability. A setting that occasionally produces an exceptional answer may still be unsuitable when the product displays only one response and cannot recognize which response succeeded.

Decoder changes also affect systems behaviour indirectly. Longer answers consume more generation steps; a policy that often reaches the maximum length can worsen latency even if its per-token computation is unchanged. Conversely, early termination may reduce cost by omitting necessary explanation. Measure complete task outcomes rather than optimizing output length alone. I would include stop-reason distribution, usable-answer rate and latency alongside content quality. These measures expose whether a supposed improvement is actually a change in how often the system declines, truncates or transfers work to the user through an unfinished response.

Choose a policy the application can explain

The strongest argument for greedy decoding is operational simplicity. If an application needs one compact label from a constrained set, introducing sampling may add variance without adding value. The strongest argument for sampling is that open-ended tasks often benefit from alternatives and need not treat the highest-scoring sequence as the only useful one. Neither argument makes temperature a truth control. I would document the intended behaviour in task language first, choose a decoder that supports it, and version the decision so an apparently small configuration adjustment receives a proportionate evaluation.

My release criterion would be a readable contract: permitted output forms, acceptable variation, stopping outcomes, resource limits and evidence from representative cases. That contract should remain meaningful if the underlying model is replaced. It gives reviewers something more concrete than whether an answer feels more natural after a parameter change. Decoding belongs in product design because it governs what users actually receive, and it belongs in engineering because its failure modes can be specified, tested and observed. Treating it as an incidental slider misses both responsibilities at once.