Quantization moves the error

Compression is also a change of numerical model

Quantization replaces a large set of representable values with a smaller one and a rule for mapping between them. That can reduce storage and unlock faster kernels, but it also changes the computation. I would review those changes at the output and decision levels, not only through the average distance between original and reconstructed weights. An error aligned with an important input direction can matter more than a larger error elsewhere. The same reconstructed matrix can therefore behave well on one workload and poorly on another without any inconsistency in the arithmetic or the evaluation.

Jacob and colleagues describe a quantization scheme and training approach designed for efficient integer-arithmetic inference. Frantar and colleagues’ GPTQ instead studies post-training weight quantization using approximate second-order information. These are different approaches to controlling the cost of approximation, not evidence that every low-bit representation has equivalent behavior. My focus here is a smaller question that can be checked by hand: how does weight rounding become output error, and what does a simple correction actually fix? The answer provides a useful review habit even when the deployed quantizer is considerably more sophisticated than the illustrative one below.

References: [1] Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference — Jacob and colleagues[2] GPTQ — Frantar and colleagues

Work through one rounding error completely

Assume a toy signed quantizer with integer values from minus seven to seven, a scale of one tenth and no zero-point offset. The weight vector is 0.26 and minus 0.14. Rounding to the nearest available values reconstructs it as 0.3 and minus 0.1. For an input vector of two and one, the original dot product is 0.38 and the reconstructed dot product is 0.5. The output error is 0.12. Both weights moved by only 0.04, but the input gives the first error twice the influence of the second.

The exact relationship is linear in this example: output error equals the dot product of the weight error and the input. A weight-only error metric cannot tell us how frequently an input activates that direction. This does not make such metrics useless; they are convenient summaries and optimization objectives. It means their interpretation depends on the input distribution and on the downstream operation. A classification boundary, a normalization layer or an accumulating residual path can make an apparently modest local error important. The numerical example is deliberately small so that this dependency is visible without treating any model’s accuracy as established.

from math import isclose
weights = [0.26, -0.14]
scale = 0.1
integers = [max(-7, min(7, round(w / scale))) for w in weights]
reconstructed = [q * scale for q in integers]
dot = lambda a, b: sum(x * y for x, y in zip(a, b))
x = [2.0, 1.0]
error = [q - w for q, w in zip(reconstructed, weights)]
correction = -dot(error, [1.0, 1.0])
assert integers == [3, -1]
assert isclose(dot(weights, x), 0.38)
assert isclose(dot(reconstructed, x), 0.50)
assert isclose(correction, -0.08)
assert isclose(dot(reconstructed, x) + correction - dot(weights, x), 0.04)

A correction has a distribution attached

Now suppose the calibration inputs have mean one in both coordinates. Under that assumption, the mean output error from the rounded weights is 0.08. Subtracting 0.08 as a bias correction removes the average error on a distribution with that mean. It does not reconstruct the original function. On the earlier input of two and one, the corrected output becomes 0.42, leaving an error of 0.04. For a different input mean, the correction can be wrong in another direction. This is an original illustrative correction, not an implementation or performance claim about GPTQ or another named quantizer.

That distinction is important when calibration data is easy to collect but not representative. A correction can make a calibration summary look excellent while moving error into rare or operationally important cases. I would retain slices for long inputs, unusual formats and decisions close to a threshold, and compare them independently. The counterargument is that exhaustive distribution coverage is impossible. That is true, but it supports a measured acceptance envelope and fallback policy, not the assumption that one average transfers everywhere. Calibration is evidence about a specified workload; it should carry that workload’s version and known limitations.

Outliers force a choice about resolution

A scale large enough to represent an extreme value leaves fewer useful levels around the many small values. A smaller scale gives those small values finer resolution but clips the outlier. Per-channel or grouped scales can reduce this conflict by separating differently distributed values, at the cost of more metadata and potentially more complicated kernels. The best granularity is therefore partly a numerical choice and partly a hardware choice. I would record the actual grouping, signed range, rounding rule and clipping rule in the model artifact rather than describing the artifact only by a nominal bit width.

Activation distributions introduce another source of variation because the values being quantized depend on the current request. A calibration set may miss an extreme activation caused by an unusual token sequence or a longer context. Dynamic scaling can respond to the current values, but it performs extra work and may change the execution path. Keeping selected operations at higher precision is a reasonable compromise when they are especially sensitive. It is not a failure to achieve ideological purity. The relevant target is a verified quality and performance envelope, including the difficult inputs that motivated the mixed-precision choice.

References: [1] Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference — Jacob and colleagues

Count the implementation, not just the bits

Four bits per weight does not imply that the complete serving process uses one eighth the memory of a thirty-two-bit baseline. Scales, offsets, packing, temporary buffers, unquantized layers, activations and caches still occupy memory. The execution engine may unpack values into a wider representation, and the available kernel may be limited by bandwidth or conversion work rather than arithmetic. I would measure resident memory and request-level latency with the actual artifact and runtime. A compressed file size is useful, but it answers a storage question rather than proving how a loaded model behaves.

Batch shape matters as well. A kernel that improves throughput for a large batch may do little for an interactive single request, while a memory reduction can enable a larger batch that changes total throughput substantially. GPTQ’s paper includes particular experimental settings; those results belong to those settings and should not be copied onto a different device as a forecast. My comparison would hold the request distribution and output policy fixed, include warm and cold behavior where relevant, and report the hardware and software versions. Numerical approximation and systems acceleration need separate evidence before combining them into one deployment claim.

References: [2] GPTQ — Frantar and colleagues

Evaluate the decisions that users receive

For a classifier, I care about changes near the action boundary and about which error types increase. For a generative model, I would compare task outcomes, supported claims and formatting reliability, while also examining token-level divergence when it helps diagnose a regression. Identical generations are not a necessary success condition for every product, but a changed refusal, tool argument or numerical result may be consequential. The evaluation contract should name those behaviors. A small average perplexity change does not automatically authorize a larger error rate in the specific operation the application performs on behalf of its user.

One useful diagnostic separates the quantized model from the surrounding decoding policy. If a tiny logit shift changes the selected token under greedy decoding, that is a real discontinuity in the product, even if the logits remain close numerically. Sampling can produce different sequences for additional reasons, so paired evaluations need controlled settings and enough examples to estimate variability. I would avoid interpreting every divergent sentence as degradation or every preserved benchmark score as equivalence. The aim is to identify which observable contracts remain reliable and which require a more precise model, a different quantizer or an explicit fallback.

Ship a reversible approximation

A quantized artifact should retain its relationship to the original weights, tokenizer, calibration procedure and runtime requirements. That makes a regression reproducible and allows rollback to a known representation. If multiple quantization variants are served, the request trace should identify the actual variant rather than only a friendly model family name. This is especially useful when a routing policy changes variants under load. Otherwise a numerical regression can masquerade as a retrieval or prompt problem because the component that changed is absent from the diagnostic record. Version identity is part of numerical accountability.

My preferred acceptance question is how quantization’s reduced representational capacity affects the supported workload. Values already on the chosen grid need not change, while rounding or clipping others can alter important outcomes. The question is whether the resulting systems benefits justify that error. The toy correction shows why that question cannot be answered by one weight-distance number: removing average error can leave a structured residual, and changing the input distribution changes the correction’s meaning. Treat the low-precision model as a separately evaluated implementation of the task, with explicit limits, rather than as a smaller file that inherited every guarantee of its source.