LoRA is a capacity choice

Start with the update, not the file size

Hu and colleagues’ LoRA freezes a pretrained weight matrix and learns an additive update represented as the product of two smaller matrices. If the original matrix maps an input of dimension k to an output of dimension d, the factors have shapes r by k and d by r. Their product has rank at most r. This is the meaningful constraint: training can move the original transformation within a restricted family of updates. A small adapter file is one consequence, but it does not describe what kinds of changes the adaptation can or cannot express.

I read the paper as an argument about efficient task adaptation, not a proof that every useful model update is low rank. The pretrained model already supplies a rich computation, and a constrained change can be enough to redirect it for a particular task. That is different from claiming the complete model has low effective rank or that the same adapter rank works everywhere. The practical question is whether the chosen update space captures the differences the target task requires, given the available examples. Rank belongs in the capacity discussion alongside target layers, data quality and the training objective.

References: [1] LoRA: Low-Rank Adaptation of Large Language Models — Hu and colleagues

Calculate exactly what gets smaller

For an illustrative square matrix with 4,096 rows and columns, a full update contains 16,777,216 parameters. A rank-eight factorization contains eight times 4,096 parameters in each factor, or 65,536 in total. That is one two-hundred-and-fifty-sixth of the full matrix, approximately 0.390625 percent. The arithmetic is attractive, but it describes the trainable update for this one matrix. It does not claim that the entire training run needs two hundred and fifty-six times less memory. The frozen base model, activations and any other trained components remain part of the workload.

Optimizer state and gradient storage can make the reduction in trainable parameters especially useful, while activation memory can still dominate for long sequences or large batches. I would account for these components separately before predicting which batch fits on a device. The adapter’s rank and the set of adapted matrices both matter: applying a small update broadly can have a different cost and effect from applying a larger update to a narrow subset. A parameter budget should therefore be accompanied by the exact module selection. Two checkpoints with the same byte count need not represent comparable adaptation capacity.

d = k = 4096
r = 8
full = d * k
adapter = r * (d + k)
assert full == 16777216 and adapter == 65536
assert full // adapter == 256
A, B, x = [3, 4], [1, 2], [2, -1]
inner = sum(a * value for a, value in zip(A, x))
update = [b * inner for b in B]
assert update == [2, 4]

Rank limits an update, not an entire behavior

In the small code example, a one-row factor first reduces the input to a single scalar, and a one-column factor maps that scalar to an output direction. Every possible update output lies along that one direction. That makes the rank restriction concrete. In a deep network, however, many adapted layers interact with nonlinearities and existing representations, so the relationship between one matrix’s rank and task behavior is not so simple. I would avoid translating rank directly into a count of concepts the model can learn. It is an algebraic constraint inside a much larger computation.

Increasing rank enlarges the representable update family, but additional capacity does not guarantee a better deployment. With limited or biased data, a larger update can fit artifacts that should not generalize. A smaller update can also underfit a task whose required changes are substantial. The right comparison holds data, evaluation and training budget sufficiently controlled to reveal that tradeoff. I would inspect learning curves and failure categories rather than choose rank from convention alone. The counterargument that default ranks often work is reasonable for a baseline; it is not evidence that the default is optimal for a consequential new task.

Quantized bases add a separate approximation

Dettmers and colleagues’ QLoRA combines a quantized frozen base model with trainable low-rank adapters. That combination makes two distinct choices: how the base computation is represented and what update family training can add. It is useful to keep those choices separate when diagnosing failures. A rank increase cannot be assumed to repair every error caused by a changed numerical representation, and a higher-precision base does not remove an adapter’s capacity limit. The paper reports a particular training approach and experiments; its results should not be treated as a blanket equivalence guarantee for arbitrary tasks and quantization settings.

I would record the base precision, quantization configuration and adapter training configuration together. An adapter learned against one representation may behave differently when attached to another, even when both artifacts share a model family name. Memory estimates also need to describe when values are dequantized and which computations use a wider precision. The frozen base is still involved in every forward pass. Calling a run parameter efficient should not obscure its activation or compute cost. That distinction helps decide whether an experiment is limited by trainable state, sequence length, data throughput or the device’s actual execution kernels.

References: [2] QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers and colleagues

Merging changes the deployment tradeoff

For a compatible linear weight representation, the learned product can be added to the base matrix before inference, avoiding a separate adapter path for that merged artifact. This is one reason LoRA is attractive operationally. The statement needs its conditions, though: a service that switches among many adapters may retain separate factors, and a quantized deployment may need to re-encode a merged weight matrix. That extra conversion can introduce numerical differences. I would test the actual merged artifact rather than assume its output is identical to every unmerged execution path under all precisions and kernels.

Serving many adapters creates an identity and scheduling problem as well as a storage benefit. Requests need to select an authorized adapter, caches must distinguish computations that depend on it, and batches may become less efficient when adapter combinations differ. A tiny checkpoint is still executable model behavior with a training provenance and version. I would treat adapter selection as part of the request contract, not a casual filename parameter. Keeping one shared base can simplify distribution, but it does not mean every downstream artifact inherits the same evaluation results or can be substituted transparently during a request.

References: [1] LoRA: Low-Rank Adaptation of Large Language Models — Hu and colleagues

Adapters do not replace evidence management

Fine-tuning is sometimes proposed as a way to make a model know the latest internal facts. An adapter can change responses, but it does not provide a citation trail, a deletion mechanism for a specific source claim or a guarantee that an old fact stops appearing. For frequently changing knowledge, I would compare adaptation with retrieval and explicit application state before choosing a training solution. An adapter may be well suited to format, style or task behavior while retrieval handles current evidence. These mechanisms can cooperate, but they answer different questions about what the system should do and why.

Evaluation should therefore include both the intended adaptation and unwanted changes elsewhere. A task-specific gain can coexist with degraded refusal behavior, poorer general responses or increased memorization of training examples. The restricted update space does not make those possibilities disappear. I would hold out examples by meaningful groups, inspect near-duplicate contamination and document the intended use. If an adapter is only validated for a narrow classification task, its small size is not a reason to expose it as a general assistant. Deployment scope should follow the demonstrated behavior, rather than the convenience of distributing the artifact.

Choose capacity with a reversible experiment

A useful experiment begins with a clear baseline and several deliberately different adaptation budgets. Vary rank or module coverage while keeping the evaluation questions stable, and report both task quality and resource consumption. Include an alternative that does not train an adapter, such as a stronger task specification or better evidence retrieval, when it addresses the same failure. This prevents training efficiency from becoming the objective in place of solving the user’s problem. The outcome may be that a small adapter is enough, that a larger update is necessary or that adaptation was aimed at the wrong boundary.

My main lesson from LoRA is to make the permitted change explicit. The method buys efficiency by representing the update in a constrained form, while the pretrained model supplies the rest of the computation. That is a powerful engineering tradeoff when the constraint matches the task. It becomes misleading only when parameter count is used as a substitute for capacity analysis, data provenance or deployment testing. I want an adapter to arrive with its base identity, target modules, rank, scaling, training data description and acceptance results, so that a compact artifact also remains an understandable one.