Small models and the cost of a decision

Define the decision before shrinking the model

I would begin with the action the system must choose, the information available at decision time and the loss associated with each kind of mistake. A model that selects between a few operational states has a different job from a general language model. It may need a narrow, stable input representation and a carefully bounded output space more than broad expressive capacity. Without that task definition, reducing parameter count risks optimizing a number that has only a loose relationship to the usefulness of the resulting system.

A useful baseline might be a threshold rule, a lookup table or a simple linear model. The learned runtime should earn its complexity against those alternatives under the same information and timing constraints. I would not compare a tiny model against an unnecessarily elaborate baseline and declare victory. The comparison needs to ask whether learning captures a relationship the simpler mechanism misses, whether that advantage persists on held-out conditions, and whether the extra uncertainty is acceptable for the action that follows the prediction.

Parameter storage is an arithmetic starting point

For an illustrative dense parameter array containing 100,000 values, four-byte storage occupies 400,000 bytes, about 390.6 KiB. Two-byte storage uses about 195.3 KiB, and one-byte storage about 97.7 KiB. This is only parameter payload. It excludes scales, biases stored separately, alignment, intermediate activations, input buffers and runtime metadata. The calculation helps locate the order of magnitude, but it should not be presented as the memory footprint of a complete application or as proof that every access will hit a particular cache.

Precision changes also alter the computation, not just the file size. LiteRT’s quantization specification describes an affine relationship between quantized integers and represented real values, with operator-specific constraints. I take that as a reminder to inspect the actual numerical contract. A narrower representation can reduce storage and bandwidth while introducing error, saturation or conversion overhead. Whether that trade-off is acceptable depends on decision boundaries: a small numerical perturbation far from a boundary may be harmless, while the same perturbation near a threshold can change the action.

References: [2] LiteRT 8-bit quantization specification

Put feature preparation inside the budget

Consider an illustrative decision path that spends 20 microseconds gathering and preparing features, 5 microseconds executing the model and 5 microseconds interpreting the result. Its total is 30 microseconds. Halving model execution reduces the total to 27.5 microseconds, a speedup of about 1.09 times rather than two times. These invented values illustrate an accounting boundary. They are not measurements of MICROJEV or another runtime. The calculation shows why an impressive kernel optimization can have a modest effect on the operation a caller actually waits for.

Feature cost can also be variable. Reading an already available counter differs from traversing a data structure, allocating an intermediate vector or waiting for another subsystem. I would document each feature’s provenance, freshness and worst plausible acquisition path. A model can be deterministic while its inputs arrive unpredictably. If the budget excludes preparation, say so explicitly and measure the excluded work separately. The goal is not to force every benchmark into one number, but to prevent a narrow measurement from being mistaken for end-to-end decision latency.

features_us, model_us, output_us = 20.0, 5.0, 5.0
before = features_us + model_us + output_us
after = features_us + model_us / 2 + output_us
print(before, after, round(before / after, 3))
# 30.0 27.5 1.091

Allocation-free is a property, not a whole latency story

Avoiding allocations in a hot path can remove one source of variable work, but it does not establish a bound on every other source. Cache misses, contention, branch behavior, operating-system scheduling and input-dependent execution can remain relevant. I would state the allocation claim with its boundary: which function, which input classes and which phase of execution. Initialization may allocate, and a fallback branch may allocate even if the common inference path does not. A precise claim is easier to test than a general assertion that the runtime is fast.

The same discipline applies to latency percentiles. A median describes the middle observation under the measured workload; it does not constrain rare slow cases. A decision loop with a deadline may care more about missed deadlines and the magnitude of overruns than about its median. I would measure the distribution under representative contention as well as in an isolated microbenchmark. The isolated result remains useful for understanding the kernel, but the loaded result answers a different question about whether the surrounding system can rely on the operation when resources are shared.

Small numerical errors can produce large behavioral changes

Suppose a hypothetical policy changes state when a score exceeds 0.5. Outputs of 0.499 and 0.501 are numerically close but lead to different actions. Evaluating average prediction error alone can therefore hide important decision errors. I would inspect behavior near operational boundaries, including how often small input perturbations or reduced precision change the chosen action. If repeated transitions are costly, the state machine may need hysteresis or a minimum dwell time, but those additions should be evaluated as part of the policy rather than treated as invisible post-processing.

For example, entering a state above 0.55 and leaving it below 0.45 creates an illustrative hysteresis band. That can reduce rapid switching, but it also delays legitimate transitions and introduces dependence on previous state. There is no universal correct band width. Choose it from the action’s costs and observed signal characteristics, then test abrupt changes as well as noisy steady inputs. A tiny model embedded in a stateful controller is a combined dynamical system; model accuracy measured on shuffled independent examples may miss the behavior users actually experience over time.

Design a useful answer when inputs are invalid

A bounded model should not be forced to produce an apparently valid decision from stale, missing or out-of-range features. I would define an input validity contract and an explicit fallback before optimizing inference. The fallback might preserve the current safe state, use a simpler rule or request a fresh observation. Which choice is appropriate depends on the application. The important part is that invalid input does not quietly become an ordinary low-confidence prediction whose downstream meaning is unclear to the consumer.

This also provides a counterargument to learning in the first place. If the available features rarely distinguish the desired actions, a small neural runtime may add complexity without improving decisions. More capacity cannot recover information that is absent at decision time. I would examine error cases by input availability and distribution shift, then compare the learned policy with a simpler abstaining baseline. A system that says it cannot decide under certain conditions may be more useful than one that always emits a class and leaves the caller to discover its limits through failures.

Benchmark the task and the system together

MLPerf Tiny is a useful primary reference because it frames small-model evaluation around defined tasks and system-level measurements rather than parameter count alone. Its benchmark design is not a certificate for an unrelated runtime. My own benchmark plan would specify the task dataset, quality requirement, input preparation boundary, hardware, compiler settings, warmup, sample count and measurement method. If energy is relevant, define how it is measured rather than inferring it from latency. Faster completion can help energy use, but the relationship is not guaranteed by a single timing result.

I would publish both a repeatable isolated measurement and a realistic integration workload where feasible. The former helps identify computational regressions; the latter exposes scheduling, data movement and feature acquisition. Include a baseline with the same output contract and evaluate quality at the chosen precision. A faster configuration that changes decisions substantially is a different trade-off, not a free optimization. Results should make that trade-off inspectable, including cases where the simplest rule remains competitive and cases where the learned model contributes something measurable.

References: [1] MLPerf Tiny Benchmark

Where MICROJEV fits, and where the evidence stops

MICROJEV is described on this site as a Rust learned decision runtime with roughly 100,000 parameters, a reported 7.4-microsecond p50 and zero hot-path allocations. Those are the supplied project figures. Hardware, workload and a complete reproducible benchmark method are not published here, so I cannot turn them into a comparative performance claim. None of the hypothetical calculations in this essay reconstruct that measurement. They explain the questions I would want answered beside any small latency number before deciding what it means for an application.

My interest is the constraint itself: how much useful decision-making can fit inside a small, predictable operation? The answer depends on the full contract, from inputs through action and recovery. A small model is attractive when its limited scope permits careful reasoning about those boundaries. It becomes less attractive when parameter count is used to distract from missing evidence or a poorly defined task. I would judge the runtime by the decisions it supports, the conditions under which it remains useful and the clarity with which it exposes the conditions where it should decline to decide.