7.4 microseconds: what a benchmark must explain

Draw the boundary before starting the clock

A request can contain input decoding, feature preparation, model execution, output conversion, queueing and transport. A timer around only model execution answers a legitimate question, but a narrower one than a timer around the request. Calling both results inference latency makes comparison ambiguous. I would describe the timed interval in executable terms: the input object already exists, the model is resident, the call begins immediately before this function, and the clock stops after this output is materialized. Everything excluded should have a reason that matches the intended use.

Consider an illustrative path with 3 µs of input preparation, 7 µs of execution and 2 µs of result handling. Its serial service time is 12 µs under these invented conditions, before any waiting or transport. Halving execution produces 8.5 µs overall, not 6 µs. That arithmetic is not a measurement of MICROJEV. It demonstrates why a fast inner operation can be both impressive and insufficient evidence about an application. Optimization priorities depend on the whole budget, while attribution depends on the individual pieces.

A clock reading is an observation with overhead

Rust's Instant is intended for elapsed-time measurement, but its documentation includes platform-dependent details and cautions about monotonic behaviour. Selecting that abstraction does not establish timer resolution, call overhead or the absence of scheduling interruptions on a particular host. For very short work, I would first measure the harness with an empty or minimal operation, then inspect the distribution rather than assuming one constant overhead can be subtracted from every sample. The subtraction itself can be misleading when the control and treatment execute different instruction paths.

Batching can reduce the relative contribution of reading the clock. An illustrative batch of 1,000 operations taking 8 milliseconds averages 8 µs per operation. It does not reveal the p50 or p99 of those operations. One 1-millisecond interruption has been diluted into the batch average, and individual slow operations cannot be reconstructed afterward. I would use batching when the question is steady-state throughput or mean operation cost, and an appropriate per-operation recording method when the question is a latency distribution. Those are complementary experiments, not interchangeable summaries.

References: [1] Rust standard library: std::time::Instant

Make the compiler do the work you mean

A benchmark whose result is never meaningfully used may measure less work than its source code appears to request. Constant inputs, repeated identical calls and aggressively inlined functions can permit transformations that are valid for the benchmark but irrelevant to the application. Google Benchmark explicitly warns that anti-optimization helpers are not a blanket prohibition on optimizing an expression. The question is whether the generated program preserves the intended experiment, not whether a familiar black-box function appears somewhere in the harness.

My illustrative harness would vary representative inputs, consume outputs, and validate correctness outside the timed interval. I would also inspect generated code for surprising elimination or repeated setup, particularly after an optimization changes the benchmark shape. There is a tradeoff: making every input unpredictable may inhibit optimizations the production application legitimately benefits from. Conversely, feeding one constant input can grant the compiler knowledge the application never has. The harness should reproduce the useful invariants of the actual workload, not mechanically maximize either unpredictability or compiler cleverness.

prepare representative inputs and a loaded model
validate outputs against a reference outside timing
for each independently started run:
    warm up according to the declared policy
    randomize the order of configurations A and B
    record samples with identical input populations
    preserve raw samples, failures and run metadata
report per-run results and the comparison uncertainty

References: [2] Google Benchmark User Guide

A million samples can still be one experiment

Successive timings from one process share conditions: CPU state, allocator history, code placement, temperature, neighbouring activity and the same loaded model. Treating every sample as independent can produce an extremely narrow uncertainty interval around a result that changes materially when the process restarts. I would distinguish within-run variation from between-run variation. Repeating complete runs, and where relevant repeating builds or machines, answers a broader question than extending a single tight loop. The correct replication level is the level at which the conclusion is supposed to generalize.

Suppose an illustrative comparison produces run medians of 7.2, 7.3 and 8.1 µs for A, and 7.1, 7.2 and 8.0 µs for B under paired conditions. B is consistently 0.1 µs lower in this tiny example, while both move by much more across conditions. That suggests a paired analysis and further replication, not a universal claim that B always finishes at 7.2 µs. Preserve the pairing, explain how runs were scheduled, and avoid pooling samples in a way that erases the variation the experiment was designed to expose.

Percentiles are statements about a population

A median describes the middle of a specified distribution. It does not state the probability of missing a deadline unless the deadline happens to be positioned relative to that distribution in a known way. In an illustrative set of 100,000 operations, a p99 near 20 µs concerns roughly the upper one percent boundary; it says little about how far the slowest thousand extend. Different quantile estimators can interpolate differently, so a serious comparison should identify its estimator and preserve enough underlying information to reproduce the reported summaries.

Outlier removal is especially dangerous when outliers are the product problem. Excluding interrupted samples might help estimate an uncontended computational cost, but it no longer describes observed wall-clock latency under those conditions. I would keep both views if both questions matter, label the exclusion rule before comparing implementations, and count excluded samples. A result that improves only after removing more difficult observations needs an explanation. Reporting p50, p95, p99, maximum and sample count is still not a substitute for explaining the workload that generated them.

The input distribution is part of the program

Repeatedly evaluating one input may keep particular memory hot and exercise only one branch pattern. Randomly generating inputs can create a different distortion if the real workload has locality, skew or structured correlations. I would define at least a representative scenario and a deliberately stressful one, then keep their results separate. An illustrative runtime might need tests for small and large inputs, common and unusual values, changing model state and concurrent callers. These scenarios answer different operational questions and should not be averaged into a reassuring number without weights.

Load generation matters too. A client that sends its next request only when the previous one completes reduces offered load precisely when the system slows. That can be a valid closed-loop user model, but it cannot automatically stand in for arrivals that continue independently. If deadlines and queueing matter, record arrival intent, actual send time, completion time and failures. Otherwise the benchmark may omit waiting that a real caller experiences. The inner execution number remains useful; it simply needs a separate load experiment before becoming a service-level claim.

Publish the experiment, not just its favourite statistic

For the reported MICROJEV figure, the missing evidence is concrete: machine and operating-system details, compiler and build flags, model and input definitions, timing boundary, warm-up policy, allocation instrumentation, sample counts, raw or suitably summarized distributions, and commands that reproduce the procedure. Zero hot-path allocations also needs a boundary: it need not include model loading, request construction, background work or every error path. Until those details exist publicly, I would keep the figure attached to the caveat rather than comparing it with differently measured systems.

The counterargument is that this is excessive ceremony for a small project. I agree that not every optimization needs a publishable statistical study. A quick local experiment can guide an implementation decision when its uncertainty is smaller than the practical difference. But the stronger the external claim, the more evidence it needs. My threshold is proportionality: enough method for another engineer to identify what was measured, reproduce the broad result and understand where it may stop applying. A compact benchmark repository can accomplish more than a much longer claim about speed.