A benchmark failure should mean there is credible evidence of a meaningful regression. I would design that decision before collecting numbers, including how the system handles noisy or inconclusive results.
A regression is a comparison under a contract
A timing number alone does not establish a regression. The claim compares a candidate with a baseline for a specified workload, environment and metric. I would store both executable identities, the input description and the harness version with the result. A baseline from a different machine or a distant period may be useful context, but it introduces additional explanations for a difference. The cleanest experiment runs both versions under comparable conditions and asks whether the candidate changes a resource or user-facing property enough to matter.
The metric should match the requirement. CPU per completed operation, peak retained memory, throughput at a latency limit and request-tail latency can respond differently to the same change. Selecting whichever metric happens to worsen after the run creates an unstable decision process. I would name a small set of primary outcomes and relevant constraints in advance. Secondary diagnostics can explain the result, but they should not silently redefine the acceptance rule. This turns performance CI into an experiment rather than a collection of numbers waiting for an interpretation.
Pair comparisons against changing conditions
If every baseline run happens first and every candidate run happens later, temperature, background load or cache state can be confounded with the code version. Randomizing or interleaving execution order helps separate the version effect from time trends. Pairing nearby runs under comparable conditions can also reduce variation from shared environmental factors. I would preserve the order in the report, since a systematic rise across the entire session is useful evidence. Randomization does not eliminate noise; it makes some misleading patterns less likely to align with the treatment.
For an illustrative set of pairs, baseline times of 100, 102 and 98 units compared with candidate times of 105, 107 and 103 suggest roughly five-percent slowdowns in each local comparison. Three pairs are not a strong statistical foundation, and the units are invented. The point is that the paired pattern is more informative than comparing one favourite baseline run with one unfortunate candidate run. The analysis should respect the experimental unit, whether that is a fresh process, a machine allocation or an entire workload execution.
| Pair | Baseline | Candidate | Relative slowdown |
|---|---|---|---|
| 1 | 100 | 105 | 5.00% |
| 2 | 102 | 107 | 4.90% |
| 3 | 98 | 103 | 5.10% |
References: [2] NIST: Completely randomized designs
Iterations are not automatically independent evidence
A benchmark framework may execute many iterations inside one process. Those iterations can share cache contents, allocator state, frequency conditions and background interference. Treating every iteration as an independent experiment can produce unjustifiably narrow uncertainty estimates. I would distinguish repetitions within a process from independent process launches and independent machine sessions. The appropriate level depends on which variation the claim needs to cover. A stable inner loop does not prove stability across deployments if process initialization and machine placement materially affect the result.
Warmup has a similar interpretive role. Excluding initialization is correct when the target is steady-state service cost, but wrong when startup latency is the product requirement. A benchmark should not discard inconvenient early observations without a declared reason. Google Benchmark offers controls for repetitions and timing, but a framework cannot decide which lifecycle phase the application promises to optimize. I would keep startup and steady-state experiments separate when both matter. Their different boundaries deserve different acceptance policies rather than one averaged number that describes neither well.
References: [1] Google Benchmark: User Guide
Practical significance belongs beside uncertainty
Suppose the team decides, illustratively, that a slowdown above two percent is materially important for a particular operation. An estimated interval from three to five percent supports a different decision from an interval spanning minus one to six percent. The latter remains compatible with both improvement and meaningful regression. An interval from minus half a percent to one percent suggests the experiment can exclude the chosen regression size under its assumptions. I would define this practical threshold before observing the candidate, rather than equate every statistically detectable difference with a product problem.
The interval method must match the data and experiment. Skewed timings, correlated repetitions and tail quantiles require more care than a generic normal approximation. A bootstrap that resamples the wrong unit can be as misleading as a formula that assumes independence. I would document the analysis and validate it on repeated unchanged-baseline comparisons. Those control experiments reveal how often the pipeline declares regressions when the code is identical, which is directly relevant to whether engineers will trust the resulting CI signal.
Many benchmarks create a multiple-comparison problem
If one hundred independent checks each falsely flag five percent of unchanged comparisons, the expected number of false flags is five. Under that simplified independence assumption, the chance of at least one false flag is 1 minus 0.95 raised to 100, about 99.4 percent. Real benchmarks are often correlated, so the exact probability changes. The calculation nevertheless explains why expanding a suite without revisiting its decision policy can make nearly every change look suspicious even when individual tests appear conventionally calibrated.
I would distinguish broad screening from a release-blocking conclusion. A first pass can identify candidates for a focused repeat with stronger controls, while primary product metrics receive a more deliberate policy. Multiple-comparison adjustments or hierarchical decisions may be appropriate, but the method should follow the purpose of the suite. Simply rerunning until a failure disappears is not a sound resolution. It selects a favourable observation and discards evidence. A repeat should have a declared stopping and aggregation rule so the result remains interpretable.
An inconclusive result is a real outcome
A noisy shared runner may be unable to distinguish a small meaningful change within the available time. The correct response can be inconclusive, followed by a dedicated run or a decision that the risk is acceptable. Forcing every result into pass or fail encourages arbitrary thresholds and repeated reruns. I would expose the effect estimate, uncertainty and reason for escalation. A candidate with a large obvious regression should not require elaborate statistics, while a borderline change deserves an experiment capable of resolving the question being asked.
The counterargument is operational simplicity: teams need fast feedback, and a sophisticated performance gate can become a maintenance project. That is a strong reason to keep the primary suite small and representative. Use inexpensive checks to detect large changes, reserve careful experiments for sensitive paths and retain a manual investigation route. The goal is not maximal statistical machinery. It is a decision process whose confidence matches the consequences. A clear coarse signal is better than a precise-looking number produced by an environment that cannot support its interpretation.
Explain the mechanism before declaring the investigation finished
Once a regression is credible, profiles and counters can help identify additional work, allocation, contention or changed code generation. The explanation should predict another observable difference and survive a targeted comparison. A rollback or smaller patch can test whether the suspected change causes the effect. I would retain both the experimental result and the mechanism, because timing alone may not reveal when the issue will recur. A workload-specific slowdown can remain hidden if the fix merely shifts the same cost to a case outside the suite.
The final artifact should name the baseline, candidate, workload, environment, execution order, analysis and decision. It should separate measured evidence from extrapolation to production. Performance CI becomes valuable when it helps engineers make reliable comparisons repeatedly, not when it promises that every timing fluctuation has a definitive explanation. The experiment's design determines which conclusion the numbers can support. Establishing that design first is the difference between a useful regression signal and a threshold that alternates between ignored failures and missed problems.
Keep unchanged-code control runs as part of maintenance. Hardware, kernels, compilers and runners evolve, so the noise model can change even while the benchmark source remains identical. Recalibrating the experiment is ordinary upkeep, and it is more defensible than gradually loosening thresholds whenever the pipeline becomes inconvenient.
Sources and further reading
- Google Benchmark: User Guide
Documents repetitions, timing choices and benchmark execution controls. The CI comparison policy and numerical examples are original recommendations.
- NIST: Completely randomized designs
Explains randomization as an experimental-design tool. This essay adapts that principle to baseline/candidate comparisons without claiming a universal statistical test.