Open-loop load tests tell a different story

Completion can secretly control the offered load

In a closed workload, a fixed population of clients performs work and starts another iteration after completing the previous one, perhaps after a think time. When responses slow down, those clients issue fewer new requests. That can be an accurate model of a bounded set of interactive users. It is a poor substitute for an arrival process that continues independently, such as incoming events or scheduled jobs. I would choose the workload model from the source of demand rather than the easiest load-generator configuration.

In an open model, arrivals are scheduled independently of previous completions. The implementation still needs enough capacity to create those arrivals, and the service still needs an admission policy. Open does not mean unlimited memory or permission to ignore rejected work. It means the intended demand is not automatically reduced by the response time of earlier requests. This distinction becomes most visible during overload, when a closed generator can quietly lower its request rate at exactly the moment an external arrival stream would build a queue.

References: [2] Grafana k6: Open and closed models

A pause exposes the missing observations

Consider an illustrative service pause lasting 100 milliseconds. Requests are intended at times zero, ten, twenty and so on through ninety milliseconds. Assume, purely to isolate the timing issue, that all ten can complete instantly when the pause ends at 100. Their delays from intended arrival are 100, 90, 80 down to 10 milliseconds, with a mean of 55 milliseconds. A single sequential client that sends at zero and waits before sending again records only the first delayed request during the pause.

That client's 100-millisecond observation is real, but it omits the nine requests that the independent schedule would have generated. The omitted experiences change the latency distribution and the amount of demand presented to the server. In an actual service, processing after the pause is not instantaneous, so accumulated work can extend the recovery interval. The example is intentionally favourable to the server. It illustrates why a test must preserve the intended arrival process before its percentile report can describe the user population the experiment claims to represent.

There are several honest latency clocks

A generator can record latency from the intended schedule time, from the actual send time or from a lower-level network event. These measurements answer different questions. If a request should start at ten milliseconds but the generator only sends it at forty, measuring from forty excludes thirty milliseconds of delay in meeting the demand schedule. wrk2 explicitly discusses this coordinated-omission problem and scheduled-time accounting. I would retain intended and actual timestamps when possible, because their difference helps distinguish generator delay from downstream service delay.

The boundary must also include failures. A request that times out or cannot be scheduled should not vanish from a report containing only successful responses. Report attempted arrivals, actual sends, admissions, completions, errors and unfinished work together. A low successful-request percentile can coexist with a large rejected population. That may be a valid admission-control outcome, but it is not evidence that all offered demand was served quickly. Latency and completion rate belong to the same experiment, even when the dashboard presents them on separate charts.

References: [1] wrk2: constant-throughput latency measurement

The generator has a capacity limit

An arrival-driven configuration does not guarantee that the client machine can produce the specified traffic. Connections, virtual users, CPU, network bandwidth and file descriptors can all become constraints. If the tool exhausts its worker pool, the experiment needs to report dropped or delayed iterations. I would measure the generator's own utilization and scheduling lag, then increase its capacity or distribute generation before attributing the entire result to the server. Otherwise the test can quietly transform into a different demand model under load.

An illustrative steady rate of 2,000 requests per second with a mean response time of 250 milliseconds requires about 500 requests in flight under stable conditions, using the same boundary for rate and time. If latency rises to two seconds while arrivals continue, the corresponding steady-state concurrency would be 4,000. During a transient, the queue evolves rather than instantly reaching that value. The calculation nevertheless explains why a generator sized for the healthy response time may run out of slots during the failure interval it is meant to investigate.

Arrival shape matters beyond the average

A perfectly periodic arrival stream and a bursty stream can have the same average rate while producing different queues. Correlated fan-out, synchronized timers and batch releases are common sources of bursts. I would model the relevant temporal structure rather than label a single requests-per-second number as production-like. A constant-rate experiment is useful for finding a capacity boundary because it is controlled. It should be supplemented when the product must absorb bursts or when synchronized clients are part of the actual environment.

The test should also preserve request cost variation and key reuse. A burst of cache hits is different from the same count of large uncached queries. Mixing payloads independently may miss a tenant whose expensive requests arrive together. I would construct a small set of named scenarios with distinct purposes: steady capacity, realistic mixture, burst recovery and dependency slowdown. Each should state its demand assumptions. One elaborate scenario that attempts to represent everything can be harder to diagnose than several focused experiments whose differences are explicit.

Correction cannot create a full counterfactual system

Some histogram techniques estimate omitted latency samples from a known expected interval. That can be valuable for understanding measurement distortion, but it cannot retroactively make the server process requests that were never sent. Missing requests would have competed for resources, triggered admission controls and potentially changed later behaviour. I would distinguish a corrected latency estimate from an actual arrival-driven experiment. The former repairs an accounting assumption; the latter exercises the queueing and overload mechanisms that the original workload failed to invoke.

The strongest case for a closed model is that it matches the real interaction. A limited worker pool polling for jobs is naturally completion-dependent. A user who waits for a page before clicking again is not an independent metronome. Testing those systems with an unrealistic open flood can exaggerate demand. My objection is therefore to an unnamed model, not to closed loops themselves. The experiment should say whose behaviour is represented and how that population reacts when responses slow down, instead of treating the tool's default as a neutral choice.

Report recovery, not just the peak

After reducing offered load, continue observing until queues drain or the service reaches a declared timeout. Record how much work remains, whether retries amplify demand and whether latency returns to its earlier distribution. A system that accepts everything during a burst may look successful until the backlog consumes the next several minutes. A system that rejects promptly can protect accepted requests while sacrificing availability for the rejected population. The experiment should make that trade visible rather than choosing whichever headline metric makes the implementation look best.

My preferred report pairs the configured arrival schedule with measured sends and outcomes over time. It includes generator saturation checks, timeout accounting and a latency boundary expressed in ordinary language. Those details make the result useful for a capacity decision. Open-loop tests often tell a harsher story because they preserve demand while the service struggles. That story is valuable when it matches the world the service must face. The goal is a faithful experiment, not an impressive throughput number produced by an accommodating client.

Finally, retain enough raw interval data to inspect the pause itself. A whole-run histogram can dilute a brief severe failure with a long healthy warmup. Splitting the run into named phases makes the overload and recovery contract reviewable without inventing a new percentile for every convenient time window after seeing the result.