CPU time is not wall time

A request can be slow without executing much

Imagine a request that spends five milliseconds parsing, ninety milliseconds waiting for a dependency and another five milliseconds preparing its response. Its elapsed duration is roughly 100 milliseconds under this simplified timeline, while its own computation accounts for ten. Optimizing parsing by half saves only 2.5 milliseconds of direct elapsed time. That may still reduce CPU demand across a large fleet, but it does not solve the waiting component. The distinction prevents a valid local optimization from being presented as a solution to a different problem.

CPU time accumulates while execution is charged to the measured process or thread. Elapsed time includes periods during which the measured work is waiting or not scheduled. Neither is intrinsically more honest; they answer different questions. I would use elapsed time for a user's deadline and CPU time for the execution resource consumed within a stated boundary. A profiler focused on CPU can correctly identify the hottest computation while remaining almost silent about why an individual request spent most of its lifetime waiting.

Parallel execution reverses the apparent inequality

It is tempting to assume CPU time must always be less than elapsed time. That is true for one thread's execution over an ordinary interval, but aggregate process CPU can accumulate across several threads running simultaneously. In an illustrative task, four workers each execute for 100 milliseconds at the same time. The wall duration can be near 100 milliseconds while aggregate CPU is near 400 milliseconds, excluding coordination and other work. The ratio indicates parallel resource consumption; it is not an impossible timer reading.

This also explains why utilization percentages need a denominator. A tool may report one fully occupied core as 100 percent, or normalize usage across all available cores. Container quotas introduce another capacity boundary, and host core count may not describe the CPU actually available to a process. I would record whether a value refers to a thread, process, container or machine and how the percentage is normalized. Comparing unlabeled percentages from different tools is a reliable way to manufacture a disagreement where only the accounting conventions differ.

Choose clocks by the property being measured

Python exposes separate clocks for elapsed performance timing and process CPU time. The small program below measures the same block with both. It intentionally sleeps so the distinction is visible, but it makes no exact timing assertion because scheduling and clock resolution vary. A monotonic elapsed clock is appropriate for durations; a civil-time clock is useful for human timestamps. Clock units do not imply accuracy, and a nanosecond-returning API does not guarantee that every nanosecond is independently observable on the underlying platform.

The process clock includes work by the measured process beyond the immediate function when other threads are active. A thread clock narrows that boundary where supported, but can miss work delegated to another thread. A child process may be accounted separately. I would therefore treat the selected clock as part of the experiment's interface, not a utility detail. The function name in a benchmark does not determine whose CPU the operating system charges. That boundary must agree with the question before the resulting number can be interpreted.

import time

wall_start = time.perf_counter_ns()
cpu_start = time.process_time_ns()
time.sleep(0.02)
cpu_ns = time.process_time_ns() - cpu_start
wall_ns = time.perf_counter_ns() - wall_start
print({'elapsed_ms': wall_ns / 1e6, 'process_cpu_ms': cpu_ns / 1e6})

References: [1] Python: time module

Waiting has several causes

A low CPU-to-elapsed ratio narrows the investigation but does not identify the cause. The request could be waiting for a socket, a lock, a timer, a connection pool or permission to run on a busy processor. Memory pressure can add stalls through reclaim and related activity. Linux's pressure-stall interfaces expose resource-specific waiting signals at system and supported cgroup boundaries. Those signals complement request traces; they do not attribute every stall to one request. I would combine them with evidence about the queue or dependency actually involved.

A runnable thread waiting for CPU is particularly easy to confuse with an I/O-bound workload. Both can have long elapsed duration relative to charged execution. The corrective actions differ: more CPU capacity or less competing work may help the former, while a faster dependency or different I/O pattern may help the latter. A blocked stack, scheduler timeline or explicit queue-wait span can discriminate between them. Optimizing the code that happens to run before the wait is usually less useful than locating the boundary that prevented progress.

References: [2] Linux kernel: Pressure Stall Information

Less CPU can still mean a slower response

Batching provides a concrete trade. An implementation may accumulate several operations before processing them together, reducing per-operation CPU through amortized setup. The first operation waits for the batch to form, so its elapsed latency can increase. Compression can similarly consume extra CPU while reducing network time, or save network bandwidth without improving a small local request. I would evaluate both resource efficiency and the deadline rather than treating one as a universal proxy for performance. The desirable trade depends on the service's constraints.

Another example is a spin loop versus blocking synchronization. Spinning can consume CPU while waiting for a condition, occasionally reducing wakeup delay when the wait is extremely short and a core is available. Blocking can release execution capacity while adding scheduler interaction. There is no general rule that lower CPU always produces lower latency. Under contention, however, spinning may deprive the producer of the CPU it needs to satisfy the condition. The experiment must include the actual contention and capacity conditions, not only an isolated happy path.

Concurrency changes what can be attributed

Subtracting process CPU readings around one asynchronous request does not isolate that request if unrelated work runs in the same process during the interval. The delta is still real, but its attribution is broader. A request that fans out across workers creates the opposite problem for a single-thread measurement: the local thread can look nearly idle while other threads do the work. I would use profiles, propagated labels or controlled workloads when precise attribution matters, and state the uncertainty when those mechanisms cannot preserve causality.

The counterargument is that aggregate process CPU is often exactly the capacity signal needed. A deployment decision may care about total CPU per completed operation under a stable workload mixture, regardless of which thread executes it. That is a valid boundary. Divide resource use by completed useful work and report errors and unfinished work too. Otherwise a build that abandons expensive requests can appear more efficient. The objective is consistent accounting, not always the smallest possible scope. Choosing the boundary explicitly makes either aggregate or fine-grained measurement defensible.

Use the difference to choose the next instrument

If elapsed duration and CPU both rise for the same controlled workload, investigate additional execution, reduced computational efficiency and changes in parallelism. If elapsed rises while CPU stays similar, investigate waiting, scheduling and dependency timing. If CPU rises while elapsed falls, the change may be purchasing latency with parallel resource consumption. These patterns are hypotheses, not diagnoses, but they organize the next measurement. I would avoid jumping straight from a single utilization number to a rewrite, a larger machine or a new concurrency model.

A useful report presents both clocks with the workload, concurrency and accounting scope. It says which waiting components were observed and which remain unexplained. The user-visible objective remains elapsed time, while resource budgets constrain how that objective can be met. Keeping the two views together makes tradeoffs legible. The error is not choosing CPU or wall time; it is letting one stand in for the other without a model of how the work runs, waits and overlaps across the system.

I would also separate measurement overhead from the operation when intervals are extremely short. Reading several clocks around a tiny function can become a significant part of the interval. That concern does not erase the conceptual distinction; it changes the measurement method, perhaps toward longer batches or sampling. The boundary should remain the same even when the instrument changes.