Latency budgets need critical paths

A request is a dependency graph

Suppose an illustrative request performs 20 milliseconds of authentication, then starts two independent reads taking 80 and 40 milliseconds, waits for both and spends ten milliseconds rendering. Its elapsed critical path is 20 plus the maximum of 80 and 40 plus ten, or 110 milliseconds. Summing all operation durations gives 150 milliseconds, which is a measure of accumulated work across these operations rather than the request's elapsed duration. The overlap changes which optimization can bring the response forward.

Halving the 40-millisecond branch to twenty does not change the 110-millisecond critical path under those assumptions. Halving the 80-millisecond branch to forty reduces it to seventy. Both changes reduce resource consumption, but only the second directly moves the join earlier. I would draw this small graph before assigning a latency target to each component. Without the graph, a team can spend substantial effort improving a branch that has plenty of slack while the dependency that actually determines completion remains untouched.

A span tree is useful but not the whole graph

Tracing records spans and their relationships, providing evidence about where operations overlap and where a request waits. A parent-child tree is not automatically a complete causal graph. Asynchronous work can have links, a join can depend on several predecessors, and an uninstrumented queue can occupy most of a gap between spans. I would verify that the trace represents the waits that control completion, rather than assume every visible nesting relationship is a strict sequential dependency. Instrumentation structure and execution structure often differ in small but consequential ways.

Clock differences between machines can also distort the apparent placement of spans. Local duration measurements and explicit dependency relationships are stronger evidence than visually comparing wall-clock timestamps across hosts without uncertainty. A negative-looking network gap may be a clock problem rather than time travel. The practical response is to preserve causality and identify unmeasured intervals, not to force every trace into a perfectly aligned waterfall. A critical-path analysis should explain how it handles missing spans, retries and clock skew before assigning precise blame to a component.

References: [1] OpenTelemetry: Traces

Queueing deserves its own budget

An operation's service time starts when it receives the resource needed to run. The user may have already waited in admission, connection-pool, executor or downstream queues. A component that reports only service time can satisfy its internal target while violating the caller's elapsed budget. I would place spans or timestamps on enqueue and start, then separate waiting from execution. This distinction also changes the remedy: a faster query and a larger connection pool do not solve the same source of delay.

For an illustrative 200-millisecond deadline, allocating 160 milliseconds to execution while ignoring 60 milliseconds of queueing guarantees failure for that timeline. A useful budget includes expected waits and reserves room for variation, transmission and final response work. The reserve is not evidence that every component may consume it independently. It belongs to the end-to-end deadline. If several layers each add their own generous retry and queue allowance, the composed request can become unbounded despite every layer having a locally reasonable timeout.

Percentiles cannot simply be added

The 99th percentile of a sum is not generally the sum of the components' 99th percentiles. The requests contributing to each component's tail may be different, and the dependence between their durations matters. Parallel joins add another complication because the relevant operation is a maximum. I would use complete request observations or an explicitly stated joint model when estimating end-to-end tails. A spreadsheet that adds independently reported component percentiles can be either pessimistic or misleading without revealing which assumptions created the result.

Correlation is particularly important during shared-resource incidents. Several branches may slow together because they use the same network path or storage cluster. Conversely, independent rare delays can make a fan-out request encounter some slow branch more often than a single branch's percentile suggests. The right evidence is the distribution of the actual critical path under the relevant workload. Component distributions remain useful for diagnosis and ownership, but their summaries cannot reconstruct all the dependency information that was discarded when the observations were aggregated.

References: [2] Google SRE: Monitoring Distributed Systems

Deadlines should travel with the work

A caller with 200 milliseconds remaining should not launch a downstream operation with a fresh one-second timeout and then forget about it. Passing a deadline or remaining budget lets the downstream boundary decide whether useful completion is still possible. I would subtract required return-path work and avoid spending the entire remaining budget on one dependency. The mechanics depend on the transport and clock model, but the principle is stable: nested calls share a user objective rather than each inventing a new duration allowance.

A remaining-time value also needs interpretation during retries. If the first attempt consumes most of the deadline, another full attempt may have little chance of producing a useful result. Admission can reject work that cannot plausibly finish, and cancellation should release resources when the caller no longer needs the answer. This does not imply that cancellation reverses an external side effect. The latency budget and effect protocol remain separate contracts. A timeout determines when an answer loses value to the caller; it does not prove what the remote system did.

Optimizing off the path can still matter

The counterargument to critical-path focus is that noncritical work consumes shared resources. Reducing the short branch in the earlier example might lower database contention enough to improve the long branch under load. It might also create capacity for more requests. I agree, which is why the dependency graph should accompany a resource model rather than replace it. The direct elapsed effect is zero in the isolated example; the indirect effect under contention is an empirical question. Keeping those claims separate produces a better experiment.

A branch can also move onto the critical path after another branch improves. Once both reads take forty milliseconds, further gains require considering both or changing the dependency structure. Optimization changes the bottleneck, so the original graph should be measured again after material changes. I would avoid permanent labels such as the slow service when they are based on one workload snapshot. What matters is the distribution of critical-path participation and the cost of reducing it under the product's current request mixture and resource constraints.

Budget decisions should change behaviour

A budget becomes useful when crossing it triggers a defined action: skip an optional enrichment, return a partial result with explicit semantics, reject new work or stop a retry. Those actions should be designed with the product, because they change what the response means. A graph can identify an expensive dependency, but it cannot decide whether that dependency is optional. I would make the required and optional branches explicit and test the response contract when each optional branch is absent or late.

The final review should follow one request from arrival through queueing, required work and response delivery. It should explain the deadline, the dominant path, the reserve and the behaviour when the budget is exhausted. Component owners can then optimize toward a shared outcome. Latency budgets fail when they become disconnected allowances that merely decorate a diagram. They work when the dependency structure and runtime decisions enforce a coherent promise about when a useful answer can reach the user.

I would validate that promise with adversarial timelines as well as normal traces: delay the shorter branch until it becomes longest, stall before execution starts, and let a retry complete after the deadline. These cases expose hidden dependencies that an average waterfall can conceal. They also show whether budget exhaustion stops resource use or merely stops the visible timer.