Profile the production shape

A profile has a population

A service might process tiny cached lookups, large uploads and occasional administrative exports. Combining them in one CPU profile answers where the sampled process spent CPU across that mixture. It does not directly answer which path makes an ordinary lookup slow, which tenant is expensive, or which request consumes the most memory. Those questions require different slices. I would record the request classes and their proportions alongside the profile, because the same executable can produce radically different hot paths without any code change at all.

The temptation is to capture the easiest workload to generate and call it representative. Repeating one cached request makes a stable picture, but may remove allocation, parsing and storage behaviour that dominate elsewhere. Conversely, profiling only the largest inputs can exaggerate an exceptional path. The right population depends on the decision. Capacity planning wants a credible aggregate mixture; improving an interactive deadline wants the requests that violate it. I would name that objective before collecting samples so the interpretation does not drift toward whichever function looks most interesting.

Request shares and resource shares differ

Consider an illustrative workload of 1,000 requests. There are 900 small requests using one millisecond of CPU each and 100 large requests using twenty milliseconds each. The small class consumes 900 milliseconds, while the large class consumes 2,000. Large requests are ten percent of arrivals but roughly 69 percent of the total 2,900 milliseconds of CPU. A profile dominated by their parsing path can therefore be entirely correct even when most users never encounter that path. The interpretation needs both denominators.

Suppose an optimization halves the large class's CPU requirement without affecting small requests. Aggregate CPU falls from 2,900 to 1,900 milliseconds for the same mixture, a reduction of about 34.5 percent. That may improve capacity substantially while leaving small-request latency unchanged at low utilization. Under contention, the smaller requests may benefit indirectly through shorter queues. I would distinguish those direct and indirect effects when proposing the change. A profile identifies resource consumption; translating that into user value requires an explicit model of how requests interact.

Sampling measures occupancy, with uncertainty

A sampling CPU profiler observes stacks at selected moments while code executes. A function occupying a substantial fraction of CPU time is more likely to appear than a very short infrequent operation. The sample count is therefore evidence about occupancy under the profiler's mechanism, not an exact invocation count. I would avoid reading a thin bar as proof that a function is cheap per call. It may be called rarely, spend time blocked, be hidden by inlining, or simply receive too few samples in a short collection.

As a rough illustrative calculation, 100 samples in a function out of 1,000 suggests a ten-percent share. If samples were independent Bernoulli observations, the standard error would be about 0.95 percentage points. Real profile samples can be correlated and biased by collection mechanisms, so that calculation is not a valid universal confidence interval. Its purpose is to show why tiny differences between short captures deserve caution. Longer independent captures, consistent workloads and agreement across evidence sources are stronger than treating every pixel change as a precise regression.

References: [1] Go: Profiling Go Programs

Choose the profile that can see the suspected cost

CPU, allocation, retained-heap, mutex and blocking profiles observe different things. A request waiting on a database may consume little CPU while suffering high elapsed latency. An allocation-heavy path may produce many short-lived objects without dominating retained memory. A mutex profile can point toward contention that a CPU graph obscures. I would choose the instrument from the hypothesis, then cross-check with request timelines. Collecting every available profile without a question can generate a large artifact set while leaving the relevant waiting boundary unobserved.

Even the word memory needs a denominator. Bytes allocated during an interval, live bytes at collection and peak resident memory answer different operational questions. A change can reduce allocation volume while retaining a larger cache, or improve CPU by trading away memory headroom. The desired outcome should specify which resource matters and under what constraint. I would keep a small resource ledger for the candidate workload, including elapsed latency, CPU, allocation and retained memory, so an apparent improvement cannot conceal an expensive transfer to another resource.

References: [2] Go runtime/pprof documentation

Attribution follows the runtime, not always the request

Background work complicates per-request interpretation. A request may enqueue compression, trigger garbage collection later or leave another worker to serialize the response. The CPU cost exists, but a stack alone may not retain the causal identity of the original request. Runtime labels and trace context can help when propagated across the actual work boundary. I would validate that propagation with a known task before relying on class-level profiles. Missing labels should remain an explicit unknown category rather than silently joining the nearest convenient request class.

Labels also carry operational costs. Unbounded tenant identifiers or request IDs can create excessive cardinality and expose information that does not belong in a diagnostic artifact. Grouping by stable workload class is often sufficient for a performance decision. Where individual attribution is required, use an approved collection and retention policy. The technical point is that more dimensions do not automatically mean better evidence. A small set of trustworthy labels is more useful than a richly annotated profile whose asynchronous work loses context halfway through the pipeline.

Production evidence still needs a controlled comparison

A production capture offers realism but also includes changing traffic, neighbours, cache warmth and background activity. A controlled replay can isolate a proposed change while preserving the features that matter. I would extract a workload description rather than blindly replay sensitive payloads: size distributions, operation mix, concurrency and relevant key reuse can often reproduce the cost structure. The replay should retain expensive edge cases and temporal bursts. Merely matching average request size does not reproduce a distribution with a long tail.

The counterargument is that such modelling is work, and a clearly dominant accidental allocation can be fixed directly. I agree when the local explanation is strong and the change is simple. Even then, verify the complete operation after the fix. A profile is excellent at proposing hypotheses but weaker at proving the final effect. If the expensive function disappears, another cost may become dominant or the workload may simply have changed. The comparison should preserve inputs and report absolute resource use, not only the new percentages.

Write down what the picture cannot tell you

A useful profile report names the binary, collection interval, runtime settings, request mixture, sample type and important exclusions. It then makes one bounded claim: for this workload, this path accounts for enough of the relevant resource to justify investigation. The proposed change should predict a measurable effect under that boundary. If an optimization removes half a path that occupies ten percent of CPU, a roughly five-percent total CPU reduction is the first-order expectation, before secondary effects. A promise of a twofold service speedup would need another explanation.

I would retain representative captures before and after the change, together with the workload description that makes them comparable. Future readers should be able to distinguish a code improvement from a shift in request population. Profiling becomes much more powerful when it is treated as an experiment on a system rather than a hunt for aesthetically unpleasant functions. The goal is a credible account of where resources went, which part can change, and why that change matters to the requests the service actually handles.

A flat profile is also informative. It may mean the workload distributes cost broadly, or that the chosen instrument cannot observe the bottleneck. That should change the investigation rather than force a culprit. Sometimes the next useful artifact is a queue timeline, a blocked-stack sample or a breakdown by payload size. The ability to stop interpreting an unhelpful graph is part of using profiles well.