A collection of timed operations is not automatically a trace. The useful structure comes from carrying causal context across the boundaries where work changes process, thread, queue or owner. I would prioritise those relationships before adding more spans, because precise durations attached to the wrong request can produce a confident but false explanation.
Trace identity connects observations across boundaries
The W3C Trace Context specification standardises fields such as traceparent so participating systems can propagate trace identity and parent information. OpenTelemetry describes propagation as the mechanism that allows telemetry from separate components to be correlated. Those formats are a shared language, not evidence that the application has instrumented every causal boundary correctly. A proxy can forward a header while an asynchronous job silently starts a new unrelated context. I would draw the execution graph and identify each place where context must be extracted, attached, injected or intentionally replaced.
A trace identifier should not be interpreted as an ordering clock, a globally authenticated user identity or proof that every span belongs to trustworthy code. Incoming context can be supplied by a caller and must be validated according to the propagation format and trust boundary. The service can choose whether to continue that context or start a new internal trace with an explicit relationship. Business authorisation must still rely on its own credentials. This separation matters because diagnostic metadata often travels farther than the original application's carefully designed authentication assumptions.
References: [1] W3C Trace Context[2] OpenTelemetry context propagation
Local context must follow concurrent execution
A global variable holding the current trace identifier is unsafe when requests overlap. Request A can set it, yield to request B and then resume with B's value. Thread-local storage helps only when execution remains tied to the relevant thread; asynchronous tasks can share a thread while representing different requests. Language and instrumentation runtimes provide context mechanisms for this reason. I would test the actual execution model rather than assuming that a helper named currentTrace remains correct across awaits, thread pools and callbacks.
The Python example uses contextvars to illustrate task-local propagation and explicit restoration of the previous value. It is not an OpenTelemetry implementation and does not generate trace identifiers or spans. Its purpose is to demonstrate the invariant: two interleaved tasks should observe their own context before and after yielding. The reset in finally matters because cleanup must occur on success and failure. In a production service, the tracing library's supported context manager should carry that responsibility, with tests at any custom boundary that bypasses the normal framework integration.
import asyncio
from contextvars import ContextVar
trace = ContextVar('trace', default=None)
async def handle(identity):
token = trace.set(identity)
try:
before = trace.get()
await asyncio.sleep(0)
after = trace.get()
return before, after
finally:
trace.reset(token)
async def main():
result = await asyncio.gather(handle('A'), handle('B'))
assert result == [('A', 'A'), ('B', 'B')]
assert trace.get() is None
print(result)
asyncio.run(main())References: [2] OpenTelemetry context propagation
Queues and batches need a causal model
An HTTP call often has an obvious parent-child relationship: one operation directly invokes another and waits for it. A queue changes that shape. A producer may finish before a consumer begins; a batch can contain messages from several traces; a retry can process the same durable message more than once. I would decide whether the instrumentation represents one long trace, a new processing trace linked to the producing context, or another supported relationship. The decision should reflect the workflow and the tracing system's capabilities, not whichever option produces the prettiest waterfall.
For a batch with multiple origins, selecting one arbitrary message as the parent can imply a causal hierarchy that does not exist. Links can preserve several relationships without pretending that all work descended from one request in a simple tree. Record stable message or operation identities in appropriately controlled attributes when they are needed for investigation, and distinguish delivery attempts from the durable business operation. A trace can then explain retries and fan-in without requiring the user to infer that two similarly named spans refer to the same underlying work.
References: [2] OpenTelemetry context propagation
Context is not permission to copy every datum
Propagation can make a small piece of metadata visible to many services, logs and vendors. That is useful for bounded routing or diagnostic fields and dangerous for secrets, personal information or arbitrary user input. W3C Trace Context includes privacy considerations, and the same discipline should apply to additional baggage. I would define an allowlist, size limits and sanitisation rules at trust boundaries. A trace ID can be enough to join controlled records without embedding the customer's email, access token or full request body in every downstream span.
Attributes also need a clear owner and meaning. A field called status can refer to HTTP status, business state or span outcome; conflating them makes cross-service queries misleading. Prefer stable names and bounded values for common dimensions, while retaining high-detail identifiers only where access and retention justify them. Truncation should be visible when it can change interpretation. Observability data is another production dataset with its own exposure and cost model, so I would avoid treating instrumentation as an unrestricted side channel exempt from the application's ordinary data-handling decisions.
References: [1] W3C Trace Context
Sampling changes what the evidence can prove
Assume illustratively that a service handles ten thousand requests per second, produces eight spans per request and averages five hundred encoded bytes per span. That is forty million bytes per second before transport and indexing overhead. Sampling one percent of requests uniformly would reduce the expected span payload to about four hundred thousand bytes per second under those assumptions. This simple budget explains why collecting everything may be impractical, but it does not establish that the retained traces represent every important failure class.
Head sampling decides early and can miss rare outcomes that are not yet known. Tail sampling can use later evidence such as errors or latency, but it needs buffering, completion heuristics and resource limits, especially for long or incomplete traces. A dataset biased toward errors is excellent for debugging and unsuitable for estimating the overall error rate without the appropriate weighting and denominator. I would describe the sampling policy alongside trace-based claims. Metrics can provide population-level counts while traces provide selected explanations, and neither should silently impersonate the other's statistical role.
References: [2] OpenTelemetry context propagation
More spans can reduce explanatory value
The counterargument to extensive tracing is that instrumentation can create overhead, noise and operational dependence on a telemetry backend. I agree that not every function deserves a span. Boundaries with meaningful waiting, resource use or failure semantics are usually more valuable than thousands of tiny implementation details. A useful span should help answer a question: which dependency consumed the deadline, where a queue delayed work or which retry attempt changed the outcome. If a span cannot be interpreted without reading the source code, its name or placement may need reconsideration.
Instrumentation should also fail in a bounded way. A slow exporter must not accumulate unlimited memory or block critical request completion indefinitely. Dropped telemetry should be counted so an empty trace view is not mistaken for an empty workload. I would test exporter failure and verify the application's intended degradation policy. The service may choose to lose some diagnostic data while preserving its main function, but that choice needs visibility. A tracing system that causes an outage under backend failure has crossed from observation into an unplanned production dependency.
References: [2] OpenTelemetry context propagation
Validate relationships with a known execution
A useful integration test sends a request through a controlled fan-out, asynchronous job and retry, then inspects the resulting relationships. Check that child operations inherit the intended context, independent requests remain separate and links preserve multiple origins where needed. Include malformed incoming headers and a task that fails before cleanup. The assertion should concern the causal graph and relevant attributes, not exact wall-clock durations that vary under scheduling. This catches failures that a test merely checking whether spans were emitted would miss.
I would consider tracing successful when a reader can reconstruct the important path of one operation and distinguish missing evidence from missing work. That requires consistent propagation, meaningful boundaries, known sampling and controlled metadata. It does not require pretending that the trace is a complete distributed execution log or that timestamp order proves causality. The strongest traces are modest about what they know and precise about how their observations connect. Context supplies that precision; additional spans help only when they preserve and explain the relationships.
References: [1] W3C Trace Context[2] OpenTelemetry context propagation
Sources and further reading
- W3C Trace Context
The primary standard defines interoperable trace context and privacy considerations. Security boundaries, graph tests and workload figures are the essay's analysis.
- OpenTelemetry context propagation
Primary documentation explains context propagation across execution boundaries. The contextvars program is an independent standard-library illustration, not SDK usage guidance.