A metric label creates a way to divide observations, and every observed combination can create another time series. That makes cardinality a resource allocation decision. I would budget dimensions according to the questions they answer, including their churn and histogram multiplier, rather than discovering the cost after an unbounded label reaches production.
Count combinations rather than label names
A metric with five labels can be inexpensive or enormous depending on the values and combinations observed. Route, method, status, region and instance may each look bounded in isolation, but their product gives a useful upper-bound planning estimate. Actual cardinality can be lower because some combinations never occur. Prometheus's instrumentation guidance warns about labels with high or unbounded cardinality and recommends moving unsuitable analyses to other systems. My interpretation is to make the expected combination space part of instrumentation review, just as memory allocations receive scrutiny in application code.
Suppose an illustrative request metric has twenty route templates, five methods, six status classes, four regions and thirty instances. If every combination occurs, that is 72,000 series before considering histogram expansion or additional metrics. Replacing a route template with the raw URL can remove the practical bound, because identifiers and query strings create new values continuously. A label named route is therefore not enough evidence of safety. The instrumentation must establish that it records a controlled template such as an operation class rather than arbitrary request text.
References: [1] Prometheus instrumentation practices
Histograms multiply the representation
A classic Prometheus histogram represents bucket counts plus sum and count series for each label set. If there are ten finite bucket boundaries, there is also the positive-infinity bucket, giving eleven bucket series and two additional series: thirteen in total. Applied to the hypothetical 72,000 label combinations, the result is 936,000 series. At a fifteen-second scrape interval, four samples per minute per series yield 3,744,000 samples per minute before considering metadata, exemplars, replication or other metrics. These are accounting figures, not a storage benchmark.
The distinction between finite buckets and total buckets prevents a common off-by-one error. Native histograms use a different representation and should not be costed by blindly applying this classic-series formula. Prometheus's histogram documentation explains the relevant data types and tradeoffs. I would record which representation is deployed before estimating cost, and include the backend's actual handling of it. The useful question is how much precision and dimensional breakdown the service needs, then how the selected representation pays for that information across the expected population.
from math import prod
label_values = [20, 5, 6, 4, 30]
combinations = prod(label_values)
finite_buckets = 10
series_per_combination = finite_buckets + 1 + 2
series = combinations * series_per_combination
samples_per_minute = series * (60 // 15)
assert combinations == 72_000
assert series == 936_000
assert samples_per_minute == 3_744_000
print(combinations, series, samples_per_minute)References: [2] Prometheus histograms and summaries
Churn can hurt even when the live set is small
A workload may expose only a few thousand active series at a moment while creating millions of distinct series across a retention window. Ephemeral instance identifiers, deployment hashes or per-job labels can produce this churn. The backend may need to retain index entries and historical data after the workload disappears. I would therefore distinguish active cardinality, newly created series per interval and retained cardinality. Looking only at the current scrape can make a high-churn system appear comfortably bounded while its storage and query costs continue to grow.
Configuration versions illustrate the tradeoff. A small number of active revisions can be useful for comparing a rollout, but putting a unique revision on every long-lived business metric can preserve a new population after every deployment. A separate bounded information metric or controlled diagnostic query may answer the same question more cheaply. The right choice depends on the backend and retention policy. I would not ban version labels categorically; I would ask how many values appear over time, whether they are needed on every measurement and when obsolete combinations stop consuming meaningful resources.
References: [1] Prometheus instrumentation practices
Choose the lowest-cost place to ask the question
Metrics are excellent for repeated aggregate questions over bounded dimensions: error rate by operation, queue depth by worker class or latency by region. A question about one customer request often fits a trace or controlled log record better than a permanent metric label containing the request ID. A question about all customers' billing behaviour may belong in an analytical dataset with explicit access and query controls. Moving the question is not losing observability; it is choosing a representation whose cost and semantics match the investigation.
The counterargument is that removing identifiers makes debugging harder. That is true when the only available tool is the metric store. The better answer is a deliberate connection between aggregate signals and detailed evidence, such as sampled exemplars or a trace identifier in an alert's investigation workflow. The aggregate metric tells the operator where and when the problem is occurring. Detailed records explain particular instances. I would preserve enough common bounded dimensions to navigate between them without forcing every high-detail attribute into a label on every continuously scraped time series.
References: [1] Prometheus instrumentation practices
Drop unwanted dimensions before paying for them
Aggregating at query time can make a dashboard look simple while the backend still ingests and stores the full high-cardinality input. If a dimension is never needed, removing it at instrumentation or ingestion prevents more cost than hiding it in a graph. The exact savings depend on where the filtering occurs: a collector can reduce backend work while the application may still allocate its local metric state. I would inspect the complete path from label construction through exporter, collector, storage and query rather than assuming one aggregation expression solves the resource problem everywhere.
Application-side state deserves special attention for libraries that keep an accumulator per label set. An unbounded label can grow memory before the first remote sample is stored. Limits should be observable rather than silently merging unrelated values into a misleading series. A controlled overflow category can be useful if its semantics are clear and a counter reports that the budget was exceeded. Another option is to reject unsafe instrumentation during review. The important property is a defined behaviour when the expected domain is violated, because user-controlled inputs rarely respect a spreadsheet's assumed maximum.
References: [1] Prometheus instrumentation practices
Precision and dimensionality compete for a budget
Histogram buckets should support the latency questions and service objectives the team actually uses. A dense set of boundaries everywhere may spend heavily without improving decisions. A sparse set that omits an important threshold may make the metric unable to answer the required question. I would start from the decision, such as the fraction of requests within a stated latency target, then choose boundaries and dimensions that support it. Percentile displays should be understood as estimates derived from the selected representation, not exact measurements of every request's rank.
A service-wide budget can reserve room for essential operational metrics and allow teams to spend additional series on justified diagnostics. Temporary investigations should have an expiry or a review date so a useful one-week dimension does not become permanent by accident. Cost estimates should include scrape frequency, retention and replication as well as series count. Two designs with equal cardinality can have very different sample volumes. The point is not to minimise all telemetry; it is to spend the available capacity on evidence that changes operational decisions.
References: [2] Prometheus histograms and summaries[1] Prometheus instrumentation practices
Review instrumentation as a production interface
For each new metric, I would ask what question it answers, who uses it, which labels are bounded and what happens when the bound is exceeded. A simple fixture can generate representative label combinations and estimate histogram expansion before deployment. Production monitoring should then compare expected and observed new-series rates, because real traffic may reveal values absent from tests. Include malformed routes, unknown status values and rapid workload creation in the test data. Those are the cases that turn a sensible metric name into an unbounded allocation pattern.
The acceptance criterion is that the telemetry remains useful and affordable as traffic, deployments and failures change. A dashboard that loads quickly today does not prove the underlying series lifecycle is healthy. Nor does a low-cardinality metric automatically provide enough detail to diagnose a problem. I would treat cardinality as a shared observability budget with explicit priorities, reviewable assumptions and feedback from actual usage. That framing keeps the discussion focused on the information the team needs, rather than turning labels into either an unrestricted convenience or a blanket prohibition.
References: [1] Prometheus instrumentation practices[2] Prometheus histograms and summaries
Sources and further reading
- Prometheus instrumentation practices
Primary guidance discusses label cardinality and appropriate instrumentation. The budget policy, lifecycle analysis and workload sizes are illustrative reasoning.
- Prometheus histograms and summaries
Primary documentation describes histogram representations and estimation tradeoffs. The calculation explicitly applies to classic histograms and is not a native-histogram cost estimate.