Mitigating Cardinality Explosion in Prometheus

A single misconfigured label — customer_id, request_uuid, or an unbounded pod_ip — can take a healthy Prometheus instance from 2 million active series to 40 million in under an hour. This is cardinality explosion: the exponential growth of unique time series caused by high-dimensionality label combinations, and it remains the single most common cause of Prometheus OOM kills in production observability stacks. Unlike a slow memory leak, cardinality explosion is combinatorial — it doesn’t degrade gracefully, it degrades catastrophically, usually during an incident when you can least afford it.

This article breaks down the TSDB internals that make cardinality explosion possible, the detection tooling required to catch it before it pages you, and the sharding and relabelling architecture needed to contain it at scale. If you are running Prometheus beyond a single-team, single-cluster footprint, this is not optional reading.

#The Bottleneck: When Label Sets Outpace TSDB Capacity

Prometheus’s storage engine indexes every unique combination of metric name and label set as a distinct time series. A metric like http_requests_total{method="GET", status="200", path="/api/v1/users"} is cheap in isolation. The problem is combinatorial: if path has 500 unique values, status has 10, and you add a well-intentioned user_id label with 100,000 distinct values, you have just generated 500,000,000 potential series from a single metric family. This is the mechanism behind virtually every cardinality explosion incident — a label that seemed reasonable in a design review turns out to be unbounded in production.

Each active series consumes memory in the head block — roughly 1-3 KB per series depending on chunk encoding and label byte size. At 10 million active series, you are looking at 10-30 GB of resident memory before you have even accounted for query working sets. Prometheus was architecturally designed as a single-node, pull-based system with a bounded cardinality assumption baked into its TSDB storage model; it does not fail safely when that assumption breaks.

#Architectural Breakdown: How Cardinality Explosion Manifests in the TSDB

#Head Block Memory Mechanics

The TSDB head block is an in-memory, mutable structure holding the last two hours of samples for every active series. Every new label combination triggers a new series entry in the head’s inverted index — a map from label name/value pairs to series references. This index is O(1) for lookups but grows linearly with unique series count, and linearly with churn rate (series created and immediately orphaned). High churn is often worse than high steady-state cardinality because it forces constant WAL (write-ahead log) writes and index mutation without any compensating query value.

#The Maths of Label Combinatorics

Cardinality is multiplicative across label dimensions, not additive. For a metric with labels L1, L2, ... Ln with cardinalities c1, c2, ... cn, worst-case series count approaches c1 × c2 × ... × cn. In practice, real-world correlation reduces this (not every combination is observed), but any label sourced from a high-entropy field — UUIDs, IP addresses, raw error messages, unbounded customer identifiers — collapses that correlation assumption and cardinality explosion follows almost immediately after deployment.

#Downstream Propagation via Remote Write

In federated architectures using Thanos, Cortex, or Grafana Mimir, cardinality explosion doesn’t stay local. Remote write forwards every active series to the long-term storage tier, meaning an uncontrolled label on a single scrape target can degrade shared multi-tenant infrastructure. This is precisely why cardinality governance needs to sit as an architectural pattern enforced at ingestion, not as a reactive cleanup task performed after a query-path outage.

#Implementation Logic: Detecting and Containing Cardinality Explosion

#Step 1 — Series Churn and Cardinality Auditing

Before applying mitigation, quantify the blast radius. Use the TSDB status endpoint and targeted PromQL to identify the offending metric families:

1# Top 10 metric names by series count
2topk(10, count by (__name__)({__name__=~".+"}))
3
4# Identify which label is driving cardinality for a specific metric
5count(count by (customer_id) (http_requests_total)) 
6
7# Check churn rate over a rolling window
8sum(rate(prometheus_tsdb_head_series_created_total[5m]))

Cross-reference against /api/v1/status/tsdb, which returns seriesCountByMetricName and labelValueCountByLabelName directly — this is the fastest path to isolating the specific label driving cardinality explosion without scanning raw samples.

#Step 2 — Relabelling at Scrape Time

Cardinality should be filtered as close to the source as possible — at scrape time, before it ever enters the head block. Use metric_relabel_configs to drop or hash high-entropy labels:

1scrape_configs:
2  - job_name: 'payments-api'
3    static_configs:
4      - targets: ['payments-api:9090']
5    metric_relabel_configs:
6      # Drop unbounded request UUID label entirely
7      - source_labels: [request_uuid]
8        regex: '.+'
9        action: drop
10      # Bucket customer_id into a fixed cardinality tier rather than dropping outright
11      - source_labels: [customer_id]
12        regex: '(.{2}).*'
13        target_label: customer_shard
14        replacement: '${1}'
15        action: replace
16      - source_labels: [customer_id]
17        action: labeldrop

This pattern preserves partial dimensionality (useful for coarse-grained debugging) while eliminating the unbounded tail that causes cardinality explosion. It is significantly cheaper than post-ingestion cleanup, since dropped labels never enter the WAL.

#Step 3 — Horizontal Sharding via Remote Write

For clusters that have already outgrown single-node Prometheus, cardinality containment shifts from prevention to isolation. Cortex and Mimir implement per-tenant ingestion limits that cap active series and reject writes exceeding the threshold, preventing one noisy team from starving shared capacity:

1limits:
2  max_global_series_per_user: 1500000
3  max_global_series_per_metric: 200000
4  ingestion_rate: 50000
5  ingestion_burst_size: 100000
6  cardinality_limit: 100000
7  per_tenant_override_period: 10s

The max_global_series_per_metric field is the direct structural defence against cardinality explosion at the metric-family level — it caps growth regardless of how many label dimensions a team introduces, converting an unbounded failure into a bounded, alertable rejection.

#Failure Modes & Edge Cases

#OOM Kill During WAL Replay

A Prometheus pod that OOM-kills mid-cardinality-explosion enters a dangerous restart loop: on boot, it must replay the WAL to reconstruct the head block. If cardinality explosion has already pushed the WAL to several gigabytes, replay time itself can exceed the pod’s liveness probe timeout, causing Kubernetes to kill it again before recovery completes. This creates a crash loop that masks the underlying root cause — engineers frequently misdiagnose this as a resource-limit problem rather than a cardinality problem.

#Cardinality Bombs from User-Controlled Labels

The most severe class of cardinality explosion originates from labels populated directly from user input — HTTP headers, raw error strings, or session tokens exposed via instrumentation middleware. These are effectively adversarial from a systems perspective: a single malformed client, bot, or misbehaving retry loop can generate millions of unique series in minutes. Any label sourced from external, non-enumerable input should be treated as untrusted and excluded from metric labels by default.

#Remote Write Backpressure Cascades

When a downstream Cortex or Mimir ingester rejects samples due to cardinality limits, misconfigured Prometheus remote-write queues will retry indefinitely, backing up the WAL and increasing local memory pressure — the mitigation designed to protect the shared tier can inadvertently destabilise the source Prometheus instance. Configure remote_write with bounded queue capacity and drop-on-failure semantics rather than infinite retry:

1remote_write:
2  - url: "https://mimir.internal/api/v1/push"
3    queue_config:
4      capacity: 10000
5      max_shards: 50
6      max_samples_per_send: 2000
7      min_backoff: 30ms
8      max_backoff: 5s

#Scaling & Security Trade-offs

Containing cardinality explosion is ultimately a governance problem layered on top of a storage-engine constraint. The following trade-offs should inform whether you invest in relabelling discipline, horizontal sharding, or both:

• Relabelling at scrape time is the cheapest mitigation (near-zero CPU overhead) but requires organisational buy-in — every team onboarding new instrumentation must follow label conventions, which does not scale past a handful of teams without automated linting.
• Per-tenant ingestion limits (Cortex/Mimir) scale well operationally but introduce a hard failure mode: writes are rejected rather than degraded, which can cause silent data loss for teams that don’t monitor discarded_samples_total.
• Recording rules for pre-aggregation reduce query-time cardinality exposure but add write amplification and a maintenance burden — every new dimension requires a corresponding rule update.
• Security exposure: unbounded labels sourced from user input (IP addresses, emails, tokens) create a data-leakage risk in addition to a cardinality risk — metrics endpoints frequently lack the access controls applied to logs, so PII inadvertently ends up in label values scraped by third-party observability vendors.
• Multi-tenant isolation: shared Prometheus/Mimir clusters without hard per-tenant caps allow a single team’s cardinality explosion to exhaust ingester memory cluster-wide — tenant-level circuit breakers are not optional at organisational scale.
• Cost implications: in managed observability platforms billed per active series, an unnoticed cardinality explosion is a direct and immediate cost incident, often before it manifests as a performance incident.