Cardinality Explosion

Cardinality explosion is the unbounded growth of unique time-series or label combinations in a metrics system, typically triggered by high-variance dimensions like user IDs, pod names, or trace IDs embedded in labels. It matters because time-series databases index on the full label set, so uncontrolled cardinality growth degrades query performance, inflates storage cost, and can crash ingestion pipelines.

Practical example

A payments service adds a `merchant_id` label to its request-duration histogram; within a day, active series jump from 50k to 4M as new merchants sign up, causing the Prometheus head block to OOM and Grafana dashboards to time out until the label is dropped via metric_relabel_configs and merchant-level detail is moved to structured logs.

Time-series databases such as Prometheus, Thanos, Cortex, and M3DB store data as a mapping from a unique combination of metric name plus label key-value pairs (the series) to a stream of timestamped samples. Internally this mapping is maintained via an inverted index, where every distinct label value creates new index entries and often a new chunk/block allocation. Cardinality is the count of distinct series; explosion occurs when a label dimension has effectively unbounded or high-variance values (request IDs, raw URLs, pod ephemeral names, customer IDs) causing the series count to grow multiplicatively across label combinations rather than linearly.

The failure mode is architectural, not just operational. Prometheus’s head block and TSDB WAL hold all active series in memory; a spike in cardinality inflates heap usage and can trigger OOM kills before any alert fires. In remote-write architectures (Cortex, Mimir, Thanos Receive), the same explosion propagates downstream, exhausting per-tenant series limits and causing ingesters to reject writes. Query-side impact is equally severe: PromQL aggregations like sum by (pod) over an exploded label force the query engine to scan and merge orders of magnitude more series than intended, causing query timeouts that look like unrelated performance regressions.

Common triggers: unbounded label values (user_id, trace_id, full URL path), Kubernetes pod/container churn without normalization, dynamically generated label keys from application code, and high-cardinality joins between metrics and exemplars.
Mitigation patterns: label whitelisting/relabeling at scrape time (metric_relabel_configs), recording rules to pre-aggregate high-cardinality dimensions into lower-cardinality rollups, per-tenant series limits with hard rejection, and routing high-cardinality data to trace/log backends (which are built for high-cardinality indexing) instead of metrics systems.
Detection: cardinality is itself observable via meta-metrics like prometheus_tsdb_head_series or Cortex’s cortex_ingester_active_series; SLOs should be defined on series growth rate, not just absolute count, since sudden slope changes are the actionable signal.

The deeper architectural lesson is that metrics, logs, and traces have fundamentally different cardinality tolerances by design — metrics systems trade high cardinality for fast aggregation, while logging and tracing systems trade slower point-lookup for near-unbounded dimensionality. Choosing the wrong telemetry backend for a high-cardinality dimension is a design error that no amount of relabeling fully fixes; it requires re-architecting what gets emitted as a metric label versus what gets attached as a log field or span attribute.