Prometheus with Kubernetes
Discover and scrape Kubernetes workloads with Prometheus while controlling target selection, cardinality, permissions and high-availability behaviour.
Integration boundary
Kubernetes service discovery exposes workload and node metadata that Prometheus can relabel into scrape targets; operators commonly package this using Prometheus Operator resources.
Supported approaches
Prometheus Operator
Declare Prometheus, ServiceMonitor, PodMonitor and rule resources through Kubernetes custom resources.
Native service discovery
Configure kubernetes_sd_configs and relabel rules directly in Prometheus configuration.
Prerequisites
- A Kubernetes cluster with metrics endpoints
- RBAC permission to discover the selected resources
- A storage and retention decision
- A label-cardinality budget
Implementation
Choose discovery boundaries
Limit namespaces, selectors and roles to the targets the Prometheus instance owns.
Declare scrape intent
Use ServiceMonitor or explicit scrape configuration with a stable port name and metrics path.
Control labels
Drop unbounded or unnecessary labels before ingestion and set sample limits where supported.
Verification
Inspect active targets
Confirm discovery, labels and scrape errors in the Prometheus targets view or API.
Query health
Check that expected jobs are up and distinguish missing targets from failing scrapes.
up{job="<job>"}Review ingestion growth
Track active series, samples and storage growth before expanding discovery.
Common failure modes
- ServiceMonitor selector does not match
- RBAC prevents discovery
- High-cardinality labels exhaust memory or storage
- Scraping every Pod without ownership boundaries
Related engineering templates
Follow Prometheus and Kubernetes failures and fixes
A useful digest about twice a week with new error references, tools, integration notes and production lessons. No daily noise.