Skip to main content
daily-triage/kubernetes-hpa-scales-pods-up-while-cpu-graphs-show-idle-capacity.md
Daily Triage Briefing

Kubernetes HPA Scales Pods Up While CPU Graphs Show Idle Capacity

Impact Summary

A fictional Kubernetes triage exercise in which a Horizontal Pod Autoscaler keeps scaling a checkout service up even though CPU dashboards show idle capacity, revealing a stale custom-metric scrape mismatch.

SeveritySEV-3
StatusResolved simulation
PlatformKubernetes & Containers
Incident TypeKubernetes & Containers
Published13 Aug 2026
Technologies Involved
Kubernetes

Simulation Label

Fictional simulation. This exercise is a bounded, invented training scenario for practising operational reasoning. No real organisation, incident, telemetry, credentials or production identifiers are involved or implied.

Operational Summary

Root Cause

The Kubernetes HPA scaled a deployment based on a custom external metric that the metrics adapter had cached during a brief real traffic spike; because the adapter's Prometheus scrape interval (5 minutes) was far longer than the HPA's sync period (15 seconds), the HPA repeatedly rescaled against the same stale value after real load had already returned to normal.

Detection Method

Evidence-led guided investigation

Current Status

Resolved simulation

Affected Services

Kubernetes

Incident Overview

A platform team runs a fictional, mid-sized Kubernetes cluster (version 1.28) hosting a checkout service behind a Horizontal Pod Autoscaler (HPA). At 14:02 the HPA scales the deployment from 6 to 18 replicas in four minutes.

The on-call engineer opens the CPU dashboard and finds node and pod CPU utilisation comfortably under 30%, well below the 70% target the HPA is meant to react to. The HPA status shows it is reacting to a custom external metric, not CPU, so the scale-up looks unjustified on the CPU graphs alone.

Investigation Options

Review the available operational moves and select the best immediate action.

A

Delete the HPA object and let Kubernetes recreate default scaling behaviour.

B

Manually scale the deployment down to match observed CPU usage and pause the HPA while investigating the metrics pipeline.

C

Restart the metrics-server pod immediately to force fresh metric collection.

D

Compare the custom metrics adapter's scrape interval against the HPA sync period, correct the mismatch, and re-verify metric freshness before trusting scale decisions again.

Tags:Kubernetes