Skip to main content
KBY Technologies Logo
Kubernetes & Containers

etcd Leader Flapping After gp3 Migration Stalls Kube API

Container orchestration incidents involving scheduling, networking, admission, resource pressure, rollouts, and cluster control planes.

Daily engineering quiz

One incident. Four plausible moves. Instant technical reasoning.

About 2 minutes
01Read the signals
02Choose a move
03Reveal reasoning

Incident brief

A multi-tenant Kubernetes platform (400 nodes, 60+ tenant teams) runs a five-member etcd cluster across three AZs backing kube-apiserver. At 13:40 UTC, infra completed a cost-optimisation migration moving etcd WAL and db volumes from local NVMe instance store to network-attached gp3 EBS volumes, enabling stateless replacement of etcd nodes on spot capacity.

At 13:55 UTC, an unrelated team deployed a new custom controller that performs full cluster-wide LIST/WATCH calls against several CRDs with resourceVersion=0. At 14:02 UTC, etcd_disk_wal_fsync_duration_seconds p99 jumps from ~9ms to 220ms on three of five members.

By 14:06 UTC, etcd_server_leader_changes_seen_total increments four times in five minutes, versus a near-zero baseline. At 14:08 UTC, kube-apiserver LIST/WATCH request_duration_seconds p99 for core/v1 pods spikes to 6.4s and the 504 rate reaches 3.8%.

By 14:11 UTC, multiple worker nodes flip to NotReady as kubelet lease renewals fail, triggering cascading pod evictions across namespaces. On-call is paged at 14:15 UTC.

Investigating dashboards shows EBS VolumeQueueLength elevated on the affected etcd members, with BurstBalance depleting toward 0% on two volumes — consistent with insufficient provisioned IOPS on the new gp3 backend under load. However, etcd_mvcc_db_total_size_in_bytes and etcd request counters show a concurrent sharp rise correlating with the 13:55 UTC controller deployment, suggesting the watch load itself may be driving CPU contention that indirectly slows fsyncs regardless of storage backend.

A third possibility is a same-day CNI security-group change affecting inter-AZ latency between etcd peers. Operational constraints: reverting the storage migration requires CAB approval and sequential re-provisioning of etcd members, which itself risks breaking quorum; only one member can be safely restarted at a time; the platform is under active SLA obligations for dozens of tenant workloads currently experiencing evictions.

Decision point

Given telemetry that plausibly supports storage degradation, watch-load overload, or network latency as the trigger, what should the on-call Staff Engineer do next to stabilise the control plane without taking an unsafe quorum-breaking action?

Choose your first move

Choose your first move

Question 1 of 1

Pick the intervention you would prioritise. Every option is plausible; the challenge is deciding what should happen first with the evidence available.

Make your initial decision, then use the deep dive to investigate the underlying failure mode.