One incident. Four plausible moves. Instant technical reasoning.
Incident brief
A multi-tenant Kubernetes platform (400 nodes, 60+ tenant teams) runs a five-member etcd cluster across three AZs backing kube-apiserver. At 13:40 UTC, infra completed a cost-optimisation migration moving etcd WAL and db volumes from local NVMe instance store to network-attached gp3 EBS volumes, enabling stateless replacement of etcd nodes on spot capacity.
At 13:55 UTC, an unrelated team deployed a new custom controller that performs full cluster-wide LIST/WATCH calls against several CRDs with resourceVersion=0. At 14:02 UTC, etcd_disk_wal_fsync_duration_seconds p99 jumps from ~9ms to 220ms on three of five members.
By 14:06 UTC, etcd_server_leader_changes_seen_total increments four times in five minutes, versus a near-zero baseline. At 14:08 UTC, kube-apiserver LIST/WATCH request_duration_seconds p99 for core/v1 pods spikes to 6.4s and the 504 rate reaches 3.8%.
By 14:11 UTC, multiple worker nodes flip to NotReady as kubelet lease renewals fail, triggering cascading pod evictions across namespaces. On-call is paged at 14:15 UTC.
Investigating dashboards shows EBS VolumeQueueLength elevated on the affected etcd members, with BurstBalance depleting toward 0% on two volumes — consistent with insufficient provisioned IOPS on the new gp3 backend under load. However, etcd_mvcc_db_total_size_in_bytes and etcd request counters show a concurrent sharp rise correlating with the 13:55 UTC controller deployment, suggesting the watch load itself may be driving CPU contention that indirectly slows fsyncs regardless of storage backend.
A third possibility is a same-day CNI security-group change affecting inter-AZ latency between etcd peers. Operational constraints: reverting the storage migration requires CAB approval and sequential re-provisioning of etcd members, which itself risks breaking quorum; only one member can be safely restarted at a time; the platform is under active SLA obligations for dozens of tenant workloads currently experiencing evictions.
Given telemetry that plausibly supports storage degradation, watch-load overload, or network latency as the trigger, what should the on-call Staff Engineer do next to stabilise the control plane without taking an unsafe quorum-breaking action?
Make your initial decision, then use the deep dive to investigate the underlying failure mode.