Incident Overview
A multi-tenant Kubernetes platform (400 nodes, 60+ tenant teams) runs a five-member etcd cluster across three AZs backing kube-apiserver. At 13:40 UTC, infra completed a cost-optimisation migration moving etcd WAL and db volumes from local NVMe instance store to network-attached gp3 EBS volumes, enabling stateless replacement of etcd nodes on spot capacity.
At 13:55 UTC, an unrelated team deployed a new custom controller that performs full cluster-wide LIST/WATCH calls against several CRDs with resourceVersion=0. At 14:02 UTC, etcd_disk_wal_fsync_duration_seconds p99 jumps from ~9ms to 220ms on three of five members.
By 14:06 UTC, etcd_server_leader_changes_seen_total increments four times in five minutes, versus a near-zero baseline. At 14:08 UTC, kube-apiserver LIST/WATCH request_duration_seconds p99 for core/v1 pods spikes to 6.4s and the 504 rate reaches 3.8%.
By 14:11 UTC, multiple worker nodes flip to NotReady as kubelet lease renewals fail, triggering cascading pod evictions across namespaces. On-call is paged at 14:15 UTC.
Investigating dashboards shows EBS VolumeQueueLength elevated on the affected etcd members, with BurstBalance depleting toward 0% on two volumes — consistent with insufficient provisioned IOPS on the new gp3 backend under load. However, etcd_mvcc_db_total_size_in_bytes and etcd request counters show a concurrent sharp rise correlating with the 13:55 UTC controller deployment, suggesting the watch load itself may be driving CPU contention that indirectly slows fsyncs regardless of storage backend.
A third possibility is a same-day CNI security-group change affecting inter-AZ latency between etcd peers. Operational constraints: reverting the storage migration requires CAB approval and sequential re-provisioning of etcd members, which itself risks breaking quorum; only one member can be safely restarted at a time; the platform is under active SLA obligations for dozens of tenant workloads currently experiencing evictions.
Investigation Options
Review the available operational moves and select the best immediate action.
Correlate EBS CloudWatch metrics (VolumeQueueLength, BurstBalance, provisioned versus consumed IOPS) for each etcd member's volume against the fsync latency and leader-change timestamps, while simultaneously reviewing etcd_mvcc_db_total_size_in_bytes and request-duration counters to isolate whether storage or watch load is driving the degradation before touching quorum.
Throttle or scale down the newly deployed controller performing full-cluster LIST/WATCH calls to reduce etcd read pressure, then observe whether fsync latency and leader-change frequency recover, without yet addressing the storage backend change.
Immediately initiate the CAB-gated rollback of etcd storage from gp3 back to local NVMe by re-provisioning all five members in sequence, accepting temporary quorum risk to restore the prior performance baseline as quickly as possible.
Force a manual leader election by restarting the current etcd leader pod to break the flapping cycle immediately, aiming to restore cluster stability before conducting deeper investigation.