Incident Overview
A regional GKE cluster (5-node etcd quorum, kube-apiserver fronting ~15,000 RPS of production traffic) underwent a cost-optimisation change overnight. Terraform migrated etcd data volumes from local-SSD to network-attached PD-balanced disks, rolled out as a canary to 2 of 5 etcd members starting 02:14 UTC.
At 02:19 UTC, etcd_disk_wal_fsync_duration_seconds p99 on the two migrated members jumped from a steady 8ms to 220ms, and has stayed elevated. Leader election counts, previously near zero for weeks, rose to 14 in the last hour. kube-apiserver p99 latency for LIST/WATCH requests climbed from 180ms to 3.2s, and several kubelets began flapping between Ready and NotReady as heartbeat updates were delayed.
VPC flow logs show inter-peer etcd network latency stable at ~0.4ms with no packet loss, and the unmigrated three etcd members show normal fsync times throughout. On-call has two competing hypotheses circulating: first, that the new PD-balanced volumes lack a dedicated IOPS/throughput reservation and are experiencing contention from co-located workloads on the same storage backend, directly causing the fsync latency and downstream leader churn; second, that a recent NIC firmware update pushed to the underlying node pool last week introduced intermittent micro-drops not visible in aggregate flow logs, and the disk migration is coincidental.
The migration Terraform run is currently paused mid-canary, with three members untouched and two already cut over; rolling forward or backward both require a maintenance window per change policy, though emergency rollback is technically permitted given active degradation. There is no approved cross-region failover runbook exercised in the last two quarters, and API server error budgets are already 40% consumed this month.
Customer-facing symptoms include intermittent 5xx responses on ingress controllers reliant on API server admission webhooks, though core workload pods remain running.
Investigation Options
Review the available operational moves and select the best immediate action.
Halt the disk migration rollout immediately, revert the two migrated etcd members back to local SSD-backed volumes, and monitor wal fsync duration and leader election counts to confirm restoration of etcd stability before resuming any further node changes.
Temporarily raise etcd --heartbeat-interval and --election-timeout on all five members to reduce election sensitivity to transient disk latency spikes, buying time to investigate root cause without triggering further quorum churn, while accepting increased failover detection latency.
Initiate a full control-plane failover to the standby region's etcd cluster to immediately restore API responsiveness, redirecting all traffic while the primary region's storage backend is investigated offline.
Restart all kube-apiserver pods to clear the backlog of pending LIST/WATCH requests and re-establish fresh connections to etcd, hoping this reduces observed request latency and stabilises the API server queue depth.