etcd Leader Election Storm After GKE Disk Type Migration

A regional GKE cluster (5-node etcd quorum, kube-apiserver fronting ~15,000 RPS of production traffic) underwent a cost-optimisation change overnight. Terraform migrated etcd data volumes from local-SSD to network-attached PD-balanced disks, rolled out as a canary to 2 of 5 etcd members starting 02:14 UTC.

At 02:19 UTC, etcd_disk_wal_fsync_duration_seconds p99 on the two migrated members jumped from a steady 8ms to 220ms, and has stayed elevated. Leader election counts, previously near zero for weeks, rose to 14 in the last hour. kube-apiserver p99 latency for LIST/WATCH requests climbed from 180ms to 3.2s, and several kubelets began flapping between Ready and NotReady as heartbeat updates were delayed.

VPC flow logs show inter-peer etcd network latency stable at ~0.4ms with no packet loss, and the unmigrated three etcd members show normal fsync times throughout. On-call has two competing hypotheses circulating: first, that the new PD-balanced volumes lack a dedicated IOPS/throughput reservation and are experiencing contention from co-located workloads on the same storage backend, directly causing the fsync latency and downstream leader churn; second, that a recent NIC firmware update pushed to the underlying node pool last week introduced intermittent micro-drops not visible in aggregate flow logs, and the disk migration is coincidental.

The migration Terraform run is currently paused mid-canary, with three members untouched and two already cut over; rolling forward or backward both require a maintenance window per change policy, though emergency rollback is technically permitted given active degradation. There is no approved cross-region failover runbook exercised in the last two quarters, and API server error budgets are already 40% consumed this month.

Customer-facing symptoms include intermittent 5xx responses on ingress controllers reliant on API server admission webhooks, though core workload pods remain running.