etcd Leader Elections Spike After Kubernetes Node Pool AZ Move

You run a self-managed three-node etcd cluster backing a 40-node production Kubernetes cluster spanning two AWS Availability Zones. Two nights ago, platform-eng completed a routine node pool migration to replace ageing m5.2xlarge control-plane nodes with m6i.2xlarge instances; one etcd member was rescheduled into a third AZ that was previously unused for control-plane workloads.

At 02:14 UTC today, PagerDuty fires on kube-apiserver p99 latency exceeding 4.2s and kubelet NotReady flapping across 11 nodes. etcd_server_leader_changes_seen_total has jumped from roughly 1 per day to 14 in the last hour. Logs show repeated "leader changed" and "failed to send out heartbeat on time" entries, alongside "slow fsync" warnings with wal_fsync_duration_seconds p99 at 340ms (baseline 8ms) on the newly migrated member.

Network telemetry shows inter-AZ RTT between the new member and the other two averaging 3.8ms, up from 0.6ms when all three were co-located. The etcd db size has also grown to 3.1GB, near the 2GB alarm threshold, with no recent defragmentation recorded.

CPU and memory on all three members remain under 40% utilisation, and no OOM events are logged. Two competing hypotheses are circulating: the on-call SRE believes the new inter-AZ network latency is pushing round-trip times past the default 1000ms election timeout margin under load, causing false leader-loss detections; the platform lead suspects the db size growth and lack of defragmentation is causing extended fsync stalls that independently trigger heartbeat timeouts, and that AZ placement is a red herring.

Compaction has not run in 9 days per etcdctl compaction history. Change freeze policy prohibits full cluster rebuilds without a change-advisory board ticket, but emergency mitigations are permitted.

Quorum must be preserved at all times; losing a second member during remediation would take the API server fully read-only.