Incident Overview
You run a self-managed three-node etcd cluster backing a 40-node production Kubernetes cluster spanning two AWS Availability Zones. Two nights ago, platform-eng completed a routine node pool migration to replace ageing m5.2xlarge control-plane nodes with m6i.2xlarge instances; one etcd member was rescheduled into a third AZ that was previously unused for control-plane workloads.
At 02:14 UTC today, PagerDuty fires on kube-apiserver p99 latency exceeding 4.2s and kubelet NotReady flapping across 11 nodes. etcd_server_leader_changes_seen_total has jumped from roughly 1 per day to 14 in the last hour. Logs show repeated "leader changed" and "failed to send out heartbeat on time" entries, alongside "slow fsync" warnings with wal_fsync_duration_seconds p99 at 340ms (baseline 8ms) on the newly migrated member.
Network telemetry shows inter-AZ RTT between the new member and the other two averaging 3.8ms, up from 0.6ms when all three were co-located. The etcd db size has also grown to 3.1GB, near the 2GB alarm threshold, with no recent defragmentation recorded.
CPU and memory on all three members remain under 40% utilisation, and no OOM events are logged. Two competing hypotheses are circulating: the on-call SRE believes the new inter-AZ network latency is pushing round-trip times past the default 1000ms election timeout margin under load, causing false leader-loss detections; the platform lead suspects the db size growth and lack of defragmentation is causing extended fsync stalls that independently trigger heartbeat timeouts, and that AZ placement is a red herring.
Compaction has not run in 9 days per etcdctl compaction history. Change freeze policy prohibits full cluster rebuilds without a change-advisory board ticket, but emergency mitigations are permitted.
Quorum must be preserved at all times; losing a second member during remediation would take the API server fully read-only.
Investigation Options
Review the available operational moves and select the best immediate action.
Pull etcd member-level metrics (wal_fsync_duration_seconds, network_peer_round_trip_time_seconds) per node and correlate against the AZ migration timestamp to confirm whether disk or network is the dominant contributor before changing cluster topology or timeouts.
Immediately restart the etcd member currently holding leadership to force a fresh election and see if elections stabilise, without first reviewing per-node fsync or RTT metrics.
Raise --heartbeat-interval and --election-timeout on all three members and roll the change out via a rolling restart, without first isolating whether the root cause is network latency, disk latency, or db size.
Run etcdctl defrag against the member showing high db size and fsync latency during the current instability window, without first confirming whether leadership is stable enough to tolerate the temporary I/O pause defrag introduces.