Incident Overview
KBY Technologies operates a five-node self-managed etcd v3.5 cluster underpinning a 340-node on-prem Kubernetes platform spread across three racks (R1-R3), each rack hosting network-attached storage via Ceph RBD for the control-plane nodes. During Saturday's 02:00-04:00 UTC maintenance window, the storage team migrated etcd data directories from local NVMe to Ceph RBD volumes to reclaim NVMe capacity for a database tier.
The migration completed at 03:40 UTC with all five members reporting healthy via etcdctl endpoint health. At 03:55 UTC, Prometheus alerts fired: etcd_server_leader_changes_seen_total incremented seven times within ten minutes, and etcd_disk_wal_fsync_duration_seconds p99 rose from a pre-migration baseline of 8ms to 240ms on members etcd-2 and etcd-4 (both R2).
Concurrently, kube-apiserver request duration p99 spiked to 12.4s for LIST operations against the core API group, and kube-controller-manager logs show repeated 'leaderelection lost' events every 60-90 seconds. Node status flapped Ready/NotReady for 22 worker nodes in R2 and R3, triggering pod evictions and a brief spike in PodDisruptionBudget violations.
Two competing hypotheses have emerged. First, the Ceph RBD backend introduces write-path latency that exceeds etcd's default 100ms election-timeout to 1s heartbeat ratio, causing false leader elections purely from disk contention.
Second, a firewall rule change bundled into the same maintenance window altered inter-rack routing, and the flapping instead reflects intermittent packet loss on the R2-R3 link affecting etcd_network_peer_round_trip_time_seconds, a metric that has not yet been pulled for comparison. Constraints: the original NVMe volumes were wiped and reprovisioned for the database migration, so an immediate storage rollback is not available without a multi-hour rebuild.
Production workloads remain live with SLA commitments, and changing etcd cluster membership or restarting members carries quorum risk given only five nodes. The network team confirms a firewall change went live at 03:30 UTC but denies any routing impact.
Investigation Options
Review the available operational moves and select the best immediate action.
Pull and compare etcd_disk_wal_fsync_duration_seconds against etcd_network_peer_round_trip_time_seconds for all five members side by side, and cross-reference against the 03:30 UTC firewall change window, before altering any etcd configuration or infrastructure.
Temporarily raise --heartbeat-interval and --election-timeout on all five etcd members to absorb the elevated fsync latency, restoring election stability while the root cause investigation continues, accepting slower failover detection as an interim trade-off.
Immediately request a full rollback of the Ceph RBD migration to reinstate the original NVMe-backed etcd volumes, even though those volumes were wiped and reprovisioned, to eliminate storage latency as a variable.
Restart the current etcd leader to force a clean election and clear the leaderelection lost errors in kube-controller-manager, without first confirming whether disk latency or network loss is driving the instability.