Incident Overview
The checkout service mesh runs Istio 1.21 with istiod managing xDS for roughly 400 Envoy sidecars across three regions. At 14:02 UTC a canary rollout pushed Envoy 1.29 to 10% of the checkout-service pods alongside a routine DestinationRule update tightening outlier detection thresholds.
By 14:15, dashboards showed p99 latency on checkout-service jumping from 180ms to 2.4s, with upstream_cx_connect_fail counters rising and grpc status DEADLINE_EXCEEDED accounting for 38% of requests. Circuit breaker overflow metrics (upstream_rq_pending_overflow) spiked across the fleet, not just the canary namespace.
At 14:22, SRE paused the rollout at 10%, but error rates continued climbing in pods still running the prior Envoy 1.28 binary, reaching 22% error rate mesh-wide by 14:30. istioctl proxy-status shows several non-canary sidecars reporting STALE for their listener configuration, with xDS ACK version numbers lagging the last pushed snapshot by two revisions. Meanwhile, istiod logs show repeated ADS push retries with 'context deadline exceeded' entries and CPU on the istiod pods sitting at 85%.
Business constraint: checkout is peak-hour critical, currently processing roughly 1,200 transactions/minute, and a full control-plane restart risks a multi-minute mesh-wide config blackout. Two competing hypotheses are live.
First, the canary Envoy 1.29 binary has a regression in connection draining logic that closes idle upstream connections prematurely, explaining the deadline errors but not why non-canary pods are also affected unless there's cross-talk via shared upstream clusters. Second, the DestinationRule/outlier-detection change pushed via the same deployment pipeline was malformed or partially applied, causing istiod to intermittently fail to converge xDS state across the entire mesh, which would explain the STALE listener status and errors appearing in pods that never received the new binary.
Rolling back the canary alone will not resolve the second hypothesis, and a blind istiod restart risks amplifying the outage given the already-elevated CPU and push retry backlog.
Investigation Options
Review the available operational moves and select the best immediate action.
Revert the most recent DestinationRule and VirtualService change to its previous git-tracked revision, reapply it, and use istioctl proxy-status to confirm all sidecars re-ACK the reconciled xDS snapshot, watching for STALE entries clearing across both canary and non-canary pods.
Roll back only the canary Envoy 1.29 image to the prior 1.28 tag, pause further rollout, and monitor error rates isolated to the canary namespace for fifteen minutes before drawing conclusions about the broader mesh-wide error spike.
Fleet-wide, increase Envoy circuit breaker max_pending_requests and per-connection timeout thresholds via a temporary EnvoyFilter to reduce cascading overflow errors while root-cause analysis continues, accepting this as a stopgap rather than a fix.
Restart all istiod control-plane pods and the backing etcd/config store cluster simultaneously, on the assumption that transient state corruption in the control plane caused the STALE xDS acknowledgements observed fleet-wide.