Skip to main content
Lexicon
Lexicon

Gray Failure

Difficulty: Advanced
2 min read

In plain English

Plain definition

Gray failure is when a server or node is technically 'up' and passes its health check, but is actually struggling to serve real traffic properly — for example, it's slow, dropping some requests, or corrupting data — while monitoring dashboards still show it as healthy.

Gray failure sits in the observability gap between fail-stop (crash, process exit, connection refused) and fully healthy. The canonical example is a node whose liveness probe returns 200 OK in 2ms because the health-check handler is served from a separate, unburdened thread pool, while the actual data path is saturated, GC-thrashing, or stuck on a slow disk I/O queue. The control plane sees green; the data plane sees red. This divergence is often called differential observability: the perspective of the monitoring system (the observer) and the perspective of the actual traffic consumer (the client) disagree on the health of the same node.

Root causes are rarely singular. Common triggers include:

  • Resource exhaustion asymmetry: CPU throttling (cgroup limits, noisy neighbor on shared hosts) that only affects request-serving threads, not the lightweight heartbeat goroutine.
  • Firmware/driver-level degradation: a NIC or NVMe device operating at reduced link speed or with rising retransmit/error counters, well below the threshold that triggers hardware alarms.
  • Non-atomic dependency failure: a downstream cache or DNS resolver that succeeds intermittently, causing p99 latency to spike without violating a simple up/down SLA.
  • Memory fragmentation or GC pause creep: the process is alive and responsive to trivial pings, but application-level request handling stalls for hundreds of milliseconds.

Detecting gray failure requires shifting from liveness-based to workload-representative health signals. Practical mitigations include synthetic canary requests that exercise the actual serving path (not a stub handler), outlier ejection in load balancers (e.g., Envoy’s outlier detection, which ejects hosts after consecutive errors or when their success rate deviates from the rest of the cluster), and cross-validation schemes where peer nodes vote on a suspect’s health rather than relying solely on self-reported status — because a gray-failed node’s own health check is precisely the signal you cannot trust. The Microsoft Research paper “Gray Failure: The Achilles’ Heel of Cloud-Scale Systems” (HotOS 2017) formalized gray failure and the notion of differential observability.

Architecturally, gray failure forces a shift away from binary circuit breakers toward graduated degradation models: weighted load shedding, request hedging (issuing a duplicate request to a second replica if the first exceeds a latency budget), and proactive quarantine of suspect nodes at reduced traffic weight rather than an all-or-nothing eviction. Ignoring this failure class is a common root cause of cascading outages, since a single gray node can silently consume a disproportionate share of retries and connection pool slots from every upstream caller, degrading the entire fleet’s tail latency while every dashboard reports 100% node availability.

About this page

Credited to

Dates

Published

Corrections

  • : Corrected during a review of published articles. Details

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

What Is Gray Failure? Definition and Example | KBY Lexicon