Circuit Breaker (Distributed Systems)

A circuit breaker is implemented as a finite state machine with three canonical states: Closed, Open, and Half-Open. In the Closed state, calls pass through normally while the breaker maintains a rolling window (count-based or time-bucketed) of success/failure/timeout outcomes. Once the error rate or consecutive failure count crosses a threshold, the breaker trips to Open, and all subsequent calls fail immediately without touching the network — this is the mechanism’s entire value proposition: it trades a guaranteed failure for an avoided timeout, freeing up threads, sockets, and queue slots that would otherwise be pinned waiting on a dependency that isn’t going to answer anyway.

• Open → Half-Open transition: after a reset timeout elapses, the breaker allows a small number of probe requests through. Success closes the circuit; failure re-opens it, often with an exponentially increasing reset interval to avoid thrashing against a dependency that is flapping.
• Sliding window semantics: naive implementations count raw failures, which misbehave under low traffic (one failure out of two calls trips the breaker) or high traffic (one failure out of ten thousand should not). Production-grade breakers (e.g., Hystrix, resilience4j, Envoy’s outlier detection) use minimum-request-volume gates and percentage-based thresholds over a rolling window to avoid both false positives at low QPS and false negatives at high QPS.
• Granularity of scope: breakers must be keyed per downstream dependency, and ideally per endpoint or shard, not globally per service — a single global breaker conflates an unrelated endpoint’s health with a hot one’s, causing unnecessary blast radius.

The architectural subtlety is that a circuit breaker is a local, client-side decision made with only partial information — it cannot distinguish between the downstream actually being down versus a network partition affecting only this caller (a classic gray failure ambiguity). This means breaker state is not authoritative cluster-wide truth; in a fleet of N callers, each maintains independent state, so a dependency can appear “up” to some callers and “down” to others simultaneously. This is by design — centralizing breaker state would reintroduce the coordination bottleneck the pattern exists to avoid — but it means breakers must be paired with proper timeout budgets and bulkheading (separate thread/connection pools per dependency) to actually deliver isolation; a breaker without pool isolation still lets one slow dependency exhaust a shared executor before it even trips.

Failure modes to design around: fallback correctness (an Open-state fallback returning stale cache or a default value must be semantically safe for the caller — silently returning zero for a balance check is worse than failing loud); thundering herd on Half-Open (if reset timers are synchronized across a large caller fleet, e.g. due to identical deploy timestamps, all instances probe simultaneously and can re-trip a barely-recovering dependency — jittering the reset timeout mitigates this); and metric pollution, where fast-failing Open-state calls get logged as errors identically to real backend errors, corrupting SLO dashboards unless explicitly tagged with a distinct “short-circuited” status. In mesh architectures, this logic is frequently pushed out of application code into the sidecar (Envoy’s outlier_detection, consecutive 5xx ejection) so that breaker behavior is uniform across polyglot services and tunable via control-plane config rather than redeploys.