Coordinated Omission

A benchmarking bias where a load generator waits for each response before sending the next request, causing slow responses to suppress the number of samples taken during stalls, making reported tail latencies falsely optimistic.

Practical example

A service with a 500ms GC pause every 10 seconds, tested with a closed-loop generator at low concurrency, reports a clean p99 because the generator simply sends fewer requests during the pause and catches up afterward — masking a latency pathology that would be visible under real, independently-arriving production traffic.

Coordinated Omission occurs whenever a load generator uses a closed-loop model: it sends a request, blocks until the response arrives (or a fixed number of in-flight slots free up), and only then issues the next request. If the system under test stalls — a GC pause, a lock contention spike, a network blip — the generator doesn’t just record that one slow request; it also fails to send the requests that should have been dispatched during the stall window. Those missing requests are never counted, so the resulting latency histogram has fewer samples during the exact interval where latency was worst. The term was popularized by Gil Tene (Azul Systems) in the context of tools like early versions of ab, JMeter, and naive wrk scripts.

The mathematically correct approach is open-loop load generation: requests are scheduled at a fixed rate (e.g., Poisson or fixed-interval arrival) irrespective of whether prior requests have completed. Each scheduled request’s latency is measured from its intended send time, not its actual send time. If the intended request couldn’t be dispatched because the client was still blocked on a prior call, that gap must be backfilled into the histogram as a correspondingly inflated latency sample — not discarded. Tools such as wrk2 and HdrHistogram-based harnesses implement this correction explicitly via constant-throughput scheduling and coordinated-omission-aware recording.

The practical impact is severe and often invisible until production incidents contradict benchmark reports:

A system with a 500ms GC pause every 10s under closed-loop testing at low concurrency may report a clean p99 — the generator simply issues fewer requests during the pause and ‘catches up’ afterward, smoothing the histogram.
Capacity planning derived from such biased benchmarks under-provisions for real traffic, where request arrival is exogenous (driven by user behavior, upstream retries, cron fan-out) and does not pause to accommodate server-side stalls.
The bias worsens as target concurrency approaches the generator’s blocking threshold — ironically, the busier and more stressed the test, the more optimistic (and wrong) the reported tail becomes.

Detecting coordinated omission requires auditing the load generator’s request-issuance model, not just its reported percentiles. Engineers should verify whether the tool schedules by intended arrival time or by completion of the previous call, and whether latency correction is applied to ‘missing’ intervals. This is directly relevant when validating SLOs, circuit breaker thresholds, or admission control tuning — any of which will be miscalibrated if the underlying tail-latency data was collected under a closed-loop harness.