Adaptive Concurrency Limiting (Gradient-Based)

Answer first: Gradient-based adaptive concurrency limiting continuously compares sampled request latency with a measured minimum-latency baseline, then raises or lowers the in-flight limit to keep queues from growing. It needs minimum sample counts, a reset strategy for the baseline, explicit overflow behavior, and observability for limit collapse.

Primary reference: Envoy adaptive concurrency API. Related KBY concepts: queueing and the utilisation knee and circuit breakers.

Adaptive concurrency limiting borrows directly from TCP congestion control theory, specifically the delay-based approach used by TCP Vegas rather than the loss-based approach of TCP Reno/CUBIC. Instead of waiting for explicit failure signals (timeouts, 503s, dropped packets), the algorithm treats rising round-trip latency as an early proxy for queuing inside the service. It maintains a rolling estimate of the minimum observed request latency (assumed to represent an uncongested baseline) and compares it against the current sampled latency. The ratio between these two values produces a gradient: a value near 1.0 indicates the system is uncongested and the limit can grow; a value pulling toward 0 indicates queuing is building and the limit must shrink. The new limit is typically computed as new_limit = old_limit * gradient + queue_size_headroom, where the headroom term (often derived from the square root of the current limit, per Netflix’s implementation) prevents the algorithm from converging to a degenerate limit of 1.

Implementations such as Netflix’s concurrency-limits library and Envoy’s Adaptive Concurrency Filter apply this per-endpoint or per-upstream-cluster, sampling latency over a sliding window (e.g., every N requests or every fixed interval) and applying exponential smoothing to avoid limit thrashing from single-sample noise. The computed limit controls how much work is admitted concurrently. Overflow behavior is implementation-specific: Envoy’s adaptive concurrency HTTP filter rejects requests above the limit, while another limiter may queue or shed work according to its own policy. Verify the configured overflow behavior rather than treating fast-fail as universal.

The mechanism is highly sensitive to what counts as a valid latency sample. Garbage collection pauses, lock contention, downstream dependency slowness, and even measurement clock jitter can all masquerade as congestion, causing the limit to collapse unnecessarily and reject otherwise-servable traffic. Systems mitigate this with minimum sample count thresholds before adjusting the limit, exclusion of samples from cold-started instances, and hysteresis windows that dampen oscillation. A related failure mode is limit collapse under bursty traffic: a sudden spike causes a latency blip, the limit shrinks aggressively, and the resulting rejections trigger client-side retries that themselves generate a secondary load spike — an adaptive analog of retry storms seen with naive circuit breakers.

• Baseline drift: the minimum-latency baseline must periodically decay/reset, otherwise a permanent infrastructure change (e.g., migrating to slower hardware) never gets reflected and the limit stays pinned artificially low.
• Composability: adaptive limits are typically layered underneath a circuit breaker and bulkhead isolation, not as a replacement — the limiter controls admitted concurrency per resource, while the breaker handles hard dependency failure.
• Cold start: newly started instances have no latency history, so implementations seed an initial conservative limit and let it ramp, similar to TCP slow start.

Architecturally, adaptive concurrency limiting shifts capacity management from a static, manually-tuned SRE artifact (load test once, hardcode a max-connections value, forget about it) into a continuously reactive control loop embedded in the data path. This matters most in environments with heterogeneous or elastic capacity — autoscaled fleets, multi-tenant shared backends, or services behind a mesh where instance sizing varies across zones — where a single static threshold is either perpetually too conservative or occasionally catastrophically wrong. The tradeoff is added control-loop complexity and a new class of tuning parameters (smoothing factors, window sizes, minimum sample counts) that themselves require observability and can fail silently if misconfigured, making it a mechanism best adopted only after static limits have demonstrably proven inadequate.