Admission Control (Load Shedding)
In plain English
Plain definition
Admission control is a technique where a service continuously measures how saturated it is (e.g. in-flight requests, queue depth, or latency) and dynamically decides how many requests it can safely accept. Instead of enforcing a fixed quota, it shrinks or grows its capacity limit in real time, rejecting excess requests as cheaply as possible before they consume execution resources, so the system degrades gracefully instead of collapsing entirely under load.
Unlike a fixed-rate limiter, which enforces an arbitrary quota (e.g., 1000 req/s) regardless of actual server health, admission control is adaptive. It continuously measures a proxy for saturation — typically in-flight request count, queue depth, or observed p99 latency — and dynamically computes a concurrency ceiling. The canonical implementation is a gradient-based algorithm derived from TCP Vegas: the system tracks a rolling minimum RTT as the ‘no queueing’ baseline and compares it to the current RTT. If current latency exceeds baseline by a computed gradient, the limit is ratcheted down; if latency is stable at or below baseline, the limit is allowed to probe upward (AIMD-style). Netflix’s concurrency-limits library and gRPC’s server-side load-based backpressure hooks are the most cited production implementations of this pattern.
The critical architectural distinction is where rejection happens relative to cost. A well-placed admission controller rejects at the cheapest possible point in the call stack — ideally before deserialization, authentication, or any downstream fanout — because the entire point is to shed load before it does work. This differs from a circuit breaker, which trips based on downstream failure rate and protects a caller from a known-bad dependency; admission control protects the callee itself from being overwhelmed regardless of downstream health. It also differs from generic queueing/backpressure, which slows producers; admission control drops requests outright, converting an availability problem into a controlled, partial-failure problem.
- Priority-aware shedding: Naive FIFO rejection under load sheds requests indiscriminately, which is catastrophic if health checks or control-plane traffic get dropped alongside low-priority batch reads. Production systems tag requests with a criticality class (e.g.,
CRITICAL,DEGRADED_EXPERIENCE,BEST_EFFORT) and shed lowest-priority traffic first, sometimes implemented as nested token buckets per class. - LIFO over FIFO under saturation: Once queue depth exceeds a threshold, serving the most recently arrived request first (LIFO) yields better tail latency than FIFO, because older queued requests have already likely exceeded the caller’s timeout and are effectively “dead work” — completing them wastes resources that could serve a fresh, still-useful request.
- Metastable failure prevention: The core justification (per the OSDI ‘Metastable Failures’ literature) is that overloaded services often don’t recover on their own even after the traffic spike subsides, because retry storms and thread-pool exhaustion create a self-sustaining high-latency equilibrium. Admission control breaks this loop by keeping per-request latency low enough that clients don’t retry in the first place.
Edge cases dominate real deployments: GC pauses or noisy-neighbor CPU steal on a host can be misread as legitimate overload, causing the gradient algorithm to clamp concurrency on a healthy service — mitigated with hysteresis windows and minimum-sample-size gating before adjusting limits. Multi-tenant systems must also guard against a single noisy tenant collapsing the shared limit; this requires per-tenant sub-limits nested inside the global admission gate. Finally, admission control interacts poorly with client-side retries unless retry budgets are coordinated — a client that blindly retries a 503 from an admission controller simply re-injects the rejected load, defeating the mechanism entirely.