Admission Control (Load Shedding)

Unlike a fixed-rate limiter, which enforces an arbitrary quota (e.g., 1000 req/s) regardless of actual server health, admission control is adaptive. It continuously measures a proxy for saturation — typically in-flight request count, queue depth, or observed p99 latency — and dynamically computes a concurrency ceiling. The canonical implementation is a gradient-based algorithm derived from TCP Vegas: the system tracks a rolling minimum RTT as the ‘no queueing’ baseline and compares it to the current RTT. If current latency exceeds baseline by a computed gradient, the limit is ratcheted down; if latency is stable at or below baseline, the limit is allowed to probe upward (AIMD-style). Netflix’s concurrency-limits library and gRPC’s server-side load-based backpressure hooks are the most cited production implementations of this pattern.

The critical architectural distinction is where rejection happens relative to cost. A well-placed admission controller rejects at the cheapest possible point in the call stack — ideally before deserialization, authentication, or any downstream fanout — because the entire point is to shed load before it does work. This differs from a circuit breaker, which trips based on downstream failure rate and protects a caller from a known-bad dependency; admission control protects the callee itself from being overwhelmed regardless of downstream health. It also differs from generic queueing/backpressure, which slows producers; admission control drops requests outright, converting an availability problem into a controlled, partial-failure problem.

• Priority-aware shedding: Naive FIFO rejection under load sheds requests indiscriminately, which is catastrophic if health checks or control-plane traffic get dropped alongside low-priority batch reads. Production systems tag requests with a criticality class (e.g., CRITICAL, DEGRADED_EXPERIENCE, BEST_EFFORT) and shed lowest-priority traffic first, sometimes implemented as nested token buckets per class.
• LIFO over FIFO under saturation: Once queue depth exceeds a threshold, serving the most recently arrived request first (LIFO) yields better tail latency than FIFO, because older queued requests have already likely exceeded the caller’s timeout and are effectively “dead work” — completing them wastes resources that could serve a fresh, still-useful request.
• Metastable failure prevention: The core justification (per the OSDI ‘Metastable Failures’ literature) is that overloaded services often don’t recover on their own even after the traffic spike subsides, because retry storms and thread-pool exhaustion create a self-sustaining high-latency equilibrium. Admission control breaks this loop by keeping per-request latency low enough that clients don’t retry in the first place.

Edge cases dominate real deployments: GC pauses or noisy-neighbor CPU steal on a host can be misread as legitimate overload, causing the gradient algorithm to clamp concurrency on a healthy service — mitigated with hysteresis windows and minimum-sample-size gating before adjusting limits. Multi-tenant systems must also guard against a single noisy tenant collapsing the shared limit; this requires per-tenant sub-limits nested inside the global admission gate. Finally, admission control interacts poorly with client-side retries unless retry budgets are coordinated — a client that blindly retries a 503 from an admission controller simply re-injects the rejected load, defeating the mechanism entirely.