Distributed Rate Limiting

A coordination pattern for enforcing a single global request quota across a horizontally scaled fleet of stateless nodes, rather than each node independently limiting traffic. It matters because naive per-instance limiting silently multiplies the effective limit by the number of replicas, defeating the policy's intent under exactly the load conditions it exists to control.

Practical example

A 50-pod API gateway enforces a 1,000 req/s-per-API-key quota using a GCRA Lua script executed atomically against a sharded Redis cluster, with each pod caching a short-lived local token allowance to avoid a Redis round trip on every request.

Distributed rate limiting solves a fundamentally different problem than single-process rate limiting: the counter or token bucket state must be visible to (or reconciled across) every node enforcing the policy, without that shared state becoming a bottleneck or single point of failure itself. The naive approach — giving each replica an independent local limit/N budget — degrades badly under uneven load distribution, and giving each replica the full limit independently defeats the policy entirely once replica count exceeds one.

Implementations generally fall into three families. Centralized counter stores (Redis, Memcached with atomic INCR, or a dedicated rate-limit service like Envoy’s ratelimit) hold authoritative counters, typically implementing the Generic Cell Rate Algorithm (GCRA) or a sliding-window log via a Lua script for atomicity. This gives strong accuracy but adds a synchronous network hop to every request path and turns the store into a shared dependency with its own availability and tail-latency profile. Sharded/local-with-sync approaches let each node maintain a local token bucket sized as a fraction of global capacity, periodically reconciling drift with a central authority or via gossip; this trades precision for reduced hot-path latency and graceful degradation if the sync channel is unavailable. CRDT-based counters (e.g., PN-Counters) allow fully decentralized, eventually-consistent aggregation without a central store, at the cost of transient overshoot during partition or high propagation delay.

Burst absorption: hybrid designs grant each node a small local burst allowance on top of a synchronized baseline, smoothing tail latency from the counter store without materially loosening the global cap.
Clock skew and window boundaries: fixed-window counters are vulnerable to boundary-doubling (2x burst at window edges); GCRA and sliding-window-log avoid this but require more state per key.
Thundering herd on the store: a hot key (e.g., a single abusive tenant) can itself become a scalability bottleneck on the counter backend, requiring key-level sharding independent of the rate-limit logic.
Fail-open vs fail-closed: when the shared store is unreachable, the system must decide whether to admit traffic unbounded (risking backend overload) or reject it (risking false-positive throttling of legitimate traffic) — this decision is a first-class architectural choice, not an afterthought.

The operational failure mode engineers most often hit is treating rate limiting as a purely local concern during initial implementation, then discovering under a load test or real incident that the effective quota scales linearly with replica count. Retrofitting global coordination after the fact typically means introducing a new stateful dependency into a previously stateless service tier, which has its own blast-radius and capacity-planning implications that need to be modeled before rollout, not during the incident that exposed the gap.