Request Coalescing (Singleflight)

Request coalescing (popularized by Go’s singleflight package but implemented independently in CDNs, ORMs, and service meshes) sits at the boundary between a cache and its origin. When N concurrent callers request the same key and none of them find a cached value, a naive implementation issues N identical origin calls simultaneously — the classic cache stampede or dogpile effect. Coalescing introduces a per-key mutex or promise registry: the first caller becomes the leader for that key, executes the actual fetch/compute, and registers a shared future. Subsequent callers for the same key observe the in-flight future and simply await its resolution instead of dispatching their own request. Once the leader completes, the result (or error) is broadcast to all waiters and the registry entry is evicted.

The implementation detail that separates correct coalescing from subtly broken coalescing is error and cancellation semantics. If the leader’s call fails, do all followers fail identically, or does one of them get promoted to retry? Most singleflight implementations propagate the exact same error/result to every waiter — which is dangerous if the error is transient and retry-worthy, because a single flaky origin response now fails an entire batch of callers instead of just one. Production-grade implementations add a forget() or key-eviction call immediately after failure so the next request re-triggers a fresh attempt rather than serving a stale error from a registry that hasn’t expired yet.

• Context propagation hazard: if the leader’s request is scoped to a specific caller’s context (deadline, trace ID, cancellation token) and that caller cancels or times out, naive coalescing will cancel the underlying fetch for every follower still waiting on it — a cross-tenant cancellation leak. Correct designs detach the shared fetch from any single caller’s lifecycle and use the union (or max) of follower deadlines.
• Partial fanout skew: coalescing collapses call volume but not necessarily latency variance — followers pay the leader’s full latency plus any dispatch/broadcast overhead, which can violate SLOs tuned for the uncoalesced fast path.
• Key granularity: too coarse a key (e.g., coalescing on endpoint rather than endpoint+params) silently merges semantically distinct requests; too fine a key defeats the purpose under high cardinality.

Architecturally, coalescing is often layered beneath a cache (Redis, Memcached, in-process LRU) as a stampede guard on miss, and is functionally related to but distinct from a Circuit Breaker — coalescing reduces redundant concurrent calls to the same key, while a circuit breaker stops calls to a failing dependency regardless of key. It’s also distinct from batching: batching aggregates different keys into one round-trip, coalescing merges identical keys into one execution. In multi-node deployments, in-process coalescing only protects a single instance; true stampede protection at scale requires a distributed lock or lease (e.g., Redis SETNX with TTL) so that N service replicas don’t each independently coalesce and still send N calls to the origin.