Singleflight maintains an in-flight map of active keys so that concurrent callers for the same key await a single leader execution instead of stampeding the origin.
TL;DR30-second simplified summary▾
When a hot cache key expires, thousands of concurrent requests miss the cache simultaneously and slam the origin database with identical queries—a failure mode known as a cache stampede or thundering herd. Singleflight solves this at the application or proxy layer using a mutex-protected in-flight map keyed by request key.
The first caller (the leader) registers a call entry containing a synchronization primitive (e.g., a WaitGroup/Promise) in the map, releases the mutex, and executes the origin fetch. Subsequent callers (followers) arriving while the key is active find the existing call entry, release the mutex, and block on its completion signal. When the leader finishes, it stores the result/error, deletes the key from the in-flight map, and wakes all waiting followers.
In production, naive singleflight has three sharp edges: (1) a single leader failure/timeout fails all N followers simultaneously unless you support Forget(key) or retries; (2) if the leader's request context cancels early, it must not abort the shared fetch for remaining followers; and (3) a singleflight group only deduplicates within one process unless paired with consistent-hash routing.
From patterns to decisions.
Collapse N concurrent cache-miss requests for the same key into 1 upstream origin call, protecting databases from thundering-herd overload.
Why Hot-Key Expiry Triggers Origin Collapse.
Suppose a hot key receives 50,000 QPS and takes 40 ms to recompute from PostgreSQL. The moment its Redis TTL expires, ~2,000 concurrent requests miss Redis within that 40 ms window and all 2,000 query PostgreSQL simultaneously, exhausting connection pools and causing cascading timeouts.
Cache Stampede Amplification
1.Stage Objective: Quantify how concurrent cache misses amplify origin QPS by orders of magnitude.
2.Level (Foundation): Suppose a hot key receives 50,000 QPS and takes 40 ms to recompute from PostgreSQL. The moment its Redis TTL expires, ~2,000 concurrent requ
3.Operational Invariant: 2,000 concurrent misses during a 40 ms origin query saturate downstream database pools.
Caption: 2,000 concurrent misses during a 40 ms origin query saturate downstream database pools.
Leader Election & Follower Fan-Out via In-Flight Map.
A SingleflightGroup holds mu (a lightweight mutex) and calls: map[key]*Call. Callers acquire mu only long enough to check/insert into calls (nanoseconds), never across the slow network I/O.
If key exists in calls, the caller increments dups, releases mu, and awaits call.done. Otherwise, it inserts a new Call, releases mu, runs fn(), deletes key from calls under mu, and signals call.done.
Singleflight Leader/Follower Flow
1.Stage Objective: Trace the exact mutex, map lookup, and WaitGroup/Promise lifecycle of a singleflight call.
2.Level (Foundation): A `SingleflightGroup` holds `mu` (a lightweight mutex) and `calls: map[key]*Call`. Callers acquire `mu` only long enough to check/insert int
3.Operational Invariant: 1 leader executes the DB query while 999 followers block on a shared completion signal.
Caption: 1 leader executes the DB query while 999 followers block on a shared completion signal.
Implementing Singleflight with Detached Contexts & Error Isolation.
Detach the leader's upstream timeout from individual caller cancellations so one impatient client cannot abort 500 waiting followers. Always wrap fn() in try/finally so the in-flight entry is cleaned up even on exception.
import threading
from typing import Any, Callable
class _Call:
def __init__(self) -> None:
self.event = threading.Event()
self.val: Any = None
self.err: Exception | None = None
class SingleflightGroup:
def __init__(self) -> None:
self._mu = threading.Lock()
self._calls: dict[str, _Call] = {}
def do(self, key: str, fn: Callable[[], Any], wait_timeout: float = 2.0) -> tuple[Any, bool]:
with self._mu:
if key in self._calls:
call = self._calls[key]
is_follower = True
else:
call = _Call()
self._calls[key] = call
is_follower = False
if is_follower:
if not call.event.wait(timeout=wait_timeout):
raise TimeoutError(f'Singleflight wait timed out for {key}')
if call.err is not None:
raise call.err
return call.val, True
try:
call.val = fn()
except Exception as exc:
call.err = exc
finally:
with self._mu:
if self._calls.get(key) is call:
del self._calls[key]
call.event.set()
if call.err is not None:
raise call.err
return call.val, FalseSafe Singleflight Execution
1.Stage Objective: Write a thread-safe singleflight coalescer that guards against leader cancellation and shared panics.
2.Level (Intermediate): Detach the leader's upstream timeout from individual caller cancellations so one impatient client cannot abort 500 waiting followers. Always
3.Operational Invariant: Bounded follower wait timeouts and finally-block cleanup prevent indefinite hangs on leader errors.
Caption: Bounded follower wait timeouts and finally-block cleanup prevent indefinite hangs on leader errors.
Mitigating Shared-Error Amplification with `Forget(key)`.
If the leader encounters a transient 503 error, all 999 followers receive that same error. Worse, if a stale negative result is cached, traffic fails repeatedly. Using Forget(key) (or only coalescing healthy responses while allowing bounded jittered retry on transient errors) limits blast radius.
Never mutate a returned slice/object in place without cloning, because all followers share the exact same pointer/reference unless copied.
Shared-Reference & Error Guard
1.Stage Objective: Prevent transient errors from failing an entire batch of followers unnecessarily.
2.Level (Intermediate): If the leader encounters a transient 503 error, all 999 followers receive that same error. Worse, if a stale negative result is cached, traf
3.Operational Invariant: Cloning mutable results and supporting Forget(key) prevent cross-request corruption and error storms.
Caption: Cloning mutable results and supporting Forget(key) prevent cross-request corruption and error storms.
Scaling Singleflight Across 100+ Stateless Nodes.
In a 100-pod Kubernetes deployment, local-only singleflight still lets 100 pods each send 1 leader request per key. Routing cache-miss traffic through a Consistent-Hash Routing Ring (e.g., Envoy Maglev/RingHash or an internal coalescer tier) funnels all requests for key to the same coalescer pod, reducing 100 origin calls down to 1.
Consistent-Hash Coalescing Tier
1.Stage Objective: Combine consistent-hash routing with local singleflight to eliminate cross-pod duplicate fetches.
2.Level (Advanced): In a 100-pod Kubernetes deployment, local-only singleflight still lets 100 pods each send 1 leader request per key. Routing cache-miss traff
3.Operational Invariant: Consistent hashing funnels identical keys to a single coalescer node for cluster-wide deduplication.
Caption: Consistent hashing funnels identical keys to a single coalescer node for cluster-wide deduplication.
Pairing Singleflight with Probabilistic Early Refresh (PER) & SWR.
Singleflight still forces callers to wait for 1 origin round-trip on expiry. Pairing Singleflight with Stale-While-Revalidate (SWR) or Probabilistic Early Refresh (XFetch) lets one background leader refresh the key before TTL=0 while all other callers receive the cached value in < 1 ms.
Singleflight + SWR + PER
1.Stage Objective: Eliminate even the single leader's miss latency on ultra-hot keys.
2.Level (Advanced): Singleflight still forces callers to wait for 1 origin round-trip on expiry. Pairing Singleflight with **Stale-While-Revalidate (SWR)** or *
3.Operational Invariant: Background singleflight refresh before expiry keeps p99 read latency flat at sub-millisecond levels.
Caption: Background singleflight refresh before expiry keeps p99 read latency flat at sub-millisecond levels.
When Singleflight Hurts: Non-Idempotent Writes & Personalized Auth.
Never coalesce requests whose key omits tenant ID, auth scope, or user permissions—doing so leaks one user's private payload to 999 followers. Also never coalesce non-idempotent mutations (POST /charge) unless keyed by an explicit client idempotency token.
Safe Key Scoping Checklist
1.Stage Objective: Avoid catastrophic security and correctness bugs from over-broad coalescing keys.
2.Level (Advanced): Never coalesce requests whose key omits tenant ID, auth scope, or user permissions—doing so leaks one user's private payload to 999 follower
3.Operational Invariant: Include tenant, auth scope, and normalized parameters in every singleflight key to prevent data leaks.
Caption: Include tenant, auth scope, and normalized parameters in every singleflight key to prevent data leaks.
Practice: Make the trade-off yourself.
Why should a Singleflight leader detach its upstream context from the first caller's request cancellation?