Your e-commerce homepage serves a global flash-deal configuration key (`deal:global:active`) cached in Redis with a fixed 60-second TTL at 50,000 req/s across 200 API pods. Every 60 seconds, the key expires instantaneously; thousands of API workers miss Redis in the same 50 ms window and fire identical heavy SQL joins (180 ms each) against PostgreSQL. The DB connection pool (200 connections) saturates, causing 504 Gateway Timeouts for 8 seconds every minute. Design a stampede-proof caching architecture that caps DB recomputation at 1 query per refresh cycle.
From patterns to decisions.
Your e-commerce homepage serves a global flash-deal configuration key (`deal:global:active`) cached in Redis with a fixed 60-second TTL at 50,000 req/s across 200 API pods. Every 60 seconds, the key expires instantaneously; thousands of API workers miss Redis in the same 50 ms window and fire identical heavy SQL joins (180 ms each) against PostgreSQL. The DB connection pool (200 connections) saturates, causing 504 Gateway Timeouts for 8 seconds every minute. Design a stampede-proof caching architecture that caps DB recomputation at 1 query per refresh cycle.
A single shared key has.
A single shared key has only 1 TTL; jitter only spreads expirations across different keys, not concurrent readers of the same key.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Any synchronous cache-aside pattern without coalescing or SWR amplifies a single DB latency spike into an outage. |
Compute misses in a 50ms window at 50k req/s
2,500 simultaneous misses.
Step 2
Recognize 2,500 * 180ms SQL queries immediately exhaust 200 DB connections.
Step 3
Note that TTL jitter does nothing when all 50k req/s hit one key.
Alternative 1
Background cron pre-warmer (works for known global keys, fails for dynamic hot keys).
Failure Mode 1
504 Gateway Timeouts cascading to retry storms from mobile/web clients.
qps = 50_000
db_latency_s = 0.180
misses_during_one_db_call = qps * db_latency_s
print(f'Concurrent misses before 1st DB call finishes: {misses_during_one_db_call:,.0f}')
XFetch triggers an early async.
XFetch triggers an early async refresh by 1 lucky reader before soft_expiry, while a SET NX PX 2000 lease ensures at most 1 pod hits PostgreSQL.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Inline refresh on the 1 winning request adds 180ms to 1 out of 3,000,000 requests; offloading to a background thread keeps even that request < 5ms. |
Step 1
Store {value, delta_s, soft_expiry} in Redis with 300s hard TTL.
Step 2
Evaluate now - delta_s * beta * math.log(random.random()) >= soft_expiry.
Step 3
If triggered, acquire SET lock NX PX 2000 and refresh asynchronously while returning value.
Alternative 1
Scheduled background updater pushing to Redis every 45s (simpler for 1 static key, doesn't generalize to arbitrary hot keys).
Failure Mode 1
Using random.random() == 0.0 causes math.log(0) ValueError; clamp to max(1e-9, random.random()).
import math, random, time, json
def get_with_xfetch(redis_client, key: str, fetch_fn, soft_ttl=60, hard_ttl=300, beta=1.0):
raw = redis_client.get(key)
now = time.time()
if raw:
entry = json.loads(raw)
should_refresh = (now - entry['delta'] * beta * math.log(random.random())) >= entry['soft_exp']
if should_refresh and redis_client.set(f'lock:{key}', '1', nx=True, px=2000):
t0 = time.time()
val = fetch_fn()
delta = time.time() - t0
redis_client.set(key, json.dumps({'val': val, 'delta': delta, 'soft_exp': time.time() + soft_ttl}), ex=hard_ttl)
return entry['val']
# Cold cache fallback: acquire lease or wait
for _ in range(10):
if redis_client.set(f'lock:{key}', '1', nx=True, px=2000):
t0 = time.time()
val = fetch_fn()
redis_client.set(key, json.dumps({'val': val, 'delta': time.time()-t0, 'soft_exp': time.time() + soft_ttl}), ex=hard_ttl)
return val
time.sleep(0.05 + random.uniform(0, 0.02))
raw = redis_client.get(key)
if raw:
return json.loads(raw)['val']
return fetch_fn()
A 2-second L1 cache cuts.
A 2-second L1 cache cuts Redis hot-shard traffic from 50,000 req/s to 100 req/s while adding at most 2 seconds to worst-case staleness (62s total <= 65s SLA).
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Pods may observe slightly different values for up to 2 seconds after an update. |
Calculate Redis QPS without L1
50,000 req/s on 1 key saturates 1 Redis core.
Step 2
Calculate Redis QPS with 2s L1 across 200 pods: 200 / 2 = 100 req/s.
Verify staleness
60s soft TTL + 2s L1 = 62s < 65s requirement.
Alternative 1
Key-replication across N Redis keys (deal:0..9) to shard 50k req/s across 10 Redis shards.
Failure Mode 1
Unbounded L1 cache causing pod OOM; always set maximumSize.
import com.github.benmanes.caffeine.cache.*;
import java.time.Duration;
LoadingCache<String, String> l1Cache = Caffeine.newBuilder()
.maximumSize(10_000)
.expireAfterWrite(Duration.ofSeconds(2))
.build(key -> fetchFromRedisWithSingleflight(key));
Implementation & Guardrails (4).
Per-pod singleflight collapses 50,000 req/s to at most 200 concurrent Redis lock attempts, and SET NX + 50ms backoff collapses those 200 to 1 PostgreSQL query.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Waiters experience ~200ms latency for one 180ms cycle on cold start instead of crashing PostgreSQL. |
Step 1
200 pods each collapse ~250 local threads via singleflight -> 200 Redis SET NX calls.
Step 2
1 pod wins the lock and runs 1 DB query (180ms); 199 pods wait 50ms and read the populated key.
Step 3
Enforce a DB-side bulkhead (Semaphore(5)) as a last-resort circuit breaker.
Alternative 1
Warming critical keys from a read-replica snapshot before shifting traffic.
Failure Mode 1
If the DB query takes > 2s (exceeding lock PX 2000), a second pod can acquire the lock; set PX to 5x p99 DB time.
import threading
db_bulkhead = threading.BoundedSemaphore(5)
def safe_query_db(fn):
if not db_bulkhead.acquire(timeout=0.2):
raise RuntimeError('DB bulkhead full - fail fast 503')
try:
return fn()
finally:
db_bulkhead.release()
Publish an invalidation event over.
Publish an invalidation event over Redis Pub/Sub to evict the 2s L1 cache across all 200 pods and overwrite (not delete) the Redis key with the cancelled state.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Overwrite + Pub/Sub gives < 50ms global invalidation without triggering a stampede. |
Step 1
Write-through the new cancelled state to Redis (avoiding a cold miss).
Broadcast `PUBLISH cache
invalidate deal:global:active` so all 200 pods evict L1 immediately.
Alternative 1
Accepting the 2-second L1 + 60s soft TTL lag.
Failure Mode 1
Redis Pub/Sub is fire-and-forget; keep the 2s L1 TTL as a safety net for missed Pub/Sub messages.
def update_deal_and_broadcast(redis_client, new_payload: dict):
redis_client.set('deal:global:active', json.dumps(new_payload), ex=300)
redis_client.publish('cache:invalidate', 'deal:global:active')
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.
- 02Trade-off tip: Pods may observe slightly different values for up to 2 seconds after an update.
- 03Watch out for: If the DB query takes > 2s (exceeding lock PX 2000), a second pod can acquire the lock; set PX to 5x p99 DB time.
Preventing Hot Key Cache Stampede — Normal → Load / Fault → Recovery
Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.
Putting Preventing Hot-Key Cache Stampedes Together
Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.
Putting Preventing Hot-Key Cache Stampedes Together
- Store
{value, delta_s, soft_expiry}in Redis with 300s hard TTL. - Evaluate
now - delta_s * beta * math.log(random.random()) >= soft_expiry. - If triggered, acquire
SET lock NX PX 2000and refresh asynchronously while returningvalue.
Interview Readiness Check
Per-pod singleflight collapses 50,000 req/s to at most 200 concurrent Redis lock attempts, and SET NX + 50ms backoff collapses those 200 to 1 PostgreSQL query.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Preventing Hot-Key Cache Stampedes
Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.
