Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Preventing Hot-Key
Cache Stampedes.

Your e-commerce homepage serves a global flash-deal configuration key (deal:global:active) cached in Redis with a fixed 60-second TTL at 50,000 req/s across 200 API pods. Every 60 seconds, the key expires instantaneously; thousands of API workers miss Redis in the same 50 ms window and fire identical heavy SQL joins (180 ms each) against PostgreSQL. The DB connection pool (200 connections) saturates, causing 504 Gateway Timeouts for 8 seconds every minute. Design a stampede-proof caching architecture that caps DB recomputation at 1 query per refresh cycle.
Start the journey5 stages
Loading diagram…
Preventing Hot-Key Cache Stampedes architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your e-commerce homepage serves a global flash-deal configuration key (`deal:global:active`) cached in Redis with a fixed 60-second TTL at 50,000 req/s across 200 API pods. Every 60 seconds, the key expires instantaneously; thousands of API workers miss Redis in the same 50 ms window and fire identical heavy SQL joins (180 ms each) against PostgreSQL. The DB connection pool (200 connections) saturates, causing 504 Gateway Timeouts for 8 seconds every minute. Design a stampede-proof caching architecture that caps DB recomputation at 1 query per refresh cycle.

From patterns to decisions.

Your e-commerce homepage serves a global flash-deal configuration key (`deal:global:active`) cached in Redis with a fixed 60-second TTL at 50,000 req/s across 200 API pods. Every 60 seconds, the key expires instantaneously; thousands of API workers miss Redis in the same 50 ms window and fire identical heavy SQL joins (180 ms each) against PostgreSQL. The DB connection pool (200 connections) saturates, causing 504 Gateway Timeouts for 8 seconds every minute. Design a stampede-proof caching architecture that caps DB recomputation at 1 query per refresh cycle.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

A single shared key has.

A single shared key has only 1 TTL; jitter only spreads expirations across different keys, not concurrent readers of the same key.

ChoiceWhat it buysWhat it costs
Trade-off 1Any synchronous cache-aside pattern without coalescing or SWR amplifies a single DB latency spike into an outage.

Compute misses in a 50ms window at 50k req/s

2,500 simultaneous misses.

Step 2

Recognize 2,500 * 180ms SQL queries immediately exhaust 200 DB connections.

Step 3

Note that TTL jitter does nothing when all 50k req/s hit one key.

Alternative 1

Background cron pre-warmer (works for known global keys, fails for dynamic hot keys).

Failure Mode 1

504 Gateway Timeouts cascading to retry storms from mobile/web clients.

qps = 50_000
db_latency_s = 0.180
misses_during_one_db_call = qps * db_latency_s
print(f'Concurrent misses before 1st DB call finishes: {misses_during_one_db_call:,.0f}')
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

XFetch triggers an early async.

XFetch triggers an early async refresh by 1 lucky reader before soft_expiry, while a SET NX PX 2000 lease ensures at most 1 pod hits PostgreSQL.

ChoiceWhat it buysWhat it costs
Trade-off 1Inline refresh on the 1 winning request adds 180ms to 1 out of 3,000,000 requests; offloading to a background thread keeps even that request < 5ms.

Step 1

Store {value, delta_s, soft_expiry} in Redis with 300s hard TTL.

Step 2

Evaluate now - delta_s * beta * math.log(random.random()) >= soft_expiry.

Step 3

If triggered, acquire SET lock NX PX 2000 and refresh asynchronously while returning value.

Alternative 1

Scheduled background updater pushing to Redis every 45s (simpler for 1 static key, doesn't generalize to arbitrary hot keys).

Failure Mode 1

Using random.random() == 0.0 causes math.log(0) ValueError; clamp to max(1e-9, random.random()).

import math, random, time, json

def get_with_xfetch(redis_client, key: str, fetch_fn, soft_ttl=60, hard_ttl=300, beta=1.0):
    raw = redis_client.get(key)
    now = time.time()
    if raw:
        entry = json.loads(raw)
        should_refresh = (now - entry['delta'] * beta * math.log(random.random())) >= entry['soft_exp']
        if should_refresh and redis_client.set(f'lock:{key}', '1', nx=True, px=2000):
            t0 = time.time()
            val = fetch_fn()
            delta = time.time() - t0
            redis_client.set(key, json.dumps({'val': val, 'delta': delta, 'soft_exp': time.time() + soft_ttl}), ex=hard_ttl)
        return entry['val']
    # Cold cache fallback: acquire lease or wait
    for _ in range(10):
        if redis_client.set(f'lock:{key}', '1', nx=True, px=2000):
            t0 = time.time()
            val = fetch_fn()
            redis_client.set(key, json.dumps({'val': val, 'delta': time.time()-t0, 'soft_exp': time.time() + soft_ttl}), ex=hard_ttl)
            return val
        time.sleep(0.05 + random.uniform(0, 0.02))
        raw = redis_client.get(key)
        if raw:
            return json.loads(raw)['val']
    return fetch_fn()
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

A 2-second L1 cache cuts.

A 2-second L1 cache cuts Redis hot-shard traffic from 50,000 req/s to 100 req/s while adding at most 2 seconds to worst-case staleness (62s total <= 65s SLA).

ChoiceWhat it buysWhat it costs
Trade-off 1Pods may observe slightly different values for up to 2 seconds after an update.

Calculate Redis QPS without L1

50,000 req/s on 1 key saturates 1 Redis core.

Step 2

Calculate Redis QPS with 2s L1 across 200 pods: 200 / 2 = 100 req/s.

Verify staleness

60s soft TTL + 2s L1 = 62s < 65s requirement.

Alternative 1

Key-replication across N Redis keys (deal:0..9) to shard 50k req/s across 10 Redis shards.

Failure Mode 1

Unbounded L1 cache causing pod OOM; always set maximumSize.

import com.github.benmanes.caffeine.cache.*;
import java.time.Duration;

LoadingCache<String, String> l1Cache = Caffeine.newBuilder()
    .maximumSize(10_000)
    .expireAfterWrite(Duration.ofSeconds(2))
    .build(key -> fetchFromRedisWithSingleflight(key));
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Per-pod singleflight collapses 50,000 req/s to at most 200 concurrent Redis lock attempts, and SET NX + 50ms backoff collapses those 200 to 1 PostgreSQL query.

ChoiceWhat it buysWhat it costs
Trade-off 1Waiters experience ~200ms latency for one 180ms cycle on cold start instead of crashing PostgreSQL.

Step 1

200 pods each collapse ~250 local threads via singleflight -> 200 Redis SET NX calls.

Step 2

1 pod wins the lock and runs 1 DB query (180ms); 199 pods wait 50ms and read the populated key.

Step 3

Enforce a DB-side bulkhead (Semaphore(5)) as a last-resort circuit breaker.

Alternative 1

Warming critical keys from a read-replica snapshot before shifting traffic.

Failure Mode 1

If the DB query takes > 2s (exceeding lock PX 2000), a second pod can acquire the lock; set PX to 5x p99 DB time.

import threading

db_bulkhead = threading.BoundedSemaphore(5)

def safe_query_db(fn):
    if not db_bulkhead.acquire(timeout=0.2):
        raise RuntimeError('DB bulkhead full - fail fast 503')
    try:
        return fn()
    finally:
        db_bulkhead.release()
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Publish an invalidation event over.

Publish an invalidation event over Redis Pub/Sub to evict the 2s L1 cache across all 200 pods and overwrite (not delete) the Redis key with the cancelled state.

ChoiceWhat it buysWhat it costs
Trade-off 1Overwrite + Pub/Sub gives < 50ms global invalidation without triggering a stampede.

Step 1

Write-through the new cancelled state to Redis (avoiding a cold miss).

Broadcast `PUBLISH cache

invalidate deal:global:active` so all 200 pods evict L1 immediately.

Alternative 1

Accepting the 2-second L1 + 60s soft TTL lag.

Failure Mode 1

Redis Pub/Sub is fire-and-forget; keep the 2s L1 TTL as a safety net for missed Pub/Sub messages.

def update_deal_and_broadcast(redis_client, new_payload: dict):
    redis_client.set('deal:global:active', json.dumps(new_payload), ex=300)
    redis_client.publish('cache:invalidate', 'deal:global:active')

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.
  • 02Trade-off tip: Pods may observe slightly different values for up to 2 seconds after an update.
  • 03Watch out for: If the DB query takes > 2s (exceeding lock PX 2000), a second pod can acquire the lock; set PX to 5x p99 DB time.
Behavioral Progression · Preventing Hot Key Cache Stampede

Preventing Hot Key Cache Stampede — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.

Synthesis / One connected design

Putting Preventing Hot-Key Cache Stampedes Together

Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.

A candidate architecture, not a universal answer

Putting Preventing Hot-Key Cache Stampedes Together

Step 1
Step 2
Step 3
  1. Store {value, delta_s, soft_expiry} in Redis with 300s hard TTL.
  2. Evaluate now - delta_s * beta * math.log(random.random()) >= soft_expiry.
  3. If triggered, acquire SET lock NX PX 2000 and refresh asynchronously while returning value.
Per-pod singleflight collapses 50,000 req/s to at most 200 concurrent Redis lock attempts, and `SET NX` + 50ms backoff collapses those 200 to 1 PostgreSQL query.
Interview Readiness Check

Per-pod singleflight collapses 50,000 req/s to at most 200 concurrent Redis lock attempts, and SET NX + 50ms backoff collapses those 200 to 1 PostgreSQL query.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Preventing Hot-Key Cache Stampedes

Combine in-process singleflight request coalescing, distributed Redis lease locks, and XFetch probabilistic early expiration to keep DB load at 1 query per refresh.