Your e-commerce catalog service serves 80,000 QPS from a 4-shard Redis 7 cluster (99.4% cache hit ratio). Every night at 00:00 UTC, a batch job pre-warms 120,000 top product pages with a uniform `EX 3600` (1-hour) TTL. At 01:00:00 UTC, all 120,000 keys expire simultaneously. Cache hit ratio plunges to 12%, 70,000 QPS hits PostgreSQL 16 (max 400 connections via PgBouncer), connection wait queues saturate in 800ms, and API gateway p99 spikes from 6ms to 12,000ms (HTTP 504). Redesign the caching, expiration, and fallback architecture to absorb key expirations smoothly with < 5% DB load variance.
From patterns to decisions.
Your e-commerce catalog service serves 80,000 QPS from a 4-shard Redis 7 cluster (99.4% cache hit ratio). Every night at 00:00 UTC, a batch job pre-warms 120,000 top product pages with a uniform `EX 3600` (1-hour) TTL. At 01:00:00 UTC, all 120,000 keys expire simultaneously. Cache hit ratio plunges to 12%, 70,000 QPS hits PostgreSQL 16 (max 400 connections via PgBouncer), connection wait queues saturate in 800ms, and API gateway p99 spikes from 6ms to 12,000ms (HTTP 504). Redesign the caching, expiration, and fallback architecture to absorb key expirations smoothly with < 5% DB load variance.
Uniform EX 3600 synchronized all.
Uniform EX 3600 synchronized all 120,000 expirations into 1 second, flipping 70,000 QPS of cache hits into simultaneous DB misses that saturated PgBouncer's 400 connections.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Uniform TTLs are simple to reason about but create deterministic traffic cliffs. |
Calculate the miss surge
80,000 QPS * 88% miss rate = 70,400 QPS hitting PostgreSQL.
Step 2
Compare against PgBouncer/DB limit (400 connections, 4,000 QPS max).
Step 3
Observe connection-pool queueing and retry amplification causing 12s 504 timeouts.
Alternative 1
Never expire keys; push updates via CDC.
Failure Mode 1
Complete DB connection exhaustion every hour on the hour.
# Anti-pattern: uniform TTL in midnight batch warmer
for product in top_120k_products:
redis.set(f"prod:{product.id}", serialize(product), ex=3600) # All expire at 01:00:00Z!
Combine jittered hard_ttl (2 *.
Combine jittered hard_ttl (2 * base_ttl + rand), logical soft_exp, and an in-process inflight Future map so concurrent misses coalesce into one DB call.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Doubles Redis TTL window (more memory) in exchange for zero latency spikes on expiration. |
Store `{"data"
val, "soft_exp": now + base_ttl + jitter} with Redis ex = base_ttl * 2 + jitter`.
Step 2
If now > soft_exp on hit, return data immediately and trigger singleflight background refresh.
Step 3
On hard miss, await the shared inflight[key] Future.
Alternative 1
Go golang.org/x/sync/singleflight or Caffeine refreshAfterWrite in Java.
Forgetting `finally
inflight.pop(key, None)` leaks futures and permanently blocks future refreshes.
import asyncio, json, random, time
inflight: dict[str, asyncio.Future] = {}
async def get_product(redis, db, key: str, base_ttl: int = 3600):
raw = await redis.get(key)
now = time.time()
if raw:
payload = json.loads(raw)
if now > payload["soft_exp"] and key not in inflight:
asyncio.create_task(_singleflight_refresh(redis, db, key, base_ttl))
return payload["data"]
return await _singleflight_refresh(redis, db, key, base_ttl)
async def _singleflight_refresh(redis, db, key: str, base_ttl: int):
if key in inflight:
return await inflight[key]
fut = asyncio.get_running_loop().create_future()
inflight[key] = fut
try:
data = await asyncio.wait_for(db.fetch_product(key), timeout=0.5)
jitter = random.randint(0, int(base_ttl * 0.2))
payload = {"data": data, "soft_exp": time.time() + base_ttl + jitter}
await redis.set(key, json.dumps(payload), ex=base_ttl * 2 + jitter)
fut.set_result(data)
return data
except Exception as exc:
fut.set_exception(exc)
raise
finally:
inflight.pop(key, None)
Implementation & Guardrails (3).
Caffeine refreshAfterWrite protects each pod's L1 cache with zero network hops, while Redis Soft-TTL shares refreshed data across all 40 pods and caps total DB rebuilds.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | L1 + L2 adds minor stale-window variance across pods (up to 30s) but cuts Redis network load by ~45%. |
Step 1
Use L1 Caffeine/LRU (short 60s TTL, ~256MB) for ultra-hot keys and Redis outage survival.
Step 2
Use Redis Soft-TTL as the shared L2 source of truth across all 40 pods.
Optionally add a Redis `SETNX lock
key 1 EX 2 inside _singleflight_refresh` so even across 40 pods only 1 pod queries DB.
Alternative 1
Redis-only with distributed SETNX refresh lease.
Failure Mode 1
Unbounded L1 cache without maximumSize causes JVM OOM kills under key-cardinality spikes.
// Java Caffeine L1 fallback with async refreshAfterWrite
LoadingCache<String, Product> l1Cache = Caffeine.newBuilder()
.maximumSize(10_000)
.refreshAfterWrite(Duration.ofSeconds(30))
.expireAfterWrite(Duration.ofSeconds(120))
.build(key -> redisOrDbFetch(key));
L1 in-memory LRU absorbs immediate.
L1 in-memory LRU absorbs immediate traffic, PgBouncer + DB semaphore caps concurrent origin queries, and a circuit breaker serves stale L1 or degraded responses when DB latency exceeds 150ms.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Serving slightly stale L1 data during a Redis reboot preserves 99.9% availability instead of cascading 504s. |
Step 1
Enable Redis AOF (appendfsync everysec) + replica failover so shard restarts rarely lose data.
Step 2
Cap per-pod DB concurrency with an asyncio.Semaphore(10) (40 pods * 10 = 400 max DB conns).
Step 3
When DB p99 > 150ms, circuit breaker opens and serves L1 stale entries.
Alternative 1
Shadow cache warming from a read replica before promoting a cold Redis shard.
Failure Mode 1
Without a semaphore, 40 pods can still open thousands of waiting coroutines against PgBouncer.
db_sem = asyncio.Semaphore(10)
async def protected_db_fetch(db, l1_cache, key):
if db_sem.locked() and key in l1_cache:
return l1_cache[key] # Shed load to L1 stale
async with db_sem:
return await asyncio.wait_for(db.fetch_product(key), timeout=0.5)
XFetch triggers a probabilistic refresh.
XFetch triggers a probabilistic refresh before TTL expiry weighted by the key's recompute duration delta, ensuring the hottest keys refresh early without waiting for an exact soft_exp boundary.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Self-adapts to slow DB queries (`delta`) with zero extra Redis locks, though `beta > 1` trades slightly more refreshes for zero misses. |
Step 1
Store delta (last DB fetch duration in seconds) alongside expiry in Redis.
Step 2
On read, evaluate time.time() - delta * beta * math.log(random.random()) >= expiry.
Step 3
Because -ln(rand()) is positive, higher delta or closer expiry increases refresh probability smoothly.
Alternative 1
Fixed soft_exp + singleflight (simpler to debug).
Failure Mode 1
If delta_sec is 0 or unrecorded, XFetch degrades to standard hard-TTL expiration.
import math, random, time
def should_xfetch_refresh(expiry: float, delta_sec: float, beta: float = 1.0) -> bool:
# Random value in (0, 1]; -ln(u) is exponential distribution
u = max(random.random(), 1e-9)
return (time.time() - delta_sec * beta * math.log(u)) >= expiry
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Break TTL synchronization with randomized jitter (
base_ttl + random(0, 20%)), coalesce concurrent misses viasingleflight, and servestale-while-revalidatefallbacks using logical expiration metadata. - 02Trade-off tip: L1 + L2 adds minor stale-window variance across pods (up to 30s) but cuts Redis network load by ~45%.
- 03Watch out for: Without a semaphore, 40 pods can still open thousands of waiting coroutines against PgBouncer.
Preventing Cache Avalanche Ttl Jitter — Normal → Load / Fault → Recovery
Break TTL synchronization with randomized jitter (basettl + random(0, 20%)), coalesce concurrent misses via singleflight, and serve stale-while-revalidate fallbacks using logical expiration metadata.
Putting Preventing Cache Avalanche with TTL Jitter Together
Break TTL synchronization with randomized jitter (`base_ttl + random(0, 20%)`), coalesce concurrent misses via `singleflight`, and serve `stale-while-revalidate` fallbacks using logical expiration metadata.
Putting Preventing Cache Avalanche with TTL Jitter Together
- Store
{"data": val, "soft_exp": now + base_ttl + jitter}with Redisex = base_ttl * 2 + jitter. - If
now > soft_expon hit, returndataimmediately and triggersingleflightbackground refresh. - On hard miss, await the shared
inflight[key]Future.
Interview Readiness Check
L1 in-memory LRU absorbs immediate traffic, PgBouncer + DB semaphore caps concurrent origin queries, and a circuit breaker serves stale L1 or degraded responses when DB latency exceeds 150ms.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Preventing Cache Avalanche with TTL Jitter
Break TTL synchronization with randomized jitter (base_ttl + random(0, 20%)), coalesce concurrent misses via singleflight, and serve stale-while-revalidate fallbacks using logical expiration metadata.
