Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Preventing Cache Avalanche
with TTL Jitter.

Your e-commerce catalog service serves 80,000 QPS from a 4-shard Redis 7 cluster (99.4% cache hit ratio). Every night at 00:00 UTC, a batch job pre-warms 120,000 top product pages with a uniform EX 3600 (1-hour) TTL. At 01:00:00 UTC, all 120,000 keys expire simultaneously. Cache hit ratio plunges to 12%, 70,000 QPS hits PostgreSQL 16 (max 400 connections via PgBouncer), connection wait queues saturate in 800ms, and API gateway p99 spikes from 6ms to 12,000ms (HTTP 504). Redesign the caching, expiration, and fallback architecture to absorb key expirations smoothly with < 5% DB load variance.
Start the journey5 stages
Loading diagram…
Preventing Cache Avalanche with TTL Jitter architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your e-commerce catalog service serves 80,000 QPS from a 4-shard Redis 7 cluster (99.4% cache hit ratio). Every night at 00:00 UTC, a batch job pre-warms 120,000 top product pages with a uniform `EX 3600` (1-hour) TTL. At 01:00:00 UTC, all 120,000 keys expire simultaneously. Cache hit ratio plunges to 12%, 70,000 QPS hits PostgreSQL 16 (max 400 connections via PgBouncer), connection wait queues saturate in 800ms, and API gateway p99 spikes from 6ms to 12,000ms (HTTP 504). Redesign the caching, expiration, and fallback architecture to absorb key expirations smoothly with < 5% DB load variance.

From patterns to decisions.

Your e-commerce catalog service serves 80,000 QPS from a 4-shard Redis 7 cluster (99.4% cache hit ratio). Every night at 00:00 UTC, a batch job pre-warms 120,000 top product pages with a uniform `EX 3600` (1-hour) TTL. At 01:00:00 UTC, all 120,000 keys expire simultaneously. Cache hit ratio plunges to 12%, 70,000 QPS hits PostgreSQL 16 (max 400 connections via PgBouncer), connection wait queues saturate in 800ms, and API gateway p99 spikes from 6ms to 12,000ms (HTTP 504). Redesign the caching, expiration, and fallback architecture to absorb key expirations smoothly with < 5% DB load variance.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Uniform EX 3600 synchronized all.

Uniform EX 3600 synchronized all 120,000 expirations into 1 second, flipping 70,000 QPS of cache hits into simultaneous DB misses that saturated PgBouncer's 400 connections.

ChoiceWhat it buysWhat it costs
Trade-off 1Uniform TTLs are simple to reason about but create deterministic traffic cliffs.

Calculate the miss surge

80,000 QPS * 88% miss rate = 70,400 QPS hitting PostgreSQL.

Step 2

Compare against PgBouncer/DB limit (400 connections, 4,000 QPS max).

Step 3

Observe connection-pool queueing and retry amplification causing 12s 504 timeouts.

Alternative 1

Never expire keys; push updates via CDC.

Failure Mode 1

Complete DB connection exhaustion every hour on the hour.

# Anti-pattern: uniform TTL in midnight batch warmer
for product in top_120k_products:
    redis.set(f"prod:{product.id}", serialize(product), ex=3600)  # All expire at 01:00:00Z!
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Combine jittered hard_ttl (2 *.

Combine jittered hard_ttl (2 * base_ttl + rand), logical soft_exp, and an in-process inflight Future map so concurrent misses coalesce into one DB call.

ChoiceWhat it buysWhat it costs
Trade-off 1Doubles Redis TTL window (more memory) in exchange for zero latency spikes on expiration.

Store `{"data"

val, "soft_exp": now + base_ttl + jitter} with Redis ex = base_ttl * 2 + jitter`.

Step 2

If now > soft_exp on hit, return data immediately and trigger singleflight background refresh.

Step 3

On hard miss, await the shared inflight[key] Future.

Alternative 1

Go golang.org/x/sync/singleflight or Caffeine refreshAfterWrite in Java.

Forgetting `finally

inflight.pop(key, None)` leaks futures and permanently blocks future refreshes.

import asyncio, json, random, time

inflight: dict[str, asyncio.Future] = {}

async def get_product(redis, db, key: str, base_ttl: int = 3600):
    raw = await redis.get(key)
    now = time.time()
    if raw:
        payload = json.loads(raw)
        if now > payload["soft_exp"] and key not in inflight:
            asyncio.create_task(_singleflight_refresh(redis, db, key, base_ttl))
        return payload["data"]
    return await _singleflight_refresh(redis, db, key, base_ttl)

async def _singleflight_refresh(redis, db, key: str, base_ttl: int):
    if key in inflight:
        return await inflight[key]
    fut = asyncio.get_running_loop().create_future()
    inflight[key] = fut
    try:
        data = await asyncio.wait_for(db.fetch_product(key), timeout=0.5)
        jitter = random.randint(0, int(base_ttl * 0.2))
        payload = {"data": data, "soft_exp": time.time() + base_ttl + jitter}
        await redis.set(key, json.dumps(payload), ex=base_ttl * 2 + jitter)
        fut.set_result(data)
        return data
    except Exception as exc:
        fut.set_exception(exc)
        raise
    finally:
        inflight.pop(key, None)
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

Caffeine refreshAfterWrite protects each pod's L1 cache with zero network hops, while Redis Soft-TTL shares refreshed data across all 40 pods and caps total DB rebuilds.

ChoiceWhat it buysWhat it costs
Trade-off 1L1 + L2 adds minor stale-window variance across pods (up to 30s) but cuts Redis network load by ~45%.

Step 1

Use L1 Caffeine/LRU (short 60s TTL, ~256MB) for ultra-hot keys and Redis outage survival.

Step 2

Use Redis Soft-TTL as the shared L2 source of truth across all 40 pods.

Optionally add a Redis `SETNX lock

key 1 EX 2 inside _singleflight_refresh` so even across 40 pods only 1 pod queries DB.

Alternative 1

Redis-only with distributed SETNX refresh lease.

Failure Mode 1

Unbounded L1 cache without maximumSize causes JVM OOM kills under key-cardinality spikes.

// Java Caffeine L1 fallback with async refreshAfterWrite
LoadingCache<String, Product> l1Cache = Caffeine.newBuilder()
    .maximumSize(10_000)
    .refreshAfterWrite(Duration.ofSeconds(30))
    .expireAfterWrite(Duration.ofSeconds(120))
    .build(key -> redisOrDbFetch(key));
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

L1 in-memory LRU absorbs immediate.

L1 in-memory LRU absorbs immediate traffic, PgBouncer + DB semaphore caps concurrent origin queries, and a circuit breaker serves stale L1 or degraded responses when DB latency exceeds 150ms.

ChoiceWhat it buysWhat it costs
Trade-off 1Serving slightly stale L1 data during a Redis reboot preserves 99.9% availability instead of cascading 504s.

Step 1

Enable Redis AOF (appendfsync everysec) + replica failover so shard restarts rarely lose data.

Step 2

Cap per-pod DB concurrency with an asyncio.Semaphore(10) (40 pods * 10 = 400 max DB conns).

Step 3

When DB p99 > 150ms, circuit breaker opens and serves L1 stale entries.

Alternative 1

Shadow cache warming from a read replica before promoting a cold Redis shard.

Failure Mode 1

Without a semaphore, 40 pods can still open thousands of waiting coroutines against PgBouncer.

db_sem = asyncio.Semaphore(10)

async def protected_db_fetch(db, l1_cache, key):
    if db_sem.locked() and key in l1_cache:
        return l1_cache[key]  # Shed load to L1 stale
    async with db_sem:
        return await asyncio.wait_for(db.fetch_product(key), timeout=0.5)
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

XFetch triggers a probabilistic refresh.

XFetch triggers a probabilistic refresh before TTL expiry weighted by the key's recompute duration delta, ensuring the hottest keys refresh early without waiting for an exact soft_exp boundary.

ChoiceWhat it buysWhat it costs
Trade-off 1Self-adapts to slow DB queries (`delta`) with zero extra Redis locks, though `beta > 1` trades slightly more refreshes for zero misses.

Step 1

Store delta (last DB fetch duration in seconds) alongside expiry in Redis.

Step 2

On read, evaluate time.time() - delta * beta * math.log(random.random()) >= expiry.

Step 3

Because -ln(rand()) is positive, higher delta or closer expiry increases refresh probability smoothly.

Alternative 1

Fixed soft_exp + singleflight (simpler to debug).

Failure Mode 1

If delta_sec is 0 or unrecorded, XFetch degrades to standard hard-TTL expiration.

import math, random, time

def should_xfetch_refresh(expiry: float, delta_sec: float, beta: float = 1.0) -> bool:
    # Random value in (0, 1]; -ln(u) is exponential distribution
    u = max(random.random(), 1e-9)
    return (time.time() - delta_sec * beta * math.log(u)) >= expiry

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Break TTL synchronization with randomized jitter (base_ttl + random(0, 20%)), coalesce concurrent misses via singleflight, and serve stale-while-revalidate fallbacks using logical expiration metadata.
  • 02Trade-off tip: L1 + L2 adds minor stale-window variance across pods (up to 30s) but cuts Redis network load by ~45%.
  • 03Watch out for: Without a semaphore, 40 pods can still open thousands of waiting coroutines against PgBouncer.
Behavioral Progression · Preventing Cache Avalanche Ttl Jitter

Preventing Cache Avalanche Ttl Jitter — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Break TTL synchronization with randomized jitter (basettl + random(0, 20%)), coalesce concurrent misses via singleflight, and serve stale-while-revalidate fallbacks using logical expiration metadata.

Synthesis / One connected design

Putting Preventing Cache Avalanche with TTL Jitter Together

Break TTL synchronization with randomized jitter (`base_ttl + random(0, 20%)`), coalesce concurrent misses via `singleflight`, and serve `stale-while-revalidate` fallbacks using logical expiration metadata.

A candidate architecture, not a universal answer

Putting Preventing Cache Avalanche with TTL Jitter Together

Step 1
Step 2
Step 3
  1. Store {"data": val, "soft_exp": now + base_ttl + jitter} with Redis ex = base_ttl * 2 + jitter.
  2. If now > soft_exp on hit, return data immediately and trigger singleflight background refresh.
  3. On hard miss, await the shared inflight[key] Future.
L1 in-memory LRU absorbs immediate traffic, PgBouncer + DB semaphore caps concurrent origin queries, and a circuit breaker serves stale L1 or degraded responses when DB latency exceeds 150ms.
Interview Readiness Check

L1 in-memory LRU absorbs immediate traffic, PgBouncer + DB semaphore caps concurrent origin queries, and a circuit breaker serves stale L1 or degraded responses when DB latency exceeds 150ms.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Preventing Cache Avalanche with TTL Jitter

Break TTL synchronization with randomized jitter (base_ttl + random(0, 20%)), coalesce concurrent misses via singleflight, and serve stale-while-revalidate fallbacks using logical expiration metadata.