Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Synchronizing Multi-Tier L1
& L2 Caches.

Your product-pricing microservice runs 40 pods, each with an L1 in-process Caffeine cache (TTL = 5 min, 100k entries, 82% hit rate) backed by an L2 Redis Cluster (TTL = 30 min, 16% hit rate) and PostgreSQL (2% miss). When a merchant updates a product price in PostgreSQL, the service deletes the key from L2 Redis (DEL). However, the other 39 pods continue serving the old price from L1 for up to 5 minutes—causing checkout price mismatches. Worse, an in-flight read on Pod 12 that read the old price before commit can write that stale value into L1 after invalidation. Design a synchronized L1/L2 invalidation and version-fencing architecture.
Start the journey5 stages
Loading diagram…
Synchronizing Multi-Tier L1 & L2 Caches architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your product-pricing microservice runs 40 pods, each with an L1 in-process Caffeine cache (TTL = 5 min, 100k entries, 82% hit rate) backed by an L2 Redis Cluster (TTL = 30 min, 16% hit rate) and PostgreSQL (2% miss). When a merchant updates a product price in PostgreSQL, the service deletes the key from L2 Redis (`DEL`). However, the other 39 pods continue serving the old price from L1 for up to 5 minutes—causing checkout price mismatches. Worse, an in-flight read on Pod 12 that read the old price before commit can write that stale value into L1 *after* invalidation. Design a synchronized L1/L2 invalidation and version-fencing architecture.

From patterns to decisions.

Your product-pricing microservice runs 40 pods, each with an L1 in-process Caffeine cache (TTL = 5 min, 100k entries, 82% hit rate) backed by an L2 Redis Cluster (TTL = 30 min, 16% hit rate) and PostgreSQL (2% miss). When a merchant updates a product price in PostgreSQL, the service deletes the key from L2 Redis (`DEL`). However, the other 39 pods continue serving the old price from L1 for up to 5 minutes—causing checkout price mismatches. Worse, an in-flight read on Pod 12 that read the old price before commit can write that stale value into L1 *after* invalidation. Design a synchronized L1/L2 invalidation and version-fencing architecture.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Pod 12 misses L1/L2 and.

Pod 12 misses L1/L2 and reads V1 from DB before Pod 3 commits V2 and calls DEL. Pod 12 then writes V1 into L2 and L1, overwriting the invalidation until TTL expires.

ChoiceWhat it buysWhat it costs
Trade-off 1Adding a 50 ms delay + double-delete (`DEL` -> sleep 50ms -> `DEL`) reduces the race window but never closes it completely.

Order the operations chronologically

Pod12_DB_Read(V1) -> Pod3_DB_Commit(V2) -> Pod3_DEL -> Pod12_Cache_Set(V1).

Step 2

Show how neither DEL nor Pub/Sub alone stops Pod 12's delayed SET.

Alternative 1

CDC (Debezium) tailing Postgres WAL to update Redis in strict commit order.

Failure Mode 1

Stale V1 remains pinned in L2 Redis for the full 30-minute TTL.

# Chronology of stale-fill race:
# t=0ms: Pod 12 reads (price=100, v=1) from Postgres
# t=5ms: Pod 3 commits (price=120, v=2) & DELs Redis
# t=8ms: Pod 12 finishes slow network call and writes (price=100, v=1) to L2 & L1!
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Use l1.

Use l1.asMap().compute() for atomic version checks in Caffeine, l1.invalidateAll() on Pub/Sub reconnect, and a Redis Lua script for L2 version fencing.

ChoiceWhat it buysWhat it costs
Trade-off 1Flushing all of L1 on reconnect causes a brief 1–2s spike in L2 Redis reads for that pod.

Step 1

Build Caffeine cache with expireAfterWrite(30, SECONDS).

Subscribe to `cache

inv via Lettuce RedisPubSubAdapter; flush L1 in subscribed()`.

Step 3

Compare version atomically in both L1 compute() and L2 Lua.

Alternative 1

In Python, use cachetools.TTLCache with threading.Lock + redis.asyncio.PubSub.

Failure Mode 1

Not running Pub/Sub on a dedicated connection blocks regular Redis commands.

Cache<String, VersionedPrice> l1 = Caffeine.newBuilder()
    .maximumSize(100_000)
    .expireAfterWrite(Duration.ofSeconds(30))
    .build();

// Version-fenced L1 write
void putL1IfNewer(String key, VersionedPrice incoming) {
    l1.asMap().compute(key, (k, existing) ->
        (existing == null || incoming.version() > existing.version()) ? incoming : existing);
}

// Pub/Sub listener
public void message(String channel, String msg) {
    String[] parts = msg.split(":");
    String key = parts[0];
    long newVer = Long.parseLong(parts[1]);
    l1.asMap().computeIfPresent(key, (k, v) -> v.version() < newVer ? null : v);
}

public void subscribed(String channel, long count) {
    l1.invalidateAll(); // Self-heal on connect/reconnect
}
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Redis Pub/Sub already exists in.

Redis Pub/Sub already exists in the L2 stack, delivers < 5 ms fan-out without per-pod consumer group management, and the 30s L1 TTL + reconnect flush eliminates the need for durable Kafka offsets.

ChoiceWhat it buysWhat it costs
Trade-off 1Redis Pub/Sub does not buffer messages during pod restarts (acceptable because restarted pods start with empty L1).

Compare operational overhead

40 unique Kafka consumer groups vs 1 Redis SUBSCRIBE.

Step 2

Verify 30s TTL bounds any missed-message window.

Alternative 1

Use Redis 6+ Client-Side Caching (CLIENT TRACKING ON BCAST) if clients support RESP3.

Failure Mode 1

Sharing one Kafka consumer group ID across 40 pods delivers each invalidation to only 1 pod instead of all 40!

# Redis Lua script for version-fenced L2 SET:
LUA_SET_IF_NEWER = """
local cur = redis.call('HGET', KEYS[1], 'version')
if not cur or tonumber(ARGV[1]) > tonumber(cur) then
    redis.call('HSET', KEYS[1], 'version', ARGV[1], 'payload', ARGV[2])
    redis.call('EXPIRE', KEYS[1], ARGV[3])
    return 1
end
return 0
"""
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Coalesce bulk invalidations in a.

Coalesce bulk invalidations in a 100 ms buffer or emit a single FLUSH_PREFIX:merchant_42 message when batch size > 500 keys.

ChoiceWhat it buysWhat it costs
Trade-off 1Coarse invalidation briefly lowers L1 hit rate for unaffected merchants during large imports.

Step 1

Detect bulk imports (> 500 keys in 100 ms).

Step 2

Publish a single FLUSH_ALL or FLUSH_MERCHANT:<id> control event instead of 200k individual messages.

Alternative 1

Maintain a merchant-level version epoch in L1 so bumping 1 epoch invalidates all keys for that merchant in O(1).

Failure Mode 1

Unthrottled 200k PUBLISH calls can hit Redis client-output-buffer-limit pubsub and disconnect slow pods.

if (updatedKeys.size() > 500) {
    redisPubSub.publish("cache:inv", "__FLUSH_ALL__:0");
} else {
    updatedKeys.forEach(k -> redisPubSub.publish("cache:inv", k + ":" + ver));
}
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Redis CLIENT TRACKING automatically pushes.

Redis CLIENT TRACKING automatically pushes invalidations only for keys a pod actually read (or prefixes via BCAST), eliminating manual PUBLISH calls, but requires RESP3 and careful Redis memory sizing for the invalidation table.

ChoiceWhat it buysWhat it costs
Trade-off 1Without `BCAST`, Redis tracks every client key in a global table, consuming extra Redis RAM.

Use `CLIENT TRACKING ON BCAST PREFIX catalog

` with RESP3.

Step 2

Remember Redis still needs Lua version-fencing for DB-to-L2 population races.

Alternative 1

Stick to explicit PUBLISH when running Redis Cluster with custom sharding or RESP2 clients.

Failure Mode 1

In Redis Cluster, CLIENT TRACKING must be enabled on every master shard independently.

// Lettuce RESP3 Broadcast Tracking
// CLIENT TRACKING ON BCAST PREFIX price:

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.
  • 02Trade-off tip: Redis Pub/Sub does not buffer messages during pod restarts (acceptable because restarted pods start with empty L1).
  • 03Watch out for: Unthrottled 200k PUBLISH calls can hit Redis client-output-buffer-limit pubsub and disconnect slow pods.
Behavioral Progression · Synchronizing Multi Tier L1 L2 Caches

Synchronizing Multi Tier L1 L2 Caches — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.

Synthesis / One connected design

Putting Synchronizing Multi-Tier L1 & L2 Caches Together

Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.

A candidate architecture, not a universal answer

Putting Synchronizing Multi-Tier L1 & L2 Caches Together

Step 1
Step 2
Step 3
  1. Build Caffeine cache with expireAfterWrite(30, SECONDS).
  2. Subscribe to cache:inv via Lettuce RedisPubSubAdapter; flush L1 in subscribed().
  3. Compare version atomically in both L1 compute() and L2 Lua.
Coalesce bulk invalidations in a 100 ms buffer or emit a single `FLUSH_PREFIX:merchant_42` message when batch size > 500 keys.
Interview Readiness Check

Coalesce bulk invalidations in a 100 ms buffer or emit a single FLUSH_PREFIX:merchant_42 message when batch size > 500 keys.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Synchronizing Multi-Tier L1 & L2 Caches

Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.