Your product-pricing microservice runs 40 pods, each with an L1 in-process Caffeine cache (TTL = 5 min, 100k entries, 82% hit rate) backed by an L2 Redis Cluster (TTL = 30 min, 16% hit rate) and PostgreSQL (2% miss). When a merchant updates a product price in PostgreSQL, the service deletes the key from L2 Redis (`DEL`). However, the other 39 pods continue serving the old price from L1 for up to 5 minutes—causing checkout price mismatches. Worse, an in-flight read on Pod 12 that read the old price before commit can write that stale value into L1 *after* invalidation. Design a synchronized L1/L2 invalidation and version-fencing architecture.
From patterns to decisions.
Your product-pricing microservice runs 40 pods, each with an L1 in-process Caffeine cache (TTL = 5 min, 100k entries, 82% hit rate) backed by an L2 Redis Cluster (TTL = 30 min, 16% hit rate) and PostgreSQL (2% miss). When a merchant updates a product price in PostgreSQL, the service deletes the key from L2 Redis (`DEL`). However, the other 39 pods continue serving the old price from L1 for up to 5 minutes—causing checkout price mismatches. Worse, an in-flight read on Pod 12 that read the old price before commit can write that stale value into L1 *after* invalidation. Design a synchronized L1/L2 invalidation and version-fencing architecture.
Pod 12 misses L1/L2 and.
Pod 12 misses L1/L2 and reads V1 from DB before Pod 3 commits V2 and calls DEL. Pod 12 then writes V1 into L2 and L1, overwriting the invalidation until TTL expires.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Adding a 50 ms delay + double-delete (`DEL` -> sleep 50ms -> `DEL`) reduces the race window but never closes it completely. |
Order the operations chronologically
Pod12_DB_Read(V1) -> Pod3_DB_Commit(V2) -> Pod3_DEL -> Pod12_Cache_Set(V1).
Step 2
Show how neither DEL nor Pub/Sub alone stops Pod 12's delayed SET.
Alternative 1
CDC (Debezium) tailing Postgres WAL to update Redis in strict commit order.
Failure Mode 1
Stale V1 remains pinned in L2 Redis for the full 30-minute TTL.
# Chronology of stale-fill race:
# t=0ms: Pod 12 reads (price=100, v=1) from Postgres
# t=5ms: Pod 3 commits (price=120, v=2) & DELs Redis
# t=8ms: Pod 12 finishes slow network call and writes (price=100, v=1) to L2 & L1!
Use l1.
Use l1.asMap().compute() for atomic version checks in Caffeine, l1.invalidateAll() on Pub/Sub reconnect, and a Redis Lua script for L2 version fencing.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Flushing all of L1 on reconnect causes a brief 1–2s spike in L2 Redis reads for that pod. |
Step 1
Build Caffeine cache with expireAfterWrite(30, SECONDS).
Subscribe to `cache
inv via Lettuce RedisPubSubAdapter; flush L1 in subscribed()`.
Step 3
Compare version atomically in both L1 compute() and L2 Lua.
Alternative 1
In Python, use cachetools.TTLCache with threading.Lock + redis.asyncio.PubSub.
Failure Mode 1
Not running Pub/Sub on a dedicated connection blocks regular Redis commands.
Cache<String, VersionedPrice> l1 = Caffeine.newBuilder()
.maximumSize(100_000)
.expireAfterWrite(Duration.ofSeconds(30))
.build();
// Version-fenced L1 write
void putL1IfNewer(String key, VersionedPrice incoming) {
l1.asMap().compute(key, (k, existing) ->
(existing == null || incoming.version() > existing.version()) ? incoming : existing);
}
// Pub/Sub listener
public void message(String channel, String msg) {
String[] parts = msg.split(":");
String key = parts[0];
long newVer = Long.parseLong(parts[1]);
l1.asMap().computeIfPresent(key, (k, v) -> v.version() < newVer ? null : v);
}
public void subscribed(String channel, long count) {
l1.invalidateAll(); // Self-heal on connect/reconnect
}
Redis Pub/Sub already exists in.
Redis Pub/Sub already exists in the L2 stack, delivers < 5 ms fan-out without per-pod consumer group management, and the 30s L1 TTL + reconnect flush eliminates the need for durable Kafka offsets.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Redis Pub/Sub does not buffer messages during pod restarts (acceptable because restarted pods start with empty L1). |
Compare operational overhead
40 unique Kafka consumer groups vs 1 Redis SUBSCRIBE.
Step 2
Verify 30s TTL bounds any missed-message window.
Alternative 1
Use Redis 6+ Client-Side Caching (CLIENT TRACKING ON BCAST) if clients support RESP3.
Failure Mode 1
Sharing one Kafka consumer group ID across 40 pods delivers each invalidation to only 1 pod instead of all 40!
# Redis Lua script for version-fenced L2 SET:
LUA_SET_IF_NEWER = """
local cur = redis.call('HGET', KEYS[1], 'version')
if not cur or tonumber(ARGV[1]) > tonumber(cur) then
redis.call('HSET', KEYS[1], 'version', ARGV[1], 'payload', ARGV[2])
redis.call('EXPIRE', KEYS[1], ARGV[3])
return 1
end
return 0
"""
Coalesce bulk invalidations in a.
Coalesce bulk invalidations in a 100 ms buffer or emit a single FLUSH_PREFIX:merchant_42 message when batch size > 500 keys.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Coarse invalidation briefly lowers L1 hit rate for unaffected merchants during large imports. |
Step 1
Detect bulk imports (> 500 keys in 100 ms).
Step 2
Publish a single FLUSH_ALL or FLUSH_MERCHANT:<id> control event instead of 200k individual messages.
Alternative 1
Maintain a merchant-level version epoch in L1 so bumping 1 epoch invalidates all keys for that merchant in O(1).
Failure Mode 1
Unthrottled 200k PUBLISH calls can hit Redis client-output-buffer-limit pubsub and disconnect slow pods.
if (updatedKeys.size() > 500) {
redisPubSub.publish("cache:inv", "__FLUSH_ALL__:0");
} else {
updatedKeys.forEach(k -> redisPubSub.publish("cache:inv", k + ":" + ver));
}
Redis CLIENT TRACKING automatically pushes.
Redis CLIENT TRACKING automatically pushes invalidations only for keys a pod actually read (or prefixes via BCAST), eliminating manual PUBLISH calls, but requires RESP3 and careful Redis memory sizing for the invalidation table.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Without `BCAST`, Redis tracks every client key in a global table, consuming extra Redis RAM. |
Use `CLIENT TRACKING ON BCAST PREFIX catalog
` with RESP3.
Step 2
Remember Redis still needs Lua version-fencing for DB-to-L2 population races.
Alternative 1
Stick to explicit PUBLISH when running Redis Cluster with custom sharding or RESP2 clients.
Failure Mode 1
In Redis Cluster, CLIENT TRACKING must be enabled on every master shard independently.
// Lettuce RESP3 Broadcast Tracking
// CLIENT TRACKING ON BCAST PREFIX price:
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.
- 02Trade-off tip: Redis Pub/Sub does not buffer messages during pod restarts (acceptable because restarted pods start with empty L1).
- 03Watch out for: Unthrottled 200k
PUBLISHcalls can hit Redisclient-output-buffer-limit pubsuband disconnect slow pods.
Synchronizing Multi Tier L1 L2 Caches — Normal → Load / Fault → Recovery
Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.
Putting Synchronizing Multi-Tier L1 & L2 Caches Together
Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.
Putting Synchronizing Multi-Tier L1 & L2 Caches Together
- Build Caffeine cache with
expireAfterWrite(30, SECONDS). - Subscribe to
cache:invvia LettuceRedisPubSubAdapter; flush L1 insubscribed(). - Compare
versionatomically in both L1compute()and L2 Lua.
Interview Readiness Check
Coalesce bulk invalidations in a 100 ms buffer or emit a single FLUSH_PREFIX:merchant_42 message when batch size > 500 keys.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Synchronizing Multi-Tier L1 & L2 Caches
Synchronize L1 and L2 caches by broadcasting invalidations over Redis Pub/Sub (with short 30–60s L1 TTLs as a safety net) and fencing L1/L2 writes with monotonic version tokens to prevent stale fill races.
