Your user profile service uses classic Cache-Aside against PostgreSQL 16 and Redis 7 (30,000 reads/sec, 800 writes/sec). Writers execute `UPDATE profiles ...` followed by `redis.del(key)`. Readers on a cache miss execute `SELECT ...` and then `redis.set(key, val, ex=3600)`. Under concurrent load, ~0.05% of profile updates (~35/day) revert to old values in Redis for up to 1 hour because a concurrent cache-miss reader fetched `v1` before the writer committed `v2`, paused on GC/network, and wrote `v1` into Redis after the writer's `DEL`. Redesign the read/write caching protocol to make stale cache repopulation impossible.
From patterns to decisions.
Your user profile service uses classic Cache-Aside against PostgreSQL 16 and Redis 7 (30,000 reads/sec, 800 writes/sec). Writers execute `UPDATE profiles ...` followed by `redis.del(key)`. Readers on a cache miss execute `SELECT ...` and then `redis.set(key, val, ex=3600)`. Under concurrent load, ~0.05% of profile updates (~35/day) revert to old values in Redis for up to 1 hour because a concurrent cache-miss reader fetched `v1` before the writer committed `v2`, paused on GC/network, and wrote `v1` into Redis after the writer's `DEL`. Redesign the read/write caching protocol to make stale cache repopulation impossible.
A reader misses Redis, reads.
A reader misses Redis, reads v1 from DB, pauses while a writer commits v2 and deletes Redis, and then wakes up to write v1 into the empty Redis key.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Bare `DEL` is simple but has no memory of `v2`, leaving the door open to delayed readers. |
T1
Reader misses Redis.
T2
Reader queries DB and gets v1.
T3
Writer commits v2 to DB and runs DEL key in Redis.
T4
Reader wakes up and runs SET key v1 EX 3600, pinning stale data for 1 hour.
Alternative 1
Write-through cache or CDC-only population.
Failure Mode 1
35+ silent data reversions per day under 30k QPS.
# Why bare DEL fails:
# Thread A (Reader): val = db.get(id) -> v1
# Thread B (Writer): db.update(id, v2); redis.delete(key)
# Thread A (Reader): redis.set(key, v1, ex=3600) # Stale v1 wins!
Use PostgreSQL UPDATE.
Use PostgreSQL UPDATE ... SET version = version + 1 RETURNING version and an atomic Redis Lua script that only writes when incoming_ver > cur_ver.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Zero distributed locks on reads while guaranteeing 100% monotonic cache state. |
Step 1
Register LUA_CAS_SET on Redis startup.
Step 2
On write, commit DB with version = version + 1 and call cas_set(key, new_ver, payload, 3600, 0) (or tombstone).
Step 3
On read miss, fetch (payload, ver) from DB and call cas_set.
Alternative 1
Delayed Double Delete (heuristic only).
Failure Mode 1
Using tonumber(cur) > tonumber(ARGV[1]) instead of >= allows an older tombstone to be overwritten by the same version.
LUA_CAS_SET = """
local cur = redis.call('HGET', KEYS[1], 'ver')
if cur and tonumber(cur) >= tonumber(ARGV[1]) then
return 0 -- Reject stale or duplicate version
end
redis.call('HSET', KEYS[1], 'ver', ARGV[1], 'data', ARGV[2], 'tomb', ARGV[3])
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))
return 1
"""
def update_profile(db, redis, cas_sha, user_id, new_fields):
row = db.execute(
"UPDATE profiles SET bio = %s, version = version + 1 WHERE id = %s RETURNING version, bio",
(new_fields['bio'], user_id)
).fetchone()
# Stamp v2 directly (or tombstone='1' with ttl=30)
redis.evalsha(cas_sha, 1, f"profile:{user_id}", row.version, row.bio, "0", 3600)
Delayed Double Delete relies on.
Delayed Double Delete relies on a heuristic sleep (500ms) that fails whenever a reader pauses longer than 500ms (e.g., GC pause or slow replica) and adds 500ms latency or async queue complexity.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Delayed Double Delete requires no DB schema changes, but leaves a probabilistic gap under >500ms stalls. |
Step 1
Compare heuristic timing (500ms window) vs deterministic version comparison (ver1 < ver2).
Step 2
Note that Delayed Double Delete also fails to order two concurrent writers (v2 vs v3) if their second DELs reorder.
Step 3
Choose Versioned Lua CAS when a version column can be added to the DB.
Alternative 1
Debezium CDC single-writer cache population.
Failure Mode 1
Process crash during the 500ms sleep skips the second DEL.
// Delayed Double Delete (heuristic fallback when DB schema cannot be altered)
public void updateWithDoubleDelete(long id, ProfileDto dto) {
db.update(id, dto);
redis.del("profile:" + id);
scheduler.schedule(() -> redis.del("profile:" + id), 500, TimeUnit.MILLISECONDS);
}
If the replica is 400ms.
If the replica is 400ms behind, the miss-reader fetches v1 even after v2 committed; if a tombstone is present, cas_set rejects v1, and the reader must fall back to the primary.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Routes only post-write misses to the primary, keeping 99%+ of reads off the primary. |
Step 1
If Redis returned tombstone=1 (meaning a recent write occurred), route the miss-read directly to the primary DB.
Alternatively, pass `cas_set` result
if cas_set returns 0, re-read Redis to get v2.
Step 3
Keep replica lag monitored and alert above 100ms.
Alternative 1
Store min_lsn in Redis and wait on replica pg_last_wal_replay_lsn().
Failure Mode 1
Reading v1 from a lagging replica when Redis was cold (no tombstone) still requires a safety TTL.
def read_profile(db_primary, db_replica, redis, cas_sha, user_id):
cached = redis.hgetall(f"profile:{user_id}")
if cached and cached.get(b'tomb') == b'0':
return cached[b'data'].decode()
# Tombstone active -> bypass lagging replica and hit primary!
db = db_primary if cached.get(b'tomb') == b'1' else db_replica
row = db.fetch_profile(user_id)
if redis.evalsha(cas_sha, 1, f"profile:{user_id}", row.version, row.bio, "0", 3600) == 0:
return redis.hget(f"profile:{user_id}", "data").decode()
return row.bio
When Debezium -> Kafka is.
When Debezium -> Kafka is the sole writer to Redis for a keyspace (or readers only read, never write back), there are no concurrent reader-writer races.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Zero race conditions, but introduces ~50-150ms eventual consistency lag after `COMMIT`. |
Step 1
Route all DB mutations through PostgreSQL WAL -> Debezium -> Kafka (partitioned by user_id).
Step 2
A single consumer per partition updates Redis sequentially in exact commit LSN order.
Step 3
Read-only API pods never SET Redis on a miss—they only query DB on miss.
Alternative 1
Keep Cache-Aside + Lua CAS when read-your-own-writes (<5ms) is mandatory.
Failure Mode 1
If an API pod also runs redis.set() on miss without Lua CAS, the race reappears.
# Single-writer CDC pipeline removes reader-SET race completely
kafka_topic: pg.public.profiles
partition_key: id # Guarantees strict LSN order per user_id
redis_ttl: 3600 # Refreshed by CDC on every commit
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Eliminate Cache-Aside read-write races using monotonic row
versionCompare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream. - 02Trade-off tip: Delayed Double Delete requires no DB schema changes, but leaves a probabilistic gap under >500ms stalls.
- 03Watch out for: Reading
v1from a lagging replica when Redis was cold (no tombstone) still requires a safety TTL.
Solving Dual Write Cache Invalidation Races — Normal → Load / Fault → Recovery
Eliminate Cache-Aside read-write races using monotonic row version Compare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream.
Putting Solving Dual-Write Cache Invalidation Races Together
Eliminate Cache-Aside read-write races using monotonic row `version` Compare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream.
Putting Solving Dual-Write Cache Invalidation Races Together
- Register
LUA_CAS_SETon Redis startup. - On write, commit DB with
version = version + 1and callcas_set(key, new_ver, payload, 3600, 0)(or tombstone). - On read miss, fetch
(payload, ver)from DB and callcas_set.
Interview Readiness Check
If the replica is 400ms behind, the miss-reader fetches v1 even after v2 committed; if a tombstone is present, cas_set rejects v1, and the reader must fall back to the primary.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Solving Dual-Write Cache Invalidation Races
Eliminate Cache-Aside read-write races using monotonic row version Compare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream.
