Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Solving Dual-Write Cache
Invalidation Races.

Your user profile service uses classic Cache-Aside against PostgreSQL 16 and Redis 7 (30,000 reads/sec, 800 writes/sec). Writers execute UPDATE profiles ... followed by redis.del(key). Readers on a cache miss execute SELECT ... and then redis.set(key, val, ex=3600). Under concurrent load, ~0.05% of profile updates (~35/day) revert to old values in Redis for up to 1 hour because a concurrent cache-miss reader fetched v1 before the writer committed v2, paused on GC/network, and wrote v1 into Redis after the writer's DEL. Redesign the read/write caching protocol to make stale cache repopulation impossible.
Start the journey5 stages
Loading diagram…
Solving Dual-Write Cache Invalidation Races architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your user profile service uses classic Cache-Aside against PostgreSQL 16 and Redis 7 (30,000 reads/sec, 800 writes/sec). Writers execute `UPDATE profiles ...` followed by `redis.del(key)`. Readers on a cache miss execute `SELECT ...` and then `redis.set(key, val, ex=3600)`. Under concurrent load, ~0.05% of profile updates (~35/day) revert to old values in Redis for up to 1 hour because a concurrent cache-miss reader fetched `v1` before the writer committed `v2`, paused on GC/network, and wrote `v1` into Redis after the writer's `DEL`. Redesign the read/write caching protocol to make stale cache repopulation impossible.

From patterns to decisions.

Your user profile service uses classic Cache-Aside against PostgreSQL 16 and Redis 7 (30,000 reads/sec, 800 writes/sec). Writers execute `UPDATE profiles ...` followed by `redis.del(key)`. Readers on a cache miss execute `SELECT ...` and then `redis.set(key, val, ex=3600)`. Under concurrent load, ~0.05% of profile updates (~35/day) revert to old values in Redis for up to 1 hour because a concurrent cache-miss reader fetched `v1` before the writer committed `v2`, paused on GC/network, and wrote `v1` into Redis after the writer's `DEL`. Redesign the read/write caching protocol to make stale cache repopulation impossible.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

A reader misses Redis, reads.

A reader misses Redis, reads v1 from DB, pauses while a writer commits v2 and deletes Redis, and then wakes up to write v1 into the empty Redis key.

ChoiceWhat it buysWhat it costs
Trade-off 1Bare `DEL` is simple but has no memory of `v2`, leaving the door open to delayed readers.

T1

Reader misses Redis.

T2

Reader queries DB and gets v1.

T3

Writer commits v2 to DB and runs DEL key in Redis.

T4

Reader wakes up and runs SET key v1 EX 3600, pinning stale data for 1 hour.

Alternative 1

Write-through cache or CDC-only population.

Failure Mode 1

35+ silent data reversions per day under 30k QPS.

# Why bare DEL fails:
# Thread A (Reader): val = db.get(id)  -> v1
# Thread B (Writer): db.update(id, v2); redis.delete(key)
# Thread A (Reader): redis.set(key, v1, ex=3600)  # Stale v1 wins!
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Use PostgreSQL UPDATE.

Use PostgreSQL UPDATE ... SET version = version + 1 RETURNING version and an atomic Redis Lua script that only writes when incoming_ver > cur_ver.

ChoiceWhat it buysWhat it costs
Trade-off 1Zero distributed locks on reads while guaranteeing 100% monotonic cache state.

Step 1

Register LUA_CAS_SET on Redis startup.

Step 2

On write, commit DB with version = version + 1 and call cas_set(key, new_ver, payload, 3600, 0) (or tombstone).

Step 3

On read miss, fetch (payload, ver) from DB and call cas_set.

Alternative 1

Delayed Double Delete (heuristic only).

Failure Mode 1

Using tonumber(cur) > tonumber(ARGV[1]) instead of >= allows an older tombstone to be overwritten by the same version.

LUA_CAS_SET = """
local cur = redis.call('HGET', KEYS[1], 'ver')
if cur and tonumber(cur) >= tonumber(ARGV[1]) then
    return 0  -- Reject stale or duplicate version
end
redis.call('HSET', KEYS[1], 'ver', ARGV[1], 'data', ARGV[2], 'tomb', ARGV[3])
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))
return 1
"""

def update_profile(db, redis, cas_sha, user_id, new_fields):
    row = db.execute(
        "UPDATE profiles SET bio = %s, version = version + 1 WHERE id = %s RETURNING version, bio",
        (new_fields['bio'], user_id)
    ).fetchone()
    # Stamp v2 directly (or tombstone='1' with ttl=30)
    redis.evalsha(cas_sha, 1, f"profile:{user_id}", row.version, row.bio, "0", 3600)
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Delayed Double Delete relies on.

Delayed Double Delete relies on a heuristic sleep (500ms) that fails whenever a reader pauses longer than 500ms (e.g., GC pause or slow replica) and adds 500ms latency or async queue complexity.

ChoiceWhat it buysWhat it costs
Trade-off 1Delayed Double Delete requires no DB schema changes, but leaves a probabilistic gap under >500ms stalls.

Step 1

Compare heuristic timing (500ms window) vs deterministic version comparison (ver1 < ver2).

Step 2

Note that Delayed Double Delete also fails to order two concurrent writers (v2 vs v3) if their second DELs reorder.

Step 3

Choose Versioned Lua CAS when a version column can be added to the DB.

Alternative 1

Debezium CDC single-writer cache population.

Failure Mode 1

Process crash during the 500ms sleep skips the second DEL.

// Delayed Double Delete (heuristic fallback when DB schema cannot be altered)
public void updateWithDoubleDelete(long id, ProfileDto dto) {
    db.update(id, dto);
    redis.del("profile:" + id);
    scheduler.schedule(() -> redis.del("profile:" + id), 500, TimeUnit.MILLISECONDS);
}
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

If the replica is 400ms.

If the replica is 400ms behind, the miss-reader fetches v1 even after v2 committed; if a tombstone is present, cas_set rejects v1, and the reader must fall back to the primary.

ChoiceWhat it buysWhat it costs
Trade-off 1Routes only post-write misses to the primary, keeping 99%+ of reads off the primary.

Step 1

If Redis returned tombstone=1 (meaning a recent write occurred), route the miss-read directly to the primary DB.

Alternatively, pass `cas_set` result

if cas_set returns 0, re-read Redis to get v2.

Step 3

Keep replica lag monitored and alert above 100ms.

Alternative 1

Store min_lsn in Redis and wait on replica pg_last_wal_replay_lsn().

Failure Mode 1

Reading v1 from a lagging replica when Redis was cold (no tombstone) still requires a safety TTL.

def read_profile(db_primary, db_replica, redis, cas_sha, user_id):
    cached = redis.hgetall(f"profile:{user_id}")
    if cached and cached.get(b'tomb') == b'0':
        return cached[b'data'].decode()
    # Tombstone active -> bypass lagging replica and hit primary!
    db = db_primary if cached.get(b'tomb') == b'1' else db_replica
    row = db.fetch_profile(user_id)
    if redis.evalsha(cas_sha, 1, f"profile:{user_id}", row.version, row.bio, "0", 3600) == 0:
        return redis.hget(f"profile:{user_id}", "data").decode()
    return row.bio
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

When Debezium -> Kafka is.

When Debezium -> Kafka is the sole writer to Redis for a keyspace (or readers only read, never write back), there are no concurrent reader-writer races.

ChoiceWhat it buysWhat it costs
Trade-off 1Zero race conditions, but introduces ~50-150ms eventual consistency lag after `COMMIT`.

Step 1

Route all DB mutations through PostgreSQL WAL -> Debezium -> Kafka (partitioned by user_id).

Step 2

A single consumer per partition updates Redis sequentially in exact commit LSN order.

Step 3

Read-only API pods never SET Redis on a miss—they only query DB on miss.

Alternative 1

Keep Cache-Aside + Lua CAS when read-your-own-writes (<5ms) is mandatory.

Failure Mode 1

If an API pod also runs redis.set() on miss without Lua CAS, the race reappears.

# Single-writer CDC pipeline removes reader-SET race completely
kafka_topic: pg.public.profiles
partition_key: id   # Guarantees strict LSN order per user_id
redis_ttl: 3600     # Refreshed by CDC on every commit

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Eliminate Cache-Aside read-write races using monotonic row version Compare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream.
  • 02Trade-off tip: Delayed Double Delete requires no DB schema changes, but leaves a probabilistic gap under >500ms stalls.
  • 03Watch out for: Reading v1 from a lagging replica when Redis was cold (no tombstone) still requires a safety TTL.
Behavioral Progression · Solving Dual Write Cache Invalidation Races

Solving Dual Write Cache Invalidation Races — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Eliminate Cache-Aside read-write races using monotonic row version Compare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream.

Synthesis / One connected design

Putting Solving Dual-Write Cache Invalidation Races Together

Eliminate Cache-Aside read-write races using monotonic row `version` Compare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream.

A candidate architecture, not a universal answer

Putting Solving Dual-Write Cache Invalidation Races Together

Step 1
Step 2
Step 3
  1. Register LUA_CAS_SET on Redis startup.
  2. On write, commit DB with version = version + 1 and call cas_set(key, new_ver, payload, 3600, 0) (or tombstone).
  3. On read miss, fetch (payload, ver) from DB and call cas_set.
If the replica is 400ms behind, the miss-reader fetches `v1` even *after* `v2` committed; if a tombstone is present, `cas_set` rejects `v1`, and the reader must fall back to the primary.
Interview Readiness Check

If the replica is 400ms behind, the miss-reader fetches v1 even after v2 committed; if a tombstone is present, cas_set rejects v1, and the reader must fall back to the primary.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Solving Dual-Write Cache Invalidation Races

Eliminate Cache-Aside read-write races using monotonic row version Compare-And-Swap (Lua CAS) in Redis or by moving cache population/invalidation to a single-writer CDC stream.