A distributed feature-flag and session-metadata store uses a Dynamo-style quorum configuration with `N=3` replicas (`R1, R2, R3`), write quorum `W=2`, and read quorum `R=2` (`R + W = 4 > 3`). During a partial rack degradation, `R3` experiences 45% packet loss and `R2` suffers an abrupt reboot, wiping its volatile memtable/unflushed WAL or recovering from a 10-minute-old EBS snapshot. Meanwhile, the coordinator has `sloppy_quorum = true` (hinted handoff) enabled so writes succeed on `{R1, HintedR4}` when `R2` and `R3` are slow. Immediately afterward, `R2` rejoins the ring and a client executes an `R=2` read that contacts `{R2, R3}`, receiving `v1` instead of the newly committed `v2`. Design the consistency protocol, recovery gating, and read-repair mechanisms to eliminate stale reads under partial replica degradation.
From patterns to decisions.
A distributed feature-flag and session-metadata store uses a Dynamo-style quorum configuration with `N=3` replicas (`R1, R2, R3`), write quorum `W=2`, and read quorum `R=2` (`R + W = 4 > 3`). During a partial rack degradation, `R3` experiences 45% packet loss and `R2` suffers an abrupt reboot, wiping its volatile memtable/unflushed WAL or recovering from a 10-minute-old EBS snapshot. Meanwhile, the coordinator has `sloppy_quorum = true` (hinted handoff) enabled so writes succeed on `{R1, HintedR4}` when `R2` and `R3` are slow. Immediately afterward, `R2` rejoins the ring and a client executes an `R=2` read that contacts `{R2, R3}`, receiving `v1` instead of the newly committed `v2`. Design the consistency protocol, recovery gating, and read-repair mechanisms to eliminate stale reads under partial replica degradation.
Both sloppy quorum (HintedR4) and.
Both sloppy quorum (HintedR4) and an amnesiac reboot of R2 reduced the number of canonical replicas holding v2 below N - R + 1 (which must be >= 2 for R=2 to guarantee intersection).
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Disabling sloppy quorum sacrifices write availability when 2 of 3 replicas fail simultaneously. |
Step 1
Show that with N=3, R=2, at least 2 canonical replicas must durably hold v2 for every 2-node read subset to intersect v2.
Step 2
With sloppy quorum, v2 landed on {R1, R4}, leaving only 1 canonical replica (R1) with v2.
Step 3
With R2 rebooting from a stale snapshot, {R2, R3} had zero intersection with R1.
Alternative 1
Use Raft/Paxos where a leader only commits after a strict majority of canonical voters fsync.
Failure Mode 1
Treating hinted handoff nodes as equivalent to canonical replicas in W ack counting.
# Quorum intersection check
N = 3
W = 2
R = 2
# Pigeonhole principle: canonical_replicas_with_v2 + R > N => must be >= 2
assert W + R > N
canonical_with_v2 = 1 # only R1 if R4 was a sloppy hint!
assert canonical_with_v2 + R <= N # intersection is NOT guaranteed (1 + 2 <= 3)
Fetch 1 full value +.
Fetch 1 full value + R-1 digests from ACTIVE canonical replicas; if digests differ, fetch all full values, pick the highest HLC/version, and synchronously repair stale replicas before returning.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Blocking read repair increases tail latency on divergent reads by 1 extra RTT, but prevents stale-read regressions on subsequent reads. |
Step 1
Filter replica list to ACTIVE nodes only (excluding CATCHUP nodes).
Step 2
Send 1 get_full and R-1 get_digest calls in parallel.
Step 3
If all digests match, return immediately (~1.2 ms).
Step 4
On mismatch, fetch full payloads, resolve winner by (hlc_ts, node_id), write winner to stale replicas, and await repair acks.
Alternative 1
Use Raft ReadIndex if strict linearizability is required without read-repair races.
Failure Mode 1
If the repair write to R3 times out due to 45% packet loss, failing the read or returning without repair must be explicitly decided per SLA.
from concurrent.futures import ThreadPoolExecutor
def quorum_read(key: str, active_replicas: list, r_quorum: int = 2):
if len(active_replicas) < r_quorum:
raise RuntimeError("Insufficient ACTIVE replicas for strict R quorum")
targets = active_replicas[:r_quorum]
with ThreadPoolExecutor(max_workers=r_quorum) as pool:
full_fut = pool.submit(targets[0].get_full, key)
digest_futs = [pool.submit(rep.get_digest, key) for rep in targets[1:]]
winner = full_fut.result(timeout=0.01)
digests = [f.result(timeout=0.01) for f in digest_futs]
if all(d == winner.digest for d in digests):
return winner.value
# Digest mismatch: fetch full values and perform blocking read repair
responses = [winner] + [rep.get_full(key) for rep in targets[1:]]
newest = max(responses, key=lambda rec: (rec.hlc_ts, rec.version))
for rep, rec in zip(targets, responses):
if rec.digest != newest.digest:
rep.repair_write(key, newest) # synchronous before returning
return newest.value
For feature flags and session.
For feature flags and session metadata (5,000 WPS, small key-value payloads), Raft/Paxos is almost always superior because it gives true linearizability without LWW clock-skew bugs or read-repair complexity.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Paxos/LWT (`SERIAL`) requires 2–4 RTTs on writes, increasing write latency from ~2ms to ~6ms, while guaranteeing linearizability. |
Compare write volume
5,000 WPS across sharded Raft ranges (e.g., 64 ranges = ~78 WPS/range) is trivial for Raft.
Step 2
Note that Raft follower reads with leader lease or ReadIndex achieve < 12 ms p99.
Step 3
Reserve leaderless Dynamo quorums for massive multi-region write-heavy workloads where AP availability outweighs strict linearizability.
Alternative 1
etcd / Consul for <10k keys; CockroachDB / Spanner for larger relational datasets.
Failure Mode 1
Mixing SERIAL writes with LOCAL_ONE reads still returns stale data.
# Example Cassandra/ScyllaDB LWT (Paxos) vs QUORUM
# For strict linearizability on critical flags:
write_consistency: SERIAL # Paxos prepare/propose
read_consistency: SERIAL # Linearizable read
sloppy_quorum: false
hinted_handoff_enabled: false
Store a unique generation_id /.
Store a unique generation_id / node_epoch in both local disk and the cluster's gossip/metadata ring; a mismatch or missing clean-shutdown marker forces CATCHUP mode.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requiring Merkle sync on every dirty restart adds 10–60s to node recovery time, but eliminates amnesiac stale reads. |
Step 1
On clean shutdown, flush memtables and write a clean_shutdown_seq to local disk.
Step 2
On boot, compare local last_committed_hlc against peers' min_committed_hlc.
Step 3
If behind by > threshold or clean_shutdown flag is absent, stay in CATCHUP and run Merkle tree sync before advertising ACTIVE.
Alternative 1
Require manual nodetool / admin token replacement for any EBS snapshot restore.
Failure Mode 1
If all 3 nodes reboot simultaneously, gating all 3 in CATCHUP deadlocks the cluster unless a majority quorum override exists.
def should_enter_catchup(local_max_hlc: int, peer_max_hlc: int, clean_shutdown: bool) -> bool:
if not clean_shutdown:
return True
# Even on clean shutdown, if node was offline > 5s, require Merkle sync
return (peer_max_hlc - local_max_hlc) > 5_000_000 # 5s in microseconds
Yes, during an in-flight write.
Yes, during an in-flight write before W=2 finishes: Client A may read v2 from R1, while Client B reads v1 from {R2, R3} before R2 receives v2—unless reads also trigger blocking read repair or use consensus.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | ABD write-back turns every divergent read into a quorum write, doubling read latency during concurrent updates. |
Trace
Write v2 reaches R1 first (not yet R2). Client A reads {R1, R2} -> sees v2 and repairs R2 (or doesn't if async).
Step 2
With blocking read repair, Client A's read repairs R2 before returning v2 to Client A, so any subsequent Client B read {R2, R3} sees v2.
Step 3
Without blocking read repair, Client B can read v1 after Client A read v2 (violating linearizability).
Alternative 1
Use Raft/Paxos leader leases for single-RTT linearizable reads.
Failure Mode 1
Assuming async read repair provides linearizability.
# ABD (Attiya-Bar-Noy-Dolev) algorithm summary:
# A quorum read must write back (phase 2) the winning value to W replicas
# BEFORE returning to the caller to guarantee linearizability in shared-memory quorums.
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Strict
R + W > Nonly guarantees overlap when replica membership is static and durable; production quorum stores require disabling sloppy quorum for strong reads, monotonic/read-your-writes tokens, and epoch-gated replica recovery. - 02Trade-off tip: Paxos/LWT (
SERIAL) requires 2–4 RTTs on writes, increasing write latency from ~2ms to ~6ms, while guaranteeing linearizability. - 03Watch out for: If all 3 nodes reboot simultaneously, gating all 3 in
CATCHUPdeadlocks the cluster unless a majority quorum override exists.
Quorum Reads Writes And Sloppy Quorums — Normal → Load / Fault → Recovery
Strict R + W > N only guarantees overlap when replica membership is static and durable; production quorum stores require disabling sloppy quorum for strong reads, monotonic/read-your-writes tokens, and epoch-gated replica recovery.
Putting Quorum Consistency & Stale Reads in Replicated Stores Together
Strict `R + W > N` only guarantees overlap when replica membership is static and durable; production quorum stores require disabling sloppy quorum for strong reads, monotonic/read-your-writes tokens, and epoch-gated replica recovery.
Putting Quorum Consistency & Stale Reads in Replicated Stores Together
- Filter replica list to
ACTIVEnodes only (excludingCATCHUPnodes). - Send 1
get_fullandR-1get_digestcalls in parallel. - If all digests match, return immediately (~1.2 ms).
- On mismatch, fetch full payloads, resolve winner by
(hlc_ts, node_id), write winner to stale replicas, and await repair acks.
Interview Readiness Check
Store a unique generation_id / node_epoch in both local disk and the cluster's gossip/metadata ring; a mismatch or missing clean-shutdown marker forces CATCHUP mode.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Quorum Consistency & Stale Reads in Replicated Stores
Strict R + W > N only guarantees overlap when replica membership is static and durable; production quorum stores require disabling sloppy quorum for strong reads, monotonic/read-your-writes tokens, and epoch-gated replica recovery.
