Your 3-node metadata control plane (Nodes A, B, C across us-east-1a, 1b, 1c) runs a Raft-backed coordination service handling 4,500 lock acquisitions/sec at p99 < 12ms. A top-of-rack BGP/ACL misconfiguration drops all TCP/UDP packets between AZ-1a (Node A) and AZ-1c (Node C), while A<->B and B<->C links remain healthy at 0.8ms RTT. Within 500ms, Node C's election timeout fires, it increments its Raft term, and Node B votes for C—disrupting Leader A, which then times out and bumps its term again, causing continuous term inflation and 100% lock-service unavailability. Redesign the failure detection, consensus election protocol, and downstream fencing to survive non-transitive partitions without livelock or split-brain writes.
From patterns to decisions.
Your 3-node metadata control plane (Nodes A, B, C across us-east-1a, 1b, 1c) runs a Raft-backed coordination service handling 4,500 lock acquisitions/sec at p99 < 12ms. A top-of-rack BGP/ACL misconfiguration drops all TCP/UDP packets between AZ-1a (Node A) and AZ-1c (Node C), while A<->B and B<->C links remain healthy at 0.8ms RTT. Within 500ms, Node C's election timeout fires, it increments its Raft term, and Node B votes for C—disrupting Leader A, which then times out and bumps its term again, causing continuous term inflation and 100% lock-service unavailability. Redesign the failure detection, consensus election protocol, and downstream fencing to survive non-transitive partitions without livelock or split-brain writes.
In standard Raft, any follower.
In standard Raft, any follower that misses heartbeats increments its term and broadcasts RequestVote; a peer receiving a higher term must update its term and force the current leader to step down, even if the leader is still connected to a majority.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Standard Raft prioritizes fast election on any higher term at the cost of vulnerability to partially connected candidates. |
Trace heartbeats from Leader A
B receives them (timer resets), C does not (A<->C dropped).
Trace C's election timeout
C increments term to T+1 and sends RequestVote(T+1) to B.
Observe B's reaction
B updates its term to T+1, which propagates to A and forces A to step down, restarting elections endlessly.
Alternative 1
Wait for a full lease expiry before accepting RequestVote (slower failover).
Failure Mode 1
100% control-plane unavailability due to continuous term bumps every 500ms.
def on_request_vote_naive(node, req_term, candidate_id):
if req_term > node.current_term:
# Naive Raft steps down immediately even if Leader A is healthy!
node.current_term = req_term
node.state = 'FOLLOWER'
node.voted_for = None
# Causes A and C to ping-pong terms indefinitely via bridge B
Combine Pre-Vote (deny votes if.
Combine Pre-Vote (deny votes if leader heartbeat was seen within min_election_timeout) with CheckQuorum (step down if < majority ACKs within election_timeout) and monotonic fencing tokens.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Adds 1 RTT to legitimate failovers (~0.8ms intra-region) in exchange for complete immunity to asymmetric term storms. |
Step 1
Add last_leader_heartbeat_ms tracking on every follower.
Step 2
Require PreVote to pass before incrementing current_term.
Step 3
Run check_quorum() on the leader every 100ms.
Step 4
Attach (term << 32) | log_index as fencing_token on all downstream DB writes.
Alternative 1
SWIM indirect probing (ping-req) for non-consensus membership rings.
Failure Mode 1
Clock jumps if time.time() is used instead of time.monotonic().
import time
def handle_pre_vote(node, next_term, last_log_idx, last_log_term):
now_ms = time.monotonic() * 1000
# Reject if we recently heard from a valid leader (prevents C disrupting A via B)
if (now_ms - node.last_leader_heartbeat_ms) < node.min_election_timeout_ms:
return {"term": node.current_term, "vote_granted": False}
if next_term <= node.current_term:
return {"term": node.current_term, "vote_granted": False}
if not node.is_log_up_to_date(last_log_idx, last_log_term):
return {"term": node.current_term, "vote_granted": False}
return {"term": node.current_term, "vote_granted": True}
def check_quorum(leader):
now_ms = time.monotonic() * 1000
active_peers = 1 + sum(
1 for ts in leader.peer_last_ack_ms.values()
if (now_ms - ts) < leader.election_timeout_ms
)
if active_peers < (leader.cluster_size // 2) + 1:
leader.step_down_to_follower()
Implementation & Guardrails (3).
ReadIndex guarantees linearizability without hardware clock assumptions at the cost of 1 quorum RTT, whereas Leader Leases serve reads locally in <0.5ms but require bounded clock drift.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | ReadIndex adds ~0.8ms p99 latency per read batch but is immune to NTP/VM pause clock skew. |
Evaluate ReadIndex
Leader records commitIndex, sends heartbeat round to confirm quorum, waits until appliedIndex >= commitIndex, and serves read.
Evaluate Leader Lease
Leader tracks lease_end = start + election_timeout / max_clock_drift_factor based on majority ACKs.
Step 3
Select ReadIndex for multi-cloud or untrusted NTP environments; select Leader Lease only when PTP/chrony bounds drift < 10ms.
Alternative 1
TrueTime / Spanner-style bounded uncertainty intervals.
Failure Mode 1
VM live-migration pauses (>500ms) can violate lease bounds if monotonic clock freezes.
def linearizable_read_index(leader, read_fn):
read_idx = leader.commit_index
# Confirm we are still leader via 1 quorum heartbeat round (0.8ms)
if not leader.confirm_quorum_heartbeat():
raise NotLeaderError("Lost quorum during ReadIndex check")
leader.wait_until_applied(read_idx)
return read_fn()
If B pauses for 600ms.
If B pauses for 600ms while A<->C is severed, A loses quorum and steps down via CheckQuorum; once B resumes, B and C can elect a new leader, and fencing tokens block any late RPC from A.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 600ms pause causes a brief ~1s unavailability window, which is strictly safer than allowing dual-leader corruption. |
Step 1
A's check_quorum() detects only 1 active node (itself) after 500ms and steps down.
Step 2
B resumes and either grants PreVote to C or starts its own PreVote, electing a new leader in term T+1.
Step 3
Storage rejects any in-flight RPC from A carrying term T < T+1.
Alternative 1
5-node Raft cluster across 3 AZs so a single node GC pause never drops quorum.
Failure Mode 1
If fencing_token comparison is omitted in storage, A's delayed write overwrites C's newer lock.
-- PostgreSQL downstream fencing guard
UPDATE distributed_locks
SET owner_id = :new_owner,
fencing_token = :new_token,
updated_at = NOW()
WHERE lock_key = :lock_key
AND fencing_token < :new_token;
Emit an N x N.
Emit an N x N pairwise reachability matrix from every node's heartbeat/SWIM probes to Prometheus and alert whenever connected_peers < N-1 even while quorum is healthy.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | O(N^2) metric cardinality is negligible for 3-7 node Raft groups but should be sampled for 100+ node SWIM rings. |
Step 1
Export raft_peer_link_up{src, dst} (0 or 1) every 1s.
Step 2
Alert on sum(raft_peer_link_up) < N*(N-1) for > 15s (catches A<->C cut immediately).
Step 3
Correlate with PreVoteDenied counters to pinpoint which pair is partitioned.
Alternative 1
AWS VPC Flow Logs / eBPF tc-drop telemetry.
Failure Mode 1
Alert fatigue if transient <5s packet drops are not debounced with a for: 15s window.
# Prometheus alert rule for asymmetric mesh degradation
groups:
- name: raft_mesh_health
rules:
- alert: AsymmetricNetworkPartitionDetected
expr: raft_peer_link_up == 0 and on(cluster) raft_has_quorum == 1
for: 15s
labels:
severity: page
annotations:
summary: "Non-transitive link failure between {{ $labels.src }} and {{ $labels.dst }}"
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.
- 02Trade-off tip: ReadIndex adds ~0.8ms p99 latency per read batch but is immune to NTP/VM pause clock skew.
- 03Watch out for: If
fencing_tokencomparison is omitted in storage, A's delayed write overwrites C's newer lock.
Asymmetric Network Partitions Split Brain — Normal → Load / Fault → Recovery
Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.
Putting Handling Asymmetric Network Partitions & Split-Brain Together
Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.
Putting Handling Asymmetric Network Partitions & Split-Brain Together
- Add
last_leader_heartbeat_mstracking on every follower. - Require
PreVoteto pass before incrementingcurrent_term. - Run
check_quorum()on the leader every100ms. - Attach
(term << 32) | log_indexasfencing_tokenon all downstream DB writes.
Interview Readiness Check
If B pauses for 600ms while A<->C is severed, A loses quorum and steps down via CheckQuorum; once B resumes, B and C can elect a new leader, and fencing tokens block any late RPC from A.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Handling Asymmetric Network Partitions & Split-Brain
Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.
