Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Handling Asymmetric Network
Partitions & Split-Brain.

Your 3-node metadata control plane (Nodes A, B, C across us-east-1a, 1b, 1c) runs a Raft-backed coordination service handling 4,500 lock acquisitions/sec at p99 < 12ms. A top-of-rack BGP/ACL misconfiguration drops all TCP/UDP packets between AZ-1a (Node A) and AZ-1c (Node C), while A<->B and B<->C links remain healthy at 0.8ms RTT. Within 500ms, Node C's election timeout fires, it increments its Raft term, and Node B votes for C—disrupting Leader A, which then times out and bumps its term again, causing continuous term inflation and 100% lock-service unavailability. Redesign the failure detection, consensus election protocol, and downstream fencing to survive non-transitive partitions without livelock or split-brain writes.
Start the journey5 stages
Handling Asymmetric Network Partitions & Split-Brain architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your 3-node metadata control plane (Nodes A, B, C across us-east-1a, 1b, 1c) runs a Raft-backed coordination service handling 4,500 lock acquisitions/sec at p99 < 12ms. A top-of-rack BGP/ACL misconfiguration drops all TCP/UDP packets between AZ-1a (Node A) and AZ-1c (Node C), while A<->B and B<->C links remain healthy at 0.8ms RTT. Within 500ms, Node C's election timeout fires, it increments its Raft term, and Node B votes for C—disrupting Leader A, which then times out and bumps its term again, causing continuous term inflation and 100% lock-service unavailability. Redesign the failure detection, consensus election protocol, and downstream fencing to survive non-transitive partitions without livelock or split-brain writes.

From patterns to decisions.

Your 3-node metadata control plane (Nodes A, B, C across us-east-1a, 1b, 1c) runs a Raft-backed coordination service handling 4,500 lock acquisitions/sec at p99 < 12ms. A top-of-rack BGP/ACL misconfiguration drops all TCP/UDP packets between AZ-1a (Node A) and AZ-1c (Node C), while A<->B and B<->C links remain healthy at 0.8ms RTT. Within 500ms, Node C's election timeout fires, it increments its Raft term, and Node B votes for C—disrupting Leader A, which then times out and bumps its term again, causing continuous term inflation and 100% lock-service unavailability. Redesign the failure detection, consensus election protocol, and downstream fencing to survive non-transitive partitions without livelock or split-brain writes.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

In standard Raft, any follower.

In standard Raft, any follower that misses heartbeats increments its term and broadcasts RequestVote; a peer receiving a higher term must update its term and force the current leader to step down, even if the leader is still connected to a majority.

ChoiceWhat it buysWhat it costs
Trade-off 1Standard Raft prioritizes fast election on any higher term at the cost of vulnerability to partially connected candidates.

Trace heartbeats from Leader A

B receives them (timer resets), C does not (A<->C dropped).

Trace C's election timeout

C increments term to T+1 and sends RequestVote(T+1) to B.

Observe B's reaction

B updates its term to T+1, which propagates to A and forces A to step down, restarting elections endlessly.

Alternative 1

Wait for a full lease expiry before accepting RequestVote (slower failover).

Failure Mode 1

100% control-plane unavailability due to continuous term bumps every 500ms.

def on_request_vote_naive(node, req_term, candidate_id):
    if req_term > node.current_term:
        # Naive Raft steps down immediately even if Leader A is healthy!
        node.current_term = req_term
        node.state = 'FOLLOWER'
        node.voted_for = None
    # Causes A and C to ping-pong terms indefinitely via bridge B
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Combine Pre-Vote (deny votes if.

Combine Pre-Vote (deny votes if leader heartbeat was seen within min_election_timeout) with CheckQuorum (step down if < majority ACKs within election_timeout) and monotonic fencing tokens.

ChoiceWhat it buysWhat it costs
Trade-off 1Adds 1 RTT to legitimate failovers (~0.8ms intra-region) in exchange for complete immunity to asymmetric term storms.

Step 1

Add last_leader_heartbeat_ms tracking on every follower.

Step 2

Require PreVote to pass before incrementing current_term.

Step 3

Run check_quorum() on the leader every 100ms.

Step 4

Attach (term << 32) | log_index as fencing_token on all downstream DB writes.

Alternative 1

SWIM indirect probing (ping-req) for non-consensus membership rings.

Failure Mode 1

Clock jumps if time.time() is used instead of time.monotonic().

import time

def handle_pre_vote(node, next_term, last_log_idx, last_log_term):
    now_ms = time.monotonic() * 1000
    # Reject if we recently heard from a valid leader (prevents C disrupting A via B)
    if (now_ms - node.last_leader_heartbeat_ms) < node.min_election_timeout_ms:
        return {"term": node.current_term, "vote_granted": False}
    if next_term <= node.current_term:
        return {"term": node.current_term, "vote_granted": False}
    if not node.is_log_up_to_date(last_log_idx, last_log_term):
        return {"term": node.current_term, "vote_granted": False}
    return {"term": node.current_term, "vote_granted": True}

def check_quorum(leader):
    now_ms = time.monotonic() * 1000
    active_peers = 1 + sum(
        1 for ts in leader.peer_last_ack_ms.values()
        if (now_ms - ts) < leader.election_timeout_ms
    )
    if active_peers < (leader.cluster_size // 2) + 1:
        leader.step_down_to_follower()
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

ReadIndex guarantees linearizability without hardware clock assumptions at the cost of 1 quorum RTT, whereas Leader Leases serve reads locally in <0.5ms but require bounded clock drift.

ChoiceWhat it buysWhat it costs
Trade-off 1ReadIndex adds ~0.8ms p99 latency per read batch but is immune to NTP/VM pause clock skew.

Evaluate ReadIndex

Leader records commitIndex, sends heartbeat round to confirm quorum, waits until appliedIndex >= commitIndex, and serves read.

Evaluate Leader Lease

Leader tracks lease_end = start + election_timeout / max_clock_drift_factor based on majority ACKs.

Step 3

Select ReadIndex for multi-cloud or untrusted NTP environments; select Leader Lease only when PTP/chrony bounds drift < 10ms.

Alternative 1

TrueTime / Spanner-style bounded uncertainty intervals.

Failure Mode 1

VM live-migration pauses (>500ms) can violate lease bounds if monotonic clock freezes.

def linearizable_read_index(leader, read_fn):
    read_idx = leader.commit_index
    # Confirm we are still leader via 1 quorum heartbeat round (0.8ms)
    if not leader.confirm_quorum_heartbeat():
        raise NotLeaderError("Lost quorum during ReadIndex check")
    leader.wait_until_applied(read_idx)
    return read_fn()
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

If B pauses for 600ms.

If B pauses for 600ms while A<->C is severed, A loses quorum and steps down via CheckQuorum; once B resumes, B and C can elect a new leader, and fencing tokens block any late RPC from A.

ChoiceWhat it buysWhat it costs
Trade-off 1600ms pause causes a brief ~1s unavailability window, which is strictly safer than allowing dual-leader corruption.

Step 1

A's check_quorum() detects only 1 active node (itself) after 500ms and steps down.

Step 2

B resumes and either grants PreVote to C or starts its own PreVote, electing a new leader in term T+1.

Step 3

Storage rejects any in-flight RPC from A carrying term T < T+1.

Alternative 1

5-node Raft cluster across 3 AZs so a single node GC pause never drops quorum.

Failure Mode 1

If fencing_token comparison is omitted in storage, A's delayed write overwrites C's newer lock.

-- PostgreSQL downstream fencing guard
UPDATE distributed_locks
SET owner_id = :new_owner,
    fencing_token = :new_token,
    updated_at = NOW()
WHERE lock_key = :lock_key
  AND fencing_token < :new_token;
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Emit an N x N.

Emit an N x N pairwise reachability matrix from every node's heartbeat/SWIM probes to Prometheus and alert whenever connected_peers < N-1 even while quorum is healthy.

ChoiceWhat it buysWhat it costs
Trade-off 1O(N^2) metric cardinality is negligible for 3-7 node Raft groups but should be sampled for 100+ node SWIM rings.

Step 1

Export raft_peer_link_up{src, dst} (0 or 1) every 1s.

Step 2

Alert on sum(raft_peer_link_up) < N*(N-1) for > 15s (catches A<->C cut immediately).

Step 3

Correlate with PreVoteDenied counters to pinpoint which pair is partitioned.

Alternative 1

AWS VPC Flow Logs / eBPF tc-drop telemetry.

Failure Mode 1

Alert fatigue if transient <5s packet drops are not debounced with a for: 15s window.

# Prometheus alert rule for asymmetric mesh degradation
groups:
  - name: raft_mesh_health
    rules:
      - alert: AsymmetricNetworkPartitionDetected
        expr: raft_peer_link_up == 0 and on(cluster) raft_has_quorum == 1
        for: 15s
        labels:
          severity: page
        annotations:
          summary: "Non-transitive link failure between {{ $labels.src }} and {{ $labels.dst }}"

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.
  • 02Trade-off tip: ReadIndex adds ~0.8ms p99 latency per read batch but is immune to NTP/VM pause clock skew.
  • 03Watch out for: If fencing_token comparison is omitted in storage, A's delayed write overwrites C's newer lock.
Behavioral Progression · Asymmetric Network Partitions Split Brain

Asymmetric Network Partitions Split Brain — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.

Synthesis / One connected design

Putting Handling Asymmetric Network Partitions & Split-Brain Together

Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.

A candidate architecture, not a universal answer

Putting Handling Asymmetric Network Partitions & Split-Brain Together

Step 1
Step 2
Step 3
Step 4
  1. Add last_leader_heartbeat_ms tracking on every follower.
  2. Require PreVote to pass before incrementing current_term.
  3. Run check_quorum() on the leader every 100ms.
  4. Attach (term << 32) | log_index as fencing_token on all downstream DB writes.
If B pauses for 600ms while A<->C is severed, A loses quorum and steps down via CheckQuorum; once B resumes, B and C can elect a new leader, and fencing tokens block any late RPC from A.
Interview Readiness Check

If B pauses for 600ms while A<->C is severed, A loses quorum and steps down via CheckQuorum; once B resumes, B and C can elect a new leader, and fencing tokens block any late RPC from A.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Handling Asymmetric Network Partitions & Split-Brain

Never rely on direct pairwise reachability for cluster health or leadership; combine Raft Pre-Vote + CheckQuorum (or SWIM indirect probing) with monotonic fencing tokens at the storage layer.