Split Brain Explained: When Two Nodes Both Think They Are Leader

Scenario context

Split brain happens when a network partition causes two nodes to simultaneously believe they are the leader — each accepting writes the other never sees. Prevent it with quorum consensus (at least ⌊N/2⌋+1 nodes must agree before leadership is granted), fencing tokens (each new leader issues a monotonically increasing token; storage layers reject writes below the current high-water mark), and STONITH (physically powe…

This walkthrough focuses on Deep Dive: Why Quorum Math Makes Two Simultaneous Majorities Impossible.

Before making a design choice, clarify:

• The reason Raft's quorum rule prevents true split brain (as opposed to the temporary confusion described above) is rooted in a simple mathematical identity.
• For a cluster of N nodes, a majority quorum requires ⌊N/2⌋ + 1 nodes.
• Two majorities drawn from the same N nodes must overlap by at least one node.

Concepts in play: consistency-replication, caching, distributed-locking, jvm-internals-and-garbage-collection

Stage 1 — The Consistency Trap: What Split Brain Really Means

• Split brain is the condition in which two or more nodes in a cluster simultaneously believe they are the authoritative leader, with no mechanism to arbitrate between them.
• Each leader accepts reads and writes independently, producing divergent state.
• When the partition heals and the nodes reconnect, the divergence is often irrecoverable without human intervention or an explicit rollback policy.
• A useful analogy: imagine two branch managers of the same bank, cut off from each other by a phone outage.

Observable outcome: A useful analogy: imagine two branch managers of the same bank, cut off from each other by a phone outage.

Stage 2 — How a Five-Node Partition Creates Two Leaders Step by Step

• Consider a five-node Raft cluster: one leader (P) and four followers (S1, S2, S3, S4).
• Then a network partition divides the cluster asymmetrically: Partition A contains: P and S1 (two nodes — a minority) Partition B contains: S2, S3, and S4 (three nodes — a majority) Here is what happens, step by step: St…
• S2, S3, and S4 stop receiving heartbeats from P.
• Each starts an election timeout countdown — a randomised timer, typically 150–300 ms.

Observable outcome: Each starts an election timeout countdown — a randomised timer, typically 150–300 ms.

Stage 3 — Visualizing Split Brain: From Partition to Rollback

• The Timeline of a Split-Brain Event and Recovery The sequence diagram below traces the full lifecycle of a split-brain event in a three-participant simplified cluster: the old leader, a newly elected leader on the other…
• After the partition heals, the old leader receives an AppendEntries RPC from the new leader carrying Term 2.
• Because Term 2 Term 1, the old leader immediately steps down, discards its divergent log entries (including the balance=500 write), and overwrites with the new leader's log.
• Client A received an ACK for a write that no longer exists.

sequenceDiagram
    participant C1 as Client A
    participant OL as "Old Leader (Term 1)"
    participant NL as "New Leader (Term 2)"
    participant C2 as Client B

    Note over OL,NL: Network partition begins
    C1->>OL: Write balance=500
    OL-->>C1: ACK (accepted in Term 1)
    C2->>NL: Write balance=750
    NL-->>C2: ACK (accepted in Term 2)
    Note over OL,NL: Both leaders hold divergent state
    Note over OL,NL: Partition heals — Term 2 propagates
    NL->>OL: AppendEntries term=2
    OL-->>NL: Step down — reverting to follower
    Note over OL: balance=500 write ROLLED BACK
    Note over NL: balance=750 is the durable truth

flowchart TD
    A["Network partition detected"] --> B{"Can node reach a quorum?"}
    B -->|"Yes — majority reachable"| C["Continue normal operation"]
    B -->|"No — minority partition"| D["Reject all client writes"]
    D --> E["Increment election timeout counter"]
    E --> F{"Election timeout expired?"}
    F -->|"No"| E
    F -->|"Yes"| G["Send RequestVote with incremented term"]
    G --> H{"Votes received from majority?"}
    H -->|"No"| I["Remain follower — cannot become leader"]
    H -->|"Yes"| J["Elected leader in new term"]
    J --> K{"STONITH configured?"}
    K -->|"Yes"| L["Fence old leader node via IPMI or PDU"]
    K -->|"No"| M["Issue new fencing token to self"]
    L --> N["Old leader cannot write — hardware isolated"]
    M --> O["Old zombie writes rejected by token high-water mark"]
    N --> P["Split brain prevented"]
    O --> P

Observable outcome: Client A received an ACK for a write that no longer exists.

Stage 4 — Real-World Incidents: When Split Brain Escaped into Production

• MongoDB's Two-Primary Bug (2012) In MongoDB's replica set implementation prior to version 2.6, a specific race condition in the election protocol could produce two simultaneous primaries under high network jitter.
• Both primaries accepted writes.
• When the bug was surfaced publicly, the MongoDB team confirmed that any data written exclusively to the deposed primary was permanently lost, as the oplog entries were simply overwritten during resync.
• The fix required both a protocol correction — tightening the quorum check — and a recommendation to all operators to use w=majority acknowledgement.

Observable outcome: The fix required both a protocol correction — tightening the quorum check — and a recommendation to all operators to use w=majority acknowledgement.

Stage 5 — Comparing Defences: Quorum, Fencing, STONITH, Consensus, and Manual Procedures

• Each split-brain defence has a different cost profile and failure envelope.
• Use this table to choose the right layer for your architecture: No single mechanism is sufficient on its own.
• Production-grade systems layer quorum writes (primary gate) with fencing tokens or STONITH (backup gate for zombie scenarios).

Observable outcome: Production-grade systems layer quorum writes (primary gate) with fencing tokens or STONITH (backup gate for zombie scenarios).

Stage 6 — The Zombie Leader: When a GC Pause Outlasts the Election Timeout

• Split brain does not require a network partition in the traditional sense.
• It can emerge from a single node's extended pause — the zombie leader scenario.
• Here is the sequence: the leader node enters a full-stop garbage collection pause.
• The JVM (or Go runtime) stops all threads, including the thread that sends Raft heartbeats.

Observable outcome: The JVM (or Go runtime) stops all threads, including the thread that sends Raft heartbeats.

End-to-end outcome

The reason Raft's quorum rule prevents true split brain (as opposed to the temporary confusion described above) is rooted in a simple mathematical identity.

Alternatives considered

Choosing Your Split Brain Defence: A Decision Guide

Choosing Your Split Brain Defence: A Decision Guide

Failure and degraded modes

What Is Split Brain? The Core Failure Mode

• Detection: A split brain is a distributed systems failure where two or more nodes simultaneously believe they are the authoritative leader — each accepting writes with no knowledge of what the other is doing.
• User or operator impact: A split brain is a distributed systems failure where two or more nodes simultaneously believe they are the authoritative leader — each accepting writes with no knowledge of what the other is doing.