Data Anomalies in Distributed Systems: Split Brain, Clock Skew, Stale Reads, and More

Scenario context

Distributed systems produce anomalies not because the code is buggy — but because physics makes perfect consistency impossible across network boundaries. Split brain, stale reads, clock skew, causality violations, and cascading failures are the canonical failure modes. Each has a distinct root cause and a targeted set of prevention patterns. This post is the index: it maps every anomaly to its cause and cure, and li…

This walkthrough focuses on Distributed Anomalies vs. SQL Transaction Anomalies.

Before making a design choice, clarify:

• The SQL transaction anomaly posts in this series (dirty read, phantom read, write skew, lost update) cover anomalies that arise from concurrent transactions within a single database .
• They are solved by choosing the right isolation level.
• The anomalies in this post are different in origin, detection, and cure.

Concepts in play: consistency-replication, database-isolation-levels-anomalies, transaction-isolation-distributed-acid, circuit-breaker

Stage 1 — The Core Principle: Consistency and Availability Cannot Coexist Under Partition

• The CAP theorem states that a distributed system can guarantee at most two of three properties: Consistency (every read receives the latest write), Availability (every request receives a response), and Partition toleran…
• Since network partitions are unavoidable in production, every distributed system must choose between consistency and availability when a partition occurs.
• Systems that prefer consistency (Raft-based clusters, Zookeeper, Spanner) refuse writes when they cannot form a quorum.
• Systems that prefer availability (Cassandra, DynamoDB, Riak) accept writes from any node and reconcile conflicts later.

Observable outcome: Systems that prefer availability (Cassandra, DynamoDB, Riak) accept writes from any node and reconcile conflicts later.

Stage 2 — The Full Anomaly Taxonomy

• The flowchart is the triage tool.
• When a production anomaly surfaces, classify it into one of the four families first.
• The classification step narrows the solution space from twelve options to two or three before you consult documentation.

flowchart TD
    A["Anomaly Detected in Production"] --> B{"Classify by family"}
    B -->|"Visibility"| C["Stale or torn data returned to reader"]
    B -->|"Ordering"| D["Events appear in wrong causal order"]
    B -->|"Availability"| E["Node falsely healthy or falsely dead"]
    B -->|"Integrity"| F["Two nodes hold conflicting values"]
    C --> C1{"Stale read or torn?"}
    C1 -->|"Stale"| C2["Bounded staleness / LSN token / Quorum read"]
    C1 -->|"Torn"| C3["WAL atomic replay / Commit marker gating"]
    D --> D1{"Clock drift or async delivery?"}
    D1 -->|"Clock drift"| D2["HLC / TrueTime / Avoid LWW"]
    D1 -->|"Async delivery"| D3["Vector clocks / Causal middleware"]
    E --> E1{"Split brain or zombie?"}
    E1 -->|"Split brain"| E2["Quorum writes / Raft terms / STONITH"]
    E1 -->|"Zombie leader"| E3["Fencing tokens / Epoch rejection"]
    F --> F1{"Merge or overwrite?"}
    F1 -->|"Overwrite acceptable"| F2["LWW with NTP discipline or HLC"]
    F1 -->|"Must preserve all"| F3["CRDTs / Application merge"]

Observable outcome: The classification step narrows the solution space from twelve options to two or three before you consult documentation.

Stage 3 — Anomaly Observable Impact in Production

Observable outcome: You can explain how “Anomaly Observable Impact in Production” changes the architecture or implementation path.

Stage 4 — Deep Dive: How Network Boundaries Break ACID

• ACID properties are straightforward in a single-node database where every operation passes through one scheduler, one memory space, and one disk.
• Distributing the same data across five nodes turns each property into a distributed protocol with new failure modes.
• Atomicity across nodes requires two-phase commit, which blocks indefinitely if the coordinator fails mid-transaction.
• Consistency requires consensus (Raft or Paxos), adding multiple network round-trips per write.

Observable outcome: Consistency requires consensus (Raft or Paxos), adding multiple network round-trips per write.

Stage 5 — Which Distributed Systems Are Exposed to Which Anomalies

Observable outcome: You can explain how “Which Distributed Systems Are Exposed to Which Anomalies” changes the architecture or implementation path.

Stage 6 — Prevention Strategy Cheatsheet

Observable outcome: You can explain how “Prevention Strategy Cheatsheet” changes the architecture or implementation path.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Distributed anomalies arise from physics — network partitions, clock drift, and asynchronous replication are unavoidable properties of distributed systems, not bugs.
• • Every anomaly belongs to one of four families: visibility (stale or torn data), ordering (causality violations), availability (split brain, zombie leader), or integrity (conflicting writes).

Alternatives considered

Distributed Anomalies vs. SQL Transaction Anomalies

The SQL transaction anomaly posts in this series (dirty read, phantom read, write skew, lost update) cover anomalies that arise from concurrent transactions within a single database .

Decision Guide: Which Anomaly Prevention Applies to Your Use Case

The right prevention strategy depends on what your system cannot afford to get wrong: - Cannot afford stale data (inventory, payments, seat reservations): Use strong consistency with quorum reads, synchronous replication, or read-your-writes tokens.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.