Key Terms in Distributed Systems: The Definitive Glossary

Scenario context

Distributed systems vocabulary is precise for a reason. Mixing up read skew and write skew costs you an interview. Confusing Snapshot Isolation with Serializable costs you a production outage. This glossary organises every critical term into concept families — with concrete scenarios, prevention strategies, and visual aids — so the vocabulary becomes second nature.

In a single-machine system, time is simple: events…

This walkthrough focuses on Ordering and Time: Why "Now" Has No Meaning in a Distributed System.

Before making a design choice, clarify:

• In a single-machine system, time is simple: events have a total order enforced by a single clock and a single CPU.
• In a distributed system, machines have independent clocks that drift, and messages arrive out of order.
• The vocabulary in this section defines how distributed systems reason about causality and ordering without trusting wall clocks.

Concepts in play: database-isolation-levels-anomalies, consistency-replication, api-gateway, caching

Stage 1 — How Vocabulary Breaks System Design Interviews (and Production Systems)

• You are 35 minutes into a system design interview.
• You have just proposed a multi-leader replication setup for a globally distributed social feed.
• The interviewer pauses and says: "Your design doesn't account for write skew.
• Walk me through how you'd prevent it." You nod.

Observable outcome: Walk me through how you'd prevent it." You nod.

Stage 2 — Consistency Anomalies: The Ways Transactions Can Lie to You

• Before you can understand isolation levels, you need to understand what they protect against.
• Each anomaly below describes a class of correctness violation that can occur when multiple transactions execute concurrently.
• Dirty Read A dirty read occurs when transaction B reads data that transaction A has written but not yet committed.
• If A rolls back, B has read data that, as far as the database is concerned, never existed.

sequenceDiagram
    participant Alice as Alice Txn
    participant DB as Database
    participant Bob as Bob Txn

    Alice->>DB: BEGIN
    Bob->>DB: BEGIN
    Alice->>DB: SELECT count(*) WHERE on_call = true
    DB-->>Alice: 2 doctors on call
    Bob->>DB: SELECT count(*) WHERE on_call = true
    DB-->>Bob: 2 doctors on call
    Alice->>DB: UPDATE doctors SET on_call=false WHERE name=Alice
    Bob->>DB: UPDATE doctors SET on_call=false WHERE name=Bob
    Alice->>DB: COMMIT
    Bob->>DB: COMMIT
    Note over DB: Invariant violated - 0 doctors on call

-- Read current value
SELECT value, version FROM counters WHERE id = 1;
-- value = 5, version = 42

-- Write only if version has not changed (no lost update possible)
UPDATE counters SET value = 6, version = 43
WHERE id = 1 AND version = 42;
-- If 0 rows affected, another transaction won the race — retry

Observable outcome: If A rolls back, B has read data that, as far as the database is concerned, never existed.

Stage 3 — Isolation Levels: PostgreSQL's Six-Way Dial Between Safety and Speed

• Isolation levels are the database's knob for trading safety against concurrency and performance.
• The SQL standard defines four levels; real databases (PostgreSQL, MySQL, Oracle) add Snapshot Isolation as a practical middle ground.
• \ Phantom reads may be prevented by gap locks in some engines (MySQL InnoDB).
• †PostgreSQL's Repeatable Read prevents lost updates and read skew via MVCC.

Observable outcome: †PostgreSQL's Repeatable Read prevents lost updates and read skew via MVCC.

Stage 4 — Replication Models and the Consistency Spectrum Across Databases

• Replication is the act of maintaining copies of the same data on multiple nodes.
• Every replicated system must answer: who accepts writes, and how quickly must replicas reflect them?
• The following diagram maps the consistency spectrum from weakest to strongest, positioned against the replication models that produce each consistency level: The spectrum above runs from eventual consistency (weakest —…
• Each step upward adds correctness guarantees but increases latency or availability cost.

flowchart TD
    A[Eventual Consistency] --> B[Causal Consistency]
    B --> C[Monotonic Reads / Read-Your-Writes]
    C --> D[Session Consistency]
    D --> E[Bounded Staleness]
    E --> F[Sequential Consistency]
    F --> G[Linearizability - Strongest]

    A -.->|Leaderless async replication| A
    D -.->|Multi-leader with per-session routing| D
    G -.->|Single-leader sync replication| G

Observable outcome: Each step upward adds correctness guarantees but increases latency or availability cost.

Stage 5 — Partitioning and Sharding: Distributing Data Across Nodes

• As data volumes grow beyond what a single node can store or serve, you partition data across multiple nodes.
• Understanding the vocabulary here prevents the most common interview stumble — conflating partitioning with sharding.
• Sharding: Partitioning is the general concept of splitting a dataset into subsets.
• Sharding is the specific technique of horizontal partitioning across multiple database nodes (physical machines or instances).

Observable outcome: Sharding is the specific technique of horizontal partitioning across multiple database nodes (physical machines or instances).

Stage 6 — Consensus and Leader Election: Deciding Who Gets the Final Say

• Consensus is the problem of getting multiple distributed nodes to agree on a single value, even when some nodes may crash or messages may be delayed.
• It underpins leader election, distributed transactions, and replicated state machines.
• Consensus: Formally, a consensus algorithm must satisfy: (1) Validity — any decided value was proposed; (2) Agreement — no two nodes decide differently; (3) Termination — every non-faulty node eventually decides.
• The FLP impossibility result proves that deterministic consensus is impossible in a fully asynchronous system with even one crash failure — which is why practical implementations use timeouts.

sequenceDiagram
    participant L1 as Leader 1 (stale)
    participant ZK as ZooKeeper
    participant L2 as Leader 2 (current)
    participant ST as Storage

    L1->>ZK: Acquire lock, receive token=33
    Note over L1: Leader 1 pauses (GC / network)
    ZK-->>L2: Leader 1 timeout - new election
    L2->>ZK: Acquire lock, receive token=34
    L2->>ST: Write data with token=34
    ST-->>L2: OK - token 34 accepted
    Note over L1: Leader 1 resumes, still thinks it has lock
    L1->>ST: Write data with token=33
    ST-->>L1: REJECTED - token 33 less than seen token 34

Observable outcome: The FLP impossibility result proves that deterministic consensus is impossible in a fully asynchronous system with even one crash failure — which is why practical implementations use timeouts.

End-to-end outcome

After working through the stages, you should be able to explain:

• These are the hard-won lessons that distinguish engineers who understand the definitions from engineers who have operated these systems at scale.
• Snapshot Isolation is not Serializable — and production bugs live in that gap.

Alternatives considered

Decision Guide: Choosing the Right Consistency and Storage Model

Use this guide when you face a design question about consistency guarantees, isolation levels, or storage engines.

Failure and degraded modes

Fault Tolerance and Recovery: Building Systems That Survive Failures

• Detection: Distributed systems must be designed with the assumption that individual components will fail.
• User or operator impact: Distributed systems must be designed with the assumption that individual components will fail.

CAP Theorem and PACELC: The Consistency–Availability Trade-off Decoded

• Detection: The CAP theorem and its successor PACELC are the theoretical frameworks behind every "which database should I use" decision in a distributed system design discussion.
• User or operator impact: The CAP theorem and its successor PACELC are the theoretical frameworks behind every "which database should I use" decision in a distributed system design discussion.

Trade-offs in Distributed Systems: The Decisions That Define Your Architecture

• Detection: Every term in this glossary exists because there is a trade-off underneath it.
• User or operator impact: Every term in this glossary exists because there is a trade-off underneath it.