System Design Core Concepts: Scalability, CAP, and Consistency

Scenario context

🚀 Scalability, the CAP Theorem, and consistency models are the three concepts that determine whether a distributed system can grow, stay reliable, and deliver correct results. Get these three right and you can reason about any system design question.

In 2011, Netflix engineers deployed Chaos Monkey — a tool that randomly terminated production servers to test system resilience. Their infrastructure stayed…

This walkthrough focuses on Understanding System Design Core Concepts.

Before making a design choice, clarify:

• In 2011, Netflix engineers deployed Chaos Monkey — a tool that randomly terminated production servers to test system resilience.
• Their infrastructure stayed up.
• Netflix had deliberately designed for AP (available + partition-tolerant): they accepted that some users might see slightly stale recommendations in exchange for the system never going down .

Concepts in play: consistency-replication, api-gateway, system-design-hld-architectures, caching

Stage 1 — How the CAP Theorem Works: Every Distributed System Makes a Choice

• CAP Theorem states that a distributed system can guarantee at most two of three properties simultaneously: - Consistency (C) — every read returns the most recent write (or an error) - Availability (A) — every request re…
• The real choice is: when a partition occurs, do you sacrifice C or A?
• Practical implication: - Banking transaction?
• Choose CP — you can't have a split-brain account balance.

Observable outcome: Choose CP — you can't have a split-brain account balance.

Stage 2 — Deep Dive: Consistency Models Unpacked

• The CAP theorem frames a binary choice, but in practice the important dimension is how you trade off consistency against availability during normal operation—not just during failures.
• 📊 Eventual Consistency Propagation This diagram demonstrates eventual consistency in action: Client 1 writes a new balance to the Primary node, which immediately acknowledges the write; meanwhile Client 2 reads from Re…
• After both replicas receive the replication message from Primary, all subsequent reads return the correct value ($500).
• The key takeaway is that in an AP system there is a window of time after a write — potentially seconds or longer — during which different readers may see different values, and application logic must be designed to toler…

sequenceDiagram
    participant C1 as Client 1 (Writer)
    participant P as Primary Node
    participant R1 as Replica 1
    participant R2 as Replica 2
    participant C2 as Client 2 (Reader)

    C1->>P: Write: balance = $500
    P-->>C1: ACK (write confirmed)
    Note over R1,R2: Async replication in progress
    C2->>R2: Read: balance?
    R2-->>C2: balance = $400 (stale read)
    P->>R1: Replicate: balance = $500
    P->>R2: Replicate: balance = $500
    Note over R1,R2: Replicas converged
    C2->>R2: Read: balance?
    R2-->>C2: balance = $500 (consistent)

stateDiagram-v2
    [*] --> Strong : CP system
    [*] --> Eventual : AP system
    [*] --> Weak : Cache only

    Strong --> Eventual : Relax for throughput
    Eventual --> Strong : Require linearizability
    Eventual --> ReadYourWrites : Scope to session
    ReadYourWrites --> Causal : Add causal ordering
    Weak --> Eventual : Add replication sync

Observable outcome: The key takeaway is that in an AP system there is a window of time after a write — potentially seconds or longer — during which different readers may see different values, and application logic must be designed to toler…

Stage 3 — System Design Decision Flow: Choosing the Right Architecture

• When designing a distributed system, the sequence of decisions follows a consistent pattern.
• Use this flow to reason through any system design problem systematically.
• This decision framework applies to every data store, cache, and message queue in your architecture.
• The same three-question pattern — consistency need, availability requirement, partition tolerance — governs each component choice independently.

graph TD
    A[Define consistency requirement] --> B{Strong consistency required?}
    B -->|Yes| C[Choose CP system: etcd, Zookeeper, PostgreSQL]
    B -->|No| D{High availability critical?}
    D -->|Yes| E[Choose AP system: Cassandra, DynamoDB]
    D -->|No| F[Choose simple single-node or CA architecture]
    C --> G[Add replication + failover]
    E --> H[Add conflict resolution + eventual sync]
    F --> I[Scale vertically first, then revisit]

Observable outcome: The same three-question pattern — consistency need, availability requirement, partition tolerance — governs each component choice independently.

Stage 4 — Real-World Applications: Applying These Concepts: E-Commerce Checkout Example

• An e-commerce checkout flow actually uses multiple consistency models simultaneously: This is the key insight: you don't choose one consistency model for your whole system .
• You partition your data by its consistency requirements.

Observable outcome: You partition your data by its consistency requirements.

Stage 5 — Little's Law: Back-of-Envelope Capacity Math

• This section applies Little's Law to back-of-envelope capacity estimation — one of the most practical calculations in both system design interviews and real production capacity planning.
• It was chosen because it provides a quantitative bridge between the scalability concepts in this post and a concrete number: given your target request rate and latency budget, how much concurrency must your system suppo…
• As you work through the example, focus on how the three variables (concurrency L, arrival rate λ, latency W) constrain each other — improving any one of them directly affects the resources you need to provision.

• $$L = \lambda \cdot W$$

  Where: - $L$ = average number of requests in the system (concurrency) - $\lambda$ = arrival rate (requests per second) - $W$ = average time spent in the system (latency in seconds) Example: if yo…

Observable outcome: $$L = \lambda \cdot W$$ Where: - $L$ = average number of requests in the system (concurrency) - $\lambda$ = arrival rate (requests per second) - $W$ = average time spent in the system (latency in seconds) Example: if yo…

Stage 6 — Implementation Approach: CAP-Aware Architecture in Practice

• The e-commerce example from this post — strong consistency for payments, eventual consistency for the product catalog — reflects a two-layer architecture that most production systems use.
• The transactional data path (inventory, payments) routes through a relational database with ACID semantics: each inventory decrement or payment record is wrapped in a database transaction that either commits fully or ro…
• Acquiring a row-level lock before reading the current stock count is what prevents the race condition.
• The eventual-consistency data path (product catalog, recommendations, user preferences) routes through a distributed cache with a configurable TTL — typically 30 to 60 seconds.

Observable outcome: The eventual-consistency data path (product catalog, recommendations, user preferences) routes through a distributed cache with a configurable TTL — typically 30 to 60 seconds.

End-to-end outcome

In 2011, Netflix engineers deployed Chaos Monkey — a tool that randomly terminated production servers to test system resilience.

Alternatives considered

Scalability: Vertical vs. Horizontal

Vertical scaling (scale up) means adding more power to a single machine — more CPU cores, more RAM, faster storage.

ACID vs. BASE: Two Ways to Think About Data Correctness

ACID (traditional relational databases): - Atomic — all operations in a transaction succeed or all fail - Consistent — the database moves from one valid state to another valid state - Isolated — concurrent transactions don't interfere with each other - Durable — committed data survives crashes BASE (most NoSQL / distributed systems): - Basically Available — availability is prioritized; partial failures are acceptable - Soft state — data can be stale; state may change without input (replication catching up) - Eventual consistency — given no new writes, all replicas will eventually converge to the same value

Decision Guide: Choosing the Right Consistency Level for Your Use Case

Decision Guide: Choosing the Right Consistency Level for Your Use Case

Failure and degraded modes

Trade-offs & Failure Modes: When Things Go Wrong: Partition Scenarios

• Detection: A network partition means some nodes can't communicate with others.
• User or operator impact: A network partition means some nodes can't communicate with others.