Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

CAP & PACELC
Consistency-Latency Trade-Offs.

A global e-commerce order and catalog service runs on a geo-distributed consensus database (3-region Raft cluster across us-east-1, eu-west-1, and ap-south-1; RTTs: US-EU 75ms, EU-IN 130ms, US-IN 195ms). Every read and write uses default linearizable quorum consistency (PC/EC). During normal operation with zero network partitions, ap-south-1 users experience 390ms+ p99 checkout latency and 150ms+ catalog read latency, breaching the 120ms p99 SLA. Redesign the data consistency and placement architecture using PACELC principles.
Start the journey5 stages
CAP & PACELC Consistency-Latency Trade-Offs architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A global e-commerce order and catalog service runs on a geo-distributed consensus database (3-region Raft cluster across `us-east-1`, `eu-west-1`, and `ap-south-1`; RTTs: US-EU 75ms, EU-IN 130ms, US-IN 195ms). Every read and write uses default linearizable quorum consistency (`PC/EC`). During normal operation with zero network partitions, `ap-south-1` users experience 390ms+ p99 checkout latency and 150ms+ catalog read latency, breaching the 120ms p99 SLA. Redesign the data consistency and placement architecture using PACELC principles.

From patterns to decisions.

A global e-commerce order and catalog service runs on a geo-distributed consensus database (3-region Raft cluster across `us-east-1`, `eu-west-1`, and `ap-south-1`; RTTs: US-EU 75ms, EU-IN 130ms, US-IN 195ms). Every read and write uses default linearizable quorum consistency (`PC/EC`). During normal operation with zero network partitions, `ap-south-1` users experience 390ms+ p99 checkout latency and 150ms+ catalog read latency, breaching the 120ms p99 SLA. Redesign the data consistency and placement architecture using PACELC principles.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Under PACELC's Else branch, linearizable.

Under PACELC's Else branch, linearizable reads must contact the Raft leaseholder (often in us-east-1) to guarantee no newer write committed, incurring full cross-continent RTT.

ChoiceWhat it buysWhat it costs
Trade-off 1Guarantees zero staleness at the cost of 195ms+ speed-of-light RTT on every read.

Step 1

Contrast CAP (partition-only) with PACELC (normal-operation Latency vs. Consistency).

Step 2

Trace a linearizable read from ap-south-1 to a us-east-1 leaseholder.

Alternative 1

Read from an asynchronous regional read replica (reduces latency but loses snapshot isolation).

Failure Mode 1

Holding thousands of worker threads open waiting on cross-region RTT.

-- Default PC/EC read: contacts remote leaseholder (195ms from Mumbai to Virginia)
SELECT sku_id, price_cents, title FROM catalog WHERE sku_id = 'SKU-9912';
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Implementation & Guardrails (2).

Use follower_read_timestamp() for catalog reads, REGIONAL BY ROW for orders, and a 5-second HLC session cookie to bypass follower reads right after a user write.

ChoiceWhat it buysWhat it costs
Trade-off 1Users see <5ms reads 99% of the time while paying a single leaseholder RTT only within 5s of their own edits.

Step 1

Configure GLOBAL / follower-read SQL for catalog queries.

Step 2

Implement a Python routing helper that checks x-last-write-ts before choosing follower-read vs. leaseholder-read.

Alternative 1

Use Redis regional read-through cache with CDC invalidation from Kafka.

Failure Mode 1

If last_write_ts cookie is stripped by a client, a user might briefly see stale data right after saving.

import time

FOLLOWER_STALENESS_SEC = 5.0

def fetch_user_profile(db, user_id: str, last_write_ts: float | None):
    # Preserve read-your-own-writes for 5s after a mutation
    if last_write_ts and (time.time() - last_write_ts) < FOLLOWER_STALENESS_SEC:
        return db.query_one(
            "SELECT user_id, address FROM users WHERE user_id = %s", (user_id,)
        )
    # Otherwise use PA/EL local follower read (<5ms)
    return db.query_one(
        "SELECT user_id, address FROM users AS OF SYSTEM TIME follower_read_timestamp() WHERE user_id = %s",
        (user_id,),
    )
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

ZONE FAILURE puts all voting.

ZONE FAILURE puts all voting replicas in one region for <10ms writes but cannot survive a full region outage; REGION FAILURE spreads voters across regions, paying cross-region RTT on writes.

ChoiceWhat it buysWhat it costs
Trade-off 1Surviving a region failure requires at least 2 of 3 voting replicas in separate regions, imposing one inter-region RTT on every commit.

Step 1

Compare write RTT (<10ms vs. 130ms) and RPO/RTO during a full AWS region failure.

Step 2

Choose REGION FAILURE for financial ledgers and ZONE FAILURE (or async DR) for replaceable session state.

Alternative 1

Outbox pattern: commit locally in <10ms and replicate asynchronously, accepting non-zero RPO on total region loss.

Failure Mode 1

Misunderstanding SURVIVE REGION FAILURE and expecting <10ms cross-region writes.

-- CockroachDB multi-region survival configuration
ALTER DATABASE ecommerce SET PRIMARY REGION "us-east-1";
ALTER DATABASE ecommerce ADD REGION "eu-west-1";
ALTER DATABASE ecommerce ADD REGION "ap-south-1";
ALTER DATABASE ecommerce SURVIVE REGION FAILURE;

ALTER TABLE orders SET LOCALITY REGIONAL BY ROW;
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Under P, PA/EL catalog follower.

Under P, PA/EL catalog follower reads in ap-south-1 continue serving 100% of reads locally, while PC/EC global inventory writes fail fast (unavailable) because ap-south-1 cannot form a 2-of-3 quorum.

ChoiceWhat it buysWhat it costs
Trade-off 1Catalog browsing remains 100% available while checkout preserves zero-oversell consistency.

Trace catalog reads

local replica serves closed-timestamp snapshots without needing quorum.

Trace global inventory writes

Raft blocks/times out to prevent split-brain overselling.

Alternative 1

Pre-allocate regional inventory quotas (e.g., 200 units reserved for ap-south-1) so regional checkouts stay available during partitions.

Failure Mode 1

No client timeout on partitioned writes causes API thread pool exhaustion.

# Graceful degradation during partition: allow browsing, queue or reject checkout cleanly
def checkout_with_partition_guard(db, sku_id: str, timeout_ms: int = 800):
    try:
        return db.execute_with_timeout("UPDATE inventory SET qty = qty - 1 WHERE sku_id = %s", (sku_id,), timeout_ms)
    except TimeoutError:
        raise ServiceUnavailable("Checkout temporarily paused in your region during network partition")
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Use regional inventory reservations (escrow/quota sharding): partition stock counts into regional buckets (ap-south-1: 500, us-east-1: 500) homed to each region's Raft leaseholder, rebalancing asynchronously.

ChoiceWhat it buysWhat it costs
Trade-off 1Achieves local write latency and partition tolerance until a single region exhausts its local quota and must borrow from another region.

Step 1

Split inventory(sku_id, region) so each region owns a local stock slice.

Step 2

Deduct from the local slice via local quorum, triggering async cross-region rebalance when local stock < 20%.

Alternative 1

Accept 130ms checkout RTT for the 5% write path since users tolerate 200ms on 'Place Order' clicks.

Failure Mode 1

Fragmented stock leaves 50 units stranded in eu-west-1 while ap-south-1 shows sold out if the rebalancer lags.

CREATE TABLE regional_inventory (
    sku_id UUID NOT NULL,
    region crdb_internal_region NOT NULL,
    available_qty INT NOT NULL CHECK (available_qty >= 0),
    PRIMARY KEY (region, sku_id)
) LOCALITY REGIONAL BY ROW;

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (PC/EC) while serving catalog/session reads via follower reads with bounded staleness (PA/EL).
  • 02Trade-off tip: Surviving a region failure requires at least 2 of 3 voting replicas in separate regions, imposing one inter-region RTT on every commit.
  • 03Watch out for: No client timeout on partitioned writes causes API thread pool exhaustion.
Behavioral Progression · Cap Pacelc Consistency Latency Tradeoffs

Cap Pacelc Consistency Latency Tradeoffs — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (PC/EC) while serving catalog/session reads via follower reads with bounded staleness (PA/EL).

Synthesis / One connected design

Putting CAP & PACELC Consistency-Latency Trade-Offs Together

Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (`PC/EC`) while serving catalog/session reads via follower reads with bounded staleness (`PA/EL`).

A candidate architecture, not a universal answer

Putting CAP & PACELC Consistency-Latency Trade-Offs Together

Step 1
Step 2
  1. Configure GLOBAL / follower-read SQL for catalog queries.
  2. Implement a Python routing helper that checks x-last-write-ts before choosing follower-read vs. leaseholder-read.
Under `P`, `PA/EL` catalog follower reads in `ap-south-1` continue serving 100% of reads locally, while `PC/EC` global inventory writes fail fast (unavailable) because `ap-south-1` cannot form a 2-of-3 quorum.
Interview Readiness Check

Under P, PA/EL catalog follower reads in ap-south-1 continue serving 100% of reads locally, while PC/EC global inventory writes fail fast (unavailable) because ap-south-1 cannot form a 2-of-3 quorum.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When does the CAP trade-off actually apply?

Key takeaway

CAP & PACELC Consistency-Latency Trade-Offs

Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (PC/EC) while serving catalog/session reads via follower reads with bounded staleness (PA/EL).