A global e-commerce order and catalog service runs on a geo-distributed consensus database (3-region Raft cluster across `us-east-1`, `eu-west-1`, and `ap-south-1`; RTTs: US-EU 75ms, EU-IN 130ms, US-IN 195ms). Every read and write uses default linearizable quorum consistency (`PC/EC`). During normal operation with zero network partitions, `ap-south-1` users experience 390ms+ p99 checkout latency and 150ms+ catalog read latency, breaching the 120ms p99 SLA. Redesign the data consistency and placement architecture using PACELC principles.
From patterns to decisions.
A global e-commerce order and catalog service runs on a geo-distributed consensus database (3-region Raft cluster across `us-east-1`, `eu-west-1`, and `ap-south-1`; RTTs: US-EU 75ms, EU-IN 130ms, US-IN 195ms). Every read and write uses default linearizable quorum consistency (`PC/EC`). During normal operation with zero network partitions, `ap-south-1` users experience 390ms+ p99 checkout latency and 150ms+ catalog read latency, breaching the 120ms p99 SLA. Redesign the data consistency and placement architecture using PACELC principles.
Under PACELC's Else branch, linearizable.
Under PACELC's Else branch, linearizable reads must contact the Raft leaseholder (often in us-east-1) to guarantee no newer write committed, incurring full cross-continent RTT.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Guarantees zero staleness at the cost of 195ms+ speed-of-light RTT on every read. |
Step 1
Contrast CAP (partition-only) with PACELC (normal-operation Latency vs. Consistency).
Step 2
Trace a linearizable read from ap-south-1 to a us-east-1 leaseholder.
Alternative 1
Read from an asynchronous regional read replica (reduces latency but loses snapshot isolation).
Failure Mode 1
Holding thousands of worker threads open waiting on cross-region RTT.
-- Default PC/EC read: contacts remote leaseholder (195ms from Mumbai to Virginia)
SELECT sku_id, price_cents, title FROM catalog WHERE sku_id = 'SKU-9912';
Implementation & Guardrails (2).
Use follower_read_timestamp() for catalog reads, REGIONAL BY ROW for orders, and a 5-second HLC session cookie to bypass follower reads right after a user write.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Users see <5ms reads 99% of the time while paying a single leaseholder RTT only within 5s of their own edits. |
Step 1
Configure GLOBAL / follower-read SQL for catalog queries.
Step 2
Implement a Python routing helper that checks x-last-write-ts before choosing follower-read vs. leaseholder-read.
Alternative 1
Use Redis regional read-through cache with CDC invalidation from Kafka.
Failure Mode 1
If last_write_ts cookie is stripped by a client, a user might briefly see stale data right after saving.
import time
FOLLOWER_STALENESS_SEC = 5.0
def fetch_user_profile(db, user_id: str, last_write_ts: float | None):
# Preserve read-your-own-writes for 5s after a mutation
if last_write_ts and (time.time() - last_write_ts) < FOLLOWER_STALENESS_SEC:
return db.query_one(
"SELECT user_id, address FROM users WHERE user_id = %s", (user_id,)
)
# Otherwise use PA/EL local follower read (<5ms)
return db.query_one(
"SELECT user_id, address FROM users AS OF SYSTEM TIME follower_read_timestamp() WHERE user_id = %s",
(user_id,),
)
ZONE FAILURE puts all voting.
ZONE FAILURE puts all voting replicas in one region for <10ms writes but cannot survive a full region outage; REGION FAILURE spreads voters across regions, paying cross-region RTT on writes.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Surviving a region failure requires at least 2 of 3 voting replicas in separate regions, imposing one inter-region RTT on every commit. |
Step 1
Compare write RTT (<10ms vs. 130ms) and RPO/RTO during a full AWS region failure.
Step 2
Choose REGION FAILURE for financial ledgers and ZONE FAILURE (or async DR) for replaceable session state.
Alternative 1
Outbox pattern: commit locally in <10ms and replicate asynchronously, accepting non-zero RPO on total region loss.
Failure Mode 1
Misunderstanding SURVIVE REGION FAILURE and expecting <10ms cross-region writes.
-- CockroachDB multi-region survival configuration
ALTER DATABASE ecommerce SET PRIMARY REGION "us-east-1";
ALTER DATABASE ecommerce ADD REGION "eu-west-1";
ALTER DATABASE ecommerce ADD REGION "ap-south-1";
ALTER DATABASE ecommerce SURVIVE REGION FAILURE;
ALTER TABLE orders SET LOCALITY REGIONAL BY ROW;
Under P, PA/EL catalog follower.
Under P, PA/EL catalog follower reads in ap-south-1 continue serving 100% of reads locally, while PC/EC global inventory writes fail fast (unavailable) because ap-south-1 cannot form a 2-of-3 quorum.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Catalog browsing remains 100% available while checkout preserves zero-oversell consistency. |
Trace catalog reads
local replica serves closed-timestamp snapshots without needing quorum.
Trace global inventory writes
Raft blocks/times out to prevent split-brain overselling.
Alternative 1
Pre-allocate regional inventory quotas (e.g., 200 units reserved for ap-south-1) so regional checkouts stay available during partitions.
Failure Mode 1
No client timeout on partitioned writes causes API thread pool exhaustion.
# Graceful degradation during partition: allow browsing, queue or reject checkout cleanly
def checkout_with_partition_guard(db, sku_id: str, timeout_ms: int = 800):
try:
return db.execute_with_timeout("UPDATE inventory SET qty = qty - 1 WHERE sku_id = %s", (sku_id,), timeout_ms)
except TimeoutError:
raise ServiceUnavailable("Checkout temporarily paused in your region during network partition")
Implementation & Guardrails (5).
Use regional inventory reservations (escrow/quota sharding): partition stock counts into regional buckets (ap-south-1: 500, us-east-1: 500) homed to each region's Raft leaseholder, rebalancing asynchronously.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Achieves local write latency and partition tolerance until a single region exhausts its local quota and must borrow from another region. |
Step 1
Split inventory(sku_id, region) so each region owns a local stock slice.
Step 2
Deduct from the local slice via local quorum, triggering async cross-region rebalance when local stock < 20%.
Alternative 1
Accept 130ms checkout RTT for the 5% write path since users tolerate 200ms on 'Place Order' clicks.
Failure Mode 1
Fragmented stock leaves 50 units stranded in eu-west-1 while ap-south-1 shows sold out if the rebalancer lags.
CREATE TABLE regional_inventory (
sku_id UUID NOT NULL,
region crdb_internal_region NOT NULL,
available_qty INT NOT NULL CHECK (available_qty >= 0),
PRIMARY KEY (region, sku_id)
) LOCALITY REGIONAL BY ROW;
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (
PC/EC) while serving catalog/session reads via follower reads with bounded staleness (PA/EL). - 02Trade-off tip: Surviving a region failure requires at least 2 of 3 voting replicas in separate regions, imposing one inter-region RTT on every commit.
- 03Watch out for: No client timeout on partitioned writes causes API thread pool exhaustion.
Cap Pacelc Consistency Latency Tradeoffs — Normal → Load / Fault → Recovery
Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (PC/EC) while serving catalog/session reads via follower reads with bounded staleness (PA/EL).
Putting CAP & PACELC Consistency-Latency Trade-Offs Together
Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (`PC/EC`) while serving catalog/session reads via follower reads with bounded staleness (`PA/EL`).
Putting CAP & PACELC Consistency-Latency Trade-Offs Together
- Configure
GLOBAL/ follower-read SQL for catalog queries. - Implement a Python routing helper that checks
x-last-write-tsbefore choosing follower-read vs. leaseholder-read.
Interview Readiness Check
Under P, PA/EL catalog follower reads in ap-south-1 continue serving 100% of reads locally, while PC/EC global inventory writes fail fast (unavailable) because ap-south-1 cannot form a 2-of-3 quorum.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When does the CAP trade-off actually apply?
CAP & PACELC Consistency-Latency Trade-Offs
Apply PACELC explicitly per workload: pin monetary/linearizable writes to regional leaseholders (PC/EC) while serving catalog/session reads via follower reads with bounded staleness (PA/EL).
