Advanced Data Structures & Algorithms

Java Backend Modernization

Beginner's Guide to Data Structures

Agentic AI Systems

Data Engineering & Spark Mastery

Low-Level Design Interview Prep

Microservices Interview Roadmap

Design Patterns Interview Roadmap

Distributed Systems Interview Roadmap

Mastering Python Development

System Design Interview Prep

Distributed Systems Refresher

AI Engineering Launchpad

Architecture Patterns for Scalable Systems

RAG & Vector Search Mastery

Production DevOps & Reliability

Enterprise Patterns Playbook

Master Coding Interviews

Coding Techniques Interview Roadmap

Data Structures Interview Roadmap

Architecture Patterns Interview Roadmap

software-engineering-principles

Migrate your Spring Boot services from thread-pool exhaustion to JDK 21 virtual threads — the practical Spring developer guide

Adapting to Virtual Threads for Spring Developers

how-it-works-internals-explained

From WAL and MVCC to fsync and isolation levels — the internals behind every ACID guarantee in PostgreSQL, MySQL, Oracle, and SQL Server.

ACID Properties Explained: How SQL Databases Guarantee Atomicity, Consistency, Isolation, and Durability

How modern NoSQL databases achieve atomicity and isolation without a single machine

ACID Transactions in Distributed Databases: DynamoDB, Cosmos DB, and Spanner Compared

llm-engineering

Is an LLM a brain in a jar? To make it truly useful, we need to give it access to the world. This guide explains RAG and Agents.

Advanced AI: Agents, RAG, and the Future of Intelligence

Production AI needs explicit routing, memory, execution, and evaluation layers rather than one loop.

AI Architecture Patterns: Routers, Planner-Worker Loops, Memory Layers, and Evaluation Guardrails

datastructures-and-algorithms

Master shortest paths, dependency resolution, and cycle detection with robust Java solutions.

Advanced Graph Algorithms: Dijkstra, Topological Sort, and Disjoint Set Union (DSU)

An LLM can talk, but an AI Agent can *act*. We explain how Agents use the ReAct framework to brow...

AI Agents Explained: When LLMs Start Using Tools

A production-first guide to picking the right vector index for recall, latency, memory, and update patterns.

ANN Index Types Explained: When to Choose Flat, HNSW, IVF, or IVF-PQ

apache-spark-engineering

Process petabytes with Python: understand Spark's execution model, DataFrame transformations, partitioning strategy, and real-time streaming.

Apache Spark for Data Engineers: RDDs, DataFrames, and Structured Streaming

ACID guarantees consistency. BASE guarantees availability. We explain the trade-offs between SQL...

BASE Theorem Explained: How it Stands Against ACID

machine-learning-fundamentals

From the query-key-value formulation to multi-head attention — the mechanism that made transformers dominate AI

Attention Mechanism Explained: How Transformers Learn to Focus

system-design-interview-prep

They all sit in front of your servers. But do you need Nginx, HAProxy, or Kong? We clarify the roles of each component.

API Gateway vs. Load Balancer vs. Reverse Proxy: What's the Difference?

architecture-patterns-for-production-systems

Mobile apps need less data than Desktop dashboards. Why serve them the same JSON? The Backend for Frontend (BFF) pattern solves this.

Backend for Frontend (BFF): Tailoring APIs for UI

BFS explores graphs level by level using a queue — the key to shortest paths and level-order problems.

BFS — Breadth-First Search: Level-by-Level Graph Exploration

Backtracking is a brute-force algorithmic technique that solves problems by trying to build a sol...

Exploring Backtracking Techniques in Data Structures

big-data-engineering

Choose ingestion, serving, and ownership patterns deliberately when data platforms start to scale.

Big Data Architecture Patterns: Lambda, Kappa, CDC, Medallion, and Data Mesh

Why traditional databases fail at scale, what the 5 Vs really mean, and a practical map of the big data ecosystem from ingestion to insights.

Big Data 101: The 5 Vs, Ecosystem, and Why Scale Breaks Everything

Not all Binary Trees are created equal. We explain Full, Complete, Perfect, and Balanced Binary T...

Exploring Different Types of Binary Trees

Master Big O notation: measure algorithm efficiency from O(1) to O(n²) with growth diagrams, worked examples, and the interview mental model.

Big O Notation Explained: Time Complexity, Space Complexity, and Why They Matter in Interviews

Binary search has five distinct patterns — classic, leftmost, rightmost, rotated arrays, and 2D matrices.

Binary Search Patterns: Five Variants Every Senior Engineer Knows

Don't panic. System Design interviews are open-ended discussions. This framework (Requirements, API, DB, Scale) will help you structure your answer.

The Ultimate Guide to Acing the System Design Interview

A beginner-friendly way to structure requirements, estimates, diagrams, and trade-offs in your first design round.

System Design Interview Basics: A Beginner-Friendly Framework for Clear Answers

A practical framework for clarifying functional scope, non-functional targets, and trade-off boundaries in interviews.

System Design Requirements and Constraints: Ask Better Questions Before You Draw

Don't guess. Calculate. We explain how to estimate QPS, Storage, and Bandwidth for your system de...

The Role of Data in Precise Capacity Estimations for System Design

The building blocks of distributed systems. Learn about Vertical vs Horizontal scaling, the CAP Theorem, and ACID vs BASE.

System Design Core Concepts: Scalability, CAP, and Consistency

Design APIs interviewers trust by focusing on resource boundaries, request contracts, and failure-safe behavior.

System Design API Design for Interviews: Contracts, Idempotency, and Pagination

The internet's traffic control system. We explain how DNS resolves names, CDNs cache content, and Load Balancers distribute traffic.

System Design Networking: DNS, CDNs, and Load Balancers

How do servers talk to each other? This guide explains the key protocols: REST vs RPC for APIs, TCP vs UDP for transport.

System Design Protocols: REST, RPC, and TCP/UDP

Learn how to choose entities, indexes, and schema evolution strategies that match real query patterns at scale.

System Design Data Modeling and Schema Evolution: Query-Driven Storage That Survives Change

The eternal debate: SQL or NoSQL? We break down ACID vs BASE, Sharding vs Replication, and when to use MongoDB vs PostgreSQL.

System Design Databases: SQL vs NoSQL and Scaling

Beyond the SQL vs NoSQL debate: A framework for matching database architecture to business requirements.

Choosing the Right Database: CAP Theorem and Practical Use Cases

Cache-Aside to Refresh-Ahead, LRU to ARC — every caching pattern, eviction policy, and distributed strategy you need for system design

System Design: Complete Guide to Caching — Patterns, Eviction, and Distributed Strategies

A practical guide to shard keys, routing, hot partitions, and how to scale storage without breaking queries.

System Design Sharding Strategy: Choosing Keys, Avoiding Hot Spots, and Resharding Safely

Design asynchronous pipelines with queues, retries, and consumer scaling that survive traffic spikes.

System Design Message Queues and Event-Driven Architecture: Building Reliable Asynchronous Systems

A clear guide to replicas, failover detection, replica lag, and the trade-offs behind reliable database designs.

System Design Replication and Failover: Keep Services Alive When a Primary Dies

A practical guide to active-passive, active-active, failover routing, and the trade-offs of serving users across regions.

System Design Multi-Region Deployment: Latency, Failover, and Consistency Across Regions

Learn how clients find services safely with registries, heartbeats, and health-aware load balancing.

System Design Service Discovery and Health Checks: Routing Traffic to Healthy Instances

Design telemetry, SLOs, and response playbooks that detect failure early and recover predictably.

System Design Observability, SLOs, and Incident Response: Operating Systems You Can Trust

A senior-level HLD for a high-throughput short-link platform with 100:1 read-write ratio.

System Design HLD Example: URL Shortener (TinyURL and Bitly)

Sub-10ms prefix lookup, ranking, typo tolerance, and language-partitioned indexes at Google scale.

System Design HLD Example: Search Autocomplete (Google/Amazon)

Interview HLD for a distributed cache with eviction, invalidation, and resilience trade-offs.

System Design HLD Example: Distributed Cache Platform

Interview-focused HLD for a scalable social feed with fan-out and ranking trade-offs.

System Design HLD Example: News Feed (Home Timeline)

An interview-ready HLD for payments focusing on correctness, idempotency, and recovery.

System Design HLD Example: Payment Processing Platform

## Scenario context

System Design interviews are collaborative whiteboard sessions, not trick-question coding tests. Follow the framework — Requirements → Estimations → API → Data Model → High-Level Architecture → Deep-Dive — and you turn vague product ideas into defensible, production-ready designs.

---



```architecture
{
  "version": 1,
  "title": "TLDR:** System Design interviews are collaborative whiteboard sessions, not trick-q…

This walkthrough focuses on **Why System Design Interviews Feel Chaotic (and How to Fix That)**.

Before making a design choice, clarify:

- Picture two candidates who received the same prompt: "Design a real-time activity feed for 500,000 events per second." Candidate A: "I'd store events in a database and add a cache layer in front of it." Candidate B: "I'…
- I'd put Redis sorted sets in front as a fan-out cache for the hot feed.
- I'd choose hybrid fan-out: push to regular users, pull for accounts with over a million followers to avoid write amplification.

**Concepts in play:** system-design-hld-architectures, caching, consistency-replication, event-driven-architecture

## Stage 1 — What Makes a Strong System Design Response

- Before diving into the framework, understand what separates a strong response from a weak one.
- Weak responses jump straight to a specific technology ("I'd use Kafka and Cassandra") before understanding requirements.
- They treat the problem as a trivia question with a known answer.
- Strong responses start by asking clarifying questions, establish constraints with back-of-envelope estimates, and justify every decision with explicit trade-offs.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** Strong responses start by asking clarifying questions, establish constraints with back-of-envelope estimates, and justify every decision with explicit trade-offs.

## Stage 2 — The Six-Step Interview Framework

- A repeatable structure turns chaos into confidence.
- Step 1: Clarify Requirements (~5 min) Ask before you design.
- Split into: - Functional requirements — What does the system need to do?
- ("Post a tweet", "Follow a user", "Generate a timeline") - Non-functional requirements — How should it behave?

*Applies **caching** to the decision above.*

```mermaid
flowchart TD
    subgraph Client
        C[Mobile / Web App]
    end
    C -->|HTTPS| LB[Load Balancer]
    LB --> GW[API Gateway - auth, rate-limit]
    GW --> TS[Tweet Service]
    GW --> US[User Service]
    GW --> FS[Feed Service]
    TS -->|Write| CASS[Cassandra Cluster]
    TS -->|Event| KAF[Kafka]
    KAF --> FAN[Fan-out Workers]
    FAN --> RED[Redis Feed Cache]
    US --> PG[PostgreSQL]
    FS --> RED

```

```mermaid
flowchart TD
    A[Receive Prompt]
    B[Clarify Requirements]
    C[Estimate Capacity]
    D[Define API]
    E[Choose Data Models]
    F[High-Level Architecture]
    G{Interviewer probes?}
    H[Deep-Dive: Scaling]
    I[Raise hardest problem]
    J[Summarise trade-offs]
    K[Invite feedback]
    
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G -->|Yes| H
    G -->|No| I
    H --> J
    I --> J
    J --> K
```

**Observable outcome:** ("Post a tweet", "Follow a user", "Generate a timeline") - Non-functional requirements — How should it behave?

## Stage 3 — Key Mechanics: Caching, Fan-out, and Hot Users

- Caching the feed Each user's feed is stored in Redis as a sorted set, where the tweet ID is the member and the tweet's creation timestamp is the score.
- This structure makes inserting a new tweet O(log N) and retrieving the most recent top-N tweets equally fast — both well within the sub-10 ms latency budget of a production feed service.
- The score-based ordering also enables cursor-based pagination naturally: slide the score window backward in time to fetch the next page without offset scanning.
- Fan-out strategies Hot user problem: A celebrity with 50 M followers cannot push-fan-out on write without overloading the system.

*Applies **consistency-replication** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Key Mechanics: Caching, Fan-out, and Hot Users - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** Fan-out strategies Hot user problem: A celebrity with 50 M followers cannot push-fan-out on write without overloading the system.

## Stage 4 — Deep Dive: How Interviewers Score System Design Answers

- Most interviewers use a rubric with four dimensions: requirements clarity (did you ask the right questions?), estimation accuracy (are numbers in the right order of magnitude?), architectural completeness (does the desi…

*Applies **event-driven-architecture** to the decision above.*

**Observable outcome:** Most interviewers use a rubric with four dimensions: requirements clarity (did you ask the right questions?), estimation accuracy (are numbers in the right order of magnitude?), architectural completeness (does the desi…

## Stage 5 — Framework Flow: From Prompt to Architecture

- This flowchart shows the recommended decision sequence for a 45-minute system design session.
- The loop at step G is critical: interviewers probe whatever they find most interesting or whatever you mentioned confidently.
- Prepare to defend every component you sketch.

*Applies **system-design-hld-architectures** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Framework Flow: From Prompt to Architecture - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

```mermaid
flowchart TD
    A[Interview Prompt Received]
    B[Clarify Requirements]
    C[Estimate Capacity]
    D[Define API]
    E[Choose Data Models]
    F[Sketch Architecture]
    G{Interviewer probes?}
    H[Deep-Dive: Scaling]
    I[Raise hardest problem]
    J[Summarise trade-offs]
    
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G -->|Yes| H
    G -->|No| I
    H --> J
    I --> J
```

**Observable outcome:** Prepare to defend every component you sketch.

## Stage 6 — Real-World Applications: What Interviewers Are Actually Evaluating

- You are not graded on getting a "correct" architecture.
- (Not skipping this is a differentiator) 2.
- Did you estimate before you designed?
- (Shows engineering maturity) 3.

*Applies **caching** to the decision above.*

**Observable outcome:** (Shows engineering maturity) 3.

## End-to-end outcome

Picture two candidates who received the same prompt: "Design a real-time activity feed for 500,000 events per second." Candidate A: "I'd store events in a database and add a cache layer in front of it." Candidate B: "I'd use Cassandra with a write-heavy schema partitioned by user id and event timestamp — because this is append-only click data at 500K events/sec, and Cassandra's LSM-tree storage handles that write pattern without contention.

## Alternatives considered

### Decision Guide: Common Interview Scenarios

Decision Guide: Common Interview Scenarios


## Failure and degraded modes

### Trade-offs & Failure Modes: Common Failure Modes and How to Discuss Them

- **Detection:** Trade-offs & Failure Modes: Common Failure Modes and How to Discuss Them
- **User or operator impact:** Trade-offs & Failure Modes: Common Failure Modes and How to Discuss Them



## Scenario context

System design interviews are not about inventing a perfect architecture on the spot. They are about showing a calm, repeatable process: clarify requirements, estimate scale, sketch a simple design, explain trade-offs, and improve it when constraints change.

---



```architecture
{
  "version": 1,
  "title": "TLDR:** System design interviews are not about inventing a perfect architecture on the spot. They are about…

This walkthrough focuses on **Why System Design Interviews Feel Hard Before They Feel Logical**.

Before making a design choice, clarify:

- Most beginners think a system design interview is a test of memory.
- They assume they must instantly recall the "right" architecture for a URL shortener, chat app, or news feed.
- That is the wrong mental model.

**Concepts in play:** system-design-hld-architectures, caching, api-gateway, consistency-replication

## Stage 1 — The Three Beginner Moves Before You Draw Anything

- Before any diagram, API, or storage choice, make three moves.
- Ask what the user can do and what the system does not need to do.
- If the prompt is "Design Instagram," you do not need to design every feature on Earth.
- You might narrow the scope to posting photos, following users, and loading a home feed.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** You might narrow the scope to posting photos, following users, and loading a home feed.

## Stage 2 — The 30-Minute Whiteboard Flow That Keeps You Organized

- Once you have the scope, use a simple flow for nearly every interview.
- Estimate rough scale with quick capacity estimation math.
- Identify the core entities and APIs with explicit API contract design and data modeling choices.
- Sketch a high-level architecture.

*Applies **caching** to the decision above.*

**Observable outcome:** Sketch a high-level architecture.

## Stage 3 — Real-World Application: Designing a Tiny URL Shortener in an Interview

- A URL shortener is one of the best beginner interview exercises because the problem is small enough to explain clearly but still exposes real design trade-offs.
- Start with the simplest product definition: - A user submits a long URL.
- - The system returns a short code.
- - Anyone with that short code can redirect to the original URL.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** - Anyone with that short code can redirect to the original URL.

## Stage 4 — Practical Example: How to Answer "Design a Rate Limiter"

- If the interviewer asks you to design a rate limiter, do not start by naming Redis and token buckets immediately.
- Start with the shape of the problem.
- Is the limit per user, per IP, per API key, or per endpoint?
- Is it 100 requests per minute or 1,000 per second?

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** Is it 100 requests per minute or 1,000 per second?

## Stage 5 — From Interview Diagram to Running System: Architecture in Practice

- Spring Boot is the standard Java framework for building production-ready microservices with embedded servers and auto-configuration.
- Docker Compose is the local orchestration tool that lets you wire a multi-service architecture — app server, database, cache — into a single configuration file, turning your whiteboard diagram into a runnable environmen…
- For the URL shortener example, the architecture described in the interview maps directly onto three services: the application server, a relational database for durable URL mappings, and a Redis cache for fast redirect r…
- The application uses the cache-aside pattern for reads — checking Redis first and falling back to the database on a miss, then populating Redis so the next request is served from cache.

*Applies **system-design-hld-architectures** to the decision above.*

```architecture
{
  "version": 1,
  "title": "From Interview Diagram to Running System: Architecture in Practice - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

**Observable outcome:** The application uses the cache-aside pattern for reads — checking Redis first and falling back to the database on a miss, then populating Redis so the next request is served from cache.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - The best first answer is usually a simple answer with clear assumptions.
- - Requirements and scale estimates matter more than naming fashionable tools.

## Alternatives considered

### Decision Guide: When to Go Simple and When to Add Scale

You do not get extra points for complexity by default.


## Failure and degraded modes

### Trade-offs & Failure Modes: Where Simple Answers Break

- **Detection:** A beginner answer becomes stronger the moment you acknowledge where it breaks.
- **User or operator impact:** A beginner answer becomes stronger the moment you acknowledge where it breaks.



## Scenario context

In system design interviews, weak answers fail early because requirements are fuzzy. Strong answers start by turning vague prompts into explicit functional scope, measurable non-functional targets, and clear trade-off boundaries before any architecture diagram appears.

> **TLDR**: If you clarify requirements well, the architecture almost chooses itself.



```architecture
{
  "version": 1,
  "title": "TLDR:** In sy…

This walkthrough focuses on **Why Requirement Clarity Is the Real Beginning of System Design**.

Before making a design choice, clarify:

- Slack assumed users were on reliable corporate networks.
- When mobile users on 3G hit the app in 2015, 40% quit within 30 seconds.
- The non-functional requirement "must load in under 3 seconds on 3G" was never written down.

**Concepts in play:** system-design-hld-architectures, caching, consistency-replication, stack-and-queue-data-structures

## Stage 1 — The Requirement Stack: Functional, Non-Functional, and Business Constraints

- A reliable way to avoid chaos is to classify requirements into layers.
- Functional requirements answer "What should the system do?" Examples: - Users can create short links.
- - Users can view a personalized feed.
- - Drivers can request rides and track status.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** - Drivers can request rides and track status.

## Stage 2 — A Practical Requirement Interview Script You Can Reuse

- Candidates often ask: "What exactly should I ask first?" Use a short script in this order: 1.
- Define the primary user journey.
- Define strict consistency boundaries.
- Here is a reusable checklist table: This script works because it does not require perfect numbers.

*Applies **caching** to the decision above.*

**Observable outcome:** Here is a reusable checklist table: This script works because it does not require perfect numbers.

## Stage 3 — Deep Dive: Translating Requirements Into Enforceable Design Decisions

- Requirement gathering is useful only if it drives specific architecture decisions.
- The translation step is where many interviews are won.
- The Internals: Requirement-to-Component Mapping Every clarified constraint should map to one or more design mechanisms.
- - Low read latency target - cache layer, denormalized read model, or edge routing.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** - Low read latency target - cache layer, denormalized read model, or edge routing.

## Stage 4 — Requirement Funnel: From Vague Prompt to Defensible Architecture

- architecture { "version": 1, "title": "Requirement Funnel: From Vague Prompt to Defensible Architecture - Subsystem Overview", "description": "Decoupled system architecture isolating ingress, stateless logic, and persis…
- If the interview starts drifting, return to the funnel and show what changed in assumptions.
- 📊 Requirements Classification Tree This classification tree shows how any system requirement maps to one of three categories.
- Functional requirements define what the system does — user actions, core operations, and API boundaries; non-functional requirements define how well it must do it — latency targets, availability SLOs, and consistency le…

*Applies **stack-and-queue-data-structures** to the decision above.*

```mermaid

```

```

flowchart TD
    A[Vague interview prompt] --> B[Clarify functional scope]
    B --> C[Capture non-functional targets]
    C --> D[Set constraints and assumptions]
    D --> E[Define out-of-scope boundaries]
    E --> F[Map constraints to components]
    F --> G[Present architecture with trade-offs]

```

**Observable outcome:** Functional requirements define what the system does — user actions, core operations, and API boundaries; non-functional requirements define how well it must do it — latency targets, availability SLOs, and consistency le…

## Stage 5 — Real-World Applications: Notification, Feed, and Checkout Systems

- The same requirement framework applies across very different domains.
- Notification platform: - Functional: send notification, view delivery status.
- - Non-functional: near-real-time delivery for push, eventual for email.
- - Constraints: provider rate limits, regional SMS regulations.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** - Constraints: provider rate limits, regional SMS regulations.

## Stage 6 — Practical Example: Requirement Breakdown for "Design a Chat System"

- Suppose the interviewer says: "Design WhatsApp." A structured response starts with narrowing: - Phase 1: one-to-one messaging only.
- - Exclude group chat, media compression, and end-to-end encryption details from MVP.
- Then define measurable assumptions: Now architecture decisions follow naturally: 1.
- Per-conversation ordering requirement - partition messages by conversation ID.

*Applies **caching** to the decision above.*

**Observable outcome:** Per-conversation ordering requirement - partition messages by conversation ID.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Requirements are architecture inputs, not interview formalities.
- - Functional, non-functional, and business constraints should be separated explicitly.

## Alternatives considered

### Decision Guide: Which Requirement Style Fits the Interview Prompt?

This decision table helps you adapt your questioning style without sounding scripted.


## Failure and degraded modes

### Trade-offs & Failure Modes: What Goes Wrong When Requirements Are Weak

- **Detection:** A strong candidate explicitly narrates these failure modes and shows how requirement discipline prevents them.
- **User or operator impact:** A strong candidate explicitly narrates these failure modes and shows how requirement discipline prevents them.



## Scenario context

Capacity estimation is the skill of back-of-the-envelope math that tells you whether your system design will survive its traffic before you write a line of code. Four numbers do most of the work: **DAU, QPS, Storage/day, and Bandwidth/day**.






```mermaid
flowchart LR
    DAU["Daily Active Users (e.g., 10M)"] --> QPS["Convert to QPS (requests/sec)"]
    QPS --> Storage["Storage/day (data written)"]
    QPS --> Ba…

This walkthrough focuses on **Reference Numbers to Memorize**.

Before making a design choice, clarify:

- 

**Concepts in play:** system-design-hld-architectures, caching, jvm-internals-and-garbage-collection, consistency-replication

## Stage 1 — Estimation Decision Flow

- Use this decision tree to structure any capacity estimation from scratch.
- Start with DAU and follow the branches to determine what infrastructure tier your system actually needs — single server, horizontally scaled, or fully distributed.
- The key decision point is the scale tier .
- A system handling 100 RPS and one handling 100,000 RPS require fundamentally different architectures — not just bigger servers, but different data stores, caching strategies, and deployment models.

*Applies **system-design-hld-architectures** to the decision above.*

```mermaid
flowchart TD
    Start[Start Estimation] --> DAU[Estimate DAU]
    DAU --> RW[Estimate Read/Write Ratio]
    RW --> QPS[Calculate QPS DAU  actions / 86400]
    QPS --> Peak[Multiply by 2-3x for peak]
    Peak --> Storage[Calculate Storage/day write QPS  record size  86400]
    Storage --> BW[Calculate Bandwidth read QPS  response size]
    BW --> Scale{Scale tier?}
    Scale -->|< 1000 RPS| Small[Single server PostgreSQL]
    Scale -->|1k-100k RPS| Med[App servers + cache + sharding]
    Scale -->|> 100k RPS| Large[Distributed system CDN + microservices]
```

**Observable outcome:** A system handling 100 RPS and one handling 100,000 RPS require fundamentally different architectures — not just bigger servers, but different data stores, caching strategies, and deployment models.

## Stage 2 — Real-World Application: Capacity Estimation Across Scale Tiers

- The best way to internalize the four-step pipeline is to run it on systems you already use every day.
- Here are three scenarios at very different scales that demonstrate how the numbers drive architecture decisions.
- Twitter/X-Scale Feed - DAU: 400M users.
- Each user reads ~10 timeline requests/day; each user writes ~0.1 tweets/day.

*Applies **caching** to the decision above.*

**Observable outcome:** Each user reads ~10 timeline requests/day; each user writes ~0.1 tweets/day.

## Stage 3 — Deep Dive: Worked Example — Design a Pastebin

- - Write QPS = (1M × 1 paste/day) / 86400 ≈ 12 writes/sec - Read QPS = 12 × 10 = 120 reads/sec - Storage: 12 writes/sec × 10 KB × 86400 = ~10 GB/day → 10 TB over 3 years.
- - Read bandwidth: 120 reads/sec × 10 KB = 1.2 MB/sec → no CDN needed at this scale.
- What this tells you: - A single PostgreSQL can comfortably handle sub-1000 writes/sec.
- - Storage backend should be durable object storage to handle 10 TB over 3 years.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

**Observable outcome:** - Storage backend should be durable object storage to handle 10 TB over 3 years.

## Stage 4 — Practical: Interview Estimation Template

- In a system design interview, the interviewer is watching whether you have a structured process — not whether you arrive at the exact right number.
- Here is a reusable five-step template you can follow verbatim: Quick Reference Decision Rules Two threshold rules that short-circuit most infrastructure architecture debates: - If storage 1 TB/day → use object storage (…
- A relational database is not designed for multi-terabyte raw ingest at sustained write rates.
- - If read bandwidth 1 GB/s → put a CDN in front.

*Applies **consistency-replication** to the decision above.*

```
Step 1 — State assumptions:
  - DAU: X million
  - Read:Write ratio: X:1
  - Average object size: X KB/MB

Step 2 — Calculate QPS:
  - Write QPS = DAU × writes_per_user / 86400
  - Read QPS = Write QPS × read_write_ratio
  - Peak QPS = Average QPS × 3

Step 3 — Calculate Storage:
  - Storage/day = write QPS × record_size × 86400
  - 5-year storage = storage/day × 365 × 5
  - Account for replication: × 3

Step 4 — Calculate Bandwidth:
  - Read bandwidth = read QPS × response_size
  - CDN threshold: > 1 GB/s read bandwidth

Step 5 — Determine scale tier:
  - < 1k RPS: single server
  - 1k-100k RPS: horizontal scaling + cache
  - > 100k RPS: distributed + CDN
```

**Observable outcome:** - If read bandwidth 1 GB/s → put a CDN in front.

## Stage 5 — Spring Boot + JMeter: From Capacity Estimates to Measured Throughput

- Spring Boot is the standard Java framework for building the HTTP services whose QPS, storage, and latency you estimate.
- Apache JMeter is the open-source load testing tool that validates those estimates against a running service — generating synthetic traffic at the calculated peak load and measuring actual throughput and p95 latency.
- The workflow: estimate peak QPS on paper → implement the endpoint → run a JMeter test plan at estimated load → compare results against your SLA targets.
- Instrument the endpoint to compare live p95/p99 against your paper estimate: JMeter test plan — simulating estimated peak load: Capacity estimate vs.

*Applies **system-design-hld-architectures** to the decision above.*

```java
// Close the estimation feedback loop: @Timed captures actual latency from live traffic
// so you can compare measured p95 against the "p95 < 50ms at 11,600 RPS" estimate.

@GetMapping("/api/users/{id}")
@Timed(value = "user.lookup.latency",
       percentiles = {0.50, 0.95, 0.99},    // exported to Prometheus/Grafana/Actuator
       description = "Cache-aside: Redis ~0.5ms hit, DB ~5ms miss")
public ResponseEntity<UserDto> getUser(@PathVariable String id) {
    // L1 — Redis: serves ~90%+ of requests at sub-millisecond cost
    UserDto cached = cache.opsForValue().get("user:" + id);
    if (cached != null) return ResponseEntity.ok(cached);

    // L2 — PostgreSQL: cache miss, fills cache for next request (5-minute TTL)
    return userRepository.findById(id)
        .map(u -> { cache.opsForValue().set("user:" + id, u, 5, MINUTES); return ResponseEntity.ok(u); })
        .orElse(ResponseEntity.notFound().build());
}
// If Actuator shows p99 > 100ms → add a read replica or extend Redis TTL
```

```xml


<ThreadGroup>
  <numThreads>1000</numThreads>         
  <rampTime>60</rampTime>               
  <duration>300</duration>              

  <HTTPSamplerProxy>
    <domain>localhost</domain>
    <port>8080</port>
    <path>/api/users/${userId}</path>   
    <method>GET</method>
  </HTTPSamplerProxy>

  
</ThreadGroup>
```

**Observable outcome:** Instrument the endpoint to compare live p95/p99 against your paper estimate: JMeter test plan — simulating estimated peak load: Capacity estimate vs.

## Stage 6 — Key Lessons from Back-of-Envelope Estimation

- Five lessons that separate engineers who nail capacity estimation from those who overcomplicate it: 1.
- The goal is the scale tier, not the exact number.
- Whether your estimate lands at 80,000 RPS or 120,000 RPS doesn't change the architecture decision: both require distributed infrastructure with a load balancer and horizontal scaling.
- A 10× error is usually tolerable; a 100× error is a design problem.

*Applies **caching** to the decision above.*

```mermaid
flowchart TD
    Start[Start: State Assumptions] --> DAU[DAU  actions/user]
    DAU --> QPS[Avg Write QPS = DAU  writes / 86400]
    QPS --> Peak[Peak QPS = Avg  2-3x]
    Peak --> RQPS[Read QPS = Write QPS  read: write ratio]
    RQPS --> Storage[Storage/day = write QPS  record size  86400]
    RQPS --> BW[Bandwidth/day = read QPS  response size]
    Storage --> Scale{Scale tier?}
    BW --> CDN{BW > 1 GB/s?}
    CDN -->|Yes| AddCDN[Add CDN layer]
    CDN -->|No| NoCDN[Direct serving OK]
    Scale -->|under 1k RPS| Small[Single server]
    Scale -->|1k-100k RPS| Med[Horizontal + cache]
    Scale -->|over 100k RPS| Large[Distributed + CDN]
```

**Observable outcome:** A 10× error is usually tolerable; a 100× error is a design problem.

## End-to-end outcome

The Role of Data in Precise Capacity Estimations for System Design

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

### Trade-offs & Failure Modes: Trade-offs, Failure Modes & Decision Guide: Estimation Mistakes

- **Detection:** Trade-offs & Failure Modes: Trade-offs, Failure Modes & Decision Guide: Estimation Mistakes
- **User or operator impact:** Trade-offs & Failure Modes: Trade-offs, Failure Modes & Decision Guide: Estimation Mistakes

## Scenario context

🚀 Scalability, the CAP Theorem, and consistency models are the three concepts that determine whether a distributed system can grow, stay reliable, and deliver correct results. Get these three right and you can reason about any system design question.

---

In 2011, Netflix engineers deployed **Chaos Monkey** — a tool that randomly terminated production servers to test system resilience. Their infrastructure stayed…

This walkthrough focuses on **Understanding System Design Core Concepts**.

Before making a design choice, clarify:

- In 2011, Netflix engineers deployed Chaos Monkey — a tool that randomly terminated production servers to test system resilience.
- Their infrastructure stayed up.
- Netflix had deliberately designed for AP (available + partition-tolerant): they accepted that some users might see slightly stale recommendations in exchange for the system never going down .

**Concepts in play:** consistency-replication, api-gateway, system-design-hld-architectures, caching

## Stage 1 — How the CAP Theorem Works: Every Distributed System Makes a Choice

- CAP Theorem states that a distributed system can guarantee at most two of three properties simultaneously: - Consistency (C) — every read returns the most recent write (or an error) - Availability (A) — every request re…
- The real choice is: when a partition occurs, do you sacrifice C or A?
- Practical implication: - Banking transaction?
- Choose CP — you can't have a split-brain account balance.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** Choose CP — you can't have a split-brain account balance.

## Stage 2 — Deep Dive: Consistency Models Unpacked

- The CAP theorem frames a binary choice, but in practice the important dimension is how you trade off consistency against availability during normal operation—not just during failures.
- 📊 Eventual Consistency Propagation This diagram demonstrates eventual consistency in action: Client 1 writes a new balance to the Primary node, which immediately acknowledges the write; meanwhile Client 2 reads from Re…
- After both replicas receive the replication message from Primary, all subsequent reads return the correct value ($500).
- The key takeaway is that in an AP system there is a window of time after a write — potentially seconds or longer — during which different readers may see different values, and application logic must be designed to toler…

*Applies **api-gateway** to the decision above.*

```mermaid
sequenceDiagram
    participant C1 as Client 1 (Writer)
    participant P as Primary Node
    participant R1 as Replica 1
    participant R2 as Replica 2
    participant C2 as Client 2 (Reader)

    C1->>P: Write: balance = $500
    P-->>C1: ACK (write confirmed)
    Note over R1,R2: Async replication in progress
    C2->>R2: Read: balance?
    R2-->>C2: balance = $400 (stale read)
    P->>R1: Replicate: balance = $500
    P->>R2: Replicate: balance = $500
    Note over R1,R2: Replicas converged
    C2->>R2: Read: balance?
    R2-->>C2: balance = $500 (consistent)
```

```mermaid
stateDiagram-v2
    [*] --> Strong : CP system
    [*] --> Eventual : AP system
    [*] --> Weak : Cache only

    Strong --> Eventual : Relax for throughput
    Eventual --> Strong : Require linearizability
    Eventual --> ReadYourWrites : Scope to session
    ReadYourWrites --> Causal : Add causal ordering
    Weak --> Eventual : Add replication sync
```

**Observable outcome:** The key takeaway is that in an AP system there is a window of time after a write — potentially seconds or longer — during which different readers may see different values, and application logic must be designed to toler…

## Stage 3 — System Design Decision Flow: Choosing the Right Architecture

- When designing a distributed system, the sequence of decisions follows a consistent pattern.
- Use this flow to reason through any system design problem systematically.
- This decision framework applies to every data store, cache, and message queue in your architecture.
- The same three-question pattern — consistency need, availability requirement, partition tolerance — governs each component choice independently.

*Applies **system-design-hld-architectures** to the decision above.*

```architecture
{
  "version": 1,
  "title": "System Design Decision Flow: Choosing the Right Architecture - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

```mermaid
graph TD
    A[Define consistency requirement] --> B{Strong consistency required?}
    B -->|Yes| C[Choose CP system: etcd, Zookeeper, PostgreSQL]
    B -->|No| D{High availability critical?}
    D -->|Yes| E[Choose AP system: Cassandra, DynamoDB]
    D -->|No| F[Choose simple single-node or CA architecture]
    C --> G[Add replication + failover]
    E --> H[Add conflict resolution + eventual sync]
    F --> I[Scale vertically first, then revisit]
```

**Observable outcome:** The same three-question pattern — consistency need, availability requirement, partition tolerance — governs each component choice independently.

## Stage 4 — Real-World Applications: Applying These Concepts: E-Commerce Checkout Example

- An e-commerce checkout flow actually uses multiple consistency models simultaneously: This is the key insight: you don't choose one consistency model for your whole system .
- You partition your data by its consistency requirements.

*Applies **caching** to the decision above.*

**Observable outcome:** You partition your data by its consistency requirements.

## Stage 5 — Little's Law: Back-of-Envelope Capacity Math

- This section applies Little's Law to back-of-envelope capacity estimation — one of the most practical calculations in both system design interviews and real production capacity planning.
- It was chosen because it provides a quantitative bridge between the scalability concepts in this post and a concrete number: given your target request rate and latency budget, how much concurrency must your system suppo…
- As you work through the example, focus on how the three variables (concurrency L, arrival rate λ, latency W) constrain each other — improving any one of them directly affects the resources you need to provision.
- $$L = \lambda \cdot W$$ Where: - $L$ = average number of requests in the system (concurrency) - $\lambda$ = arrival rate (requests per second) - $W$ = average time spent in the system (latency in seconds) Example: if yo…

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** $$L = \lambda \cdot W$$ Where: - $L$ = average number of requests in the system (concurrency) - $\lambda$ = arrival rate (requests per second) - $W$ = average time spent in the system (latency in seconds) Example: if yo…

## Stage 6 — Implementation Approach: CAP-Aware Architecture in Practice

- The e-commerce example from this post — strong consistency for payments, eventual consistency for the product catalog — reflects a two-layer architecture that most production systems use.
- The transactional data path (inventory, payments) routes through a relational database with ACID semantics: each inventory decrement or payment record is wrapped in a database transaction that either commits fully or ro…
- Acquiring a row-level lock before reading the current stock count is what prevents the race condition.
- The eventual-consistency data path (product catalog, recommendations, user preferences) routes through a distributed cache with a configurable TTL — typically 30 to 60 seconds.

*Applies **api-gateway** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Implementation Approach: CAP-Aware Architecture in Practice - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

**Observable outcome:** The eventual-consistency data path (product catalog, recommendations, user preferences) routes through a distributed cache with a configurable TTL — typically 30 to 60 seconds.

## End-to-end outcome

In 2011, Netflix engineers deployed Chaos Monkey — a tool that randomly terminated production servers to test system resilience.

## Alternatives considered

### Scalability: Vertical vs. Horizontal

Vertical scaling (scale up) means adding more power to a single machine — more CPU cores, more RAM, faster storage.

### ACID vs. BASE: Two Ways to Think About Data Correctness

ACID (traditional relational databases): - Atomic — all operations in a transaction succeed or all fail - Consistent — the database moves from one valid state to another valid state - Isolated — concurrent transactions don't interfere with each other - Durable — committed data survives crashes BASE (most NoSQL / distributed systems): - Basically Available — availability is prioritized; partial failures are acceptable - Soft state — data can be stale; state may change without input (replication catching up) - Eventual consistency — given no new writes, all replicas will eventually converge to the same value

### Decision Guide: Choosing the Right Consistency Level for Your Use Case

Decision Guide: Choosing the Right Consistency Level for Your Use Case


## Failure and degraded modes

### Trade-offs & Failure Modes: When Things Go Wrong: Partition Scenarios

- **Detection:** A network partition means some nodes can't communicate with others.
- **User or operator impact:** A network partition means some nodes can't communicate with others.



Start with the system design interview loop, requirements, capacity estimates, and core scalability trade-offs.

Interview Foundations

## Scenario context

In system design interviews, API design is not a list of HTTP verbs. It is a contract strategy: clear resource boundaries, stable request and response shapes, pagination, idempotency, error semantics, and versioning decisions that survive scale and failures.

> **TLDR**: Good API design reduces ambiguity for clients and prevents operational incidents when traffic grows.

In 2015, Stripe renamed the `amount` field to…

This walkthrough focuses on **Why API Design Is an Architecture Decision, Not a Syntax Exercise**.

Before making a design choice, clarify:

- In 2015, Stripe renamed the amount field to amount cents in their charge response object — a clarification that made semantic sense internally.
- They had an active changelog and sent developer notifications.
- It still silently broke integrations for hundreds of third-party apps and payment platforms that parsed the response without strict field validation.

**Concepts in play:** system-design-hld-architectures, caching, git-internals-and-versioning, api-gateway

## Stage 1 — The API Contract Checklist You Should Apply in Every Interview

- You can use a reusable checklist to keep API design systematic.
- Define the resource model and identifiers.
- Define the core operations per resource.
- Define request and response fields with explicit constraints.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** Define request and response fields with explicit constraints.

## Stage 2 — Deep Dive: Translating Product Behavior Into Stable API Contracts

- API contracts are where product semantics become system boundaries.
- The Internals: Validation, Idempotency Store, and Backward Compatibility At runtime, robust API services usually implement these internal mechanisms: - Request validation layer for schema and semantic rules.
- - Idempotency key store for safe retried writes.
- - Serialization logic with explicit field defaults.

*Applies **caching** to the decision above.*

**Observable outcome:** - Serialization logic with explicit field defaults.

## Stage 3 — API Lifecycle Flow From Client Request to Stable Response

- This flowchart captures the API contract philosophy: consistent validation, safe retries, and predictable responses.
- 📊 REST API Request Lifecycle This sequence diagram traces the full lifecycle of a single REST API request from the client through the gateway and into the order service, highlighting the idempotency key check and the d…
- The flow shows that the API gateway handles cross-cutting concerns (auth, rate limiting) while the service layer owns business validation and deduplication.
- Take away: every hop in this chain is an opportunity for failure, which is why idempotency at the service boundary prevents duplicate side effects from network retries.

*Applies **git-internals-and-versioning** to the decision above.*

```mermaid
flowchart TD
    A[Client request] --> B[Schema and semantic validation]
    B --> C{Idempotency key present?}
    C -->|Yes| D[Check idempotency store]
    D --> E{Seen before?}
    E -->|Yes| F[Return previous response]
    E -->|No| G[Execute business transaction]
    C -->|No| G
    G --> H[Persist result and emit event]
    H --> I[Return structured response]

```

```mermaid
sequenceDiagram
    participant C as Client
    participant G as API Gateway
    participant S as Order Service
    participant DB as Database
    C->>G: POST /v1/orders
    G->>G: Auth and rate limit
    G->>S: Forward request
    S->>S: Validate schema
    S->>DB: Check idempotency key
    DB-->>S: Key not found
    S->>DB: Write order record
    DB-->>S: Commit OK
    S-->>G: 201 Created
    G-->>C: 201 + order_id

```

**Observable outcome:** Take away: every hop in this chain is an opportunity for failure, which is why idempotency at the service boundary prevents duplicate side effects from network retries.

## Stage 4 — Real-World Applications: Payments, Feeds, and Internal Microservices

- Payments API: idempotency is non-negotiable because duplicate charges are business-critical incidents.
- Timeline/feed API: pagination and filtering matter most, because reads dominate and data size grows continuously.
- Internal microservice APIs: strict schemas, backward compatibility, and explicit error contracts reduce coordination cost between teams.
- Different domains stress different contract dimensions, but the checklist remains stable.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** Different domains stress different contract dimensions, but the checklist remains stable.

## Stage 5 — Practical Example: Design a Create-and-List Orders API

- Suppose your interview prompt includes order creation and order history.
- You can propose: - POST /v1/orders with idempotency key.
- - GET /v1/orders/{order id} for direct lookup.
- - GET /v1/orders?customer id=...&cursor=...&limit=...

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** - GET /v1/orders?customer id=...&cursor=...&limit=...

## Stage 6 — Self-Documenting API Contracts: Design Approach

- A production REST API benefits from machine-readable contract documentation that stays in sync with the implementation.
- The key principle is that the contract specification should be derived from the code rather than maintained as a separate manual artifact — ensuring that the documented API always matches what the service actually does.
- In practice, this means annotating each endpoint with its intended summary, description, request parameters, and expected response codes at the point of definition.
- A create-order endpoint would declare that it accepts an idempotency key header, requires a valid request body, and returns 201 on success, 409 on duplicate key conflict, 429 on rate limit, and 422 on malformed input.

*Applies **caching** to the decision above.*

**Observable outcome:** A create-order endpoint would declare that it accepts an idempotency key header, requires a valid request body, and returns 201 on success, 409 on duplicate key conflict, 429 on rate limit, and 422 on malformed input.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - API design is about long-lived contracts, not endpoint naming alone.
- - Idempotency and pagination should be first-class concerns in write and list APIs.

## Alternatives considered

### Decision Guide: REST, RPC, and Contract Complexity

If you need protocol-level trade-offs, pair this post with System Design Protocols: REST, RPC, and TCP/UDP.


## Failure and degraded modes

### Pattern 1: Resource-first endpoint design

- **Detection:** Instead of action-heavy endpoints like /createOrder , design around resources: - POST /orders - GET /orders/{order id} - GET /orders?customer id=...
- **User or operator impact:** Instead of action-heavy endpoints like /createOrder , design around resources: - POST /orders - GET /orders/{order id} - GET /orders?customer id=...

### Pattern 2: Idempotent writes for retry safety

- **Detection:** Client retries are inevitable during network failures.
- **User or operator impact:** Client retries are inevitable during network failures.

### Pattern 3: Cursor-based pagination

- **Detection:** Offset pagination ( page=1000 ) becomes slow and unstable at scale.
- **User or operator impact:** Offset pagination ( page=1000 ) becomes slow and unstable at scale.

### Pattern 4: Structured errors

- **Detection:** Avoid free-form strings as your main failure contract.
- **User or operator impact:** Avoid free-form strings as your main failure contract.

### Trade-offs & Failure Modes: How API Contracts Break at Scale

- **Detection:** The best interview answer names at least one failure mode and one mitigation tied to API contract design.
- **User or operator impact:** The best interview answer names at least one failure mode and one mitigation tied to API contract design.



## Scenario context

When you hit a URL, **DNS** translates the name to an IP, **CDNs** serve static assets from the edge nearest to you, and **Load Balancers** spread traffic across many servers so no single machine becomes a bottleneck. These three layers are the traffic control system of the modern internet.

---

In 2021, GitHub went dark for nearly two hours. The cause was a misconfigured BGP route that blackholed traffic to GitHub…

This walkthrough focuses on **Three Layers That Stand Between a User and Your Server**.

Before making a design choice, clarify:

- In 2021, GitHub went dark for nearly two hours.
- The cause was a misconfigured BGP route that blackholed traffic to GitHub's DNS servers — meaning no request could resolve github.com to an IP address at all, let alone reach an application server.
- The outage was not a code bug or a database crash; it was a failure in the three infrastructure layers that execute before your application code ever runs.

**Concepts in play:** caching, system-design-hld-architectures, load-balancing, jvm-internals-and-garbage-collection

## Stage 1 — The Building Blocks: What Each Component Does

- Every web request to your application routes through three core network layers before reaching your code.
- DNS (Domain Name System) is the phone book of the internet: it translates human-readable hostnames like api.example.com into machine-readable IP addresses.
- Without DNS, users would need to memorize IP addresses directly to reach your service.
- CDN (Content Delivery Network) is a distributed cache of static assets.

*Applies **caching** to the decision above.*

**Observable outcome:** CDN (Content Delivery Network) is a distributed cache of static assets.

## Stage 2 — DNS: The Internet's Phone Book

- DNS maps example.com → 192.0.2.1 .
- The resolution chain has four hops: 1.
- Recursive resolver (usually your ISP or 8.8.8.8) receives the query.
- Root server directs to the TLD server.

*Applies **system-design-hld-architectures** to the decision above.*

```mermaid
sequenceDiagram
    participant B as Browser
    participant OS as OS Cache
    participant RR as Recursive Resolver
    participant Root as Root NS (.)
    participant TLD as TLD NS (.com)
    participant Auth as Authoritative NS

    B->>OS: Resolve api.example.com
    OS-->>B: Cache miss
    B->>RR: Query api.example.com
    RR->>Root: Where is .com?
    Root-->>RR: TLD NS address for .com
    RR->>TLD: Where is example.com?
    TLD-->>RR: Auth NS for example.com
    RR->>Auth: A record for api.example.com?
    Auth-->>RR: 203.0.113.10 (TTL 300s)
    RR-->>B: 203.0.113.10 (cached per TTL)

```

**Observable outcome:** Root server directs to the TLD server.

## Stage 3 — CDNs: Bringing Content Closer to Users

- A CDN is a globally distributed cache.
- When a user requests /static/logo.png , the CDN serves it from an edge PoP (Point of Presence) in their city rather than your origin server in a distant datacenter.
- CDN hit ratio is the fraction of requests served from cache: $$ ext{Avg Latency} = r \cdot L ext{edge} + (1 - r) \cdot L ext{origin}$$ where $r$ is the hit ratio.
- Even improving $r$ from 0.80 to 0.95 dramatically reduces effective latency.

*Applies **load-balancing** to the decision above.*

**Observable outcome:** Even improving $r$ from 0.80 to 0.95 dramatically reduces effective latency.

## Stage 4 — Load Balancers: Distributing Traffic Intelligently

- A load balancer sits in front of your server pool and routes incoming requests: Layer 4 vs Layer 7: Routing algorithms: Health checks probe each backend (HTTP /health , TCP ping) on a configurable interval.
- Unhealthy servers are removed from rotation without human intervention.
- Load imbalance metric: $$ ext{Imbalance} = rac{\max( ext{server QPS})}{ ext{avg}( ext{server QPS})}$$ A ratio near 1.0 means excellent distribution; above 2.0 is a warning sign.
- 📊 CDN Cache Miss and Hit Flow This sequence diagram contrasts a CDN cache miss on the first request with a cache hit on the second request for the same static asset.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```mermaid
flowchart TD
    A[User Browser] -->|DNS lookup returns LB IP| B[Load Balancer L7]
    B -->|Route /api| C[App Server 1]
    B -->|Route /api| D[App Server 2]
    B -->|Route /static| E[CDN Edge Node]
    C --> F[Origin DB]
    D --> F

```

```mermaid
sequenceDiagram
    participant C as Client
    participant E as CDN Edge PoP
    participant O as Origin Server

    C->>E: GET /static/logo.png
    Note over E: Cache MISS (first request)
    E->>O: Forward to origin
    O-->>E: 200 OK + logo.png payload
    E-->>C: 200 OK (served + cached at edge)
    Note over E: Cached for max-age=86400s

    C->>E: GET /static/logo.png
    Note over E: Cache HIT
    E-->>C: 200 OK (from edge, ~5ms)
    Note over O: Origin not contacted on hit

```

**Observable outcome:** 📊 CDN Cache Miss and Hit Flow This sequence diagram contrasts a CDN cache miss on the first request with a cache hit on the second request for the same static asset.

## Stage 5 — Deep Dive: Inside a DNS Resolution and Cache Lifecycle

- When you type a URL, the OS checks its local DNS cache first.
- On a miss, the query travels to a recursive resolver, which fans out to root → TLD → authoritative nameserver.
- Each hop adds a few milliseconds — but the result is cached per TTL, so repeat visits skip all hops.
- This is why TTL tuning matters: a 24-hour TTL means stale IPs persist for 24 hours after a server change.

*Applies **caching** to the decision above.*

**Observable outcome:** This is why TTL tuning matters: a 24-hour TTL means stale IPs persist for 24 hours after a server change.

## Stage 6 — The Full Request Journey: DNS → CDN → Load Balancer → App

- This diagram shows how a single user interaction splits into two parallel paths: the API request travels through DNS resolution to the load balancer and then to an app server, while the static asset request is intercept…
- Notice that the CDN request never touches the load balancer or app server — this is the core value proposition of a CDN, removing an entire class of requests from the origin.
- The combined effect of DNS caching, edge caching, and load-balanced app servers is what allows a modest backend cluster to serve millions of users globally.

*Applies **system-design-hld-architectures** to the decision above.*

```mermaid
sequenceDiagram
    participant U as User Browser
    participant R as Recursive Resolver
    participant A as Authoritative NS
    participant LB as Load Balancer
    participant CDN as CDN Edge
    participant App as App Server

    U->>R: Resolve example.com
    R->>A: Query A record
    A-->>R: Returns LB IP
    R-->>U: LB IP (cached per TTL)
    U->>LB: GET /api/data
    LB->>App: Forward request
    App-->>LB: JSON response
    LB-->>U: JSON response
    U->>CDN: GET /static/logo.png
    CDN-->>U: Serve from edge cache
```

**Observable outcome:** The combined effect of DNS caching, edge caching, and load-balanced app servers is what allows a modest backend cluster to serve millions of users globally.

## End-to-end outcome

In 2021, GitHub went dark for nearly two hours.

## Alternatives considered

### Decision Guide: When to Use What

Decision Guide: When to Use What


## Failure and degraded modes

### Trade-offs & Failure Modes

- **Detection:** Trade-offs & Failure Modes
- **User or operator impact:** Trade-offs & Failure Modes



## Scenario context

🎯 Use REST (HTTP + JSON) for public, browser-facing APIs where interoperability matters. Choose gRPC (HTTP/2 + Protobuf) for internal microservice communication when latency counts. Under the hood, TCP guarantees reliable ordered delivery; UDP trades reliability for speed. Know which layer you're operating at and pick the right tool for each job.

---

gRPC — Google's Remote Procedure Call framework — transfers the…

This walkthrough focuses on **The Protocol Stack: What Talks to What**.

Before making a design choice, clarify:

- gRPC — Google's Remote Procedure Call framework — transfers the same payload as REST in 60–80% fewer bytes by using binary Protobuf encoding instead of verbose JSON text.
- For a service handling one million RPCs per second, that compression gap is roughly the difference between $12,000 and $20,000 per month in bandwidth costs.
- Protocol choice is a real engineering and business decision, not a stylistic preference.

**Concepts in play:** api-gateway, system-design-hld-architectures, load-balancing, stack-and-queue-data-structures

## Stage 1 — How REST Actually Works: Request → Response

- A REST API treats everything as a resource identified by a URL.
- You act on resources using HTTP verbs.
- A GET on /users/42 fetches that user; a POST to /users creates a new one; PUT replaces the entire resource at a given URL; PATCH applies a partial update; and DELETE removes it.
- The HTTP response includes a status code (200 OK, 404 Not Found, 500 Internal Server Error) and usually a JSON body.

*Applies **api-gateway** to the decision above.*

```mermaid
sequenceDiagram
    participant C as Client
    participant LB as Load Balancer
    participant API as API Server
    participant DB as Database

    C->>LB: GET /users/42 (HTTPS)
    LB->>API: Route to available server
    API->>DB: SELECT * WHERE id=42
    DB-->>API: User row
    API-->>LB: 200 OK {id:42, name:"Alice"}
    LB-->>C: 200 OK {id:42, name:"Alice"}
    Note over C,LB: GET is cacheable (Cache-Control)

    C->>LB: POST /users {name:"Bob"}
    LB->>API: Route to server
    API->>DB: INSERT INTO users ...
    DB-->>API: New id=43
    API-->>LB: 201 Created Location:/users/43
    LB-->>C: 201 Created Location:/users/43

```

**Observable outcome:** The HTTP response includes a status code (200 OK, 404 Not Found, 500 Internal Server Error) and usually a JSON body.

## Stage 2 — Deep Dive: How Protobuf Binary Encoding Works

- Protobuf serializes each field as a compact (field number, wire type) tag followed by the value.
- Field names are never sent over the wire — only numeric tags defined in the .proto file.
- A string "hello" in JSON is 7 bytes; in Protobuf it is 5 bytes plus a 1-byte tag.
- For messages sent millions of times per day, this 3–10× size reduction translates directly into bandwidth savings and lower serialization latency.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** For messages sent millions of times per day, this 3–10× size reduction translates directly into bandwidth savings and lower serialization latency.

## Stage 3 — How gRPC Works: Contracts and Binary Efficiency

- gRPC starts with a Protobuf schema ( .proto file) that defines services and message types.
- A service definition declares each RPC method by name, specifying its request type and return type — for example, a GetUser unary call that takes a user ID and returns a User message, or a ListUsers server-streaming cal…
- Message types declare each field with a name, a scalar type (int32, string, bool, etc.), and a unique field number.
- That field number — not the name — is what travels over the wire.

*Applies **load-balancing** to the decision above.*

**Observable outcome:** That field number — not the name — is what travels over the wire.

## Stage 4 — WebSockets: Persistent Bidirectional TCP

- REST is request-response: the client asks, the server answers.
- But what if the server needs to push data to the client without being asked?
- WebSockets upgrade an HTTP connection to a persistent, full-duplex channel.
- Once the handshake completes, both sides can send messages independently at any time.

*Applies **stack-and-queue-data-structures** to the decision above.*

```mermaid
sequenceDiagram
    Client->>Server: HTTP Upgrade: websocket
    Server-->>Client: 101 Switching Protocols
    Server-->>Client: {"event": "price_update", "value": 142.50}
    Client->>Server: {"action": "subscribe", "symbol": "AAPL"}
    Server-->>Client: {"event": "price_update", "value": 143.10}

```

**Observable outcome:** Once the handshake completes, both sides can send messages independently at any time.

## Stage 5 — Protocol Decision Flow

- Use this flowchart to select the right protocol for each service boundary in your system.
- Choose the outermost layer first: if external clients or browsers need access, REST is the safest default.
- For internal high-throughput paths, gRPC's binary encoding reduces payload size by 3–10× compared to JSON.

*Applies **api-gateway** to the decision above.*

```mermaid
flowchart TD
    A[New Service Interface] --> B{External or public-facing?}
    B -->|Yes| C[REST + JSON over HTTPS]
    B -->|No| D{Need real-time streaming?}
    D -->|Server push to browser| E[WebSocket]
    D -->|Internal bidirectional| F[gRPC bidirectional streaming]
    D -->|No streaming needed| G{Latency and throughput critical?}
    G -->|Yes| H[gRPC + Protobuf over HTTP/2]
    G -->|No| C
```

**Observable outcome:** For internal high-throughput paths, gRPC's binary encoding reduces payload size by 3–10× compared to JSON.

## Stage 6 — Practical Protocol Selection Checklist

- This checklist translates the REST-vs-gRPC-vs-UDP decision framework from the protocol sections into a concrete set of diagnostic questions you answer for each service boundary in your system.
- It was structured as a per-protocol checklist because the most common mistake is picking a single protocol for an entire application — in practice, a well-designed system uses different protocols at different layers.
- As you work through each section, answer honestly: if your answers for the gRPC block are all "yes," that boundary is a genuine gRPC candidate; if even one answer is "no," the tooling overhead may not be justified by th…
- For REST: - Is the API public or consumed by third-party developers?

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** For REST: - Is the API public or consumed by third-party developers?

## End-to-end outcome

gRPC — Google's Remote Procedure Call framework — transfers the same payload as REST in 60–80% fewer bytes by using binary Protobuf encoding instead of verbose JSON text.

## Alternatives considered

### REST vs. gRPC: The API Protocol Showdown

These are the two dominant choices for service-to-service communication.

### TCP vs. UDP: The Transport Layer Choice

REST and gRPC both run over TCP .

### Decision Guide: Protocol Selection by Use Case

Decision Guide: Protocol Selection by Use Case


## Failure and degraded modes

### Trade-offs & Failure Modes: Protocol Trade-offs

- **Detection:** simplicity: JSON is easy to debug with curl or browser DevTools, but verbose at scale — large payloads raise bandwidth costs and serialization latency on hot paths.
- **User or operator impact:** simplicity: JSON is easy to debug with curl or browser DevTools, but verbose at scale — large payloads raise bandwidth costs and serialization latency on hot paths.



## Scenario context

A **Reverse Proxy** hides your servers and handles caching/SSL. A **Load Balancer** spreads traffic across server instances. An **API Gateway** manages API concerns — auth, rate limiting, routing, and protocol translation. Modern tools (Nginx, AWS ALB, Kong) often combine all three, but understanding what each layer does independently is essential for system design.






**Core API Gateway responsibilities:**

| Fe…

This walkthrough focuses on **Deep Dive: Why Rate Limiting Belongs at the Gateway, Not the Load Balancer**.

Before making a design choice, clarify:

- Reverse proxies and L4 load balancers are stateless — they route packets without knowing anything about the user making the request.
- An API Gateway is stateful per consumer: it tracks request counts against rate-limit quotas, validates JWT claims, and maps API keys to consumer identities.
- This per-user state is what makes rate limiting possible at the gateway layer but not at the load balancer — which sees only connections, not identities.

**Concepts in play:** load-balancing, api-gateway, rate-limiting, api-security-oauth2

## Stage 1 — Core Mechanics: What Each Layer Actually Does to Requests

- Understanding the mechanics helps you choose the right tool and configure it correctly.
- Reverse proxy mechanics: The proxy receives the client's request and opens a new connection to the backend on the client's behalf.
- The backend only ever sees the proxy's IP.
- SSL is terminated at the proxy — the backend connection can be plain HTTP internally.

*Applies **load-balancing** to the decision above.*

```mermaid
flowchart LR
    TR[Incoming Traffic] --> LB[Load Balancer L4]
    TR --> RP[Reverse Proxy L7]
    TR --> GW[API Gateway L7+]
    LB --> SV[Backend Servers]
    RP --> SV
    GW --> AU[Auth + Rate Limit]
    AU --> MS[Microservices]
```

**Observable outcome:** SSL is terminated at the proxy — the backend connection can be plain HTTP internally.

## Stage 2 — How a Request Travels Through All Three Layers

- The three layers handle distinctly different concerns as the request descends: 1.
- Reverse Proxy handles transport: SSL decryption, static caching, DDoS absorption.
- Load Balancer handles distribution: which instance handles this request, and is that instance healthy?
- API Gateway handles application policy: who is this user, are they allowed, which service should respond?

*Applies **api-gateway** to the decision above.*

```mermaid
flowchart TD
    Client[Browser / Mobile Client] --> RP[Reverse Proxy Nginx  SSL, cache, compress]
    RP -->|Cache miss or dynamic| LB[Load Balancer AWS ALB  distribute traffic]
    LB --> GW[API Gateway Kong  auth, rate limit, route]
    GW --> US[User Service : 8080]
    GW --> OS[Order Service : 8081]
    GW --> PS[Payment Service : 8082]
    US -->|health 200| LB
    OS -->|health 200| LB
    PS -->|health 503| LB
    LB -.->|removed from pool| PS
```

**Observable outcome:** API Gateway handles application policy: who is this user, are they allowed, which service should respond?

## Stage 3 — Real-World Applications: Real-World Deployment Patterns

- Most production stacks do not cleanly separate these three layers into distinct products — they use tools that collapse two or three roles, but the logical separation still matters for debugging and ownership: Netflix u…
- Each layer is independently operable — an Nginx config change doesn't touch auth policy, and a Zuul plugin update doesn't affect load-balancing weights.
- Stripe runs a similar topology: Nginx terminates SSL and absorbs volumetric floods, HAProxy distributes to API pods, and their internal gateway enforces the API key validation and per-customer rate limits that power the…
- The clean separation means their platform team can own rate-limiting policy without touching the Nginx configurations owned by the infra team.

*Applies **rate-limiting** to the decision above.*

**Observable outcome:** The clean separation means their platform team can own rate-limiting policy without touching the Nginx configurations owned by the infra team.

## Stage 4 — Practical Configuration Guide

- Scenario: Adding rate limiting to a new endpoint without touching application code.
- No code change to the Order Service.
- The gateway enforces both auth and rate limiting declaratively.
- Scenario: Checking if a server is actually removed from the load balancer pool.

*Applies **api-security-oauth2** to the decision above.*

```yaml
# Kong declarative config — add rate limiting to /v1/orders
services:
  - nam
e: order-service
    url: http://orders:8081
    routes:
      - nam
e: orders-route
        paths: ["/v1/orders"]
    plugins:
      - nam
e: rate-limiting
        config:
          minute: 60        # 60 requests per minute per consumer
          policy: redis     # shared state across gateway instances
      - nam
e: jwt           # auth before rate limiting
```

```bash
# AWS ALB — list target health for a target group
aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:...

# Output shows which instances are "healthy", "unhealthy", or "draining"
```

**Observable outcome:** Scenario: Checking if a server is actually removed from the load balancer pool.

## Stage 5 — What to Learn Next

- - System Design Networking: DNS, CDNs, and Load Balancers - System Design Core Concepts: Scalability, CAP, and Consistency - Webhooks Explained: Don't Call Us, We'll Call You

*Applies **load-balancing** to the decision above.*

**Observable outcome:** - System Design Networking: DNS, CDNs, and Load Balancers - System Design Core Concepts: Scalability, CAP, and Consistency - Webhooks Explained: Don't Call Us, We'll Call You

## Stage 6 — Spring Cloud Gateway: Routing, Rate Limiting, and Auth Filters in Java

- Spring Cloud Gateway (SCG) is a reactive, non-blocking API gateway built on Spring WebFlux that implements the API Gateway layer described in this post — routing, rate limiting, authentication, and circuit breaking — wi…
- Routes are declared in application.yml (or a Java DSL); filters attach cross-cutting policy to each route.
- SCG integrates natively with Spring Security for JWT validation, Redis for distributed rate-limit counters, and Resilience4j for circuit breaking.
- The Order Service and User Service require zero code changes — rate limiting, circuit breaking, and JWT extraction are enforced entirely at the gateway layer via configuration.

*Applies **api-gateway** to the decision above.*

```yaml
# application.yml — Spring Cloud Gateway declarative route configuration
spring:
  cloud:
    gateway:
      routes:
        # Route 1: user-service — auth + rate limiting
        - id: user-service
          uri: lb://user-service          # lb:// resolves via Eureka/Kubernetes
          predicates:
            - Path=/v1/users/**
          filters:
            - nam
e: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 100   # 100 req/sec baseline
                redis-rate-limiter.burstCapacity: 200   # allows short bursts
                key-resolver: "#{@userKeyResolver}"     # rate-limit per user ID

        # Route 2: order-service — circuit breaker + header rewrite
        - id: order-service
          uri: lb://order-service
          predicates:
            - Path=/v1/orders/**
          filters:
            - nam
e: CircuitBreaker
              args:
                name: orderCB
                fallbackUri: forward:/fallback/orders
            - AddRequestHeader=X-Gateway-Source, spring-cloud-gateway
            - StripPrefix=1                              # remove /v1 before forwarding
```

```java
// KeyResolver — rate-limit by authenticated user extracted from JWT claim
@Bean
public KeyResolver userKeyResolver() {
    return exchange -> exchange.getPrincipal()
        .map(java.security.Principal::getName)
        .defaultIfEmpty("anonymous");
}

// Fallback controller — called when circuit breaker trips
@RestController
public class FallbackController {
    @GetMapping("/fallback/orders")
    public ResponseEntity<Map<String, String>> orderFallback() {
        return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .body(Map.of("error", "Order service is temporarily unavailable"));
    }
}
```

**Observable outcome:** The Order Service and User Service require zero code changes — rate limiting, circuit breaking, and JWT extraction are enforced entirely at the gateway layer via configuration.

## End-to-end outcome

Reverse proxies and L4 load balancers are stateless — they route packets without knowing anything about the user making the request.

## Alternatives considered

### Decision Guide: Choosing the Right Layer: Decision Guide

Decision Guide: Choosing the Right Layer: Decision Guide


## Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.


Learn the request path, API contracts, protocols, and routing layers before drawing boxes.

APIs and Networking

## Scenario context

In system design interviews, data modeling is where architecture meets reality. A good model starts from query patterns, chooses clear entity boundaries, defines indexes deliberately, and includes a schema evolution path so the system can change without breaking reads and writes.

> **TLDR**: If your schema does not match your dominant queries, no amount of caching will save the design.



```architecture
{
  "versi…

This walkthrough focuses on **Why Data Modeling Decides Whether the Architecture Actually Works**.

Before making a design choice, clarify:

- Twitter's 2022 migration from MySQL to Manhattan for timelines required schema evolution across 100+ services.
- One wrong migration wiped user timelines for 30 minutes.
- Schema evolution patterns are the discipline that prevents this.

**Concepts in play:** database-indexing, caching, vector-databases-rag, system-design-hld-architectures

## Stage 1 — Query-Driven Modeling: The Five Inputs You Need Before Choosing Tables

- Before you pick SQL vs NoSQL, normalize vs denormalize, or partition strategy, gather five inputs.
- Top read queries by frequency and latency sensitivity.
- Top write operations by correctness requirements.
- Relationship patterns (one-to-many, many-to-many, graph-like).

*Applies **database-indexing** to the decision above.*

**Observable outcome:** Relationship patterns (one-to-many, many-to-many, graph-like).

## Stage 2 — Core Modeling Decisions: Entities, Keys, Indexes, and Denormalization

- Entity boundaries Start with core domain entities and ownership: - User - Order - OrderItem - Payment Clear boundaries reduce accidental coupling and make migrations safer.
- Key selection Primary keys should support write distribution and identity stability.
- Secondary keys should serve dominant reads.
- Index strategy Indexes speed reads but slow writes and consume storage.

*Applies **caching** to the decision above.*

**Observable outcome:** Index strategy Indexes speed reads but slow writes and consume storage.

## Stage 3 — Deep Dive: How Schema Evolution Prevents Product Growth From Breaking Production

- A static schema is a myth in growing systems.
- New product features, analytics requirements, and compliance constraints force schema evolution.
- The Internals: Expand-Contract Migrations and Backfill Strategy A safe migration pattern is usually "expand-contract": 1.
- Add new nullable columns or new tables (expand).

*Applies **vector-databases-rag** to the decision above.*

```mermaid
flowchart TD
    A[Need schema change?] --> B{Breaking change?}
    B -->|No: add field| C[Expand: add nullable col]
    B -->|Yes: rename/remove| D[Version the schema]
    C --> E[Dual-write old and new]
    D --> E
    E --> F[Backfill historical rows]
    F --> G{Backfill complete?}
    G -->|No| F
    G -->|Yes| H[Switch reads to new field]
    H --> I[Contract: remove old field]

```

**Observable outcome:** Add new nullable columns or new tables (expand).

## Stage 4 — Query-to-Model Workflow for Interview-Grade Data Design

- This flow lets you explain data modeling as a lifecycle, not a one-time DDL event.

*Applies **system-design-hld-architectures** to the decision above.*

```mermaid
flowchart TD
    A[List top queries] --> B[Define entities and ownership]
    B --> C[Choose keys and constraints]
    C --> D[Add indexes for dominant reads]
    D --> E[Validate write cost and consistency]
    E --> F[Plan schema evolution path]
    F --> G[Monitor query drift and adjust]

```

**Observable outcome:** This flow lets you explain data modeling as a lifecycle, not a one-time DDL event.

## Stage 5 — Real-World Applications: Feeds, Checkout, and Multi-Tenant SaaS

- Social feed product: - Read-heavy timelines.
- - Time-ordered queries by user.
- - Often denormalized read stores for latency.
- Checkout and order management: - Strict correctness for inventory and payment linkage.

*Applies **database-indexing** to the decision above.*

**Observable outcome:** Checkout and order management: - Strict correctness for inventory and payment linkage.

## Stage 6 — Practical Example: Modeling Orders for Growth Without Rewrites

- Suppose an e-commerce interview prompt asks for order history, order details, and basic analytics.
- A practical first model: - orders(order id, customer id, status, created at, total amount) - order items(order id, item id, quantity, price) - payments(payment id, order id, status, provider ref, created at) Access patt…
- Add shipping eta field as nullable.
- Dual-write to legacy and new shipment metadata for one release.

*Applies **caching** to the decision above.*

```mermaid
erDiagram
    ORDERS_V1 {
        bigint order_id PK
        bigint customer_id
        string status
        decimal total_amount
        timestamp created_at
        timestamp estimated_delivery
    }
    ORDERS_V2 {
        bigint order_id PK
        bigint customer_id
        string status
        decimal total_amount
        timestamp created_at
        timestamp shipping_eta
    }
    ORDER_ITEMS {
        bigint item_id PK
        bigint order_id FK
        int quantity
        decimal price
    }
    ORDERS_V2 ||--o{ ORDER_ITEMS : has
```

**Observable outcome:** Dual-write to legacy and new shipment metadata for one release.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Query patterns should drive schema decisions.
- - Indexes are performance tools with real write and storage costs.

## Alternatives considered

### Decision Guide: Normalize, Denormalize, or Split Read Models?

When in doubt, start with correctness in the write model, then optimize read paths with controlled denormalization.


## Failure and degraded modes

### Trade-offs & Failure Modes: Common Modeling Mistakes at Scale

- **Detection:** Interviewers value candidates who acknowledge these costs early instead of treating schemas as static diagrams.
- **User or operator impact:** Interviewers value candidates who acknowledge these costs early instead of treating schemas as static diagrams.



## Scenario context

SQL gives you ACID guarantees and powerful relational queries; NoSQL gives you horizontal scale and flexible schemas. The real decision is not "which is better" — it is "which trade-offs align with your workload." Understanding replication, sharding, and the CAP theorem helps you pick correctly every time.

---



```architecture
{
  "version": 1,
  "title": "TLDR:** SQL gives you ACID guarantees and powerful relati…

This walkthrough focuses on **SQL and NoSQL Engines: What Is Actually Inside**.

Before making a design choice, clarify:

- SQL internals SQL engines parse → plan → optimize → execute.
- - B-Tree indexes for range queries.
- - Hash indexes for equality lookups.

**Concepts in play:** consistency-replication, database-indexing, btree-vs-lsmtree, caching

## Stage 1 — Key Properties That Define Each Category

- Before choosing a database, understand four foundational properties that set SQL and NoSQL apart.
- Schema enforcement : SQL requires every row to match the table schema at write time.
- NoSQL is schema-on-read — the collection stores any document shape, and your application code interprets the structure at query time.
- This is powerful for rapid iteration but requires data quality enforcement at the application layer.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** This is powerful for rapid iteration but requires data quality enforcement at the application layer.

## Stage 2 — Scaling Strategies: Vertical, Horizontal, Replication, Sharding

- architecture { "version": 1, "title": "Scaling Strategies: Vertical, Horizontal, Replication, Sharding - Storage Topology", "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and tran…
- A write enters the master's Write-Ahead Log, is applied to the in-memory MemTable, flushed to disk as an SSTable, and then propagated via a replication stream to each replica node, which applies the same changes in orde…
- The key insight is that replicas always trail the primary by some replication lag — reads from a replica may return slightly stale data, which is acceptable for social feeds but unacceptable for financial transactions.
- Replication lag is the time difference between master and replica: $$\Delta = t ext{master} - t ext{replica}$$ A small $\Delta$ is acceptable for social feeds; it is catastrophic for financial ledgers.

*Applies **database-indexing** to the decision above.*

```mermaid

```

```

graph TD
    subgraph Master
        A[Write Request] --> B[Write-Ahead Log]
        B --> C[Apply to MemTable]
        C --> D[Flush to SSTable / Commit]
    end
    subgraph Replicas
        D --> E[Replication Stream]
        E --> F[Apply to Replica Log]
        F --> G[Update Replica State]
    end
    G --> H[Read Request from Replica]

```

**Observable outcome:** Replication lag is the time difference between master and replica: $$\Delta = t ext{master} - t ext{replica}$$ A small $\Delta$ is acceptable for social feeds; it is catastrophic for financial ledgers.

## Stage 3 — Deep Dive: How Write-Ahead Logs and LSM-Trees Protect Data

- Both SQL and NoSQL engines guard against data loss using append-only logs before modifying data files.
- SQL engines use a Write-Ahead Log (WAL) : every change is written sequentially to a log before being applied to B-Tree pages, enabling crash recovery.
- NoSQL LSM-tree engines write first to an in-memory MemTable , flush to immutable SSTables on disk, and periodically compact them.

*Applies **btree-vs-lsmtree** to the decision above.*

**Observable outcome:** NoSQL LSM-tree engines write first to an in-memory MemTable , flush to immutable SSTables on disk, and periodically compact them.

## Stage 4 — Decision Flow: Selecting the Right Database

- Use this flowchart as a first-pass guide when choosing a database for a new service.
- Most production systems combine multiple databases — for example, PostgreSQL for transactional data alongside Redis for caching.
- Start here and refine with your specific read/write ratio and consistency requirements.
- 📊 Read-Your-Writes Consistency in Replicated DB This sequence diagram illustrates the read-your-writes consistency hazard in a replicated database.

*Applies **caching** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Decision Flow: Selecting the Right Database - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

```mermaid
flowchart TD
    A[Define your workload] --> B{Need ACID transactions?}
    B -->|Yes| C[SQL: PostgreSQL / MySQL]
    B -->|No| D{Write-heavy at massive scale?}
    D -->|Yes| E[Wide-column NoSQL: Cassandra]
    D -->|No| F{Schema changes frequently?}
    F -->|Yes| G[Document NoSQL: MongoDB]
    F -->|No| H{Ultra-fast key lookups needed?}
    H -->|Yes| I[Key-Value Store: Redis / DynamoDB]
    H -->|No| C

```

**Observable outcome:** 📊 Read-Your-Writes Consistency in Replicated DB This sequence diagram illustrates the read-your-writes consistency hazard in a replicated database.

## Stage 5 — Choosing Your Database: A Worked Example

- This example models an orders entity in an e-commerce system side-by-side in PostgreSQL and MongoDB to make the SQL-vs-NoSQL trade-off concrete and visible in real code.
- The orders entity is the canonical choice because it sits at the boundary: it demands multi-table joins and ACID guarantees for accurate reporting (SQL's strengths) yet benefits from document embedding when every read n…
- As you read, focus on how the SQL version normalises items into a separate table — enabling flexible queries at the cost of joins — while the MongoDB version embeds items directly in the order document, enabling a singl…
- SQL representation (PostgreSQL): In the normalized SQL model, customer names and order totals live in separate tables linked by a foreign key on user id .

*Applies **consistency-replication** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Choosing Your Database: A Worked Example - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** SQL representation (PostgreSQL): In the normalized SQL model, customer names and order totals live in separate tables linked by a foreign key on user id .

## Stage 6 — Real-World Applications by Workload Type

- 

*Applies **database-indexing** to the decision above.*

**Observable outcome:** You can explain how “Real-World Applications by Workload Type” changes the architecture or implementation path.

## End-to-end outcome

SQL internals SQL engines parse → plan → optimize → execute.

## Alternatives considered

### The Library vs. the Drop Box: SQL and NoSQL Intuition

By 2013, they had 10 billion pins, 100 million users, and a 4-second page load time .

### Decision Guide: SQL vs. NoSQL at a Glance

Decision Guide: SQL vs. NoSQL at a Glance

### Schema Evolution: SQL vs. NoSQL in Practice

One of the most tangible day-to-day differences between SQL and NoSQL is how they handle schema changes as product requirements evolve.


## Failure and degraded modes

### Trade-offs & Failure Modes: Trade-offs and Failure Modes

- **Detection:** Split-brain scenario (master-master setup): 1.
- **User or operator impact:** Split-brain scenario (master-master setup): 1.



## Scenario context

Database selection is a trade-off between consistency, availability, and scalability. By using the CAP Theorem as a compass and matching your data access patterns to the right storage engine (Relational, Document, KV, or Wide-Column), you can build systems that scale without collapsing under architectural debt.



```architecture
{
  "ok": true,
  "diagram": {
    "version": 1,
    "title": "Database Selection Trade…

This walkthrough focuses on **Real-World Applications: Scenario Matching**.

Before making a design choice, clarify:

- Case Study 1: The E-commerce Product Catalog - Data: Product names, descriptions, images, reviews.
- - Pattern: Read-heavy, semi-structured, frequent schema changes (new product attributes).
- - Choice: MongoDB (Document Store) .

**Concepts in play:** caching, consistency-replication, database-indexing, database-sharding

## Stage 1 — The Database Selection Trap

- Imagine you are the founding engineer at a new startup building a global IoT sensor platform.
- You need to ingest millions of data points per second from environmental sensors around the world.
- Your team is comfortable with PostgreSQL, so you spin up a large RDS instance.
- It works beautifully for the first week.

*Applies **caching** to the decision above.*

```architecture
{
  "version": 1,
  "title": "The Database Selection Trap - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** It works beautifully for the first week.

## Stage 2 — Core Mechanics: Sharding and Partitioning

- How do databases actually scale?
- They all eventually run out of room on a single disk.
- The mechanic for solving this is Partitioning (often called Sharding).
- Vertical Partitioning: Putting different tables on different servers (e.g., Users table on DB1, Orders table on DB2).

*Applies **consistency-replication** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Core Mechanics: Sharding and Partitioning - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** Vertical Partitioning: Putting different tables on different servers (e.g., Users table on DB1, Orders table on DB2).

## Stage 3 — Visualizing the Flow of Database Sharding

- architecture { "version": 1, "title": "Visualizing the Flow of Database Sharding - Storage Topology", "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.…
- The application doesn't need to know where the data is; it sends the request to a Proxy.
- Based on the Shard Key (in this case, the user's region), the Proxy routes the request to the correct physical database instance.
- This allows the system to scale infinitely by simply adding more shards.

*Applies **database-indexing** to the decision above.*

```mermaid

```

```

graph TD
    App[Application Layer] --> Proxy[Database Proxy/Router]
    Proxy -->|Shard Key: US| DB_US[Shard 1: US Region]
    Proxy -->|Shard Key: EU| DB_EU[Shard 2: EU Region]
    Proxy -->|Shard Key: ASIA| DB_ASIA[Shard 3: ASIA Region]
    
    subgraph Storage_Layer
        DB_US
        DB_EU
        DB_ASIA
    end
```

**Observable outcome:** This allows the system to scale infinitely by simply adding more shards.

## Stage 4 — Deep Dive: CAP Theorem and The Partition Choice

- The CAP Theorem is the fundamental law of distributed databases.
- 🛡️ The Internals: CP vs AP - CP (Consistency + Partition Tolerance): If the network breaks, the database will stop accepting writes to ensure that it never serves an incorrect value.
- MongoDB and HBase are typically CP.
- - AP (Availability + Partition Tolerance): If the network breaks, the database keeps working, but nodes might temporarily disagree.

*Applies **database-sharding** to the decision above.*

**Observable outcome:** - AP (Availability + Partition Tolerance): If the network breaks, the database keeps working, but nodes might temporarily disagree.

## Stage 5 — Advanced Concepts: NewSQL and Distributed SQL

- A new category called NewSQL (like TiDB or Google Spanner ) attempts to provide the best of both worlds: the SQL interface and ACID transactions of a relational DB with the horizontal scale of NoSQL.
- They achieve this using consensus algorithms like Paxos or Raft to manage state across many nodes.

*Applies **caching** to the decision above.*

**Observable outcome:** They achieve this using consensus algorithms like Paxos or Raft to manage state across many nodes.

## Stage 6 — Practical Example: TiDB (NewSQL)

- TiDB is the leading open-source Distributed SQL database.
- It looks like MySQL to your app but scales like Cassandra.
- Example 1: Horizontal Scaling In a traditional DB, you'd be stuck.
- In TiDB, you just add more TiKV nodes.

*Applies **consistency-replication** to the decision above.*

```bash
# Adding storage capacity in TiDB is a one-command operation
tiup cluster scale-out my-cluster tikv-node-info.yaml
```

```sql
-- This transaction is distributed across multiple storage nodes
-- but remains atomic and consistent.
START TRANSACTION;
UPDATE accounts SET balance = balance - 100 WHERE id = 'Sarah';
UPDATE accounts SET balance = balance + 100 WHERE id = 'James';
COMMIT;
```

**Observable outcome:** In TiDB, you just add more TiKV nodes.

## End-to-end outcome

After working through the stages, you should be able to explain:

- Vertical scaling (bigger RDS instance) takes you further than you think and is much simpler.
- Your code still expects a certain structure.
- - SQL for complexity and integrity.
- - NoSQL for scale and flexibility.

## Alternatives considered

### The Basics of SQL vs. NoSQL

The most common divide is between Relational (SQL) and Non-Relational (NoSQL) databases.

### Decision Guide: The Database Compass

Decision Guide: The Database Compass


## Failure and degraded modes

### Why Your Database Choice is a Trade-off Decision

- **Detection:** In modern system design, there is no "best" database.
- **User or operator impact:** In modern system design, there is no "best" database.

### Trade-offs & Failure Modes

- **Detection:** Denormalized: SQL thrives on normalization (no duplicate data).
- **User or operator impact:** Denormalized: SQL thrives on normalization (no duplicate data).



## Scenario context

Caching is the single highest-leverage performance tool in distributed systems. This guide covers every read/write pattern (Cache-Aside through Refresh-Ahead), every eviction policy (LRU through ARC), cache invalidation pitfalls, thundering herd mitigations, multi-level topologies, and how Twitter, Netflix, and Facebook run caches at planetary scale. Read once; reference forever.

---



```architecture
{
  "version…

This walkthrough focuses on **Why Your Database Is Begging for a Cache**.

Before making a design choice, clarify:

- Every relational database has the same story: it starts fast, traffic grows, queries get slower, you add indexes, you tune joins, and eventually you hit the wall — the database is doing too much I/O for too many concurr…
- The standard answer is a cache.
- A cache is a key-value store that lives in RAM.

**Concepts in play:** caching, consistency-replication, system-design-hld-architectures, database-indexing

## Stage 1 — How Caches Fit Into the Read and Write Path

- Before diving into patterns, ground yourself in what a cache is not: it is not a database.
- Caches are: - Volatile — data survives only as long as the cache process is running (unless persistence is explicitly configured, as in Redis RDB/AOF modes).
- - Bounded — caches have a fixed memory budget and cannot grow infinitely.
- When the budget fills, the eviction policy decides what to drop.

*Applies **caching** to the decision above.*

**Observable outcome:** When the budget fills, the eviction policy decides what to drop.

## Stage 2 — Every Cache Pattern Explained: From Cache-Aside to Refresh-Ahead

- There are six canonical caching patterns.
- Each solves a different problem and carries different consistency trade-offs.
- A typical production system mixes them — Cache-Aside for general reads, Write-Through for high-value writes, and Refresh-Ahead for peak-traffic hot keys.
- Pattern 1: Cache-Aside (Lazy Loading) The most common pattern.

*Applies **consistency-replication** to the decision above.*

```mermaid
flowchart TD
    Request[Incoming Request] --> CheckCache{Check Cache}
    CheckCache -->|Hit| ReturnCached[Return cached data instantly]
    CheckCache -->|Miss| QueryDB[Query Database]
    QueryDB --> PopulateCache[Write result to cache with TTL]
    PopulateCache --> ReturnFresh[Return fresh data to caller]

```

```mermaid
flowchart LR
    subgraph WT[Write-Through]
        WT_App[Application] -->|Write| WT_Cache[Cache]
        WT_Cache -->|Sync write| WT_DB[Database]
        WT_DB -->|Acknowledge| WT_App
    end
    subgraph WB[Write-Back]
        WB_App[Application] -->|Write| WB_Cache[Cache]
        WB_Cache -->|Acknowledge immediately| WB_App
        WB_Cache -.->|Async flush later| WB_DB[Database]
    end

```

**Observable outcome:** Pattern 1: Cache-Aside (Lazy Loading) The most common pattern.

## Stage 3 — Deep Dive: Eviction Policies, Invalidation, and the Thundering Herd

- The Internals: LRU, LFU, ARC, SLRU, and How They Actually Work When the cache is full and a new key must be inserted, the eviction policy selects the victim to remove.
- Choosing the wrong policy wastes memory, degrades hit rates, and in pathological cases (scan pollution), can evict your entire working set.
- LRU — Least Recently Used The most widely deployed policy.
- Internally implemented as a doubly linked list + hash map : the hash map provides O(1) key lookup; the linked list maintains access recency from most-recent (head) to least-recent (tail).

*Applies **system-design-hld-architectures** to the decision above.*

```mermaid
flowchart LR
    Head[HEAD - Most Recent] --> N1[Key C] --> N2[Key A] --> N3[Key F] --> Tail[TAIL - Evict First]

```

**Observable outcome:** Internally implemented as a doubly linked list + hash map : the hash map provides O(1) key lookup; the linked list maintains access recency from most-recent (head) to least-recent (tail).

## Stage 4 — Cache Topologies: Local, Distributed, CDN, and Multi-Level Hierarchies

- Caches exist at multiple levels in a real system.
- Understanding the latency and consistency characteristics of each level is essential for designing the right topology for your workload.
- The four standard layers, ordered from fastest to slowest, are shown below.
- Each hop in the diagram represents an order-of-magnitude latency jump: L1 local cache (nanoseconds), L2 distributed cache (1–2 ms), database (10–50 ms), CDN edge (20–80 ms to nearest PoP vs.

*Applies **database-indexing** to the decision above.*

```mermaid
flowchart LR
    Client[Client Request] --> CDN[CDN Edge Cache]
    CDN -->|Cache miss| LB[Load Balancer]
    LB --> App[App Server]
    App --> L1[L1 Local Cache]
    L1 -->|Miss| L2[L2 Distributed Cache]
    L2 -->|Miss| DB[Database]
```

**Observable outcome:** Each hop in the diagram represents an order-of-magnitude latency jump: L1 local cache (nanoseconds), L2 distributed cache (1–2 ms), database (10–50 ms), CDN edge (20–80 ms to nearest PoP vs.

## Stage 5 — How Twitter, Netflix, Facebook, and Cloudflare Cache at Scale

- Twitter — Redis Sorted Sets for Timeline Assembly Twitter's home timeline is one of the most complex caching problems at scale.
- Assembling a feed from tens of thousands of followed accounts in real time would require expensive fan-in joins across hundreds of database shards on every page load.
- Instead, Twitter pre-computes each user's timeline and caches it in Redis as a sorted set (scores = tweet timestamps).
- Opening the app reads at most 800 timeline entries from Redis — no database joins, purely cache reads answering in sub-millisecond time.

*Applies **caching** to the decision above.*

**Observable outcome:** Opening the app reads at most 800 timeline entries from Redis — no database joins, purely cache reads answering in sub-millisecond time.

## Stage 6 — Sizing and Tuning Your Cache Before It Breaks in Production

- Cache sizing is not optional — a cache that runs out of memory at peak load starts evicting aggressively, hit rate collapses, and the database absorbs a sudden 10× read spike exactly when you can least afford it.
- Step 1 — Measure the working set size.
- The working set is the set of keys that receive at least one access per TTL window.
- Start by logging distinct cache keys accessed over a 5-minute window at peak traffic.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** Start by logging distinct cache keys accessed over a 5-minute window at peak traffic.

## End-to-end outcome

After working through the stages, you should be able to explain:

- Fix the slow query before you cache it.
- A 500 ms query cached for 5 minutes still hammers the database every 5 minutes — and with thundering herd potential every time the TTL fires.

## Alternatives considered

### Selecting a Cache Pattern by Scenario

Selecting a Cache Pattern by Scenario

### Selecting an Eviction Policy by Workload

Selecting an Eviction Policy by Workload

### Selecting a Cache Topology by Need

Selecting a Cache Topology by Need


## Failure and degraded modes

### Consistency vs. Performance: The Core Caching Trade-off

- **Detection:** Every caching decision sits on the consistency–performance axis.
- **User or operator impact:** Every caching decision sits on the consistency–performance axis.



## Scenario context

Sharding means splitting one logical dataset across multiple physical databases so no single node carries all the data and traffic. The hard part is not adding more nodes. The hard part is choosing a shard key that keeps data balanced and queries practical.

> **TLDR**: If your database is too big for one node, sharding is the next scaling step. The wrong shard key turns that scaling step into a permanent operationa…

This walkthrough focuses on **Why Sharding Appears Right After the Simple Database Stops Being Enough**.

Before making a design choice, clarify:

- In an interview, "we can shard later" sounds reasonable only if you understand what that sentence really means.
- It means the current system may eventually outgrow a single storage node in one of three ways: - The dataset no longer fits comfortably on one machine.
- - Write throughput overloads the primary.

**Concepts in play:** database-sharding, caching, consistency-replication, jvm-internals-and-garbage-collection

## Stage 1 — The Three Common Sharding Patterns and What They Optimize

- There are three common beginner-friendly sharding patterns.
- Range sharding: rows are split by value ranges such as user IDs 1-1M on shard A, 1M-2M on shard B.
- It is intuitive but can create hot spots if new writes cluster in one range.
- Hash sharding: a hash of the shard key distributes data more evenly.

*Applies **database-sharding** to the decision above.*

```architecture
{
  "version": 1,
  "title": "The Three Common Sharding Patterns and What They Optimize - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** Hash sharding: a hash of the shard key distributes data more evenly.

## Stage 2 — How a Shard Key Determines Where Every Record Lives

- The shard key is the field or composite of fields used to decide placement.
- In practice, the shard key is often more important than the number of shards.
- Good shard keys usually have three qualities: 1.
- High cardinality so records spread out naturally.

*Applies **caching** to the decision above.*

**Observable outcome:** High cardinality so records spread out naturally.

## Stage 3 — The Data Path: Router to Shard to Rebalancing

- This is the architecture you should keep in your head during an interview.
- The router is only fast when the request includes a routing key.
- Otherwise the system begins to behave like a distributed scatter-gather query engine.
- 📊 Hash Sharding Key Distribution This diagram shows how customer id values are distributed across four database nodes using a modulo hash function.

*Applies **consistency-replication** to the decision above.*

```mermaid
flowchart TD
    C[Client Request] --> R[Shard Router]
    R --> K{Shard key present?}
    K -->|Yes| S1[Route to single shard]
    K -->|No| F[Fan out to many shards]
    S1 --> DB1[(Shard A)]
    F --> DB1
    F --> DB2[(Shard B)]
    F --> DB3[(Shard C)]
    DB1 --> M[Merge results]
    DB2 --> M
    DB3 --> M

```

```mermaid
flowchart LR
    K[customer_id] --> H[HASH_MOD 4]
    H --> S0["Shard 0 (hash=0)"]
    H --> S1["Shard 1 (hash=1)"]
    H --> S2["Shard 2 (hash=2)"]
    H --> S3["Shard 3 (hash=3)"]
    S0 --> D0[(DB Node 0)]
    S1 --> D1[(DB Node 1)]
    S2 --> D2[(DB Node 2)]
    S3 --> D3[(DB Node 3)]

```

**Observable outcome:** 📊 Hash Sharding Key Distribution This diagram shows how customer id values are distributed across four database nodes using a modulo hash function.

## Stage 4 — Real-World Applications: Tenants, Event Streams, and Social Products

- Discord (message sharding at billions of messages): Discord shards message history by channel id across Cassandra clusters.
- Their rule of thumb: 1 logical shard cluster per ~1,000 guilds on average.
- The edge case that breaks this rule: a single guild with 500,000 concurrent users (a major game publisher's community server, for example) creates a hot partition that no even-distribution key can prevent.
- Discord's solution for "celebrity guilds" is explicit manual overrides — routing oversized guilds to dedicated Cassandra nodes rather than placing them on the standard consistent-hash ring.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

**Observable outcome:** Discord's solution for "celebrity guilds" is explicit manual overrides — routing oversized guilds to dedicated Cassandra nodes rather than placing them on the standard consistent-hash ring.

## Stage 5 — Practical Example: Sharding an Orders Table Without Breaking Queries

- Imagine an e-commerce platform whose orders table has become too large for one primary database.
- You have two obvious shard-key choices: - order id - customer id If most queries are "show this customer's order history," then customer id is a better choice because each user's history stays local to one shard.
- If most queries are point lookups by order number, order id may be simpler.
- Now consider operations: - Customer support wants order history by customer.

*Applies **database-sharding** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Practical Example: Sharding an Orders Table Without Breaking Queries - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** Now consider operations: - Customer support wants order history by customer.

## Stage 6 — Apache ShardingSphere and Spring: Custom Shard Routing in Java

- Apache ShardingSphere is an open-source data sharding and routing middleware that integrates with Spring Boot as a transparent JDBC or proxy layer, intercepting SQL queries and routing them to the correct physical shard…
- How it solves the problem: ShardingSphere removes the need to write a custom shard router for every query.
- You declare sharding rules in configuration — the shard key, the hash algorithm, and the physical database mappings — and ShardingSphere rewrites SQL at runtime, transparently routing each query to the correct physical…
- A query that includes customer id in its WHERE clause routes to exactly one shard; a query without a routing key fans out to all shards and ShardingSphere merges the partial results before returning them to the caller.

*Applies **caching** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Apache ShardingSphere and Spring: Custom Shard Routing in Java - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** A query that includes customer id in its WHERE clause routes to exactly one shard; a query without a routing key fans out to all shards and ShardingSphere merges the partial results before returning them to the caller.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Sharding is a data-placement decision before it is a scaling decision.
- - The shard key is the heart of the design.

## Alternatives considered

### Decision Guide: When Should You Actually Shard?

In many interviews, read replicas, caching, or async pipelines are the better first answer.


## Failure and degraded modes

### Trade-offs & Failure Modes: Where Sharding Gets Painful

- **Detection:** This is why the best interview answer often delays sharding until a clear signal appears.
- **User or operator impact:** This is why the best interview answer often delays sharding until a clear signal appears.



Move into databases, schemas, indexes, caching, sharding, and consistency choices.

Storage and Data Modeling

## Scenario context

Message queues and event-driven architecture let services communicate asynchronously, absorb bursty traffic, and isolate failures. The core design challenge is not adding a queue — it is defining delivery semantics, retry behavior, and idempotent consumers.






Queues are not a default replacement for APIs. They are a pressure-relief boundary when synchronous chains become fragile under burst traffic or partial ou…

This walkthrough focuses on **When to Use Event-Driven Queues and When Not To**.

Before making a design choice, clarify:

- When to use - Work is naturally asynchronous: notifications, enrichment, indexing, billing reconciliation.
- - Producer traffic is bursty, but outcomes can be delayed by seconds or minutes.
- - You need durable handoff between independently scaled services.

**Concepts in play:** message-queues, stack-and-queue-data-structures, event-driven-architecture, consistency-replication

## Stage 1 — How the Pattern Works: Producer, Broker, Consumer, Recovery

- The practical flow is simple to explain and strict to implement.
- Producer publishes an event with stable schema and idempotency key.
- Broker persists and routes events by topic/partition.
- Consumer processes event side effects.

*Applies **message-queues** to the decision above.*

**Observable outcome:** Consumer processes event side effects.

## Stage 2 — Deep Dive: Internals That Make or Break Async Reliability

- Internals: Ordering, Acknowledgment, and Schema Evolution Ordering is partition-scoped, not global.
- If one business entity needs strict order, all events for that entity must land on the same partition key.
- Acknowledgment strategy determines correctness: 1.
- If you ack before step 3, crashes can lose work.

*Applies **stack-and-queue-data-structures** to the decision above.*

**Observable outcome:** If you ack before step 3, crashes can lose work.

## Stage 3 — Event Pipeline Flow: Publish, Process, Retry, and Recover

- This is the minimum viable reliability loop.
- If any node is missing, incident load shifts to manual cleanup.
- Every path ends in a safe ack or DLQ route — there is no silent discard.
- 📊 Pub-Sub: Publisher to Subscribers This pub-sub sequence diagram shows how a single OrderCreated event published to a topic fans out simultaneously to two independent subscriber services, each of which processes the e…

*Applies **event-driven-architecture** to the decision above.*

```mermaid



flowchart TD
    A[Producer validates and publishes event] --> B[Broker topic partition]
    B --> C[Consumer reads event]
    C --> D{Idempotency key already processed?}
    D -->|Yes| E[Ack and skip duplicate]
    D -->|No| F[Execute side effect]
    F --> G{Success?}
    G -->|Yes| H[Record dedupe key then ack]
    G -->|No| I[Retry with backoff]
    I --> J{Retry limit reached?}
    J -->|No| C
    J -->|Yes| K[Route to DLQ and alert owner]

```

```mermaid
sequenceDiagram
    participant P as Publisher
    participant T as Topic
    participant S1 as EmailService
    participant S2 as InventoryService
    P->>T: Publish OrderCreated event
    T->>S1: Deliver to EmailService
    T->>S2: Deliver to InventoryService
    S1-->>T: Ack
    S2-->>T: Ack

```

**Observable outcome:** 📊 Pub-Sub: Publisher to Subscribers This pub-sub sequence diagram shows how a single OrderCreated event published to a topic fans out simultaneously to two independent subscriber services, each of which processes the e…

## Stage 4 — Real-World Applications: Flash-Sale Checkout Under Hard Constraints

- Scenario constraints: - 70k checkouts/minute during a 10-minute flash sale.
- - Payment provider has 2% transient timeout rate.
- - Inventory updates must be per-item ordered.
- - Duplicate charge rate must remain under 0.01%.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** - Duplicate charge rate must remain under 0.01%.

## Stage 5 — Lessons From Running Async Systems in Production

- - Queue adoption is only successful when correctness rules are explicit before the first message is sent.
- - Idempotent consumers are mandatory for at-least-once delivery — not optional.
- - Partition strategy is a business decision, not only a scaling decision.
- Order guarantees map to business entities.

*Applies **message-queues** to the decision above.*

**Observable outcome:** Order guarantees map to business entities.

## End-to-end outcome

When to use - Work is naturally asynchronous: notifications, enrichment, indexing, billing reconciliation.

## Alternatives considered

### Decision Guide: Queues vs Synchronous Calls in Architecture Reviews

The retry and dead-letter semantics in AMQP are expressed through the acknowledgment itself.


## Failure and degraded modes

### When to use

- **Detection:** - Work is naturally asynchronous: notifications, enrichment, indexing, billing reconciliation.
- **User or operator impact:** - Work is naturally asynchronous: notifications, enrichment, indexing, billing reconciliation.

### When not to use

- **Detection:** - A user-facing action needs immediate, deterministic completion.
- **User or operator impact:** - A user-facing action needs immediate, deterministic completion.

### Trade-offs & Failure Modes: Trade-offs and Failure Modes: Queue-Centric Design Risks

- **Detection:** Trade-offs & Failure Modes: Trade-offs and Failure Modes: Queue-Centric Design Risks
- **User or operator impact:** Trade-offs & Failure Modes: Trade-offs and Failure Modes: Queue-Centric Design Risks



## Scenario context

Replication means keeping multiple copies of your data so the system can survive machine, process, or availability-zone failures. Failover is the coordinated act of promoting a healthy replica, rerouting traffic, and recovering without corrupting data.

> **TLDR**: If one database going down takes your product down, replication is the next design step. Failover is how you make that redundancy usable.






**How it…

This walkthrough focuses on **Lessons Learned**.

Before making a design choice, clarify:

- - Replication and failover are separate ideas: one copies data, the other restores authority.
- - Read replicas help scale reads, but they do not guarantee fresh data.
- - Asynchronous replication reduces write latency but can lose the newest acknowledged data during a crash.

**Concepts in play:** consistency-replication, system-design-hld-architectures, transaction-isolation-distributed-acid

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Replication and failover are separate ideas: one copies data, the other restores authority.
- - Read replicas help scale reads, but they do not guarantee fresh data.

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.


## Scenario context

Multi-region deployment means running the same system across more than one geographic region so users get lower latency and the business can survive a regional outage. The design challenge is no longer just scaling compute. It is coordinating routing, data replication, and failover without confusing users or losing writes.

> **TLDR**: A backup region sounds simple in interviews, but the real work is deciding where…

This walkthrough focuses on **Why One Region Eventually Becomes a Business Risk**.

Before making a design choice, clarify:

- Single-region architecture is usually the right starting point.
- It keeps operations simple, reduces data coordination, and minimizes cost while the product is still finding its footing.
- Eventually, though, one region becomes a product and business risk.

**Concepts in play:** consistency-replication, caching, canary-deployments, system-design-hld-architectures

## Stage 1 — How Routing, Replication, and Failover Work Across Regions

- Multi-region design has three independent decisions.
- Decision 1: How do users reach a region?
- Common answers include GeoDNS, Anycast, or an edge network that routes users to the nearest healthy region.
- Decision 2: Where are writes accepted?

*Applies **consistency-replication** to the decision above.*

```architecture
{
  "version": 1,
  "title": "How Routing, Replication, and Failover Work Across Regions - Storage Topology",
  "description": "Multi-tier caching and replicated storage pipeline ensuring fast reads and transactional consistency.",
  "view": "data-flow",
  "pattern": "cache-aside",
  "direction": "LR",
  "groups": [
    { "id": "compute", "label": "Application Workers", "tone": "blue" },
    { "id": "cache-tier", "label": "Volatile Memory Tier", "tone": "amber" },
    { "id": "storage-tier", "label": "Persistent Disk Tier", "tone": "green" }
  ],
  "nodes": [
    { "id": "worker", "component": "workflow-service", "label": "Worker Process", "group": "compute" },
    { "id": "mem", "component": "redis", "label": "In-Memory Cache", "group": "cache-tier", "badge": "Hot Keys" },
    { "id": "primary", "component": "postgres", "label": "Primary Database", "group": "storage-tier", "badge": "Write Master" }
  ],
  "edges": [
    { "from": "worker", "to": "mem", "kind": "read", "label": "Fast Read" },
    { "from": "worker", "to": "primary", "kind": "write", "label": "Transactional Commit" }
  ],
  "takeaway": {
    "title": "Storage Access Pattern",
    "body": "In-memory caching handles high-frequency read spikes, reserving disk transactions strictly for state modifications."
  }
}
```

**Observable outcome:** Decision 2: Where are writes accepted?

## Stage 2 — Real-World Applications: Global SaaS, Media Platforms, and Enterprise APIs

- Stripe (active-active across three AWS regions): Stripe runs payment APIs with writes accepted in multiple regions simultaneously using Paxos-based consensus for write coordination.
- The result: RPO ≈ 0, RTO < 30 seconds for a regional failover.
- Before this architecture existed, a single us-east-1 disruption meant 2–5 minutes of degraded payment acceptance — minutes that directly cost revenue.
- Cloudflare (anycast, not GeoDNS): Rather than relying on GeoDNS that shifts traffic only after a health-check TTL expires, Cloudflare announces the same IP from 300+ PoPs via BGP anycast.

*Applies **caching** to the decision above.*

**Observable outcome:** Cloudflare (anycast, not GeoDNS): Rather than relying on GeoDNS that shifts traffic only after a health-check TTL expires, Cloudflare announces the same IP from 300+ PoPs via BGP anycast.

## Stage 3 — Practical Example: Taking a User Settings Service to Two Regions

- Imagine a user settings service that currently runs in one US region.
- Most traffic now comes from North America and Europe.
- The product wants faster European reads and better disaster recovery.
- The first strong design is not active-active writes.

*Applies **canary-deployments** to the decision above.*

**Observable outcome:** The first strong design is not active-active writes.

## Stage 4 — Configuration Management for Multi-Region Deployments

- In a multi-region deployment, each region typically runs the same application image but with region-specific configuration: different database connection strings, different replication role designations (primary versus…
- Managing these differences through environment-specific configuration rather than code branches is a key operational principle — it means the same tested artifact can be deployed anywhere without recompilation or code c…
- A centralized configuration service can serve the appropriate configuration slice to each region at startup, driven by a profile or environment label that is injected at deploy time.
- The primary region's configuration points its data connections at the local write-authoritative database and marks the region as accepting writes.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** The primary region's configuration points its data connections at the local write-authoritative database and marks the region as accepting writes.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Multi-region is a business decision as much as a technical one.
- - Active-passive is usually the best first answer for resilience.

## Alternatives considered

### Active-Passive vs Active-Active: The Two Big Deployment Families

Most interview discussions about multi-region begin with one of two families.

### Decision Guide: When Should You Introduce Multi-Region?

In other words, multi-region is not a maturity badge.


## Failure and degraded modes

### The Request Path Before and After a Regional Failure

- **Detection:** In normal operation, the system may route most traffic to Region A while Region B stays warm.
- **User or operator impact:** In normal operation, the system may route most traffic to Region A while Region B stays warm.

### Trade-offs & Failure Modes: The Cost of Global Reach

- **Detection:** This is why "just add another region" is not a good interview answer by itself.
- **User or operator impact:** This is why "just add another region" is not a good interview answer by itself.



## Scenario context

Service discovery is how clients find the right service instance at runtime, and health checks are how systems decide whether an instance should receive traffic. Together, they turn dynamic infrastructure from guesswork into deterministic routing.

> **TLDR**: If you scale beyond static IPs, discovery plus health-aware routing becomes a core reliability primitive.

In small systems, service communication can start w…

This walkthrough focuses on **Why Service Discovery Is the Invisible Backbone of Modern Systems**.

Before making a design choice, clarify:

- In small systems, service communication can start with fixed hostnames and static configuration.
- That model breaks quickly once autoscaling, rolling deploys, and multi-zone failover enter the picture.
- In production, service instances come and go all day: - New instances launch during traffic spikes.

**Concepts in play:** api-gateway, database-indexing, caching, canary-deployments

## Stage 1 — The Two Discovery Models You Must Distinguish in Interviews

- Service discovery usually appears in one of two patterns.
- Client-side discovery: the client queries a service registry and chooses a backend instance directly.
- This is common in microservice SDKs where clients include load-balancing logic.
- Server-side discovery: the client calls a stable endpoint (for example, a load balancer or API gateway), and that component resolves healthy backends.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** Server-side discovery: the client calls a stable endpoint (for example, a load balancer or API gateway), and that component resolves healthy backends.

## Stage 2 — How Discovery and Health Checks Work End-to-End

- A robust discovery path is usually a loop, not a one-time lookup.
- Service instance starts and registers itself with metadata.
- Registry stores endpoint, zone, version, and status.
- Clients or proxies query for candidate instances.

*Applies **database-indexing** to the decision above.*

**Observable outcome:** Clients or proxies query for candidate instances.

## Stage 3 — Discovery Flow: Registration to Health-Aware Routing

- This model captures the key principle: discovery and health checks are continuous control loops, not setup-time configuration.
- 📊 Service Registration and Client Discovery This sequence diagram traces the full lifecycle of service registration and client-driven discovery.
- OrderService registers with the registry and sends heartbeats every 5 seconds to maintain its healthy status; when a Client queries for available endpoints, the registry returns only healthy instances, and the Client ro…
- Discovery is not a one-time lookup — it is a continuous health-maintained contract that ensures clients never route to stale or unhealthy backends.

*Applies **caching** to the decision above.*

```mermaid



flowchart TD
    A[Instance boots] --> B[Register with service registry]
    B --> C[Heartbeat and metadata updates]
    C --> D{Healthy and ready?}
    D -->|Yes| E[Add to routing pool]
    D -->|No| F[Exclude from routing pool]
    E --> G[Client or proxy resolves target]
    G --> H[Request served]
    F --> I[Recovery or restart]
    I --> C

```

```mermaid
sequenceDiagram
    participant S as OrderService
    participant R as Service Registry
    participant C as Client
    S->>R: Register: host port tags
    R-->>S: ACK registration
    S->>R: Heartbeat every 5s
    C->>R: Discover order-service
    R-->>C: Return healthy endpoints
    C->>S: Route request
    S-->>C: Response

```

**Observable outcome:** Discovery is not a one-time lookup — it is a continuous health-maintained contract that ensures clients never route to stale or unhealthy backends.

## Stage 4 — Real-World Applications: API Gateways, Payments, and Internal Platforms

- HashiCorp Consul at scale: Consul's gossip protocol propagates health changes across a cluster in ~200ms on typical LAN deployments.
- The deregister critical service after field automatically removes services that remain unhealthy beyond a configurable window — preventing stale endpoints from silently accumulating in the registry.
- When a service registers with Consul, it provides its name, port, and metadata tags alongside a health check definition.
- The check specifies the HTTP path to poll, the polling interval, the per-check timeout, and — critically — a window after which Consul automatically deregisters a service that remains in a critical state.

*Applies **canary-deployments** to the decision above.*

**Observable outcome:** The check specifies the HTTP path to poll, the polling interval, the per-check timeout, and — critically — a window after which Consul automatically deregisters a service that remains in a critical state.

## Stage 5 — Practical Example: Evolving a Checkout Service Beyond Static Backends

- Imagine a checkout service initially routed via hardcoded backend IPs.
- Problems appear during traffic spikes: - New app instances launch but receive no traffic.
- - One bad instance still receives requests for minutes.
- - Rolling deploys create intermittent errors from stale endpoint lists.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** - Rolling deploys create intermittent errors from stale endpoint lists.

## Stage 6 — Dynamic Registration and Health-Aware Discovery in Production

- Spring Cloud Netflix Eureka is a client-side service registry built into the Spring Cloud ecosystem; Spring Cloud Consul provides the same programming model backed by HashiCorp Consul's gossip-based registry.
- Both integrate with a discovery client abstraction and a health indicator framework to expose liveness and readiness semantics to the control plane automatically.
- How it works in practice: A microservice opts into dynamic discovery by declaring itself as a discovery client at startup.
- From that point, registration and heartbeat renewal happen automatically — the developer does not manage registry calls manually.

*Applies **database-indexing** to the decision above.*

**Observable outcome:** From that point, registration and heartbeat renewal happen automatically — the developer does not manage registry calls manually.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Service discovery is a control-plane capability, not just a DNS trick.
- - Health checks must distinguish process liveness from real request readiness.

## Alternatives considered

### Decision Guide: Choosing a Discovery Strategy

When unsure in interviews, start with server-side discovery for simpler client behavior, then discuss where client-side control may be worth the complexity.


## Failure and degraded modes

### The Internals: Registries, Heartbeats, TTLs, and Routing Metadata

- **Detection:** Most systems maintain a control plane with these pieces: - Registry store for service instances and metadata.
- **User or operator impact:** Most systems maintain a control plane with these pieces: - Registry store for service instances and metadata.

### Performance Analysis: Lookup Latency, Convergence Time, and Flapping

- **Detection:** Discovery systems are often judged by three metrics.
- **User or operator impact:** Discovery systems are often judged by three metrics.

### Trade-offs & Failure Modes: Where Discovery Can Go Wrong

- **Detection:** The interview-quality answer always includes one sentence like: "I would define clear health semantics and failure thresholds before tuning load-balancer algorithms."
- **User or operator impact:** The interview-quality answer always includes one sentence like: "I would define clear health semantics and failure thresholds before tuning load-balancer algorithms."



## Scenario context

Observability is how you understand system behavior from telemetry, SLOs are explicit reliability targets, and incident response is the execution model when those targets are at risk. Together, they convert operational chaos into measurable, repeatable decision-making.

> **TLDR**: If your architecture has no observability and no SLOs, you do not have reliability engineering, only hopeful monitoring.

Many system de…

This walkthrough focuses on **Why Reliability Conversations Fail Without Observability and SLOs**.

Before making a design choice, clarify:

- Many system design answers stop at infrastructure choices: load balancers, replicas, caches, queues.
- Those components matter, but they do not tell you when users are actually suffering.
- Reliability is fundamentally an outcomes problem: - Are requests succeeding for users?

**Concepts in play:** api-gateway, system-design-hld-architectures, database-indexing, tree-data-structures

## Stage 1 — The Observability Pillars and SLO Vocabulary You Should Use Precisely

- A practical observability model includes: - Metrics for trend and alert thresholds.
- - Logs for event context and forensic detail.
- - Traces for request path latency and dependency attribution.
- SLO language adds decision clarity: - SLI (Service Level Indicator): measured behavior (for example, request success rate).

*Applies **api-gateway** to the decision above.*

**Observable outcome:** SLO language adds decision clarity: - SLI (Service Level Indicator): measured behavior (for example, request success rate).

## Stage 2 — How Telemetry and SLOs Drive Incident Prioritization

- A healthy reliability loop often looks like this: 1.
- Instrument critical user journeys with metrics and traces.
- Define SLOs on user-impacting paths.
- Alert on burn-rate of error budget, not raw noise.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** Alert on burn-rate of error budget, not raw noise.

## Stage 3 — Deep Dive: The Mechanics of Incident-Ready Reliability Engineering

- The Internals: Telemetry Pipelines, Correlation IDs, and Ownership Observability architecture usually has these layers: - Instrumented applications emitting metrics, logs, and traces.
- - Collection agents and pipelines.
- - Storage/index systems with retention policies.
- - Query and dashboard surfaces.

*Applies **database-indexing** to the decision above.*

**Observable outcome:** - Query and dashboard surfaces.

## Stage 4 — Reliability Loop: Measure, Detect, Respond, Improve

- This loop reflects mature operations: reliability is iterative and continuously measured, not fixed once at deploy time.
- 📊 Alert Lifecycle: Metric Spike to Incident The diagram traces the full alert lifecycle from a metrics anomaly to resolution.
- An error rate spike triggers the Alert Engine to evaluate the burn-rate rule; when it fires a FastBurn alert, PagerDuty pages the on-call engineer, who opens an incident channel, inspects traces and logs, and mitigates…
- Each handoff has a clearly defined owner, turning a noisy signal into a resolved incident with structured, predictable escalation rather than reactive firefighting.

*Applies **tree-data-structures** to the decision above.*

```mermaid
flowchart TD
    A[Instrument services] --> B[Collect metrics logs traces]
    B --> C[Evaluate SLI and SLO windows]
    C --> D{Error budget burn high?}
    D -->|No| E[Continue monitoring]
    D -->|Yes| F[Trigger incident response]
    F --> G[Mitigate and restore service]
    G --> H[Post-incident review and action items]
    H --> A

```

```mermaid
sequenceDiagram
    participant M as Metrics
    participant A as Alert Engine
    participant P as PagerDuty
    participant OC as On-Call Engineer
    participant INC as Incident Channel
    M->>A: Error rate spike
    A->>A: Check burn-rate rule
    A->>P: Fire: FastBurn alert
    P->>OC: Page on-call
    OC->>INC: Open incident
    OC->>M: Inspect traces and logs
    OC->>INC: Mitigate and resolve
    INC->>OC: Post-incident review

```

**Observable outcome:** Each handoff has a clearly defined owner, turning a noisy signal into a resolved incident with structured, predictable escalation rather than reactive firefighting.

## Stage 5 — Real-World Applications: Checkout APIs, Search, and Platform Services

- Google SRE (the burn-rate algebra): Google's SRE book formalized error budgets with concrete math.
- A 99.9% monthly SLO allows 43.8 minutes of downtime per month .
- At a 14× normal error rate , one hour of weekly budget burns in just 5 minutes — the threshold that justifies waking someone up at 3 AM.
- At a 1× steady-state rate , the monthly budget exhausts in 30 days — a ticket, not a page.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** At a 1× steady-state rate , the monthly budget exhausts in 30 days — a ticket, not a page.

## Stage 6 — Practical Example: Designing SLOs for an Orders API

- Suppose you run an orders API with these user-critical outcomes: - Place order successfully.
- - Retrieve order status quickly.
- A practical first SLO set: Incident policy example: 1.
- If fast burn-rate exceeds threshold, page primary on-call.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** If fast burn-rate exceeds threshold, page primary on-call.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Observability is useful only when tied to user-impacting objectives.
- - SLOs convert reliability debates into measurable trade-offs.

## Alternatives considered

### Decision Guide: How Much Observability Is Enough?

In interview settings, prioritize user-impacting SLIs first.


## Failure and degraded modes

### Trade-offs & Failure Modes: Common Observability Mistakes

- **Detection:** Strong interview answers include both the technical and organizational side of incident response.
- **User or operator impact:** Strong interview answers include both the technical and organizational side of incident response.



Add queues, replication, failover, observability, and multi-region thinking.

Async Work and Reliability

## Scenario context

A URL shortener is a read-heavy system (100:1 ratio) that maps long URLs to short, unique aliases. The core scaling challenge is generating unique IDs without database contention—solved using a **Range-Based ID Generator** or a **Distributed Counter** with Base62 encoding. A 301/302 redirect strategy combined with aggressive Redis caching ensures sub-10ms response times.


```architecture
{
  "version": 1,
  "title"…

This walkthrough focuses on **Lessons Learned from Production URL Shortener Systems**.

Before making a design choice, clarify:

- In production deployments, the top 1% of links receive over 95% of redirect traffic.
- Standard LRU caching handles this well under steady state, but viral link events — a celebrity post, a product launch — can cause sudden cache misses on previously cold links.
- Production teams implement traffic spike detection that proactively pre-warms cache entries for links crossing a sudden request-rate threshold, preventing the cold-start penalty during viral moments.

**Concepts in play:** caching, system-design-hld-architectures, distributed-locking, event-driven-architecture

## Stage 1 — Key Takeaways: URL Shortener System Design

- - A URL shortener is a read-heavy system (100:1 read/write ratio).
- Optimize every design decision for the redirect read path first; write performance is secondary.
- - Base62 encoding with 6 characters covers 56 billion unique URLs — sufficient for any real-world deployment over the next 47 years.
- - Range-Based Allocation solves the distributed counter bottleneck: each server claims a block of IDs from ZooKeeper and increments an in-memory counter locally, with zero per-request network coordination.

*Applies **caching** to the decision above.*

**Observable outcome:** - Range-Based Allocation solves the distributed counter bottleneck: each server claims a block of IDs from ZooKeeper and increments an in-memory counter locally, with zero per-request network coordination.

## End-to-end outcome

After working through the stages, you should be able to explain:

- In production deployments, the top 1% of links receive over 95% of redirect traffic.
- Standard LRU caching handles this well under steady state, but viral link events — a celebrity post, a product launch — can cause sudden cache misses on previously cold links.
- - A URL shortener is a read-heavy system (100:1 read/write ratio).
- Optimize every design decision for the redirect read path first; write performance is secondary.

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.


## Scenario context

Search autocomplete must respond in sub-10ms to feel "instant." The core trade-off is **Latency vs. Data Freshness**: we use an offline pipeline (Spark) to pre-calculate prefix-to-suggestion mappings and store them in **Redis Sorted Sets** (or a specialized Trie) for near-instant retrieval, while sacrificing real-time updates for most queries.

Imagine you are typing in a search bar: "best res..."
You expect the sys…

This walkthrough focuses on **The "Keystroke" Performance Gap**.

Before making a design choice, clarify:

- Imagine you are typing in a search bar: "best res..." You expect the system to suggest "best restaurants" before you even finish the word "restaurants." An average person types at about 60-80 words per minute, which mea…
- In a poorly designed system, every keystroke triggers a SQL query: SELECT query FROM search logs WHERE query LIKE 'best res%' ORDER BY count DESC LIMIT 5 .
- Under the weight of 100,000 concurrent users, this LIKE query will crush your database.

**Concepts in play:** caching, tree-data-structures, database-indexing, system-design-hld-architectures

## Stage 1 — Search Autocomplete: Use Cases & Requirements

- Actors - Search User: Expects instant, relevant suggestions as they type.
- - Data Scientist / Analyst: Fine-tunes ranking algorithms based on click-through rates.
- - Admin: Manages blacklists for inappropriate or harmful suggestions.
- Functional Requirements - Prefix Matching: Given a prefix (e.g., "ap"), return the top 5-10 most popular suggestions (e.g., "apple", "amazon").

*Applies **caching** to the decision above.*

**Observable outcome:** Functional Requirements - Prefix Matching: Given a prefix (e.g., "ap"), return the top 5-10 most popular suggestions (e.g., "apple", "amazon").

## Stage 2 — Basics: The Trie Data Structure

- At the heart of most autocomplete systems is a Trie (Prefix Tree).
- A Trie is a tree-like data structure where each node represents a character.
- - To find suggestions for "app", you traverse the tree: a - p - p .
- - Every node can store the top 10 most popular queries that pass through it.

*Applies **tree-data-structures** to the decision above.*

**Observable outcome:** - Every node can store the top 10 most popular queries that pass through it.

## Stage 3 — Mechanics: Distribution & Frequency Aggregation

- We don't compute suggestions on-the-fly.
- Instead, we use a Data Ingestion Pipeline : 1.
- Log Collection: Every time a user submits a full search, we log it to a message queue (Kafka).
- Aggregation: A background job (Spark/Flink) aggregates these logs hourly/daily to calculate the "Frequency" of each query.

*Applies **database-indexing** to the decision above.*

**Observable outcome:** Aggregation: A background job (Spark/Flink) aggregates these logs hourly/daily to calculate the "Frequency" of each query.

## Stage 4 — Estimations & Design Goals

- The Math of Search - Daily Active Users (DAU): 100 Million.
- - Total Autocomplete Requests: 100M 10 20 = 20 Billion per day .
- - Average QPS: 20B / 86,400 $\approx 230,000$ requests per second.
- Design Goal: The system must be stateless and horizontally scalable .

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** Design Goal: The system must be stateless and horizontally scalable .

## Stage 5 — High-Level Design: The Two-Path Architecture

- The system is split into the "Fast Path" (Serving) and the "Slow Path" (Analytics).
- The diagram shows the two-path architecture clearly.
- The Fast Path (top) serves user keystrokes directly from the Redis Sorted Set cache with a single ZREVRANGEBYSCORE call — the entire backend lookup completes in under 2ms.
- The Slow Path (bottom) records completed searches to Kafka asynchronously, aggregates frequency counts in Spark on an hourly or daily schedule, and rebuilds the suggestion index in Redis.

*Applies **caching** to the decision above.*

```architecture
{
  "version": 1,
  "title": "High-Level Design: The Two-Path Architecture - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

```mermaid
graph TD
    User((User)) --> LB[Load Balancer]
    LB --> AS[Autocomplete Service]
    
    subgraph Serving_Fast_Path
        AS --> RC[(Suggestion Cache: Redis)]
    end
    
    subgraph Ingestion_Slow_Path
        AS --> MQ[Kafka]
        MQ --> Agg[Aggregation Service: Spark]
        Agg --> PDB[(Search History: Postgres)]
        Agg --> IB[Index Builder]
        IB --> RC
    end
```

**Observable outcome:** The Slow Path (bottom) records completed searches to Kafka asynchronously, aggregates frequency counts in Spark on an hourly or daily schedule, and rebuilds the suggestion index in Redis.

## Stage 6 — Real-World Autocomplete Systems: Google, Amazon, and Elasticsearch

- Google Suggest operates at 230,000 QPS with suggestions updated within 15 minutes of trending events.
- Google's implementation uses a distributed Trie sharded by geographic region and query language.
- Each Trie shard serves a subset of the prefix space (e.g., all prefixes starting with characters A–M in English for the US West Coast region).
- This limits Trie size to under 10GB per shard while achieving sub-1ms traversal latency within the data center.

*Applies **tree-data-structures** to the decision above.*

**Observable outcome:** This limits Trie size to under 10GB per shard while achieving sub-1ms traversal latency within the data center.

## End-to-end outcome

Imagine you are typing in a search bar: "best res..." You expect the system to suggest "best restaurants" before you even finish the word "restaurants." An average person types at about 60-80 words per minute, which means there are only about 100-150 milliseconds between keystrokes.

## Alternatives considered

### The three hardest engineering problems in search autocomplete are choosing the right data structure for sub-10ms prefix lookup, handling typos without a full-text index, and personalizing results without violating the latency budget.

The three hardest engineering problems in search autocomplete are choosing the right data structure for sub-10ms prefix lookup, handling typos without a full-text index, and personalizing results without violating the latency budget.

### Internals: Trie vs. Redis Sorted Set — When Each Wins

A Trie (prefix tree) stores suggestions in a tree where each node represents a character.

### Performance Analysis: Offline Aggregation Pipeline and Index Freshness Budget

The serving latency of under 2ms is achievable only because the suggestion index is pre-computed offline .

### Decision Guide: Serving Strategy by System Scale

Typo tolerance should never be in the critical serving path.


## Failure and degraded modes

### Trade-offs and Failure Modes in Search Autocomplete Design

- **Detection:** Zero-Result Prefix: When a user types a prefix that has no matching entry in the suggestion index (either because it is genuinely rare or because the index hasn't been updated yet), the system must g…
- **User or operator impact:** Zero-Result Prefix: When a user types a prefix that has no matching entry in the suggestion index (either because it is genuinely rare or because the index hasn't been updated yet), the system must g…

### Lessons Learned from Autocomplete Failures at Scale

- **Detection:** A Single Hot Key Took Down the Suggestion Service During a Product Launch.
- **User or operator impact:** A Single Hot Key Took Down the Suggestion Service During a Product Launch.



## Scenario context

Distributed caches trade strict consistency for sub-millisecond read latency, using consistent hashing to scale horizontally without causing database-shattering "cache stampedes" during cluster rebalancing.

Instagram's primary database once served user profile reads at 28,000 QPS until a single viral post triggered a "cache miss storm." This storm drove the database to 95% CPU utilization in seconds, threatening to…

This walkthrough focuses on **Distributed Cache: Use Cases & Requirements**.

Before making a design choice, clarify:

- A distributed cache is an in-memory, key-value store shared across multiple servers.
- Functional Requirements - Standard Operations : Support GET(key) , SET(key, value, TTL) , and DELETE(key) .
- - TTL (Time-To-Live) : Support per-key expiration to ensure data doesn't stay stale forever.

**Concepts in play:** caching, system-design-hld-architectures, consistency-replication, hash-tables-and-bloom-filters

## Stage 1 — Basics: Baseline Architecture

- In a distributed environment, the "Cache" is not one server, but a cluster of servers.
- Cache Client : A library integrated into the application that handles the logic of where to find a key.
- Cache Nodes : Independent servers (like Redis or Memcached instances) that store data in RAM.
- Consistency Mechanism : Logic to ensure that when a database record is updated, the corresponding cache entry is invalidated or updated.

*Applies **caching** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Basics: Baseline Architecture - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

**Observable outcome:** Consistency Mechanism : Logic to ensure that when a database record is updated, the corresponding cache entry is invalidated or updated.

## Stage 2 — Mechanics: Key Logic

- Consistent Hashing The core challenge is: Which node stores key "user:123"?
- - Naive approach : hash(key) % N .
- If you add a node (N becomes N+1), almost every key's location changes.
- This causes a 100% cache miss rate — a "Cache Stampede." - Consistent Hashing : Keys and nodes are mapped onto a logical circle (the "Ring").

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** This causes a 100% cache miss rate — a "Cache Stampede." - Consistent Hashing : Keys and nodes are mapped onto a logical circle (the "Ring").

## Stage 3 — Estimations & Design Goals

- - QPS : 10M reads/sec, 1M writes/sec.
- - Storage : 100 nodes 128GB RAM = 12.8TB total capacity.
- - Object Size : Average 2KB per object.
- A 1% drop in hit rate at 10M QPS adds 100,000 queries per second to the database.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** A 1% drop in hit rate at 10M QPS adds 100,000 queries per second to the database.

## Stage 4 — High-Level Design

- architecture { "version": 1, "title": "High-Level Design - Subsystem Overview", "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.", "view": "high-level", "patte…
- The application checks the Cache Client Library first; the library hashes the key and consults the Consistent Hash Ring to determine which physical node owns that key.
- On a miss (step 6), the application queries the database directly, and the result is written back to the owning cache node (step 8).
- This backfill ensures that subsequent reads for the same key are served from cache without touching the database.

*Applies **hash-tables-and-bloom-filters** to the decision above.*

```mermaid

```

```

graph LR
    App[Application Service] -->|1. Get Key| Client[Cache Client Library]
    Client -->|2. Hash Key| Ring((Consistent Hash Ring))
    Ring -->|3. Route| NodeA[Node A]
    Ring -->|4. Route| NodeB[Node B]
    Ring -->|5. Route| NodeC[Node C]
    
    App -->|6. Miss| DB[(Database)]
    DB -->|7. Backfill| Client
    Client -->|8. Set Key| NodeA
```

**Observable outcome:** This backfill ensures that subsequent reads for the same key are served from cache without touching the database.

## Stage 5 — Deep Dive: Consistent Hashing Ring Internals and LRU Eviction Performance Under Scale

- Two mechanisms dominate the engineering complexity of a distributed cache: determining which node owns a key without causing re-hashing storms when topology changes, and deciding which keys to evict when RAM is full whi…
- Internals: The Consistent Hash Ring and Virtual Nodes Naive modulo sharding — hash(key) % N — has a fatal flaw: when the node count N changes by even 1, almost every key's owning node changes.
- At 10M QPS this produces a complete cache miss storm for tens of seconds, hammering the database to destruction.
- Consistent hashing places both keys and nodes on a conceptual ring using a hash function.

*Applies **caching** to the decision above.*

**Observable outcome:** Consistent hashing places both keys and nodes on a conceptual ring using a hash function.

## Stage 6 — Real-World Cache Strategies: How Twitter, Facebook, and Netflix Use Distributed Caches

- Twitter's Twemproxy is a lightweight cache proxy that sits between application servers and Redis/Memcached instances.
- Twemproxy handles consistent hashing, connection multiplexing, and automatic failover transparently from the application.
- The key insight Twitter solved was connection fan-out: 10,000 application server threads each opening direct connections to 100 cache nodes creates 1,000,000 concurrent connections.
- Twemproxy reduces this to 100 connections from the proxy pool.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** Twemproxy reduces this to 100 connections from the proxy pool.

## End-to-end outcome

A distributed cache is an in-memory, key-value store shared across multiple servers.

## Alternatives considered

### Decision Guide: Redis vs. Memcached and Choosing the Right Cache Pattern

The most common interview mistake is proposing Write-Through caching for all workloads.


## Failure and degraded modes

### Trade-offs and Failure Modes in Distributed Cache Design

- **Detection:** Cache Stampede (Thundering Herd): When a popular key expires, dozens or thousands of simultaneous requests all miss the cache and hit the database concurrently.
- **User or operator impact:** Cache Stampede (Thundering Herd): When a popular key expires, dozens or thousands of simultaneous requests all miss the cache and hit the database concurrently.

### Lessons Learned from Cache Failures at Scale

- **Detection:** The Stampede That Took Down Instagram's Database.
- **User or operator impact:** The Stampede That Took Down Instagram's Database.



## Scenario context

A news feed system builds personalized timelines by combining content publishing, graph relationships, and ranking. The scalability crux is the fan-out amplified write path: a single celebrity post can trigger 100M writes. A hybrid fan-out strategy—push for normal users, pull for celebrities—is the industry standard for 99.99% availability.

Imagine it’s 2013. Twitter is growing at a breakneck pace. Most users have…

This walkthrough focuses on **The Katy Perry Problem**.

Before making a design choice, clarify:

- Twitter is growing at a breakneck pace.
- Most users have a few hundred followers, and the system handles them easily by "pushing" new tweets into their home timelines at write-time.
- She has 100 million followers.

**Concepts in play:** system-design-hld-architectures, caching, graph-traversal-algorithms, event-driven-architecture

## Stage 1 — News Feed: Use Cases, Actors, and Scale Requirements

- Actors - Publisher / Author: Creates and posts content (text, media).
- - Reader / Follower: Consumes a ranked home feed of followed users' posts.
- - System: Handles fan-out, ranking, and timeline materialization.
- Functional Requirements - Post Creation: Users can POST content with text/media.

*Applies **system-design-hld-architectures** to the decision above.*

**Observable outcome:** Functional Requirements - Post Creation: Users can POST content with text/media.

## Stage 2 — Basics of News Feed Architecture

- At its heart, a news feed is a many-to-many relationship pipeline .
- Unlike a simple blog where one post is read by everyone, a feed is a personalized "inbox" for every user.
- Apache Kafka is the decoupling mechanism between post creation and fan-out.
- The posts.created topic is partitioned by author id , ensuring that all posts from one author are processed by the same fan-out worker partition and arrive in creation order.

*Applies **caching** to the decision above.*

**Observable outcome:** The posts.created topic is partitioned by author id , ensuring that all posts from one author are processed by the same fan-out worker partition and arrive in creation order.

## End-to-end outcome

After working through the stages, you should be able to explain:

- The Celebrity Threshold Requires Continuous Calibration.
- Setting the celebrity threshold too low pushes too many accounts into pull fan-out, degrading read performance for mid-tier influencers whose follower lists still take seconds to query.
- - A news feed is a read-heavy, write-amplified system.
- The core design tension is between write-time cost (fan-out to millions of followers) and read-time cost (merging posts from all followed authors on every page load).

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.


## Scenario context

Payment systems optimize for correctness first, then throughput. This guide covers idempotency, double-entry ledgers, and reconciliation.

Stripe processes over 250 million API requests per day, and every single payment must be idempotent: a user clicking "Pay" twice must never produce two charges, even when the first HTTP call timed out and no confirmation reached the client. The Knight Capital incident of 2012 — w…

This walkthrough focuses on **Payment Processing: Use Cases & Requirements**.

Before making a design choice, clarify:

- A robust payment platform must handle several distinct user journeys and operational requirements.
- In an interview, defining these boundaries early is critical.
- Functional Requirements - Idempotent Authorization : Ensure that a payment request is processed exactly once, regardless of retries.

**Concepts in play:** system-design-hld-architectures, caching, consistency-replication, jvm-internals-and-garbage-collection

## Stage 1 — Basics: Baseline Architecture

- At its core, a payment system is a coordinator between three entities: the User (Client) , the Internal Ledger (Source of Truth) , and the External Payment Service Provider (PSP) .
- The baseline architecture consists of: 1.
- Payment API : The entry point that receives requests and orchestrates the flow.
- Idempotency Store : A fast, key-value store (like Redis) to track request keys.

*Applies **system-design-hld-architectures** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Basics: Baseline Architecture - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

**Observable outcome:** Idempotency Store : A fast, key-value store (like Redis) to track request keys.

## Stage 2 — Mechanics: Key Logic

- The most critical logic in a payment system is the Persist-Before-Call pattern combined with Double-Entry Bookkeeping .
- The Persist-Before-Call Pattern To prevent lost payments, you must record the intent to pay in your database before calling the external PSP.
- This ensures that if the call to the PSP times out or the server crashes, you have a record to reconcile against later.
- Double-Entry Bookkeeping Money is never simply "updated" in a row.

*Applies **caching** to the decision above.*

```mermaid
sequenceDiagram
    participant U as User
    participant API as Payment API
    participant Redis as Idempotency Store
    participant DB as Ledger DB
    participant PSP as Payment Provider
    U->>API: POST /payments with Idempotency-Key
    API->>Redis: Check existing key
    Redis-->>API: Not found — proceed
    API->>DB: Write PENDING intent
    DB-->>API: Intent persisted
    API->>PSP: Authorize charge with intent_id
    PSP-->>API: Authorization success
    API->>DB: Update status to AUTHORIZED
    API->>Redis: Cache response with 24h TTL
    API-->>U: Payment confirmed
```

**Observable outcome:** Double-Entry Bookkeeping Money is never simply "updated" in a row.

## Stage 3 — Estimations & Design Goals

- Before diving into the high-level design, let's establish the scale.
- - Throughput : 1,000 Transactions Per Second (TPS) average, 5,000 TPS peak (e.g., Black Friday).
- - Storage : Each transaction record is ~2KB.
- At 1,000 TPS, that's 2MB/sec, or ~172GB/day.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** At 1,000 TPS, that's 2MB/sec, or ~172GB/day.

## Stage 4 — High-Level Design

- The following diagram illustrates the orchestrated flow of a payment transaction.
- The flow enforces the Persist-Before-Call pattern: the Payment API writes a PENDING intent record to the PostgreSQL ledger (step 2) before calling the external PSP (step 3).
- This ensures that even if the network drops between steps 3 and 4, the system has a record to reconcile against.
- The idempotency check in Redis (step 1) prevents a duplicate upstream call if the client retries with the same key.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```architecture
{
  "version": 1,
  "title": "High-Level Design - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

```mermaid
graph TD
    User[Customer/App] -->|POST /payments| API[Payment API]
    API -->|1. Check/Lock Key| Redis[(Idempotency Redis)]
    API -->|2. Create Intent| Ledger[(PostgreSQL Ledger)]
    API -->|3. Authorize| PSP[Payment Provider - Stripe/Adyen]
    PSP -->|4. Response| API
    API -->|5. Update Status| Ledger
    API -->|6. Cache Result| Redis
    Ledger -->|7. CDC Event| Kafka{Kafka}
    Kafka -->|8. Notify| Webhook[Webhook Service]
    Webhook -->|9. Callback| Merchant[Merchant Server]
    Recon[Reconciliation Job] -->|Daily Sync| PSP
    Recon -->|Verify| Ledger
```

**Observable outcome:** The idempotency check in Redis (step 1) prevents a duplicate upstream call if the client retries with the same key.

## Stage 5 — Deep Dive: Double-Entry Bookkeeping, Idempotency Key Lifecycle, and the Reconciliation Engine

- Three mechanisms define correctness in a payment system: double-entry bookkeeping that makes every financial state change auditable, idempotency that prevents duplicate charges from network retries, and reconciliation t…
- Internals: Double-Entry Bookkeeping and Why Integer Cents Matter In a double-entry ledger, money is never simply "updated in a row." Every movement of value produces at minimum two ledger entries: a debit from one accou…
- When a customer pays $100 to a merchant, the system writes: DEBIT $100 from the customer's account, CREDIT $100 to the merchant's escrow account.
- The fundamental accounting invariant is that the sum of all debits must equal the sum of all credits across the entire ledger at all times.

*Applies **system-design-hld-architectures** to the decision above.*

```mermaid
flowchart TD
    A["Request Received with Idempotency-Key"] --> B{"Key Exists in Redis?"}
    B -->|Yes| C["Return Cached Response Immediately"]
    B -->|No| D["Acquire Distributed Lock — 30s TTL"]
    D --> E["Write PENDING Intent to Ledger"]
    E --> F["Call External PSP"]
    F --> G{"PSP Response Received?"}
    G -->|Success| H["Update Ledger to AUTHORIZED"]
    G -->|Failure| I["Update Ledger to FAILED"]
    H --> J["Cache Success Response — 24h TTL"]
    I --> J
    J --> K["Release Lock and Return to Caller"]
```

**Observable outcome:** The fundamental accounting invariant is that the sum of all debits must equal the sum of all credits across the entire ledger at all times.

## Stage 6 — Real-World Payment Architectures: Stripe, PayPal, and Square

- Stripe built its idempotency key system as a first-class API primitive from day one.
- Every write endpoint in the Stripe API accepts an Idempotency-Key header.
- Stripe stores the idempotency key, the request fingerprint (URL + body hash), and the response body in a dedicated idempotency database with a 24-hour retention window.
- Critically, Stripe validates that the same idempotency key is not used with a different request body — preventing a class of bugs where a client reuses a key across conceptually different operations.

*Applies **caching** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Real-World Payment Architectures: Stripe, PayPal, and Square - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

**Observable outcome:** Critically, Stripe validates that the same idempotency key is not used with a different request body — preventing a class of bugs where a client reuses a key across conceptually different operations.

## End-to-end outcome

A robust payment platform must handle several distinct user journeys and operational requirements.

## Alternatives considered

### Decision Guide: Choosing the Right Payment Guarantee Level for Each Operation

Decision Guide: Choosing the Right Payment Guarantee Level for Each Operation


## Failure and degraded modes

### Trade-offs and Failure Modes in Payment System Design

- **Detection:** The Double-Charge Failure Mode: The API calls the PSP, the PSP charges the card and returns success, but the network drops before the response reaches the API.
- **User or operator impact:** The Double-Charge Failure Mode: The API calls the PSP, the PSP charges the card and returns success, but the network drops before the response reaches the API.

### Lessons Learned from Payment System Failures

- **Detection:** The Knight Capital Incident: No Idempotency Guard on Algorithmic Orders.
- **User or operator impact:** The Knight Capital Incident: No Idempotency Guard on Algorithmic Orders.



Practice full designs from simple to harder product surfaces.

Case Studies

Move from fundamentals to real design use cases: requirements, APIs, storage, async workflows, distributed trade-offs, and operations.