System Design HLD Example: Distributed Cache Platform

Scenario context

Distributed caches trade strict consistency for sub-millisecond read latency, using consistent hashing to scale horizontally without causing database-shattering "cache stampedes" during cluster rebalancing.

Instagram's primary database once served user profile reads at 28,000 QPS until a single viral post triggered a "cache miss storm." This storm drove the database to 95% CPU utilization in seconds, threatening to…

This walkthrough focuses on Distributed Cache: Use Cases & Requirements.

Before making a design choice, clarify:

• A distributed cache is an in-memory, key-value store shared across multiple servers.
• Functional Requirements - Standard Operations : Support GET(key) , SET(key, value, TTL) , and DELETE(key) .
• • TTL (Time-To-Live) : Support per-key expiration to ensure data doesn't stay stale forever.

Concepts in play: caching, system-design-hld-architectures, consistency-replication, hash-tables-and-bloom-filters

Stage 1 — Basics: Baseline Architecture

• In a distributed environment, the "Cache" is not one server, but a cluster of servers.
• Cache Client : A library integrated into the application that handles the logic of where to find a key.
• Cache Nodes : Independent servers (like Redis or Memcached instances) that store data in RAM.
• Consistency Mechanism : Logic to ensure that when a database record is updated, the corresponding cache entry is invalidated or updated.

Observable outcome: Consistency Mechanism : Logic to ensure that when a database record is updated, the corresponding cache entry is invalidated or updated.

Stage 2 — Mechanics: Key Logic

• Consistent Hashing The core challenge is: Which node stores key "user:123"?
• • Naive approach : hash(key) % N .
• If you add a node (N becomes N+1), almost every key's location changes.
• This causes a 100% cache miss rate — a "Cache Stampede." - Consistent Hashing : Keys and nodes are mapped onto a logical circle (the "Ring").

Observable outcome: This causes a 100% cache miss rate — a "Cache Stampede." - Consistent Hashing : Keys and nodes are mapped onto a logical circle (the "Ring").

Stage 3 — Estimations & Design Goals

• • QPS : 10M reads/sec, 1M writes/sec.
• • Storage : 100 nodes 128GB RAM = 12.8TB total capacity.
• • Object Size : Average 2KB per object.
• A 1% drop in hit rate at 10M QPS adds 100,000 queries per second to the database.

Observable outcome: A 1% drop in hit rate at 10M QPS adds 100,000 queries per second to the database.

Stage 4 — High-Level Design

• architecture { "version": 1, "title": "High-Level Design - Subsystem Overview", "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.", "view": "high-level", "patte…
• The application checks the Cache Client Library first; the library hashes the key and consults the Consistent Hash Ring to determine which physical node owns that key.
• On a miss (step 6), the application queries the database directly, and the result is written back to the owning cache node (step 8).
• This backfill ensures that subsequent reads for the same key are served from cache without touching the database.

graph LR
    App[Application Service] -->|1. Get Key| Client[Cache Client Library]
    Client -->|2. Hash Key| Ring((Consistent Hash Ring))
    Ring -->|3. Route| NodeA[Node A]
    Ring -->|4. Route| NodeB[Node B]
    Ring -->|5. Route| NodeC[Node C]

    App -->|6. Miss| DB[(Database)]
    DB -->|7. Backfill| Client
    Client -->|8. Set Key| NodeA

Observable outcome: This backfill ensures that subsequent reads for the same key are served from cache without touching the database.

Stage 5 — Deep Dive: Consistent Hashing Ring Internals and LRU Eviction Performance Under Scale

• Two mechanisms dominate the engineering complexity of a distributed cache: determining which node owns a key without causing re-hashing storms when topology changes, and deciding which keys to evict when RAM is full whi…
• Internals: The Consistent Hash Ring and Virtual Nodes Naive modulo sharding — hash(key) % N — has a fatal flaw: when the node count N changes by even 1, almost every key's owning node changes.
• At 10M QPS this produces a complete cache miss storm for tens of seconds, hammering the database to destruction.
• Consistent hashing places both keys and nodes on a conceptual ring using a hash function.

Observable outcome: Consistent hashing places both keys and nodes on a conceptual ring using a hash function.

Stage 6 — Real-World Cache Strategies: How Twitter, Facebook, and Netflix Use Distributed Caches

• Twitter's Twemproxy is a lightweight cache proxy that sits between application servers and Redis/Memcached instances.
• Twemproxy handles consistent hashing, connection multiplexing, and automatic failover transparently from the application.
• The key insight Twitter solved was connection fan-out: 10,000 application server threads each opening direct connections to 100 cache nodes creates 1,000,000 concurrent connections.
• Twemproxy reduces this to 100 connections from the proxy pool.

Observable outcome: Twemproxy reduces this to 100 connections from the proxy pool.

End-to-end outcome

A distributed cache is an in-memory, key-value store shared across multiple servers.

Alternatives considered

Decision Guide: Redis vs. Memcached and Choosing the Right Cache Pattern

The most common interview mistake is proposing Write-Through caching for all workloads.

Failure and degraded modes

Trade-offs and Failure Modes in Distributed Cache Design

• Detection: Cache Stampede (Thundering Herd): When a popular key expires, dozens or thousands of simultaneous requests all miss the cache and hit the database concurrently.
• User or operator impact: Cache Stampede (Thundering Herd): When a popular key expires, dozens or thousands of simultaneous requests all miss the cache and hit the database concurrently.

Lessons Learned from Cache Failures at Scale

• Detection: The Stampede That Took Down Instagram's Database.
• User or operator impact: The Stampede That Took Down Instagram's Database.