System Design: Complete Guide to Caching — Patterns, Eviction, and Distributed Strategies

Scenario context

Caching is the single highest-leverage performance tool in distributed systems. This guide covers every read/write pattern (Cache-Aside through Refresh-Ahead), every eviction policy (LRU through ARC), cache invalidation pitfalls, thundering herd mitigations, multi-level topologies, and how Twitter, Netflix, and Facebook run caches at planetary scale. Read once; reference forever.

flowchart LR
    subgraph WT[Write-Through]
        WT_App[Application] -->|Write| WT_Cache[Cache]
        WT_Cache -->|Sync write| WT_DB[Database]
        WT_DB -->|Acknowledge| WT_App
    end
    subgraph WB[Write-Back]
        WB_App[Application] -->|Write| WB_Cache[Cache]
        WB_Cache -->|Acknowledge immediately| WB_App
        WB_Cache -.->|Async flush later| WB_DB[Database]
    end

Observable outcome: Pattern 1: Cache-Aside (Lazy Loading) The most common pattern.

Stage 3 — Deep Dive: Eviction Policies, Invalidation, and the Thundering Herd

• The Internals: LRU, LFU, ARC, SLRU, and How They Actually Work When the cache is full and a new key must be inserted, the eviction policy selects the victim to remove.
• Choosing the wrong policy wastes memory, degrades hit rates, and in pathological cases (scan pollution), can evict your entire working set.
• LRU — Least Recently Used The most widely deployed policy.
• Internally implemented as a doubly linked list + hash map : the hash map provides O(1) key lookup; the linked list maintains access recency from most-recent (head) to least-recent (tail).

flowchart LR
    Head[HEAD - Most Recent] --> N1[Key C] --> N2[Key A] --> N3[Key F] --> Tail[TAIL - Evict First]

Observable outcome: Internally implemented as a doubly linked list + hash map : the hash map provides O(1) key lookup; the linked list maintains access recency from most-recent (head) to least-recent (tail).

Stage 4 — Cache Topologies: Local, Distributed, CDN, and Multi-Level Hierarchies

• Caches exist at multiple levels in a real system.
• Understanding the latency and consistency characteristics of each level is essential for designing the right topology for your workload.
• The four standard layers, ordered from fastest to slowest, are shown below.
• Each hop in the diagram represents an order-of-magnitude latency jump: L1 local cache (nanoseconds), L2 distributed cache (1–2 ms), database (10–50 ms), CDN edge (20–80 ms to nearest PoP vs.

flowchart LR
    Client[Client Request] --> CDN[CDN Edge Cache]
    CDN -->|Cache miss| LB[Load Balancer]
    LB --> App[App Server]
    App --> L1[L1 Local Cache]
    L1 -->|Miss| L2[L2 Distributed Cache]
    L2 -->|Miss| DB[Database]

Observable outcome: Each hop in the diagram represents an order-of-magnitude latency jump: L1 local cache (nanoseconds), L2 distributed cache (1–2 ms), database (10–50 ms), CDN edge (20–80 ms to nearest PoP vs.

Stage 5 — How Twitter, Netflix, Facebook, and Cloudflare Cache at Scale

• Twitter — Redis Sorted Sets for Timeline Assembly Twitter's home timeline is one of the most complex caching problems at scale.
• Assembling a feed from tens of thousands of followed accounts in real time would require expensive fan-in joins across hundreds of database shards on every page load.
• Instead, Twitter pre-computes each user's timeline and caches it in Redis as a sorted set (scores = tweet timestamps).
• Opening the app reads at most 800 timeline entries from Redis — no database joins, purely cache reads answering in sub-millisecond time.

Observable outcome: Opening the app reads at most 800 timeline entries from Redis — no database joins, purely cache reads answering in sub-millisecond time.

Stage 6 — Sizing and Tuning Your Cache Before It Breaks in Production

• Cache sizing is not optional — a cache that runs out of memory at peak load starts evicting aggressively, hit rate collapses, and the database absorbs a sudden 10× read spike exactly when you can least afford it.
• Step 1 — Measure the working set size.
• The working set is the set of keys that receive at least one access per TTL window.
• Start by logging distinct cache keys accessed over a 5-minute window at peak traffic.

Observable outcome: Start by logging distinct cache keys accessed over a 5-minute window at peak traffic.

End-to-end outcome

After working through the stages, you should be able to explain:

• Fix the slow query before you cache it.
• A 500 ms query cached for 5 minutes still hammers the database every 5 minutes — and with thundering herd potential every time the TTL fires.

Alternatives considered

Selecting a Cache Pattern by Scenario

Selecting a Cache Pattern by Scenario

Selecting an Eviction Policy by Workload

Selecting an Eviction Policy by Workload

Selecting a Cache Topology by Need

Selecting a Cache Topology by Need

Failure and degraded modes

Consistency vs. Performance: The Core Caching Trade-off

• Detection: Every caching decision sits on the consistency–performance axis.
• User or operator impact: Every caching decision sits on the consistency–performance axis.