Redis Maxmemory Eviction Thrashing on Volatile-LRU Under Write Surges

System Design Interview•~30 min read•medium

Redis instance configured with maxmemory 16 GB and volatile-lru eviction policy.

1. Understand Requirements & Scope

Redis instance configured with maxmemory 16 GB and volatile-lru eviction policy. Failure cascade dynamic: 100k QPS → TTL Expiration → Cache Miss Herd → Origin Slam → 💥 DB Meltdown.

Functional Requirements:

Operational Invariant: "Unless dedicated strictly as a durable data store, configure caching Redis instances with allkeys-lru/lfu eviction."
Production Hardening: Configure eviction policy to allkeys-lru (or allkeys-lfu), ensuring that least recently used keys are eligible for eviction regardless of TTL flags, and set strict memory usage ale
Prevent cascading failure by isolating downstream degradation at the ingress boundary.

2. Back-of-the-Envelope Capacity Estimation

• Incident Scenario Baseline: Redis instance configured with maxmemory 16 GB and volatile-lru eviction policy.

• Failure Cascade Path: 100k QPS → TTL Expiration → Cache Miss Herd → Origin Slam → 💥 DB Meltdown

• Sub-millisecond rejection / shed latency for overloaded requests

3. High-Level Architecture Flow

Configure eviction policy to allkeys-lru (or allkeys-lfu), ensuring that least recently used keys are eligible for eviction regardless of TTL flags, and set strict memory usage alert thresholds at 75%. Invariant: "Unless dedicated strictly as a durable data store, configure caching Redis instances with allkeys-lru/lfu eviction."

Ingress → Bounded Admission Controller → Adaptive Concurrency / Queue Gate → Protected Worker Engine; Overload Shed Path → HTTP 429 / Circuit Fallback (<0.1ms).

Fig 1. Architecture data flow and component topology.

4. Production Implementation & Code

// HARDENED: Singleflight mutex coalescing with probabilistic early refresh
public byte[] getProduct(String key) {
    // XFetch algorithm evaluates early background refresh before TTL expiration
    if (shouldProbabilisticEarlyRefresh(key, beta, delta)) {
        backgroundPool.submit(() -> refreshCacheKey(key));
    }
    // Singleflight ensures exactly ONE upstream DB fetch executes per in-flight key
    return singleflightGroup.execute(key, () -> database.queryProduct(key));
}

Hardened Implementation Sequence

Ingress Boundary Defense: Configure eviction policy to allkeys-lru (or allkeys-lfu), ensuring that least recently used keys are eligible for eviction regardless of TTL flags, a
Failure Containment: Eliminate unconstrained buffering and propagate physical backpressure.
Invariant Enforcement: Unless dedicated strictly as a durable data store, configure caching Redis instances with allkeys-lru/lfu eviction.

5. Key Decisions & Operational Mitigations

Architectural Decisions & Trade-Offs

•Ingress Admission: Naive unconstrained buffering causes queue bloat and GC exhaustion. Mitigation: Bounded ingress gate with fast non-blocking shedding.
•Resource Strategy: Static capacity allocations collapse under brownouts. Mitigation: Dynamic adaptive limits scaling to real-time physical bounds.
•Outage Impact: Cascading lock convoys and memory saturation. Mitigation: Isolated failure domains keep P99 latency flat and protected.

Operational Hazards & Mitigations

•Cascading Failure: A developer writes 5 GB of ephemeral session data without setting a TTL (no expiry). Since keys have no TTL, volatile-lru cannot evict them. When memory fills, Redis throws 'OOM command not allowed when used memory > maxmemory', crashing write flows.
•Breakdown Chain: 100k QPS → TTL Expiration → Cache Miss Herd → Origin Slam → 💥 DB Meltdown
•Shedding Failure: Without fast rejection at the boundary, queued dead work triggers multi-second GC pauses, CPU thrashing, and cascading 504 Gateway Timeouts.

Interactive Knowledge Check • Question 1 of 3↓

Interactive Practice Quiz•Question 1 of 3

intermediate

Single-Thread Execution Bottleneck

Why does Redis execute single-threaded command processing faster than traditional multi-threaded locked data structures?