Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Preventing Redis Big-Key
Event Loop Freezes.

Your Redis 7 primary serves 85,000 OPS at 0.8ms p99 latency. A feature-flag/catalog cache stores 2.5 million entries inside a single Redis Hash (catalog:items, ~480 MB RSS). Whenever a daily refresh job runs DEL catalog:items or its 24-hour TTL expires, all clients experience 820ms-1,200ms latency spikes, triggering 200ms SocketTimeoutExceptions and cascading application retries. Redesign the invalidation mechanism and data layout so cache refreshes and expirations never stall the Redis event loop above 2ms.
Start the journey5 stages
Loading diagram…
Preventing Redis Big-Key Event Loop Freezes architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your Redis 7 primary serves 85,000 OPS at 0.8ms p99 latency. A feature-flag/catalog cache stores 2.5 million entries inside a single Redis Hash (`catalog:items`, ~480 MB RSS). Whenever a daily refresh job runs `DEL catalog:items` or its 24-hour TTL expires, all clients experience 820ms-1,200ms latency spikes, triggering 200ms SocketTimeoutExceptions and cascading application retries. Redesign the invalidation mechanism and data layout so cache refreshes and expirations never stall the Redis event loop above 2ms.

From patterns to decisions.

Your Redis 7 primary serves 85,000 OPS at 0.8ms p99 latency. A feature-flag/catalog cache stores 2.5 million entries inside a single Redis Hash (`catalog:items`, ~480 MB RSS). Whenever a daily refresh job runs `DEL catalog:items` or its 24-hour TTL expires, all clients experience 820ms-1,200ms latency spikes, triggering 200ms SocketTimeoutExceptions and cascading application retries. Redesign the invalidation mechanism and data layout so cache refreshes and expirations never stall the Redis event loop above 2ms.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Redis's aeMain event loop is.

Redis's aeMain event loop is single-threaded; DEL synchronously walks and frees all 2.5M hash entries before processing the next queued command.

ChoiceWhat it buysWhat it costs
Trade-off 1Synchronous DEL reclaims memory immediately, whereas UNLINK defers reclamation by a few milliseconds to bio_lazy_free.

Step 1

Recognize that Redis executes commands sequentially on one main thread.

Step 2

Note that DEL is O(N) for collection types (Hash, Set, ZSet, List), where N = 2,500,000.

Step 3

Connect the 820ms stall to 200ms client SocketTimeoutExceptions and retry amplification.

Alternative 1

Chunked HSCAN + HDEL in batches of 500 fields.

Failure Mode 1

200ms client timeouts trigger application retries, multiplying load when Redis unfreezes.

# Anti-pattern: blocks Redis aeMain for ~820ms on 2.5M fields
# redis_client.delete("catalog:items")

# Safe: O(1) unlink (~0.08ms), memory freed in bio_lazy_free thread
# redis_client.unlink("catalog:items")
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Shard into 64 buckets via.

Shard into 64 buckets via CRC32, swap an active_ver pointer atomically, and reclaim old buckets with UNLINK (or chunked HSCAN+HDEL for legacy cleanup).

ChoiceWhat it buysWhat it costs
Trade-off 164 buckets keep each key ~7.5 MB, eliminating big-key stalls on both reads and TTL expirations.
Trade-off 2Requires 2x memory (~960 MB) briefly during Blue-Green refresh.

Step 1

Compute bucket = zlib.crc32(item_id.encode()) % 64.

Step 2

Write new version buckets in 500-entry pipelines, flip active_ver, and UNLINK old buckets.

Step 3

Use HSCAN COUNT 500 + HDEL with 1ms sleep for safe in-place legacy big-key removal.

Alternative 1

Storing each item as an independent top-level string key (catalog:item:{id}) if per-item TTLs are needed.

Failure Mode 1

Forgetting to set lazyfree-lazy-expire yes still causes stalls if a bucket's TTL expires.

import time
import zlib
from typing import Dict, Iterable

NUM_BUCKETS = 64

def bucket_key(version: str, item_id: str) -> str:
    bucket = zlib.crc32(item_id.encode("utf-8")) % NUM_BUCKETS
    return f"catalog:{version}:{bucket:02d}"

def refresh_catalog_blue_green(r, new_ver: str, old_ver: str, items: Iterable[tuple[str, str]]) -> None:
    pipe = r.pipeline(transaction=False)
    count = 0
    for item_id, payload in items:
        pipe.hset(bucket_key(new_ver, item_id), item_id, payload)
        count += 1
        if count % 500 == 0:
            pipe.execute()
    if count % 500 != 0:
        pipe.execute()

    r.set("catalog:active_ver", new_ver)
    old_keys = [f"catalog:{old_ver}:{b:02d}" for b in range(NUM_BUCKETS)]
    r.unlink(*old_keys)

def chunked_delete_big_hash(r, key: str, batch_size: int = 500) -> None:
    cursor = 0
    while True:
        cursor, data = r.hscan(key, cursor=cursor, count=batch_size)
        if data:
            r.hdel(key, *data.keys())
        if cursor == 0:
            break
        time.sleep(0.001)
    r.unlink(key)
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

64 sharded Hashes use ~40%.

64 sharded Hashes use ~40% less RAM via listpack/dict overhead sharing and allow fast versioned UNLINK, whereas individual keys are required only when items need independent TTLs.

ChoiceWhat it buysWhat it costs
Trade-off 1Top-level keys support per-item EXPIRE; sharded Hashes save ~240 MB RAM and simplify Blue-Green version swaps.

Compare memory overhead

2.5M top-level keys add 2.5M redisObject + dictEntry headers (~240 MB extra overhead).

Check TTL granularity

catalog refreshes expire together, so bucket-level TTL + version pointer is ideal.

Alternative 1

1,024 smaller Hashes using listpack encoding (hash-max-listpack-entries 512) for even tighter memory.

Failure Mode 1

Deleting 2.5M individual keys requires SCAN + UNLINK loops instead of 64 keys.

# 2.5M top-level keys: ~96 bytes metadata overhead each = ~240 MB extra RAM
# 64 sharded Hashes: 64 top-level keys, shared dict overhead, O(1) version pointer swap
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Enable replica-lazy-flush yes in redis.

Enable replica-lazy-flush yes in redis.conf so the replica flushes its old dataset asynchronously in bio_lazy_free before loading the incoming RDB.

ChoiceWhat it buysWhat it costs
Trade-off 1Async flush on replica briefly doubles memory until bio_lazy_free finishes.

Step 1

Identify that during full resync, a replica must flush its entire existing keyspace before loading the primary's RDB.

Step 2

With replica-lazy-flush no, that flush blocks the replica's main thread for seconds, failing health checks.

Alternative 1

Diskless replication (repl-diskless-sync yes) to speed up replica catch-up.

Failure Mode 1

Read replicas serving traffic during resync fail readiness probes if flush blocks synchronously.

# Required redis.conf directives:
# lazyfree-lazy-eviction yes
# lazyfree-lazy-expire yes
# lazyfree-lazy-server-del yes
# replica-lazy-flush yes
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Run redis-cli --bigkeys -i 0.

Run redis-cli --bigkeys -i 0.1 against a read replica (never the primary) or sample MEMORY USAGE key SAMPLES 5 during off-peak hours.

ChoiceWhat it buysWhat it costs
Trade-off 1Running `--bigkeys` on a replica with `-i 0.1` adds <1% CPU overhead and zero primary impact.

Step 1

Target a read replica with -i 0.1 (100ms sleep every 100 SCAN iterations).

Step 2

Monitor LATENCY LATEST and INFO commandstats for del/expire max_usec spikes.

Step 3

Alert when any key exceeds 10 MB or 10,000 elements.

Alternative 1

Offline RDB analysis (rdbtools) on a nightly backup snapshot.

Failure Mode 1

Running MEMORY USAGE without SAMPLES 5 on a 2.5M-element key performs an O(N) full traversal.

# Safe replica-only sampling check
def sample_key_bytes(replica_conn, key: str) -> int:
    return replica_conn.memory_usage(key, samples=5) or 0

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Replace DEL with async UNLINK, enable lazyfree-lazy-expire/eviction, and shard multi-million-entry collections into 64 hash buckets (< 8 MB each) or delete in 500-field HSCAN+HDEL batches.
  • 02Trade-off tip: Top-level keys support per-item EXPIRE; sharded Hashes save ~240 MB RAM and simplify Blue-Green version swaps.
  • 03Watch out for: Read replicas serving traffic during resync fail readiness probes if flush blocks synchronously.
Behavioral Progression · Redis Big Key Event Loop Freezes

Redis Big Key Event Loop Freezes — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Replace DEL with async UNLINK, enable lazyfree-lazy-expire/eviction, and shard multi-million-entry collections into 64 hash buckets (< 8 MB each) or delete in 500-field HSCAN+HDEL batches.

Synthesis / One connected design

Putting Preventing Redis Big-Key Event Loop Freezes Together

Replace DEL with async UNLINK, enable lazyfree-lazy-expire/eviction, and shard multi-million-entry collections into 64 hash buckets (< 8 MB each) or delete in 500-field HSCAN+HDEL batches.

A candidate architecture, not a universal answer

Putting Preventing Redis Big-Key Event Loop Freezes Together

Step 1
Step 2
Step 3
  1. Compute bucket = zlib.crc32(item_id.encode()) % 64.
  2. Write new version buckets in 500-entry pipelines, flip active_ver, and UNLINK old buckets.
  3. Use HSCAN COUNT 500 + HDEL with 1ms sleep for safe in-place legacy big-key removal.
Enable `replica-lazy-flush yes` in redis.conf so the replica flushes its old dataset asynchronously in bio_lazy_free before loading the incoming RDB.
Interview Readiness Check

Enable replica-lazy-flush yes in redis.conf so the replica flushes its old dataset asynchronously in bio_lazy_free before loading the incoming RDB.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Preventing Redis Big-Key Event Loop Freezes

Replace DEL with async UNLINK, enable lazyfree-lazy-expire/eviction, and shard multi-million-entry collections into 64 hash buckets (< 8 MB each) or delete in 500-field HSCAN+HDEL batches.