Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Fixing Redis Fork
Copy-on-Write Memory Bloat.

A production Redis 7 primary holds a 24 GB dataset on a 32 GB RAM Linux host and handles 15,000 writes/sec (each mutating ~64 bytes across random keys). Whenever an hourly BGSAVE RDB snapshot (taking 90 seconds) or BGREWRITEAOF triggers fork(), the child process shares pages with the parent via Copy-on-Write (CoW). Within 70 seconds of fork(), Redis RSS balloons from 24 GB to >31.5 GB and the Linux kernel OOM-kills the Redis primary. Inspection shows Transparent Huge Pages (THP) is set to always. Design the OS memory, Redis persistence, and automated CoW-guardrail architecture to prevent OOM kills while preserving durable backups.
Start the journey5 stages
Loading diagram…
Fixing Redis Fork Copy-on-Write Memory Bloat architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A production Redis 7 primary holds a 24 GB dataset on a 32 GB RAM Linux host and handles 15,000 writes/sec (each mutating ~64 bytes across random keys). Whenever an hourly `BGSAVE` RDB snapshot (taking 90 seconds) or `BGREWRITEAOF` triggers `fork()`, the child process shares pages with the parent via Copy-on-Write (CoW). Within 70 seconds of `fork()`, Redis RSS balloons from 24 GB to >31.5 GB and the Linux kernel OOM-kills the Redis primary. Inspection shows Transparent Huge Pages (THP) is set to `always`. Design the OS memory, Redis persistence, and automated CoW-guardrail architecture to prevent OOM kills while preserving durable backups.

From patterns to decisions.

A production Redis 7 primary holds a 24 GB dataset on a 32 GB RAM Linux host and handles 15,000 writes/sec (each mutating ~64 bytes across random keys). Whenever an hourly `BGSAVE` RDB snapshot (taking 90 seconds) or `BGREWRITEAOF` triggers `fork()`, the child process shares pages with the parent via Copy-on-Write (CoW). Within 70 seconds of `fork()`, Redis RSS balloons from 24 GB to >31.5 GB and the Linux kernel OOM-kills the Redis primary. Inspection shows Transparent Huge Pages (THP) is set to `always`. Design the OS memory, Redis persistence, and automated CoW-guardrail architecture to prevent OOM kills while preserving durable backups.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Linux fork() Copy-on-Write operates at.

Linux fork() Copy-on-Write operates at page granularity; with 2 MB THP, a 64-byte mutation forces the kernel to copy an entire 2 MB page (512x more than 4 KB pages).

ChoiceWhat it buysWhat it costs
Trade-off 14 KB pages eliminate CoW amplification and fork() page-table latency spikes.

Step 1

Trace fork() marking all page-table entries read-only.

Step 2

Calculate worst-case CoW over 90s at 15,000 writes/sec: 5.4 GB with 4 KB pages vs. ~24 GB (100% dataset) with 2 MB THP.

Step 3

Show why 24 GB + 24 GB > 32 GB host RAM triggers the OOM killer.

Alternative 1

Disabling persistence completely (loses durability and still risks CoW during replica full resyncs).

Failure Mode 1

OOM-killer terminates Redis primary during peak write traffic.

def worst_case_cow_gb(writes_per_sec: int, duration_s: int, page_kb: int, dataset_gb: float) -> float:
    touched_gb = (writes_per_sec * duration_s * page_kb) / (1024 * 1024)
    return min(dataset_gb, touched_gb)
# 4 KB pages -> 5.15 GB; 2 MB THP -> capped at full 24.0 GB!
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Verify THP=[never] at the OS.

Verify THP=[never] at the OS level, check rdb_last_cow_size and used_memory_rss against 80% host RAM, and only then issue BGSAVE.

ChoiceWhat it buysWhat it costs
Trade-off 1Aborting BGSAVE when RAM > 80% prevents an immediate primary outage, alerting operators to scale/shard before backups fall behind.

Step 1

Read /sys/kernel/mm/transparent_hugepage/enabled and fail if [never] is not selected.

Step 2

Query INFO memory (used_memory_rss, total_system_memory) and INFO persistence (rdb_last_cow_size, rdb_bgsave_in_progress).

Step 3

Estimate required headroom as rss + max(last_cow, 0.25 * rss) and abort BGSAVE if >= 80% of host RAM.

Alternative 1

Running BGSAVE exclusively on a dedicated 64 GB backup replica.

Failure Mode 1

If total_system_memory is inside a cgroup container, ensure Redis maxmemory is sized to the container cgroup limit, not the bare-metal host.

from pathlib import Path

def is_thp_disabled(thp_path: str = "/sys/kernel/mm/transparent_hugepage/enabled") -> bool:
    text = Path(thp_path).read_text().strip()
    return "[never]" in text

def safe_trigger_bgsave(r) -> str:
    if not is_thp_disabled():
        return "ABORT_THP_ENABLED"

    mem = r.info("memory")
    pers = r.info("persistence")
    if pers.get("rdb_bgsave_in_progress", 0) == 1:
        return "SKIP_BGSAVE_IN_PROGRESS"

    rss = mem["used_memory_rss"]
    total_ram = mem["total_system_memory"]
    last_cow = pers.get("rdb_last_cow_size", 0)
    estimated_cow = max(last_cow, int(rss * 0.25))

    if (rss + estimated_cow) > (total_ram * 0.80):
        return f"ABORT_LOW_RAM_HEADROOM:rss={rss},cow={estimated_cow},ram={total_ram}"

    r.bgsave()
    return "BGSAVE_STARTED"
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

At 15,000 writes/sec over 90s.

At 15,000 writes/sec over 90s with 4 KB pages, CoW still adds up to 5.4 GB (24 + 5.4 = 29.4 GB), leaving <2.6 GB for the Linux kernel, page cache, and network buffers.

ChoiceWhat it buysWhat it costs
Trade-off 150-55% RAM utilization trades hardware efficiency for guaranteed OOM immunity.

Step 1

Add 24 GB RSS + 5.4 GB worst-case 4 KB CoW = 29.4 GB.

Step 2

Account for OS/kernel/socket buffers (~2-3 GB) and jemalloc fragmentation (~1.1x).

Step 3

Conclude that 32 GB is insufficient; either upgrade to 64 GB or shard to 2x 12 GB.

Alternative 1

Using Redis Enterprise / Valkey or diskless replica-only backups with strict primary maxmemory <= 65%.

Failure Mode 1

Memory fragmentation (mem_fragmentation_ratio > 1.2) silently pushes RSS above maxmemory.

# 24 GB RSS + 5.4 GB (4 KB CoW) + 2.5 GB OS/buffers = 31.9 GB -> Razor-thin OOM risk
# Rule of thumb: maxmemory <= 55% of host RAM when fork() is possible
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

With default vm.

With default vm.overcommit_memory = 0, Linux estimates that fork() might need 24 GB + 24 GB = 48 GB of virtual memory and rejects fork() with ENOMEM on a 32 GB host.

ChoiceWhat it buysWhat it costs
Trade-off 1Allows `fork()` to succeed without pre-reserving 2x RAM, relying on 4 KB CoW + 50% maxmemory to stay within physical RAM.

Step 1

Observe Linux's heuristic overcommit check during fork().

Step 2

Set vm.overcommit_memory = 1 so fork() succeeds and allocates physical pages lazily on CoW faults.

Alternative 1

Adding swap (discouraged for Redis due to severe latency spikes on page-in).

Failure Mode 1

Leaving vm.overcommit_memory = 0 causes silent Can't save in background: fork: Cannot allocate memory errors.

# /etc/sysctl.d/99-redis.conf
# vm.overcommit_memory = 1
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Enable repl-diskless-sync yes on the.

Enable repl-diskless-sync yes on the primary so the forked child streams the RDB directly over the socket to the replica without hitting disk, and size repl-backlog-size generously (e.g., 1 GB) to avoid unnecessary full resyncs.

ChoiceWhat it buysWhat it costs
Trade-off 1Larger replication backlog uses 1 GB static RAM but prevents expensive full-resync forks on 30-second network blips.

Step 1

Set repl-backlog-size 1gb so brief network blips use partial resync (PSYNC) with zero fork().

Step 2

Set repl-diskless-sync yes and repl-diskless-sync-delay 5 to batch multiple joining replicas into one fork().

Alternative 1

Promoting an existing replica or seeding from an S3 RDB snapshot where supported.

Failure Mode 1

Default 1 MB repl-backlog-size overflows in <1 second at 15,000 writes/sec, forcing a full resync on any disconnect.

# redis.conf replication tuning:
# repl-diskless-sync yes
# repl-diskless-sync-delay 5
# repl-backlog-size 1gb

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Disable Transparent Huge Pages (echo never > /sys/kernel/mm/transparent_hugepage/enabled), cap Redis maxmemory at <= 55% of host RAM, monitor rdb_last_cow_size, and gate BGSAVE during heavy write bursts.
  • 02Trade-off tip: 50-55% RAM utilization trades hardware efficiency for guaranteed OOM immunity.
  • 03Watch out for: Leaving vm.overcommit_memory = 0 causes silent Can't save in background: fork: Cannot allocate memory errors.
Behavioral Progression · Redis Fork Cow Memory Bloat Thp

Redis Fork Cow Memory Bloat Thp — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Disable Transparent Huge Pages (echo never > /sys/kernel/mm/transparenthugepage/enabled), cap Redis maxmemory at <= 55% of host RAM, monitor rdblastcowsize, and gate BGSAVE during heavy write bursts.

Synthesis / One connected design

Putting Fixing Redis Fork Copy-on-Write Memory Bloat Together

Disable Transparent Huge Pages (`echo never > /sys/kernel/mm/transparent_hugepage/enabled`), cap Redis `maxmemory` at <= 55% of host RAM, monitor `rdb_last_cow_size`, and gate BGSAVE during heavy write bursts.

A candidate architecture, not a universal answer

Putting Fixing Redis Fork Copy-on-Write Memory Bloat Together

Step 1
Step 2
Step 3
  1. Read /sys/kernel/mm/transparent_hugepage/enabled and fail if [never] is not selected.
  2. Query INFO memory (used_memory_rss, total_system_memory) and INFO persistence (rdb_last_cow_size, rdb_bgsave_in_progress).
  3. Estimate required headroom as rss + max(last_cow, 0.25 * rss) and abort BGSAVE if >= 80% of host RAM.
With default `vm.overcommit_memory = 0`, Linux estimates that `fork()` might need 24 GB + 24 GB = 48 GB of virtual memory and rejects `fork()` with ENOMEM on a 32 GB host.
Interview Readiness Check

With default vm.overcommit_memory = 0, Linux estimates that fork() might need 24 GB + 24 GB = 48 GB of virtual memory and rejects fork() with ENOMEM on a 32 GB host.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Fixing Redis Fork Copy-on-Write Memory Bloat

Disable Transparent Huge Pages (echo never > /sys/kernel/mm/transparent_hugepage/enabled), cap Redis maxmemory at <= 55% of host RAM, monitor rdb_last_cow_size, and gate BGSAVE during heavy write bursts.