A Python ML inference service runs on Kubernetes (16 GB container memory limit, cgroup v2) using a Gunicorn master process + 8 worker processes, plus a shared Redis sidecar in the same pod. When a single worker leaks 9 GB of memory on an oversized payload, the Linux OOM killer terminates the Redis sidecar or Gunicorn master process (`SIGKILL`, exit 137) instead of the leaking worker, taking down the entire pod and dropping all healthy in-flight requests. Diagnose how `oom_badness()` selects victims and design a deterministic memory-isolation and OOM protection strategy.
From patterns to decisions.
A Python ML inference service runs on Kubernetes (16 GB container memory limit, cgroup v2) using a Gunicorn master process + 8 worker processes, plus a shared Redis sidecar in the same pod. When a single worker leaks 9 GB of memory on an oversized payload, the Linux OOM killer terminates the Redis sidecar or Gunicorn master process (`SIGKILL`, exit 137) instead of the leaking worker, taking down the entire pod and dropping all healthy in-flight requests. Diagnose how `oom_badness()` selects victims and design a deterministic memory-isolation and OOM protection strategy.
oom_badness() scores (% of cgroup.
oom_badness() scores (% of cgroup memory used * 10) + oom_score_adj; lowering oom_score_adj below its initial value requires CAP_SYS_RESOURCE, which Kubernetes drops.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Raising worker `oom_score_adj` works in unprivileged containers without security risk. |
Show the formula
score = (RSS + swap + pagetables) / mem_limit * 1000 + oom_score_adj.
Step 2
Explain that unprivileged processes can raise oom_score_adj (e.g., 0 -> +300) in child workers.
Alternative 1
Run each worker in its own Kubernetes pod (1 worker/pod) if memory isolation must be absolute.
Failure Mode 1
Writing -500 to /proc/self/oom_score_adj inside a standard container fails with PermissionError (EACCES).
def oom_score(rss_gb: float, limit_gb: float, adj: int) -> int:
base = int((rss_gb / limit_gb) * 1000)
return max(0, min(1000, base + adj))
print("Master:", oom_score(1.5, 16.0, 0))
print("Worker (with +300 adj):", oom_score(4.0, 16.0, 300))
Use when_ready for gc.
Use when_ready for gc.freeze(), post_fork to write 300 to /proc/self/oom_score_adj, and post_request to recycle workers exceeding 1.8 GB RSS.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Reading `/proc/self/statm` takes ~5 microseconds per request—negligible for ML inference. |
Step 1
Freeze GC objects in master before fork.
Step 2
Set oom_score_adj = 300 in worker.
Step 3
Check /proc/self/statm RSS in post_request and SIGTERM self if > 1.8 GB.
Alternative 1
Use resource.setrlimit(resource.RLIMIT_AS, ...) (though virtual address space includes mmapped files, making RSS checks cleaner).
Failure Mode 1
Using RLIMIT_AS on PyTorch/CUDA apps often fails because CUDA reserves large virtual address ranges.
import gc, os, signal
preload_app = True
workers = 8
MAX_WORKER_RSS_BYTES = 1800 * 1024 * 1024
def when_ready(server):
gc.freeze()
def post_fork(server, worker):
with open("/proc/self/oom_score_adj", "w") as f:
f.write("300")
def post_request(worker, req, environ, resp):
with open("/proc/self/statm") as f:
rss_pages = int(f.read().split()[1])
if rss_pages * os.sysconf("SC_PAGE_SIZE") > MAX_WORKER_RSS_BYTES:
os.kill(os.getpid(), signal.SIGTERM)
Move Redis to its own.
Move Redis to its own pod. Even with Guaranteed QoS, sharing a node/pod lifecycle means ML pod restarts or node evictions unnecessarily flush the cache.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Moving Redis out adds ~0.2 ms network RTT compared to `localhost`, which is negligible for ML inference. |
Step 1
Compare Pod-level cgroup sharing vs separate Deployment.
Step 2
Note requests == limits gives Guaranteed QoS (oom_score_adj = -997 at kubelet level).
Alternative 1
Keep Redis in-pod only if using Guaranteed QoS on every container AND separate memory limits.
Failure Mode 1
Leaving Redis in a Burstable pod lets kubelet assign it a high oom_score_adj.
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "4Gi"
cpu: "2"
Implementation & Guardrails (4).
memory.oom.group = 1 treats the entire cgroup as an indivisible unit, killing the Gunicorn master and all 8 workers simultaneously on any OOM.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `memory.oom.group=0` allows Gunicorn to survive and respawn the single killed worker. |
Step 1
Verify /sys/fs/cgroup/memory.oom.group is 0 for multi-process worker containers.
Step 2
Keep 1 only for tightly coupled single-unit workloads like JVM + child helpers.
Alternative 1
Use systemd slice per worker if running on bare-metal VMs.
Failure Mode 1
Pod-wide exit 137 when only 1 of 8 workers leaked.
with open("/sys/fs/cgroup/memory.oom.group") as f:
assert f.read().strip() == "0", "memory.oom.group=1 will kill the parent master!"
A container OOM kill produces.
A container OOM kill produces reason: OOMKilled (exit code 137) and increments container_oom_events_total in cAdvisor/kernel dmesg, whereas a child-only OOM leaves the pod running (status: Running) while logging SIGKILL in Gunicorn master logs.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Worker-level OOMs are invisible to Kubernetes pod status unless exported by the master process. |
Step 1
Check kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}.
Step 2
Alert on Gunicorn worker SIGKILL exit codes (-9) in application logs.
Alternative 1
Scrape memory.events (oom_kill counter) via node_exporter/cAdvisor.
Failure Mode 1
Silent worker respawn loops masking a deterministic memory leak.
# Alert rules:
# 1. Pod-level OOM: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
# 2. Worker-level OOM: rate(gunicorn_worker_sigkill_total[5m]) > 0
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Protect critical parent/supervisor processes with
oom_score_adj = -500..-1000, isolate workers in dedicated cgroup v2 slices (memory.high+memory.oom.group), and enforce per-worker RSS limits. - 02Trade-off tip: Moving Redis out adds ~0.2 ms network RTT compared to
localhost, which is negligible for ML inference. - 03Watch out for: Pod-wide exit 137 when only 1 of 8 workers leaked.
Linux Oom Killer Container Limits — Normal → Load / Fault → Recovery
Protect critical parent/supervisor processes with oomscoreadj = -500..-1000, isolate workers in dedicated cgroup v2 slices (memory.high + memory.oom.group), and enforce per-worker RSS limits.
Putting Understanding the Linux OOM Killer in Containers Together
Protect critical parent/supervisor processes with `oom_score_adj = -500..-1000`, isolate workers in dedicated cgroup v2 slices (`memory.high` + `memory.oom.group`), and enforce per-worker RSS limits.
Putting Understanding the Linux OOM Killer in Containers Together
- Freeze GC objects in master before fork.
- Set
oom_score_adj = 300in worker. - Check
/proc/self/statmRSS inpost_requestandSIGTERMself if > 1.8 GB.
Interview Readiness Check
memory.oom.group = 1 treats the entire cgroup as an indivisible unit, killing the Gunicorn master and all 8 workers simultaneously on any OOM.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Understanding the Linux OOM Killer in Containers
Protect critical parent/supervisor processes with oom_score_adj = -500..-1000, isolate workers in dedicated cgroup v2 slices (memory.high + memory.oom.group), and enforce per-worker RSS limits.
