Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Preventing Pod OOMKilled (Exit
Code 137) Outages.

A high-throughput Java 21 gRPC/Netty microservice running in Kubernetes is configured with -Xmx4g and container resources.limits.memory=4608Mi. Under peak traffic (12,000 RPS with 64 KB payloads), pods suddenly crash with Exit Code 137 (OOMKilled) even though Prometheus JVM heap metrics show heap usage at only 2.8 GiB (70%) and zero OutOfMemoryError exceptions in application logs. Design a memory-budgeting, off-heap bounding, and observability strategy to eliminate silent cgroup OOM kills.
Start the journey5 stages
Loading diagram…
Preventing Pod OOMKilled (Exit Code 137) Outages architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A high-throughput Java 21 gRPC/Netty microservice running in Kubernetes is configured with -Xmx4g and container resources.limits.memory=4608Mi. Under peak traffic (12,000 RPS with 64 KB payloads), pods suddenly crash with Exit Code 137 (OOMKilled) even though Prometheus JVM heap metrics show heap usage at only 2.8 GiB (70%) and zero OutOfMemoryError exceptions in application logs. Design a memory-budgeting, off-heap bounding, and observability strategy to eliminate silent cgroup OOM kills.

From patterns to decisions.

A high-throughput Java 21 gRPC/Netty microservice running in Kubernetes is configured with -Xmx4g and container resources.limits.memory=4608Mi. Under peak traffic (12,000 RPS with 64 KB payloads), pods suddenly crash with Exit Code 137 (OOMKilled) even though Prometheus JVM heap metrics show heap usage at only 2.8 GiB (70%) and zero OutOfMemoryError exceptions in application logs. Design a memory-budgeting, off-heap bounding, and observability strategy to eliminate silent cgroup OOM kills.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

The Linux kernel cgroup OOM.

The Linux kernel cgroup OOM killer enforces memory.max against the entire process RSS (Heap + Metaspace + DirectByteBuffers + Thread Stacks + Native arenas + Page Cache), not just -Xmx.

ChoiceWhat it buysWhat it costs
Trade-off 1Enabling NativeMemoryTracking=summary adds ~5–10% overhead to native allocations, so use summary rather than detail in production.

Check kubectl describe pod for Reason

OOMKilled and Exit Code: 137.

Step 2

Inspect container_memory_working_set_bytes vs jvm_memory_used_bytes{area="heap"}.

Step 3

Run jcmd 1 VM.native_memory summary to measure off-heap allocations.

Alternative 1

Use async-profiler in malloc/natmem mode to flamegraph native allocations.

Failure Mode 1

Waiting for a JVM OutOfMemoryError log fails because kernel SIGKILL terminates the process immediately.

import java.lang.management.BufferPoolMXBean;
import java.lang.management.ManagementFactory;

public class OffHeapMetrics {
    public static void logDirectMemory() {
        for (BufferPoolMXBean pool : ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class)) {
            System.out.printf("pool=%s count=%d usedMiB=%.2f capacityMiB=%.2f%n",
                pool.getName(), pool.getCount(),
                pool.getMemoryUsed() / (1024.0 * 1024.0),
                pool.getTotalCapacity() / (1024.0 * 1024.0));
        }
    }
}
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Set Guaranteed QoS at 8192Mi.

Set Guaranteed QoS at 8192Mi, cap Heap at 3584m, DirectMemory at 1024m, Metaspace at 512m, Thread stacks at 512k, and set MALLOC_ARENA_MAX=2.

ChoiceWhat it buysWhat it costs
Trade-off 1Using 3584m heap inside an 8 GiB pod leaves 4.5 GiB for off-heap and native safety, trading slightly higher GC frequency for zero OOM kills.

Step 1

Set requests.memory == limits.memory == 8192Mi.

Explicitly bound -Xmx, -XX

MaxDirectMemorySize, -XX:MaxMetaspaceSize, and -Xss.

Step 3

Set MALLOC_ARENA_MAX=2 and Netty maxOrder=9.

Alternative 1

Switch from glibc to jemalloc via LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2.

Failure Mode 1

Leaving MaxDirectMemorySize unset causes it to default to 3584m, silently allowing 7.1 GiB+ before counting threads or Metaspace.

resources:
  requests:
    memory: "8192Mi"
    cpu: "2000m"
  limits:
    memory: "8192Mi"
env:
  - name: MALLOC_ARENA_MAX
    value: "2"
  - name: JAVA_TOOL_OPTIONS
    value: >- 
      -Xms3584m -Xmx3584m -XX:MaxMetaspaceSize=512m
      -XX:MaxDirectMemorySize=1024m -Xss512k
      -XX:ReservedCodeCacheSize=256m -XX:NativeMemoryTracking=summary
      -Dio.netty.allocator.maxOrder=9 -XX:+ExitOnOutOfMemoryError
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

MaxRAMPercentage only scales the JVM.

MaxRAMPercentage only scales the JVM heap and leaves MaxDirectMemorySize equal to heap, which routinely blows past container limits in Netty/gRPC services.

ChoiceWhat it buysWhat it costs
Trade-off 1Explicit flags must be updated if container limits change, whereas MaxRAMPercentage auto-scales but is unsafe for off-heap-heavy apps.

Step 1

Calculate worst-case native + off-heap footprint independently of heap.

Step 2

Subtract off-heap + 20% headroom from limits.memory to derive -Xmx.

Alternative 1

Compute JVM flags dynamically in an entrypoint script using cgroup memory.max minus fixed off-heap reserves.

Failure Mode 1

MaxRAMPercentage=75% on a 4 GiB pod sets heap to 3 GiB and implicit MaxDirectMemorySize to 3 GiB (6 GiB total).

// Verify effective runtime limits at startup
long maxHeapMiB = Runtime.getRuntime().maxMemory() / (1024 * 1024);
System.out.println("Configured Max Heap MiB: " + maxHeapMiB);
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Apply gRPC/Netty concurrency limits and.

Apply gRPC/Netty concurrency limits and AUTO_READ=false backpressure so excess requests receive 429/RESOURCE_EXHAUSTED rather than allocating unbounded direct buffers.

ChoiceWhat it buysWhat it costs
Trade-off 1Shedding load at 80% direct memory returns fast 429/503 errors to callers instead of letting all pods crash simultaneously.

Step 1

Configure Netty ChannelOption.WRITE_BUFFER_WATER_MARK (low=32KB, high=64KB).

Step 2

Reject incoming requests with gRPC RESOURCE_EXHAUSTED when in-flight memory crosses 80% of MaxDirectMemorySize.

Alternative 1

Configure Netty unpooled heap buffers (-Dio.netty.noPreferDirect=true) at the cost of extra memory copies and GC pressure.

Failure Mode 1

Slow-reading clients cause outbound write queues to accumulate direct buffers unless WriteBufferWaterMark is enforced.

ServerBootstrap b = new ServerBootstrap();
b.childOption(ChannelOption.WRITE_BUFFER_WATER_MARK,
    new WriteBufferWaterMark(32 * 1024, 64 * 1024));
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Track the linear slope of.

Track the linear slope of container_memory_working_set_bytes minus jvm_memory_used_bytes{area="heap"} and alert on sustained positive drift across GC cycles.

ChoiceWhat it buysWhat it costs
Trade-off 1Long-window derivative alerts avoid false positives from short traffic spikes while catching slow leaks hours before OOM.

Step 1

Scrape container_memory_working_set_bytes and jvm_buffer_pool_used_bytes every 15s.

Step 2

Compare jcmd VM.native_memory baseline vs diff after 24 hours.

Step 3

Alert if non-heap RSS grows > 50 MiB/hour over 6 hours.

Alternative 1

Run periodic jcmd VM.native_memory summary.diff via an automated diagnostic sidecar.

Failure Mode 1

Alerting only on raw working_set > 85% misses slow leaks until minutes before an outage.

- alert: PodOffHeapMemoryLeak
  expr: deriv((container_memory_working_set_bytes{container="app"} - jvm_memory_used_bytes{area="heap"})[6h:5m]) > 14000
  for: 30m
  labels:
    severity: warning

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Budget container memory limits across Heap + Metaspace + MaxDirectMemorySize + Thread Stacks + Native overhead, cap glibc arenas with MALLOC_ARENA_MAX=2, and use Guaranteed QoS.
  • 02Trade-off tip: Explicit flags must be updated if container limits change, whereas MaxRAMPercentage auto-scales but is unsafe for off-heap-heavy apps.
  • 03Watch out for: Slow-reading clients cause outbound write queues to accumulate direct buffers unless WriteBufferWaterMark is enforced.
Behavioral Progression · Preventing Pod Oomkilled Exit Code 137

Preventing Pod Oomkilled Exit Code 137 — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Budget container memory limits across Heap + Metaspace + MaxDirectMemorySize + Thread Stacks + Native overhead, cap glibc arenas with MALLOCARENAMAX=2, and use Guaranteed QoS.

Synthesis / One connected design

Putting Preventing Pod OOMKilled (Exit Code 137) Outages Together

Budget container memory limits across Heap + Metaspace + MaxDirectMemorySize + Thread Stacks + Native overhead, cap glibc arenas with MALLOC_ARENA_MAX=2, and use Guaranteed QoS.

A candidate architecture, not a universal answer

Putting Preventing Pod OOMKilled (Exit Code 137) Outages Together

Step 1
Step 2
Step 3
  1. Set requests.memory == limits.memory == 8192Mi.
  2. Explicitly bound -Xmx, -XX:MaxDirectMemorySize, -XX:MaxMetaspaceSize, and -Xss.
  3. Set MALLOC_ARENA_MAX=2 and Netty maxOrder=9.
Apply gRPC/Netty concurrency limits and AUTO_READ=false backpressure so excess requests receive 429/RESOURCE_EXHAUSTED rather than allocating unbounded direct buffers.
Interview Readiness Check

Apply gRPC/Netty concurrency limits and AUTO_READ=false backpressure so excess requests receive 429/RESOURCE_EXHAUSTED rather than allocating unbounded direct buffers.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Preventing Pod OOMKilled (Exit Code 137) Outages

Budget container memory limits across Heap + Metaspace + MaxDirectMemorySize + Thread Stacks + Native overhead, cap glibc arenas with MALLOC_ARENA_MAX=2, and use Guaranteed QoS.