Spark Executor Sizing: Memory Model, Core Tuning, and GC Strategy

Scenario context

Spark executor OOMs are almost never caused by insufficient total cluster RAM — they are caused by misallocating memory across five distinct JVM regions while ignoring GC behavior and memoryOverhead. Master the UnifiedMemoryManager model, apply the 3–5 cores-per-executor rule, choose G1GC for batch and ZGC for streaming, and always account for off-heap overhead. Getting these numbers right simultaneously cuts your…

This walkthrough focuses on The Incident That Stumped a Senior Team.

Before making a design choice, clarify:

• The on-call engineer got paged at 2:47 AM.
• The nightly aggregation job — a critical revenue-reporting pipeline that joined 180 GB of transaction events against a 40 GB product catalog — had failed again.
• The YARN logs showed the usual message: Container killed on request.

Concepts in play: spark-data-engineering, tree-data-structures, jvm-internals-and-garbage-collection, caching

Stage 1 — Inside the Executor JVM: Five Memory Regions You Must Know

• When you set spark.executor.memory=16g , you are not giving Spark 16 GB of usable working memory.
• You are setting the JVM heap size — and Spark then carves that heap into four named regions before your first task runs.
• There is also a fifth region that lives outside the JVM heap entirely: memoryOverhead .
• Together these five regions explain almost every OOM failure mode.

graph TD
    A[Executor JVM Process] --> B[Reserved Memory - 300 MB fixed]
    A --> C[Usable Heap - heap minus 300 MB]
    A --> H[Off-Heap Memory - Tungsten optional]
    A --> I[Memory Overhead - outside JVM heap]
    C --> D[User Memory - 40 pct of usable heap]
    C --> E[Unified Spark Memory Pool - 60 pct of usable heap]
    E --> F[Execution Memory - shuffle sort and join buffers]
    E --> G[Storage Memory - RDD cache and broadcasts]
    I --> J[Netty shuffle buffers]
    I --> K[Python worker process memory]
    I --> L[JVM stack and native libs]

Observable outcome: Together these five regions explain almost every OOM failure mode.

Stage 2 — How UnifiedMemoryManager Orchestrates Memory Allocation and Eviction

• The Internals Before Spark 1.6, memory management used the StaticMemoryManager : Execution and Storage had fixed, non-overlapping allocations.
• If execution was idle, storage could not use its quota, and vice versa.
• This led to chronic under-utilization — a job that needed 90% execution memory for a large sort could only use its fixed 60% allocation while the other 40% (storage quota) sat empty.
• The UnifiedMemoryManager , introduced in Spark 1.6 and the default since then, replaced static partitions with a single shared pool and a soft boundary.

flowchart TD
    Driver["Driver Node"]
    Planner["Query Planner"]
    Optimizer["Catalyst Optimizer"]
    Executor["Executor Task"]

    Driver -->|"Spark Execution"| Planner
    Planner --> Optimizer
    Optimizer -->|"Optimized Plan"| Executor
    Executor -->|"Result"| Driver

Observable outcome: The UnifiedMemoryManager , introduced in Spark 1.6 and the default since then, replaced static partitions with a single shared pool and a soft boundary.

Stage 3 — The Production Sizing Formula: Deriving Optimal Config from Cluster Specs

• The academic configuration advice — "try 4 cores per executor and 8 GB of memory" — fails in production because it ignores the interaction between core count, HDFS throughput, container overhead, and the memoryOverhead…
• Here is a deterministic formula derived from Cloudera's HDFS throughput research, YARN container overhead measurements, and the UnifiedMemoryManager model above.
• Inputs: N nodes, M GB RAM per node, C cores per node.
• Step 1 — Establish Available Resources Per Node YARN's NodeManager and the OS need resources to function.

flowchart TD
    A[Cluster: N nodes, M GB RAM, C cores] --> B[Subtract 1 core and 1 GB per node for OS and YARN]
    B --> C{Workload Type}
    C -->|Batch ETL - HDFS or S3| D[executor.cores = 4 or 5]
    C -->|Structured Streaming| E[executor.cores = 2 or 3]
    C -->|Large Shuffle or ML| F[executor.cores = 3 or 4]
    D --> G[executors per node = floor of available cores per executor cores]
    E --> G
    F --> G
    G --> H[raw memory per executor = available RAM divided by executors per node]
    H --> I[memoryOverhead = max of 10 pct raw memory and 384 MB]
    I --> J[executor.memory = raw memory minus memoryOverhead - round down to GB]
    J --> K[Total executors = N times executors per node minus 1 for driver]
    K --> L[Set driver.memory to 4 to 8 GB - or match estimated collect result size plus 2 GB]

Observable outcome: Step 1 — Establish Available Resources Per Node YARN's NodeManager and the OS need resources to function.

Stage 4 — What Happens When Execution Memory Runs Out: The Spill Cascade

• Most engineers learn about memory pressure from error messages.
• Understanding the mechanism that precedes those messages — the sequence of decisions Spark makes when a task cannot get the memory it needs — turns reactive firefighting into proactive sizing.
• The diagram below traces the complete path from a task's memory request through the eviction, wait, and spill decision tree.
• The diagram reveals three distinct recovery strategies that Spark attempts in order before resorting to disk.

flowchart TD
    A[Task requests execution memory] --> B{Spark pool has capacity?}
    B -->|Yes| C[Allocate from unified Spark pool]
    B -->|No| D{Storage occupying above storageFraction floor?}
    D -->|Yes| E[Evict LRU cached blocks from storage region]
    D -->|No| F[Task waits up to spark.memory.timeout - default 60s]
    E --> C
    F --> G{Memory freed by completing tasks?}
    G -->|Yes| C
    G -->|No - timeout exceeded| H[Spill intermediate data to local disk]
    C --> I[Execute in-memory shuffle or sort or join]
    H --> J[Serialize and write spill files to local disk]
    J --> K[Merge spill files during shuffle read phase]
    I --> L[Task completes - memory released to pool]
    K --> L

Observable outcome: The diagram reveals three distinct recovery strategies that Spark attempts in order before resorting to disk.

Stage 5 — How Production Teams at Scale Have Applied This

• The Fintech Data Platform Rewrite A fintech company running a 200-executor Spark cluster on AWS EMR had a nightly risk aggregation job joining 500 GB of transaction data.
• The original executor configuration was spark.executor.memory=8g, spark.executor.cores=2 .
• The job ran nightly for 6 hours and failed roughly 40% of the time with OOMKilled containers.
• The diagnosis: with 2 cores per executor, they had 4× as many executor JVM processes as necessary, each with an 8 GB heap that was 40% User Memory (3.2 GB) — but the join operation was pure Execution Memory work, and ea…

Observable outcome: The diagnosis: with 2 cores per executor, they had 4× as many executor JVM processes as necessary, each with an 8 GB heap that was 40% User Memory (3.2 GB) — but the join operation was pure Execution Memory work, and ea…

Stage 6 — Walkthrough: Sizing a 10-Node YARN Cluster for a 500 GB ETL Job

• This walkthrough applies the sizing formula to a specific, realistic cluster and works through every decision point.
• The scenario is a nightly ETL pipeline that reads 500 GB of Parquet files from HDFS, joins them against a 5 GB dimension table via broadcast join, aggregates by 8 dimensions, and writes results back to HDFS.
• The workload is pure batch — no streaming latency requirements.
• Cluster specs: 10 worker nodes.

flowchart TD
    Input["Input Stream"]
    Batch1["Micro-Batch [t, t+Δt)"]
    Process["Processing"]
    Output["Output"]

    Input -->|"Split by interval"| Batch1
    Batch1 --> Process
    Process --> Output
    Output -->|"Repeat"| Input

Observable outcome: Cluster specs: 10 worker nodes.

End-to-end outcome

After working through the stages, you should be able to explain:

• Doubling memory never fixes a misallocation.
• The most common first response to OOMKilled is increasing spark.executor.memory .
• • Five regions, not one heap: spark.executor.memory sets the JVM heap, but Spark allocates five distinct regions inside and outside it — Reserved (300 MB fixed), User Memory, Execution Memory, Storage Memory, and Memory…
• OOM failures happen in one specific region; you must diagnose which one.

Alternatives considered

When to Change Which Knob: A Decision Guide for Executor Tuning

Use this table when diagnosing a Spark performance problem.

Failure and degraded modes

Fat Executor vs. Thin Executor

• Detection: The tension between large executors (many cores, large heap) and small executors (few cores, small heap) is one of the most common sources of Spark performance debates.
• User or operator impact: The tension between large executors (many cores, large heap) and small executors (few cores, small heap) is one of the most common sources of Spark performance debates.

G1GC vs. ZGC for Executor Heaps

• Detection: G1GC is the default and appropriate choice for most batch workloads.
• User or operator impact: G1GC is the default and appropriate choice for most batch workloads.