Spark Architecture: Driver, Executors, DAG Scheduler, and Task Scheduler Explained

Scenario context

Spark's architecture is a precise chain of responsibility. The Driver converts user code into a DAG, the DAGScheduler breaks it into stages at shuffle boundaries, the TaskScheduler dispatches tasks to Executors respecting data locality, and the Cluster Manager handles raw resource allocation. Understanding what each component does — and what kills it — is the difference between tuning Spark intentionally and tuning…

This walkthrough focuses on Why Spark's Architecture Is the Missing Mental Model for Every Data Engineer.

Before making a design choice, clarify:

• Spark is often introduced as "distributed pandas" or "fast MapReduce." Both descriptions are technically accurate and pedagogically dangerous.
• They invite engineers to treat Spark like a local computation library with a bigger machine behind it — which is exactly the mental model that produces the OOM story above.
• Spark is a distributed execution engine with a layered architecture designed around a specific division of labor.

Concepts in play: spark-data-engineering, caching, jvm-internals-and-garbage-collection, api-gateway

Stage 1 — Forty-Five Minutes of Work, One Error, No Answers

• A data engineer on a mid-sized e-commerce team submits a Spark job that reads 200 GB of clickstream events, aggregates them by country and device type, and writes a summary table.
• The job runs for 45 minutes and then fails with: The engineer checks executor memory: 4 GB.
• They bump it to 8 GB and resubmit.
• At this point they have spent three hours on a job that should take twenty minutes, and every configuration change is a guess.

java.lang.OutOfMemoryError: GC overhead limit exceeded
  at org.apache.spark.sql.execution.aggregate.HashAggregateExec...

Observable outcome: At this point they have spent three hours on a job that should take twenty minutes, and every configuration change is a guess.

Stage 2 — The Five Components Every Spark Job Runs Through

• Every Spark application, regardless of size or cluster manager, passes through the same five components.
• Before examining any one of them in depth, it helps to see how they relate.
• The diagram below maps all major participants in a running Spark application.
• Notice the two distinct zones: everything inside the Driver JVM (planning and coordination) and everything in the Executor JVMs (data processing).

graph TD
    UserCode[User Application Code]
    SS[SparkSession]
    Driver[Driver JVM Process]
    DAG[DAGScheduler]
    Task[TaskScheduler]
    BE[SchedulerBackend]
    CM[Cluster Manager]
    WN1[Worker Node 1]
    WN2[Worker Node 2]
    WN3[Worker Node 3]
    E1[Executor JVM]
    E2[Executor JVM]
    E3[Executor JVM]
    Th1[Task Threads]
    Th2[Task Threads]
    Th3[Task Threads]

    UserCode --> SS
    SS --> Driver
    Driver --> DAG
    DAG --> Task
    Task --> BE
    BE --> CM
    CM --> WN1
    CM --> WN2
    CM --> WN3
    WN1 --> E1
    WN2 --> E2
    WN3 --> E3
    E1 --> Th1
    E2 --> Th2
    E3 --> Th3

Observable outcome: Notice the two distinct zones: everything inside the Driver JVM (planning and coordination) and everything in the Executor JVMs (data processing).

Stage 3 — How the Driver and Schedulers Convert User Code Into Distributed Tasks

• The Driver is the entry point for your Spark application.
• When you run spark-submit my job.py , a JVM process starts on the driver node, runs your main() (or Python equivalent via Py4J), and creates the SparkSession.
• From that point, the Driver acts as the central coordinator for the entire distributed computation.
• From SparkSession Call to Logical Plan The moment you call a DataFrame transformation, SparkSession captures it as a lazy logical plan — a description of what you want, not how to execute it.

sequenceDiagram
    participant App as User Code
    participant SS as SparkSession
    participant Cat as Catalyst Optimizer
    participant DAG as DAGScheduler

    App->>SS: df.groupBy(country).count().show()
    SS->>Cat: Build logical plan from transformation chain
    Cat->>Cat: Apply predicate pushdown and column pruning
    Cat->>Cat: Choose physical plan - partial aggregate strategy
    Cat->>DAG: Submit optimized RDD DAG
    DAG-->>App: Job submitted - execution begins

graph TD
    subgraph Stage1[Stage 1 - Tokenize and Partial Count]
        A[Read Input Partitions from Storage]
        B[flatMap - split lines into words]
        C[map - emit word and count 1]
        A --> B --> C
    end
    subgraph Stage2[Stage 2 - Shuffle Merge and Write]
        D[Shuffle Read - fetch word buckets by hash]
        E[reduceByKey - sum counts per word]
        F[Write output partitions]
        D --> E --> F
    end
    Stage1 --> Shuffle[Wide Transformation Boundary - reduceByKey causes shuffle]
    Shuffle --> Stage2

Observable outcome: From SparkSession Call to Logical Plan The moment you call a DataFrame transformation, SparkSession captures it as a lazy logical plan — a description of what you want, not how to execute it.

Stage 4 — Deep Dive: How Executors Manage Memory and How the Shuffle Layer Actually Works

• The Internals: Unified Memory Model, Shuffle Files, and Task Concurrency Each Executor is a separate JVM process on a worker node.
• It receives serialized tasks from the TaskScheduler, deserializes them, runs them against data in memory or storage, and either returns results to the Driver or writes shuffle output for downstream stages.
• Every executor JVM has a configurable heap ( --executor-memory ).
• Spark divides this heap into three regions: The Spark Memory Region is shared dynamically between Execution Memory and Storage Memory.

graph TD
    Heap[Executor JVM Heap - set by executor-memory]
    Reserved[Reserved Memory - 300 MB fixed - JVM and Spark internals]
    SparkMem[Spark Memory Region - 60 pct of remaining heap]
    UserMem[User Memory Region - 40 pct of remaining heap]
    ExecMem[Execution Memory - joins, sorts, aggregations, shuffle buffers]
    StoreMem[Storage Memory - cached RDDs, DataFrames, broadcast variables]

    Heap --> Reserved
    Heap --> SparkMem
    Heap --> UserMem
    SparkMem --> ExecMem
    SparkMem --> StoreMem

Observable outcome: Spark divides this heap into three regions: The Spark Memory Region is shared dynamically between Execution Memory and Storage Memory.

Stage 5 — End-to-End Flow: What Happens When You Call df.groupBy("country").count().show()

• Every component described above participates in executing that single line.
• The sequence below traces the complete path from user call to printed results, touching every layer of the architecture.
• The moment .show() is called, lazy evaluation ends.
• Catalyst fires and rewrites the plan — for this groupBy().count() , it generates a two-phase aggregation: partial counts per partition (Stage 1) then a merge (Stage 2).

sequenceDiagram
    participant App as User Code
    participant Driver as Driver JVM
    participant DAG as DAGScheduler
    participant Task as TaskScheduler
    participant CM as Cluster Manager
    participant Exec as Executor JVMs

    App->>Driver: df.groupBy(country).count().show()
    Driver->>Driver: Catalyst builds and optimizes plan
    Driver->>DAG: Submit job with optimized RDD DAG
    DAG->>DAG: Identify shuffle boundary after groupBy
    DAG->>DAG: Create Stage 1 - partial aggregate per partition
    DAG->>Task: Submit Stage 1 task set - 400 tasks
    Task->>CM: Request executor slots
    CM-->>Task: Executor endpoints returned
    Task->>Exec: Serialize and send tasks with data location
    Exec->>Exec: Deserialize task - read partition - hash-aggregate locally
    Exec->>Exec: Write shuffle map files to local disk
    Exec-->>Task: Task complete - shuffle output location reported to Driver
    Task-->>DAG: All Stage 1 tasks complete
    DAG->>DAG: Create Stage 2 tasks - one per shuffle partition
    DAG->>Task: Submit Stage 2 task set - 200 tasks
    Task->>Exec: Send Stage 2 tasks with shuffle fetch locations
    Exec->>Exec: Fetch shuffle buckets from Stage 1 executors
    Exec->>Exec: Merge partial aggregates - final count per country
    Exec-->>Task: Stage 2 tasks complete
    Task-->>DAG: All stages done
    DAG-->>Driver: Job complete
    Driver->>Driver: Collect result rows from executors
    Driver-->>App: Print rows to console

Observable outcome: Catalyst fires and rewrites the plan — for this groupBy().count() , it generates a two-phase aggregation: partial counts per partition (Stage 1) then a merge (Stage 2).

Stage 6 — Choosing Your Cluster Manager and Sizing Spark Resources

• Spark supports three cluster managers, each with a different operational context.
• The choice affects resource allocation behavior, but not Spark's internal DAG or task execution logic.
• Sizing decisions that depend on architecture understanding: When to use fixed vs.
• Dynamic allocation is the right default for shared clusters where jobs have variable parallelism across stages — a job with 10 stages of varying task counts wastes cluster resources holding 100 executors during the sing…

Observable outcome: Dynamic allocation is the right default for shared clusters where jobs have variable parallelism across stages — a job with 10 stages of varying task counts wastes cluster resources holding 100 executors during the sing…

End-to-end outcome

Spark is often introduced as "distributed pandas" or "fast MapReduce." Both descriptions are technically accurate and pedagogically dangerous.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

Failure Modes and the Recovery Limits of Each Component

• Detection: Understanding which failures are recoverable and which are terminal is the most operationally useful aspect of Spark architecture knowledge.
• User or operator impact: Understanding which failures are recoverable and which are terminal is the most operationally useful aspect of Spark architecture knowledge.