Spark on Kubernetes: Operator, Dynamic Allocation, and Production Monitoring

Scenario context

Running Spark on Kubernetes replaces YARN's static queue model with a container-native, elastically-scaled execution environment. The kubeflow Spark Operator manages SparkApplication CRDs through a reconciliation loop that creates driver and executor pods, while Dynamic Resource Allocation and pod templates give platform teams granular control over where pods land and how resources are consumed. Prometheus ServiceMo…

This walkthrough focuses on When Four Hundred Jobs Are Waiting: Why YARN's Queue Model Was Never Built for Elasticity.

Before making a design choice, clarify:

• It is a Tuesday morning at a mid-sized data platform team.
• The on-call engineer opens Ambari and finds the YARN resource manager showing 20 Spark applications in RUNNING state and 400 in the ACCEPTED queue.
• Downstream reports that should have refreshed by 7 AM are still stale at 9 AM.

Concepts in play: spark-data-engineering, stack-and-queue-data-structures, caching, event-driven-architecture

Stage 1 — How the Spark Operator Turns a Kubernetes Cluster into a Managed Spark Platform

• The kubeflow Spark Operator — hosted at github.com/kubeflow/spark-operator — implements the Kubernetes Operator pattern for Apache Spark.
• The pattern itself is straightforward: extend the Kubernetes API with a Custom Resource Definition, then run a controller that watches for changes to that resource and reconciles actual cluster state toward the desired…
• The Spark Operator makes the custom resource a SparkApplication , and the controller knows how to translate that spec into driver pods, executor pods, Kubernetes services, and the RBAC bindings that let the driver pod i…
• The SparkApplication CRD The SparkApplication is the unit of work.

flowchart TD
    A[kubectl apply SparkApplication CRD] --> B[Operator detects resource - state NEW]
    B --> C[Create Driver Service and ConfigMap]
    C --> D[Create Driver Pod - state SUBMITTED]
    D --> E{Driver Pod Running?}
    E -->|Yes| F[Driver JVM initializes SparkContext - state RUNNING]
    E -->|No - quota exceeded| G[Pod stays Pending - operator times out]
    F --> H[Driver requests executor pods via K8s API]
    H --> I[Executor pods scheduled on assigned nodes]
    I --> J[Tasks execute on executor threads]
    J --> K{All stages complete?}
    K -->|Yes| L[State COMPLETED - pods released]
    K -->|Driver OOMKill| M[State FAILED - restart policy evaluated]

Observable outcome: The SparkApplication CRD The SparkApplication is the unit of work.

Stage 2 — Deep Dive: The Two Schedulers, the Memory Safety Model, and DRA Scaling Mathematics

• The Internals When a Spark application runs on Kubernetes, two completely independent schedulers operate simultaneously, and their interaction determines both performance and correctness.
• Understanding this two-scheduler model is prerequisite knowledge for diagnosing any Spark-on-K8s production issue.
• The Kubernetes scheduler runs inside the control plane.
• Its job is pod placement: given a pod with declared resource requests, find a node where the requested CPU and memory fit within the node's allocatable capacity after accounting for already-running pods.

pod_request_memory = M_heap + M_oh

M_oh = max(M_heap × 0.10, 384 MiB)

Observable outcome: Its job is pod placement: given a pod with declared resource requests, find a node where the requested CPU and memory fit within the node's allocatable capacity after accounting for already-running pods.

Stage 3 — Building a Production Multi-Tenant Spark Platform on Kubernetes

• A shared Kubernetes cluster running Spark for multiple teams requires deliberate architectural choices to prevent one team's runaway jobs from evicting another team's drivers and to give the platform team visibility int…
• Namespace Isolation and RBAC The standard model is one Kubernetes namespace per team, with a single spark-operator namespace hosting the operator controller.
• The operator is granted a ClusterRole that allows it to watch SparkApplication CRDs across all namespaces and manage pods, services, and config maps within each team namespace.
• Each team namespace gets its own ServiceAccount used by the driver pod when it calls the Kubernetes API to request executor pods.

flowchart LR
    subgraph OP[spark-operator namespace]
        CTL[Operator Controller]
    end
    subgraph TA[team-alpha namespace]
        QA[ResourceQuota]
        SA_A[ServiceAccount]
        CRD_A[SparkApplication CRDs]
    end
    subgraph TB[team-beta namespace]
        QB[ResourceQuota]
        SA_B[ServiceAccount]
        CRD_B[SparkApplication CRDs]
    end
    subgraph NP[Node Pools]
        STD[Standard compute nodes]
        SPOT[Spot nodes]
        GPU[GPU nodes]
    end
    CTL -->|watches and reconciles| CRD_A
    CTL -->|watches and reconciles| CRD_B
    CRD_A -->|nodeSelector| STD
    CRD_A -->|toleration| SPOT
    CRD_B -->|toleration| SPOT
    CRD_B -->|toleration| GPU

Observable outcome: Each team namespace gets its own ServiceAccount used by the driver pod when it calls the Kubernetes API to request executor pods.

Stage 4 — Visualizing the Spark Operator Lifecycle and Dynamic Allocation Flow

• The two diagrams in this section complement each other: the first shows the coarse-grained state machine for a SparkApplication from kubectl apply to pod cleanup; the second shows the fine-grained DRA loop that runs con…
• The lifecycle diagram below traces every state transition the operator controller manages.
• The branches at Driver Pod State and Job Outcome capture the two most operationally important failure points: driver pod scheduling failures (usually caused by ResourceQuota exhaustion or missing node capacity) and driv…
• The DRA scale loop runs independently of the coarse-grained lifecycle.

flowchart TD
    A[kubectl apply SparkApplication CRD] --> B[Operator detects new resource]
    B --> C[Create Driver Service and ConfigMap]
    C --> D[Create Driver Pod]
    D --> E{Driver Pod State}
    E -->|Pending| F[K8s Scheduler places Driver on node]
    F --> G[Driver Pod Running]
    E -->|Running| G
    G --> H[Driver calls K8s API to request executor pods]
    H --> I[Operator creates Executor Pods]
    I --> J{Executor Pod State}
    J -->|Pending| K[K8s Scheduler places Executors on nodes]
    K --> L[Executors register with Driver]
    J -->|Running| L
    L --> M[Spark tasks execute across Executors]
    M --> N{Job completes?}
    N -->|All stages succeed| O[SparkApplication COMPLETED]
    N -->|Driver OOMKill or exception| P[SparkApplication FAILED]
    O --> Q[Operator cleans up all pods and resources]
    P --> R{Restart policy allows retry?}
    R -->|Yes| B
    R -->|No| Q

flowchart TD
    A[Spark task queue has pending tasks] --> B[ExecutorAllocationManager evaluates backlog]
    B --> C[Request new executors via K8s API]
    C --> D[Operator creates new Executor Pods]
    D --> E{Node capacity available?}
    E -->|Yes| F[Pod scheduled and transitions to Running]
    E -->|No| G[Cluster Autoscaler provisions new node]
    G --> F
    F --> H[Executor JVM starts and registers with Driver]
    H --> I[Tasks assigned to new executor]
    I --> J{Executor idle beyond timeout?}
    J -->|No| I
    J -->|Yes| K[DRA sends DecommissionExecutor RPC]
    K --> L[Executor completes current tasks]
    L --> M[Shuffle blocks transferred to External Shuffle Service]
    M --> N[Executor Pod terminates gracefully]
    N --> O[Cluster Autoscaler may scale down empty node]

Observable outcome: The DRA scale loop runs independently of the coarse-grained lifecycle.

Stage 5 — How Spotify, Apple, and Lyft Run Spark on Kubernetes in Production

• The migration from YARN to Kubernetes for Spark is not an experiment — it is the direction of the industry.
• Several large-scale data platforms have published their architectures and lessons publicly.
• Spotify runs one of the largest data platforms on Kubernetes, processing petabytes of event streams daily.
• Their migration was driven by the same isolation problem: different teams sharing YARN clusters led to noisy-neighbor effects where one team's poorly-tuned job could cause GC storms across shared NodeManagers.

flowchart TD
    A[Application running - tasks queued] --> B{Pending tasks exceed threshold?}
    B -->|Yes| C[DRA scale-up trigger fires]
    C --> D[Driver requests new executor pods from K8s API]
    D --> E[K8s scheduler places pods on available nodes]
    E --> F[New executors register with driver]
    F --> G[Pending tasks assigned to new executors]
    G --> H{Executors idle beyond executorIdleTimeout?}
    H -->|Yes| I[DRA scale-down trigger fires]
    I --> J[Driver decommissions idle executors gracefully]
    J --> K[Executor pods released - nodes freed]
    H -->|No| G
    B -->|No| H

Observable outcome: Their migration was driven by the same isolation problem: different teams sharing YARN clusters led to noisy-neighbor effects where one team's poorly-tuned job could cause GC storms across shared NodeManagers.

Stage 6 — Choosing Between YARN, Kubernetes, and Managed Spark for Your Platform

• The decision between YARN, Kubernetes, and managed services (Databricks, EMR Serverless, GCP Dataproc Serverless) depends on organizational maturity, workload patterns, and infrastructure ownership model.
• The most common mistake in this decision is assuming that "switching to Kubernetes" is equivalent to adopting a managed service.
• Running the Spark Operator on Kubernetes still requires platform engineering expertise for cluster sizing, autoscaler configuration, image management, RBAC governance, and observability setup.
• The payoff is complete control over the execution environment and the ability to co-locate Spark with other workloads on shared infrastructure.

sequenceDiagram
    participant User
    participant K8s API
    participant Operator Controller
    participant Driver Pod
    participant Executor Pods
    User->>K8s API: kubectl apply SparkApplication CRD
    K8s API->>Operator Controller: Watch event - new SparkApplication
    Operator Controller->>K8s API: Create Driver Service and ConfigMap
    Operator Controller->>K8s API: Create Driver Pod
    K8s API->>Driver Pod: Schedule and start driver container
    Driver Pod->>K8s API: Request executor pods via ServiceAccount
    K8s API->>Executor Pods: Schedule executor containers on nodes
    Executor Pods->>Driver Pod: Register with SparkContext
    Driver Pod->>Executor Pods: Dispatch tasks
    Executor Pods->>Driver Pod: Report task results
    Driver Pod->>K8s API: Release executor pods on job completion

Observable outcome: The payoff is complete control over the execution environment and the ability to co-locate Spark with other workloads on shared infrastructure.

End-to-end outcome

After working through the stages, you should be able to explain:

• Always set memoryOverhead explicitly.
• The default 10% rule was calibrated for YARN where memory enforcement is gradual.
• • The Spark Operator replaces YARN's ResourceManager with a Kubernetes controller loop that reconciles SparkApplication CRDs into driver pods, executor pods, services, and config maps.
• Jobs are declared as Kubernetes resources and managed by the same kubectl toolchain as the rest of your platform.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

Every architecture decision involves trade-offs, and the YARN-to-Kubernetes migration is no exception. The table below captures the most operationally significant differences:

• Detection: Every architecture decision involves trade-offs, and the YARN-to-Kubernetes migration is no exception. The table below captures the most operationally significant differences:
• User or operator impact: Every architecture decision involves trade-offs, and the YARN-to-Kubernetes migration is no exception. The table below captures the most operationally significant differences:

Failure Modes

• Detection: Because the driver hosts the SparkContext, the DAGScheduler, the metadata for all active broadcast variables, and the result collection buffer for collect() calls, it is prone to memory pressure on l…
• User or operator impact: Because the driver hosts the SparkContext, the DAGScheduler, the metadata for all active broadcast variables, and the result collection buffer for collect() calls, it is prone to memory pressure on l…