Partitioning in Spark: HashPartitioner, RangePartitioner, and Custom Strategies

Scenario context

Spark's partition count and partitioning strategy are the two levers that determine whether a job scales linearly or crumbles under data growth. HashPartitioner distributes keys by hash modulo — fast and uniform for well-distributed keys, catastrophic for hot keys. RangePartitioner samples the dataset to find natural boundaries, preserving sort order at the cost of an extra pass. Custom partitioners route by domain…

This walkthrough focuses on The Pipeline That Ran Fine at 200 Partitions and Collapsed When Data Grew 5x.

Before making a design choice, clarify:

• A retail analytics team had a daily aggregation pipeline: read 50 GB of order events from HDFS, group by store id , sum revenue per store, and write a summary table.
• With the default 200 shuffle partitions the job ran in 8 minutes.
• Six months later, after an acquisition, the dataset grew to 250 GB.

Concepts in play: spark-data-engineering, database-sharding, jvm-internals-and-garbage-collection, llm-agents-and-langgraph

Stage 1 — What a Partition Is and How Spark Decides the Initial Count

• A partition is the fundamental unit of parallelism in Spark.
• It is a contiguous slice of data that one task processes on one executor thread.
• If you have 200 partitions, Spark creates exactly 200 tasks for that stage, and each task runs independently on whichever executor has a free thread.
• No communication happens between tasks within a stage — they are entirely parallel.

flowchart TD
    A[Input files on S3 or HDFS] --> B["File split by maxPartitionBytes - default 128 MB"]
    B --> C["Read-time partitions - count driven by file size"]
    C --> D[Map-side transformations - filter select withColumn]
    D --> E{Wide transformation?}
    E -->|Yes - groupBy join repartition| F["Shuffle boundary - new partition count from shuffle.partitions"]
    E -->|No - narrow only| G[Same partition count passes through]
    F --> H[Post-shuffle partitions - default 200]
    H --> I{Another wide transformation?}
    I -->|Yes| F
    I -->|No| J[Write output - coalesce or use current count]

Observable outcome: No communication happens between tasks within a stage — they are entirely parallel.

Stage 2 — How HashPartitioner, RangePartitioner, and Custom Strategies Assign Keys to Partitions

• HashPartitioner: The Default Strategy HashPartitioner is Spark's default for all shuffle operations.
• The assignment formula is deceptively simple: For every row, Spark extracts the grouping or join key, calls Java's hashCode() method on it, takes the absolute value, and divides by the target partition count.
• The remainder is the partition assignment.
• This happens independently on every executor during the shuffle write phase, which means no coordination is needed — each task can compute the destination partition locally.

partitionId = Math.abs(key.hashCode()) % numPartitions

flowchart LR
    A[Input row with grouping key] --> B["Compute key.hashCode()"]
    B --> C["Take absolute value"]
    C --> D["Modulo numPartitions"]
    D --> E{Key distribution uniform?}
    E -->|Yes| F[Rows spread evenly across partitions]
    E -->|Hot key dominates| G[Many rows hash to same bucket]
    G --> H[One partition receives most data]
    H --> I[One task runs 10x to 100x longer than others]
    F --> J[All tasks finish in similar time]

Observable outcome: This happens independently on every executor during the shuffle write phase, which means no coordination is needed — each task can compute the destination partition locally.

Stage 3 — Deep Dive into How Spark Partitioners Work Internally

• The Internals of RangePartitioner The sampling pass in RangePartitioner is more sophisticated than a simple random sample.
• Spark uses a reservoir sampling algorithm (specifically a variant of Algorithm R) to draw a fixed-size sample from a stream of unknown length without loading the full dataset into memory.
• Each sampled element is held in a reservoir of size k = 20 numPartitions .
• As the stream progresses, each new element has a probability of k / n (where n is the count of elements seen so far) of replacing a randomly chosen element already in the reservoir.

flowchart TD
    Driver["Driver Node"]
    Planner["Query Planner"]
    Optimizer["Catalyst Optimizer"]
    Executor["Executor Task"]

    Driver -->|"Spark Execution"| Planner
    Planner --> Optimizer
    Optimizer -->|"Optimized Plan"| Executor
    Executor -->|"Result"| Driver

Observable outcome: As the stream progresses, each new element has a probability of k / n (where n is the count of elements seen so far) of replacing a randomly chosen element already in the reservoir.

Stage 4 — How Data Flows Through HashPartitioner Versus RangePartitioner

• The diagram below traces a single shuffle operation from input data through key extraction, partition assignment, and task execution.
• The two paths diverge at the partitioner choice, and the critical difference — uniform distribution versus hot-key collapse — is visible in the task duration column at the bottom.
• The hash path creates straggler task N — a single task holding 41 GB of data that extends the stage duration from 18 seconds to 47 minutes, while 49 other executors sit idle waiting for it to finish.
• The range path produces uniform 2.5 GB partitions because the boundaries were derived from the actual key distribution, not from a mathematical hash function that has no knowledge of frequency.

graph TD
    A[Input Dataset - 500 GB order events] --> B[Map Phase - each executor reads its partitions]
    B --> C[Key Extraction - extract store_id from each row]
    C --> D{Partitioner Choice}

    D -->|HashPartitioner| E[Compute Math.abs store_id.hashCode mod 200]
    D -->|RangePartitioner| F[Reservoir sample 4000 keys - find 199 boundaries]

    E --> G[Partition 0 - 2.3 GB uniform stores]
    E --> H[Partition 1 - 41 GB hot acquired stores]
    E --> I[Partition 2 - 2.1 GB uniform stores]

    F --> J[Partition 0 - stores with low revenue range - 2.5 GB]
    F --> K[Partition 1 - stores with mid revenue range - 2.5 GB]
    F --> L[Partition 2 - stores with high revenue range - 2.5 GB]

    G --> M[Task duration: 18 sec]
    H --> N[Task duration: 47 min - STRAGGLER]
    I --> O[Task duration: 16 sec]

    J --> P[Task duration: 21 sec]
    K --> Q[Task duration: 23 sec]
    L --> R[Task duration: 20 sec]

Observable outcome: The range path produces uniform 2.5 GB partitions because the boundaries were derived from the actual key distribution, not from a mathematical hash function that has no knowledge of frequency.

Stage 5 — Choosing the Right Partitioning Strategy for Your Use Case

• Use this table to match the data scenario and workload pattern to the correct partitioning approach.
• The "Symptom in Spark UI" column is the diagnostic signal that confirms you have reached the right row.

Observable outcome: The "Symptom in Spark UI" column is the diagnostic signal that confirms you have reached the right row.

Stage 6 — Diagnosing Partition Skew Using the Spark UI Stage Detail View

• This section walks through the exact UI navigation and metric interpretation for a live skew diagnosis.
• The goal is to identify whether a slow stage is caused by too few partitions, partition skew, or a configuration mismatch — each has a distinct signature.
• Navigate to the affected stage.
• Open the Spark UI at http://<driver-host :4040 .

Observable outcome: Open the Spark UI at http://<driver-host :4040 .

End-to-end outcome

After working through the stages, you should be able to explain:

• Lesson 1: Adding executors never fixes skew.
• The most common incorrect response to a skewed stage is "give it more resources." More executors do not help because the bottleneck is a single task that cannot be parallelized further without changing the partitioning…

Alternatives considered

repartition() vs coalesce()

Both operations change partition count, but they behave fundamentally differently: repartition(N) triggers a full shuffle .

Salting to Cure Hot Keys

When a few keys dominate the data distribution, the fix is salting : artificially increasing key cardinality by appending a random integer to the hot key before the first shuffle, then removing it after partial aggregation.

Bucketing for Pre-Partitioned Joins

Bucketing is a write-time optimization that pre-partitions and pre-sorts data so that Spark can skip the shuffle on future joins and groupBys.

HashPartitioner vs RangePartitioner vs Custom: Every Choice Has a Cost

The most expensive mistake in Spark partitioning is the repartition–coalesce confusion : teams routinely call repartition(200) after every transformation to "normalize" partition count, paying a full shuffle each time.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.