Caching and Persistence in Spark: Storage Levels and When to Use Them

Scenario context

Calling cache() or persist() does not immediately store anything — Spark caches lazily at the first action, partition by partition, managed by a per-executor BlockManager. When memory fills up, LRU eviction silently drops or spills partitions. Choosing the right StorageLevel and knowing when NOT to cache is what separates a genuine 5× speedup from an OOM-driven cluster reboot.

Observable outcome: persist(StorageLevel.MEMORY ONLY) is the explicit way to get in-memory-only caching on a DataFrame.

Stage 2 — Inside BlockManager: How Spark Physically Stores Cached Data

• Caching in Spark is implemented through the BlockManager , a storage subsystem running inside every executor JVM.
• Understanding its architecture explains why caching behaviour can feel unpredictable without this knowledge.
• The BlockManager topology Every running Spark application has one BlockManager per executor, plus a coordinating BlockManagerMaster running inside the Driver JVM.
• The master does not store data itself — it maintains a registry of which executor holds which blocks and routes lookup requests.

graph TD
    Driver[Driver JVM]
    BMM[BlockManagerMaster]
    BM1[Executor 1 BlockManager]
    BM2[Executor 2 BlockManager]
    MS1[MemoryStore - Executor 1]
    DS1[DiskStore - Executor 1]
    MS2[MemoryStore - Executor 2]
    DS2[DiskStore - Executor 2]
    NBTS[NettyBlockTransferService]

    Driver --> BMM
    BMM -->|block registry| BM1
    BMM -->|block registry| BM2
    BM1 --> MS1
    BM1 --> DS1
    BM2 --> MS2
    BM2 --> DS2
    BM1 -->|direct block fetch| NBTS
    BM2 -->|direct block fetch| NBTS

Observable outcome: The master does not store data itself — it maintains a registry of which executor holds which blocks and routes lookup requests.

Stage 3 — Deep Dive: Caching Internals and Memory Competition

• The Internals of BlockManager The BlockManagerMaster uses an RPC endpoint ( BlockManagerMasterEndpoint ) to receive registration and update messages from executor BlockManagers.
• Every time an executor caches, evicts, or removes a block, it sends an asynchronous message to the master.
• The master's registry is the single source of truth for block locations across the entire application.
• Block replication for fault tolerance is handled at write time.

Executor Heap
└── Reserved Memory (300 MB fixed)
└── Usable Memory = (Heap − 300 MB) × spark.memory.fraction  [default: 0.6]
    └── Initial Storage Region = Usable Memory × spark.memory.storageFraction  [default: 0.5]
    └── Remaining = Initial Execution Region

Observable outcome: Block replication for fault tolerance is handled at write time.

Stage 4 — Where Caching Actually Pays Off in Production

• Caching delivers genuine value in four specific production patterns.
• Outside these patterns, the costs often exceed the benefits.
• Iterative machine learning algorithms Algorithms like gradient descent, k-means clustering, and alternating least squares (ALS) for collaborative filtering make dozens to hundreds of passes over the same training datase…
• Without caching, every iteration reloads the data from object storage — adding seconds or minutes per pass to a training loop that runs hundreds of passes.

Observable outcome: Without caching, every iteration reloads the data from object storage — adding seconds or minutes per pass to a training loop that runs hundreds of passes.

Stage 5 — Choosing a Storage Level for Your Use Case

• Use this reference table to pick the right storage level before reaching for cache() by default: A useful rule of thumb: start with MEMORY AND DISK SER plus Kryo as the default for any new caching decision.
• It is almost universally more memory-efficient than deserialized storage, and the disk fallback prevents the silent-drop failure mode of MEMORY ONLY .
• Upgrade to MEMORY ONLY only after profiling confirms that deserialization CPU cost is a bottleneck, which is rare.

Observable outcome: Upgrade to MEMORY ONLY only after profiling confirms that deserialization CPU cost is a bottleneck, which is rare.

Stage 6 — Reading the Spark UI Storage Tab to Catch Over-Caching

• The Spark UI's Storage tab is the primary diagnostic tool for caching decisions.
• It is often overlooked — many engineers check the Jobs and Stages tabs but ignore Storage entirely.
• That oversight allows over-caching, eviction thrashing, and memory fragmentation to go undetected.
• The Storage tab shows one row per cached RDD or DataFrame.

Observable outcome: The Storage tab shows one row per cached RDD or DataFrame.

End-to-end outcome

After working through the stages, you should be able to explain:

• Lesson 1: Always check the Storage tab, not just the Stages tab.
• A job that completes "successfully" with a Fraction Cached of 0.3 is recomputing 70% of its supposedly-cached data.

Alternatives considered

Cache Hit vs. Cache Miss: The Full Execution Path

The two paths a Spark task can take when it needs a cached partition are quite different in cost.

Failure and degraded modes

Deserialized vs. serialized: speed vs. memory footprint

• Detection: Deserialized storage ( MEMORY ONLY , MEMORY AND DISK ) keeps data as Java objects directly in the JVM heap.
• User or operator impact: Deserialized storage ( MEMORY ONLY , MEMORY AND DISK ) keeps data as Java objects directly in the JVM heap.

When NOT to cache

• Detection: Not every DataFrame benefits from caching.
• User or operator impact: Not every DataFrame benefits from caching.