Spark Structured Streaming: Micro-Batch vs Continuous Processing

Scenario context

<!--

A fintech team runs payment fraud detection with a well-tuned Spark batch job. Every 15 minutes it reads a day's worth of transaction events from S3, scores them against a model, and writes flagged payments to a PostgreSQL alerts table. It works. The false-positive rate is low, the job completes in under three minutes, and the pipeline has been stable for months.

Then the fraud team reviews a case where a sto…

This walkthrough focuses on The 15-Minute Gap: How a Fraud Team Discovered They Needed Real-Time Streaming.

Before making a design choice, clarify:

• A fintech team runs payment fraud detection with a well-tuned Spark batch job.
• Every 15 minutes it reads a day's worth of transaction events from S3, scores them against a model, and writes flagged payments to a PostgreSQL alerts table.
• The false-positive rate is low, the job completes in under three minutes, and the pipeline has been stable for months.

Concepts in play: spark-data-engineering, event-driven-architecture, jvm-internals-and-garbage-collection, data-warehousing-lakehouse

Stage 1 — Streams as Unbounded Tables: The Core Abstraction of Structured Streaming

• Structured Streaming reframes how engineers think about data in motion.
• In the traditional batch model, a dataset is a finite, bounded table: you read it all, compute results, and write output.
• In Structured Streaming, a data stream is modelled as an unbounded table — a table that never stops growing because new rows are constantly being appended by the source (a Kafka topic, a file directory, a socket).
• Reading and Writing a Stream To consume a stream, you use readStream on a SparkSession .

Observable outcome: Reading and Writing a Stream To consume a stream, you use readStream on a SparkSession .

Stage 2 — Inside MicroBatchExecution: How Every Trigger Fires a Batch Job

• The default and most commonly used execution engine in Structured Streaming is MicroBatchExecution .
• Understanding it moves you from "I follow the docs" to "I can debug production incidents." The StreamExecution Thread When you call writeStream.start() , Spark starts a dedicated StreamExecution thread for the query.
• This thread runs in a loop and is managed by StreamingQueryManager , a singleton component in the SparkSession that tracks all active streaming queries, enforces resource policies, and provides the awaitAnyTermination()…
• The loop proceeds in four phases for every trigger interval: 1.

flowchart TD
    A["Trigger fires"] --> B["Poll sources for new offsets"]
    B --> C["Write end offsets to offset WAL"]
    C --> D["Execute Spark batch job over offset range"]
    D --> E["Write committed epoch to commit WAL"]
    E --> F{"More data or trigger?"}
    F -- Yes --> B
    F -- No --> G["Query idle - waiting for next trigger"]

<checkpointDir>/
  offsets/        # WAL: planned end offsets per micro-batch epoch
  commits/        # WAL: committed epochs after successful batch
  state/          # State store data for stateful operators (windows, dedup, joins)
  metadata        # Query ID, Spark version, config snapshot

Observable outcome: The loop proceeds in four phases for every trigger interval: 1.

Stage 3 — How Structured Streaming Guarantees Correctness at Scale

• The Internals of MicroBatchExecution MicroBatchExecution extends StreamExecution , which is the base class for all Spark streaming execution engines.
• The object hierarchy is worth knowing when reading stack traces or Spark source code: Epoch-based offset tracking is how Structured Streaming achieves determinism.
• Each micro-batch is assigned a monotonically increasing epoch number (the batch ID).
• The offset log and commit log are both indexed by this epoch number.

StreamingQueryManager  (1 per SparkSession)
  └── StreamingQuery   (1 per active stream)
        └── StreamExecution  (abstract)
              └── MicroBatchExecution  (default engine)
              └── ContinuousExecution  (experimental engine)

flowchart TD
    A["Measure processingTime vs triggerInterval"] --> B{"processingTime > triggerInterval?"}
    B -- No --> C["Pipeline healthy - trigger interval is fine"]
    B -- Yes --> D{"inputRows > processedRows per second?"}
    D -- No --> E["Transient spike - monitor but no action"]
    D -- Yes --> F["Backlog is growing"]
    F --> G["Check maxOffsetsPerTrigger setting"]
    G --> H{"Resource bottleneck?"}
    H -- Yes --> I["Add executors or increase executor memory"]
    H -- No --> J["Reduce shuffle partitions or simplify stateful ops"]

Observable outcome: The offset log and commit log are both indexed by this epoch number.

Stage 4 — The End-to-End Streaming Pipeline: From Kafka to Sink

• The following diagram shows how data flows through a Structured Streaming pipeline backed by Kafka and writing to Delta Lake.
• Each component maps to a concrete class or abstraction in Spark's codebase.
• The offset WAL (top-right loop) is written before the batch executes.
• The commit WAL (bottom loop) is written after the batch succeeds.

graph TD
    A[Kafka Topic - Source] --> B[KafkaSource - offset polling]
    B --> C[MicroBatchExecution - trigger loop]
    C --> D[Offset WAL - offsets dir]
    D --> C
    C --> E[Spark Batch Job - Catalyst plan]
    E --> F[State Store - windows and dedup]
    F --> G[DataStreamWriter - writeStream]
    G --> H[Delta Lake Sink - transactional write]
    H --> I[Commit WAL - commits dir]
    I --> C
    C --> J[StreamingQueryManager - lifecycle]
    J --> C

Observable outcome: The commit WAL (bottom loop) is written after the batch succeeds.

Stage 5 — Real-World Patterns: Kafka, Incremental File ETL, and Delta Lake Sinks

• Kafka Source with Structured Streaming Kafka is the most common source for Structured Streaming production deployments.
• The KafkaSource maps each Kafka partition to a Spark task and tracks {topic, partition, offset} triples as the offset state.
• The source guarantees that each offset range is processed exactly once across restarts.
• A critical operational detail: Spark does not commit offsets back to Kafka's consumer offsets topic by default.

Observable outcome: A critical operational detail: Spark does not commit offsets back to Kafka's consumer offsets topic by default.

Stage 6 — Choosing the Right Streaming Execution Mode for Your Pipeline

• Use this table to map your pipeline's constraints to the appropriate execution model.
• A few rules of thumb that will save a post-mortem: Do not choose Continuous processing for stateful work.
• It will either not compile or silently produce incorrect results for operators like groupBy , window , and dropDuplicates .
• Do not tune your way out of an undersized cluster.

flowchart TD
    A["Define latency requirement"] --> B{"Latency > 1 minute?"}
    B -- Yes --> C["AvailableNow trigger - batch-style incremental ETL"]
    B -- No --> D{"Latency 1s to 60s?"}
    D -- Yes --> E{"Complex stateful ops?"}
    E -- Yes --> F["Micro-Batch + RocksDB state store"]
    E -- No --> G["Micro-Batch default trigger"]
    D -- No --> H{"Sub-100ms with full state?"}
    H -- Yes --> I["Apache Flink"]
    H -- No --> J["Spark Continuous (experimental - stateless only)"]

Observable outcome: Do not tune your way out of an undersized cluster.

End-to-end outcome

After working through the stages, you should be able to explain:

• Lesson 1: The checkpoint is not optional — treat it like a database.
• Teams routinely point the checkpoint location at a local filesystem or ephemeral NFS mount for development convenience and then forget to change it before deploying to production.
• TLDR: Spark Structured Streaming models a data stream as an unbounded table and executes queries over it using the micro-batch engine ( MicroBatchExecution ).
• Each trigger fires a Spark batch job over a bounded offset range.

Alternatives considered

These three execution models represent genuinely different tradeoffs — not just a dial turned from slow to fast.

These three execution models represent genuinely different tradeoffs — not just a dial turned from slow to fast.

Micro-Batch (Structured Streaming default)

The micro-batch engine achieves latency in the range of 100ms to several seconds depending on trigger configuration and cluster size.

Continuous Processing (Spark experimental)

Continuous processing ( Trigger.Continuous("1 second") ) eliminates the micro-batch boundary.

Apache Flink

Flink's streaming runtime is a true event-at-a-time engine.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.