Kafka and Spark Structured Streaming: Building a Production Pipeline

Scenario context

An analytics platform at a mid-sized fintech company needs to process 500,000 payment events per second from a Kafka cluster. The team starts with a straightforward approach: a hand-rolled Python consumer that polls Kafka, deserializes each message, and writes records to S3. It works in staging with 1,000 events per second. On day one in production, it falls apart completely.

The consumer cannot drain partitions…

This walkthrough focuses on The 500K-Event Problem: When a Naive Kafka Consumer Falls Apart.

Before making a design choice, clarify:

• An analytics platform at a mid-sized fintech company needs to process 500,000 payment events per second from a Kafka cluster.
• The team starts with a straightforward approach: a hand-rolled Python consumer that polls Kafka, deserializes each message, and writes records to S3.
• It works in staging with 1,000 events per second.

Concepts in play: spark-data-engineering, event-driven-architecture, jvm-internals-and-garbage-collection, data-warehousing-lakehouse

Stage 1 — Kafka as a Source and Sink in Structured Streaming: Subscription Modes and Message Schema

• Before examining how Spark manages a Kafka connection, it is worth understanding what Kafka actually hands Structured Streaming.
• Every message read from Kafka arrives as a row in a fixed schema — regardless of what the original producer put into the value bytes.
• The Kafka Source Row Schema When you read from Kafka as a Structured Streaming source, every micro-batch DataFrame has exactly these columns: The critical point: key and value arrive as raw binary.
• You are responsible for deserialization.

Observable outcome: You are responsible for deserialization.

Stage 2 — How Spark Structured Streaming Controls Kafka Reads, Rate-Limits, and Delivers Exactly-Once

• The Kafka–Spark integration is more than a simple poll loop.
• Three mechanics distinguish it from naive consumers: offset tracking via checkpoint files (not Kafka consumer groups), micro-batch rate limiting, and end-to-end exactly-once delivery through idempotent sinks.
• startingOffsets: Where the Query Begins When a streaming query starts for the first time, Spark needs to know where to begin reading each partition.
• The startingOffsets option governs this: - "earliest" — read from the oldest available offset on each partition.

Observable outcome: The startingOffsets option governs this: - "earliest" — read from the oldest available offset on each partition.

Stage 3 — Deep Dive: How Spark Owns Kafka Offset State and How to Tune Pipeline Performance

• The Internals of Kafka Offset Management in Spark One of the most common misunderstandings in Spark–Kafka pipelines is the assumption that Spark uses Kafka's consumer group commit mechanism to track progress.
• Spark manages its own offset state through the checkpoint directory, completely independent of Kafka's consumer group offset tracking.
• The checkpoint directory for a streaming query contains a sources/0/ subdirectory (for the first source) that holds one JSON file per micro-batch.
• Each file is a KafkaSourceOffset — a snapshot of the per-partition offset ranges consumed in that batch.

{"apache-kafka-source":{"payments-prod":{"0":12345,"1":67890,"2":45012}}}

flowchart TD
    Driver["Driver Node"]
    Planner["Query Planner"]
    Optimizer["Catalyst Optimizer"]
    Executor["Executor Task"]

    Driver -->|"Spark Execution"| Planner
    Planner --> Optimizer
    Optimizer -->|"Optimized Plan"| Executor
    Executor -->|"Result"| Driver

Observable outcome: Each file is a KafkaSourceOffset — a snapshot of the per-partition offset ranges consumed in that batch.

Stage 4 — The Complete Kafka-to-Spark Pipeline: From Partition Read to Checkpoint Commit

• The diagram below traces the full lifecycle of a single micro-batch in a Kafka–Spark pipeline.
• The flow moves from Kafka topic partitions through Spark's internal offset tracking machinery, through transformation and deserialization, out to either a Delta Lake table or a downstream Kafka topic, and finally to the…
• Read this diagram from top to bottom.
• Kafka partitions are polled in parallel by the Spark Kafka source.

graph TD
    KP1[Kafka Partition 0]
    KP2[Kafka Partition 1]
    KP3[Kafka Partition N]
    KS[Spark Kafka Source]
    OT[Offset Tracker reads KafkaSourceOffset from checkpoint]
    RB[Rate Limiter applies maxOffsetsPerTrigger per partition]
    MB[Micro-Batch Execution Engine]
    DS[Deserialization - JSON or Avro from_json or from_avro]
    TX[Transformation - filter, join, aggregate, watermark]
    DL[Delta Lake Sink - transactional write]
    KOut[Kafka Sink - idempotent producer]
    CP[Checkpoint WAL - commit offset range]

    KP1 --> KS
    KP2 --> KS
    KP3 --> KS
    KS --> OT
    OT --> RB
    RB --> MB
    MB --> DS
    DS --> TX
    TX --> DL
    TX --> KOut
    DL --> CP
    KOut --> CP

Observable outcome: Kafka partitions are polled in parallel by the Spark Kafka source.

Stage 5 — Where Spark + Kafka Integration Powers Real Systems

• Real-Time ETL to Delta Lake The most common production pattern is Kafka acting as the ingestion buffer and Delta Lake as the analytical store.
• Events arrive in Kafka with raw JSON or Avro payloads.
• Spark reads micro-batches, deserializes and validates each record, applies business-level transformations (currency normalization, user ID resolution, event deduplication), and writes structured Parquet files into a Del…
• Delta's transaction log gives the downstream analytics team a consistent read view even while Spark is actively writing.

Observable outcome: Delta's transaction log gives the downstream analytics team a consistent read view even while Spark is actively writing.

Stage 6 — Streaming Engine Selection: Mapping Throughput, Latency, and State Complexity to the Right Tool

• Use this table as a first-pass decision guide.
• For each row, pick the cell that matches your primary constraint, then verify that the trade-offs in that column are acceptable for your secondary constraints.

Observable outcome: For each row, pick the cell that matches your primary constraint, then verify that the trade-offs in that column are acceptable for your secondary constraints.

End-to-end outcome

After working through the stages, you should be able to explain:

• Checkpoint corruption is an operational reality, not a theoretical edge case.
• In containerized deployments where checkpoint directories live on network-attached storage (S3, ADLS), partial writes during instance termination can corrupt the WAL.
• TLDR: Spark Structured Streaming integrates with Kafka through a checkpoint-owned offset management system that is independent of Kafka's consumer group mechanism.
• Configure maxOffsetsPerTrigger to balance throughput against latency, align Kafka partition count with executor parallelism, use idempotent producers for exactly-once Kafka sink semantics, and monitor both Spark Streami…

Alternatives considered

Spark Structured Streaming vs Kafka Streams vs Flink: Choosing the Right Processing Engine

No single streaming engine dominates every scenario.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.