Kappa Architecture: Streaming-First Data Pipelines

Scenario context

Kappa architecture replaces Lambda's batch + speed dual codebases with a single streaming pipeline backed by a replayable Kafka log. Reprocessing becomes replaying from offset 0. One codebase, no drift. >

TLDR: Kappa is the right call when your team is streaming-native and can't afford the logic-drift tax of two parallel pipelines.

Three months after your team shipped a Lambda pipeline, a product manager flag…

This walkthrough focuses on Lambda vs Kappa Decision Guide.

Before making a design choice, clarify:

• Deciding rule of thumb : If your team would reach for Spark first for any new data job, stay with Lambda.
• If your team would reach for Flink or Kafka Streams first, Kappa will be net-simpler.

Concepts in play: event-driven-architecture, data-warehousing-lakehouse, spark-data-engineering, git-internals-and-versioning

Stage 1 — E-Commerce Clickstream Pipeline with Apache Flink (PyFlink)

• The following PyFlink job implements the core Kappa pipeline: consuming raw click events from Kafka, computing 30-minute session windows per user, and writing session summaries back to Kafka.
• This same job — with offset reset to earliest and a new consumer group — is the reprocessing job.
• To switch from live to reprocessing mode , change exactly two lines: 1.
• set group id("kappa-session-v3") — new consumer group so offsets start fresh 2.

from pyflink.datastream import StreamExecutionEnvironment, TimeCharacteristic
from pyflink.datastream.connectors.kafka import (
    KafkaSource,
    KafkaSink,
    KafkaRecordSerializationSchema,
    DeliveryGuarantee,
)
from pyflink.common import WatermarkStrategy, Duration, SimpleStringSchema
from pyflink.common.serialization import SimpleStringSchema
from pyflink.datastream.window import EventTimeSessionWindows
from pyflink.datastream.functions import AggregateFunction
import json
from datetime import datetime

# --- Stream execution environment ---
env = StreamExecutionEnvironment.get_execution_environment()
env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
env.set_parallelism(4)

# --- Kafka source (live mode: latest; reprocess mode: earliest) ---
OFFSET_RESET = "latest"   # Change to "earliest" for reprocessing

kafka_source = (
    KafkaSource.builder()
    .set_bootstrap_servers("kafka:9092")
    .set_topics("clickstream.raw")
    .set_group_id("kappa-session-v2")        # New group ID for reprocessing
    .set_starting_offsets(
        # KafkaOffsetsInitializer.earliest() for reprocessing
        # KafkaOffsetsInitializer.latest() for live
        __import__("pyflink.common", fromlist=["KafkaOffsetsInitializer"])
        .KafkaOffsetsInitializer.latest()
    )
    .set_value_only_deserializer(SimpleStringSchema())
    .build()
)

# --- Parse and assign event-time watermarks ---
def parse_click(raw: str):
    """Returns (user_id, event_type, ts_ms) or None on parse error."""
    try:
        ev = json.loads(raw)
        return (ev["user_id"], ev["event_type"], int(ev["timestamp_ms"]))
    except Exception:
        return None

raw_stream = env.from_source(
    kafka_source,
    WatermarkStrategy
        .for_bounded_out_of_orderness(Duration.of_seconds(30))
        .with_timestamp_assigner(
            lambda event, _: event[2] if event else 0
        ),
    "Clickstream Source",
)

click_stream = (
    raw_stream
    .map(parse_click)
    .filter(lambda x: x is not None)
)

# --- Session window aggregation: 30-minute gap ---
class SessionAggregator(AggregateFunction):
    def create_accumulator(self):
        return {"views": 0, "add_to_cart": 0, "purchases": 0, "start_ts": None, "end_ts": None}

    def add(self, event, acc):
        _, event_type, ts = event
        acc["views"] += 1 if event_type == "view" else 0
        acc["add_to_cart"] += 1 if event_type == "add_to_cart" else 0
        acc["purchases"] += 1 if event_type == "purchase" else 0
        acc["start_ts"] = min(acc["start_ts"] or ts, ts)
        acc["end_ts"] = max(acc["end_ts"] or ts, ts)
        return acc

    def get_result(self, acc):
        return json.dumps(acc)

    def merge(self, acc_a, acc_b):
        return {
            "views": acc_a["views"] + acc_b["views"],
            "add_to_cart": acc_a["add_to_cart"] + acc_b["add_to_cart"],
            "purchases": acc_a["purchases"] + acc_b["purchases"],
            "start_ts": min(acc_a["start_ts"] or 0, acc_b["start_ts"] or 0),
            "end_ts": max(acc_a["end_ts"] or 0, acc_b["end_ts"] or 0),
        }

session_summaries = (
    click_stream
    .key_by(lambda x: x[0])                    # key by user_id
    .window(EventTimeSessionWindows.with_gap(Duration.of_minutes(30)))
    .aggregate(SessionAggregator())
)

# --- Kafka sink: write session summaries ---
kafka_sink = (
    KafkaSink.builder()
    .set_bootstrap_servers("kafka:9092")
    .set_record_serializer(
        KafkaRecordSerializationSchema.builder()
        .set_topic("sessions.v2")              # New topic for reprocessed output
        .set_value_serialization_schema(SimpleStringSchema())
        .build()
    )
    .set_delivery_guarantee(DeliveryGuarantee.EXACTLY_ONCE)
    .build()
)

session_summaries.sink_to(kafka_sink)
env.execute("Kappa-Clickstream-Session-Pipeline-v2")

Observable outcome: set group id("kappa-session-v3") — new consumer group so offsets start fresh 2.

Stage 2 — Apache Flink: The Engine That Makes Kappa Architecture Viable at Scale

• Apache Flink is the open-source stream processing framework most commonly paired with Kappa deployments.
• It was built from the ground up to handle the properties that Kappa requires: exactly-once semantics, event-time processing, and stateful computation with durable state backends.
• • Bronze layer (raw ingest): A Flink job consumes from the source Kafka topic and writes raw events to a Bronze Kafka topic or Delta Lake table — no transformation, schema-on-read, all fields preserved.
• This is the append-only truth log.

Observable outcome: This is the append-only truth log.

End-to-end outcome

After working through the stages, you should be able to explain:

• Start with retention sizing, not job design.
• The most common Kappa failure is running out of Kafka retention mid-replay.
• • Kappa Architecture eliminates Lambda's batch layer by treating a replayable Kafka log as the universal source of truth for both real-time and historical computation.
• • Reprocessing in Kappa = deploying a new version of the streaming job with a new consumer group at offset 0, writing to a new output, and atomically swapping the serving layer when caught up.

Alternatives considered

Lambda vs Kappa Decision Guide

Deciding rule of thumb : If your team would reach for Spark first for any new data job, stay with Lambda.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.