Stream Processing Pipeline Pattern: Stateful Real-Time Data Products

Scenario context

Stream pipelines succeed when event-time semantics, state management, and replay strategy are designed together — and Kafka Streams lets you build all three directly inside your Spring Boot service.

Stripe's real-time fraud detection processes each transaction in under 50 ms. Fraud detection that runs on yesterday's batch data misses a card-cloning attack tonight. Stream processing handles events as they arrive.

I…

This walkthrough focuses on From Batch Jobs to Live Data: Why Stream Processing Exists.

Before making a design choice, clarify:

• Imagine your e-commerce platform's revenue dashboard refreshes every six hours.
• A surge of fraudulent orders hits a specific region at 2:14 AM.
• Your batch job picks it up at 8:00 AM — six hours too late to block the chargebacks.

Concepts in play: event-driven-architecture, jvm-internals-and-garbage-collection, spark-data-engineering, api-gateway

Stage 1 — Framework Choices for Java Stream Processing

• Four mature options exist for JVM teams.
• The right choice depends on your operational model and existing stack.
• It runs as a library inside your application — no separate cluster to provision, operate, or scale.
• It offers exactly-once semantics via Kafka transactions and exposes an interactive state store API that makes building a live REST query endpoint trivial.

Observable outcome: It offers exactly-once semantics via Kafka transactions and exposes an interactive state store API that makes building a live REST query endpoint trivial.

Stage 2 — Event Time, Windows, and State: The Three Pillars of Stream Processing

• processing time Event time is when the event actually occurred — recorded in the event payload itself.
• Processing time is when the event arrives at your pipeline.
• Mobile clients, network partitions, and consumer lag mean these can diverge by seconds to minutes.
• A pipeline relying only on processing time assigns late-arriving events to the wrong window.

flowchart LR
    subgraph Tumbling
        T1[0-60s] --> T2[60-120s]
        T2 --> T3[120-180s]
    end
    subgraph Sliding
        S1[0-60s]
        S2[30-90s]
        S3[60-120s]
    end
    subgraph Session
        SE1[events grouped by idle gap]
    end

Observable outcome: A pipeline relying only on processing time assigns late-arriving events to the wrong window.

Stage 3 — Live Revenue Topology: How the Data Flows

• The complete Kafka Streams topology for this post's scenario — every order event in, aggregated revenue per region per 5-minute window out: Re-keying by region is the critical step: it ensures all events for the same re…
• Without this step, the same region's events would be spread across partitions and each partition would produce a partial — and incorrect — sum.

flowchart LR
    A([order-events topic]) --> B["selectKey (by region)"]
    B --> C[groupByKey]
    C --> D["windowedBy (5-min tumbling)"]
    D --> E["aggregate (sum revenueCents)"]
    E --> F[(revenue-store WindowStore)]
    F --> G[toStream]
    G --> H([revenue-by-region topic])
    F -. "interactive query" .-> I[REST /revenue/{region}/current]

Observable outcome: Without this step, the same region's events would be spread across partitions and each partition would produce a partial — and incorrect — sum.

Stage 4 — Java Implementation: Live Revenue Aggregation with Kafka Streams

• This example demonstrates a complete, production-shaped Kafka Streams topology implementing the live revenue aggregation scenario described earlier in this post.
• It was chosen because Kafka Streams runs as a library inside a standard JVM service—no separate cluster to operate—making it the most accessible entry point for teams already running Kafka.
• Read the code top-to-bottom as a pipeline: Maven dependencies first, domain events next, then the topology definition that wires source topic → key selection → windowed aggregation → sink topic.
• Maven dependencies Domain events Kafka Streams topology Spring Boot application.yml configuration Querying the state store via REST Because the revenue-store is materialized, you can query it directly from a REST contro…

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-streams</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>

public record OrderEvent(String orderId, String region, long revenueCents, Instant eventTime) {}
public record RegionRevenueSummary(String region, long totalRevenueCents, Instant windowStart, Instant windowEnd) {}

Observable outcome: Maven dependencies Domain events Kafka Streams topology Spring Boot application.yml configuration Querying the state store via REST Because the revenue-store is materialized, you can query it directly from a REST contro…

Stage 5 — Deep Dive: Late Arrivals, Watermarks, and Exactly-Once Guarantees

• Internals: Grace Periods, Late Arrivals, and Watermarks TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)) is the strictest window policy: any event arriving after its window's close is silently dropped .
• For a live dashboard this is often acceptable — a 30-second late order is minor noise.
• For financial correctness, add a grace period: This holds the window open an extra 30 seconds, absorbing minor network delays before emitting the final aggregate.
• A region revenue window that drops orders arriving 30 seconds late can undercount by 2–5% during high-traffic bursts — a material discrepancy in a live revenue report reviewed by finance.

.windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))

Observable outcome: A region revenue window that drops orders arriving 30 seconds late can undercount by 2–5% during high-traffic bursts — a material discrepancy in a live revenue report reviewed by finance.

Stage 6 — Real-World Applications: Where Stream Pipelines Are Applied

Observable outcome: You can explain how “Real-World Applications: Where Stream Pipelines Are Applied” changes the architecture or implementation path.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Event-time is almost always the right choice.
• Processing-time windows produce correct-looking but subtly wrong results during any consumer lag event or network hiccup.

Alternatives considered

Decision Guide: Choosing the Right Framework for Your Pipeline

Decision Guide: Choosing the Right Framework for Your Pipeline

Failure and degraded modes

Trade-offs & Failure Modes: Failure Modes in Stream Pipelines

• Detection: Trade-offs & Failure Modes: Failure Modes in Stream Pipelines
• User or operator impact: Trade-offs & Failure Modes: Failure Modes in Stream Pipelines