Kafka Event Streaming Architecture: Consumer Groups, Partition Offsets, and Exactly-Once Semantics

Order events must stay ordered per customer without losing messages on failure

An e-commerce platform emits order.created, payment.captured, and inventory.reserved events. Payment and inventory services must observe those events in a predictable order for a given order ID, survive broker restarts, and scale consumers horizontally. Kafka provides the log, partitions, and consumer groups that make that possible.

This scenario traces one order event from producer batching through broker append to consumer offset commit.

sequenceDiagram
    title Kafka Partition Replication & Offset Commit Lifecycle
    participant Producer as Order service
    participant Broker as Partition leader
    participant Follower as Follower replica
    participant Consumer as Payment worker
    Producer->>Broker: Produce batch (key=order_id)
    Broker->>Follower: replicate segment
    Broker-->>Producer: offset acknowledged
    Consumer->>Broker: fetch from partition
    Consumer->>Consumer: process payment
    Consumer->>Broker: commit offset

Stage 1 — Choose the partition key and batch records

The producer hashes order_id so every event for the same order lands in one partition. That preserves order for a single key while allowing parallel partitions across the topic. event-driven-architecture frames the topic as the durable contract between services.

• Partition key selection is a correctness decision, not only a throughput knob.
• Batching in the producer accumulator amortizes network round trips.
• Hot keys can create partition hotspots if skew is ignored.

Observable outcome: Events for one order remain ordered within a partition.

Stage 2 — Replicate and acknowledge writes

The partition leader appends records to its log segment and waits for in-sync replicas according to acks and min.insync.replicas. consistency-replication explains why operators tune these settings against their durability promise.

Observable outcome: Producers receive an offset only after the configured replication guarantee is met.

Stage 3 — Consume, process, and commit offsets

Consumers in the same group divide partitions among members. After processing a batch, a consumer commits its offset to __consumer_offsets. distributed-locking analogies apply during rebalance: only one active consumer owns a partition assignment at a time.

Observable outcome: Restarted consumers resume from the last committed offset; duplicate processing is possible unless idempotent handlers are used.

End-to-end outcome

Downstream services scale independently while the log retains a replayable history. Operators monitor consumer lag, rebalance storms, and hot partitions.

Alternatives considered

RabbitMQ queues

Flexible routing but no built-in replayable partitioned log for the same retention semantics.

Failure and degraded modes

Consumer rebalance during deploy

Partition reassignment pauses consumption briefly; cooperative protocols reduce stop-the-world impact.

Hot partition

A skewed key distribution overloads one broker while others stay idle.

• Kafka’s unit of scale and ordering is the partition: keying strategy determines both parallelism and hot-spot risk.
• Consumers advance offsets; at-least-once delivery is the default mental model unless you build stronger idempotency or transactional consume/produce.
• Replication and the ISR define durability — tune acks and min ISR to the loss you can accept on broker failure.
• Rebalances pause work; cooperative assignment and sticky membership reduce deploy-time disruption.
• Retention turns the log into a replayable system of record; design compaction and consumer lag alerts around that property.