Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Kafka Exactly-Once Processing
& Zombie Fencing.

A real-time fraud-scoring pipeline reads from Kafka payments.raw, scores events, and atomically writes results to payments.scored while committing consumer offsets using Kafka Transactions (EOS). During a Kubernetes pod eviction and rebalance, a stalled pod hangs in a 25-second JVM GC pause and wakes up while its replacement pod is already processing the same partition. Operators observe both ProducerFencedException crashes and duplicate records appearing in a downstream microservice. Design a deterministic transactional.id mapping, zombie-fencing lifecycle, and consumer isolation configuration that guarantees zero duplicates across rebalances and GC pauses.
Start the journey5 stages
Loading diagram…
Kafka Exactly-Once Processing & Zombie Fencing architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A real-time fraud-scoring pipeline reads from Kafka `payments.raw`, scores events, and atomically writes results to `payments.scored` while committing consumer offsets using Kafka Transactions (`EOS`). During a Kubernetes pod eviction and rebalance, a stalled pod hangs in a 25-second JVM GC pause and wakes up while its replacement pod is already processing the same partition. Operators observe both `ProducerFencedException` crashes and duplicate records appearing in a downstream microservice. Design a deterministic `transactional.id` mapping, zombie-fencing lifecycle, and consumer isolation configuration that guarantees zero duplicates across rebalances and GC pauses.

From patterns to decisions.

A real-time fraud-scoring pipeline reads from Kafka `payments.raw`, scores events, and atomically writes results to `payments.scored` while committing consumer offsets using Kafka Transactions (`EOS`). During a Kubernetes pod eviction and rebalance, a stalled pod hangs in a 25-second JVM GC pause and wakes up while its replacement pod is already processing the same partition. Operators observe both `ProducerFencedException` crashes and duplicate records appearing in a downstream microservice. Design a deterministic `transactional.id` mapping, zombie-fencing lifecycle, and consumer isolation configuration that guarantees zero duplicates across rebalances and GC pauses.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Implementation & Guardrails (1).

initTransactions() increments producer_epoch for a given transactional.id, fencing older epochs, while read_committed hides aborted records downstream.

ChoiceWhat it buysWhat it costs
Trade-off 1`read_committed` adds latency equal to the upstream commit interval (~200 ms) because consumers wait for the LSO to advance.

Step 1

Explain that transactional.id maps to a persistent (PID, epoch) in __transaction_state.

Step 2

Show how initTransactions() bumps epoch and aborts open transactions from the previous epoch.

Step 3

Explain that read_uncommitted (Kafka's default) reads uncommitted/aborted bytes before markers arrive.

Alternative 1

Using Kafka Streams with processing.guarantee=exactly_once_v2, which manages transactional.id and fencing automatically.

Failure Mode 1

Leaving isolation.level at default read_uncommitted exposes downstream services to aborted zombie writes.

// Mandatory downstream consumer config for EOS
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "fraud-alert-service");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

A Java EOS loop that.

A Java EOS loop that ties output records and consumer offsets into a single Kafka transaction, aborting/seeking on transient errors and closing on ProducerFencedException.

ChoiceWhat it buysWhat it costs
Trade-off 1Using `consumer.groupMetadata()` (KIP-447) simplifies single-producer EOS, while per-partition producers isolate LSO stalls.

Step 1

Configure deterministic transactional.id and enable.idempotence=true.

Step 2

Call producer.initTransactions() once before polling.

Inside the poll loop

beginTransaction(), send(), sendOffsetsToTransaction(offsets, consumer.groupMetadata()), commitTransaction().

Step 4

On ProducerFencedException, close immediately; on KafkaException, abort and reset consumer offsets.

Alternative 1

Apache Flink KafkaSink with DeliveryGuarantee.EXACTLY_ONCE.

Failure Mode 1

Calling abortTransaction() after ProducerFencedException throws another fencing error instead of shutting down.

producer.initTransactions();
while (running) {
    ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(200));
    if (records.isEmpty()) continue;
    try {
        producer.beginTransaction();
        Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
        for (ConsumerRecord<String, byte[]> rec : records) {
            byte[] scored = scoreFraud(rec.value());
            producer.send(new ProducerRecord<>("payments.scored", rec.key(), scored));
            offsets.put(new TopicPartition(rec.topic(), rec.partition()), new OffsetAndMetadata(rec.offset() + 1));
        }
        producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
        producer.commitTransaction();
    } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException fatal) {
        producer.close(Duration.ZERO);
        consumer.close();
        throw fatal;
    } catch (KafkaException transientEx) {
        producer.abortTransaction();
        resetToCommittedOffsets(consumer);
    }
}
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

EOS v2 (KIP-447) uses one.

EOS v2 (KIP-447) uses one transactional producer per worker thread instead of per partition, reducing broker connection and memory overhead by 10-50x.

ChoiceWhat it buysWhat it costs
Trade-off 1EOS v2 drastically cuts broker memory and socket count; EOS v1 isolates a slow partition from delaying other partitions' commits.

Step 1

EOS v1 required N_partitions producers per instance, creating thousands of open sockets and __transaction_state entries.

Step 2

EOS v2 passes consumer.groupMetadata() so the coordinator validates consumer generation IDs during sendOffsetsToTransaction.

Alternative 1

Idempotent consumer deduplication in PostgreSQL when end-to-end EOS includes an external DB.

Failure Mode 1

Using EOS v2 without static group.instance.id or deterministic instance IDs breaks epoch continuity across pod restarts.

// EOS v2: one deterministic transactional.id per static consumer instance
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "fraud-scorer-" + staticInstanceId);
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, staticInstanceId);
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Lower transaction.

Lower transaction.timeout.ms to 15000 ms and enable static membership (group.instance.id) + session.timeout.ms=10000 so stalled transactions abort quickly.

ChoiceWhat it buysWhat it costs
Trade-off 115s `transaction.timeout.ms` bounds worst-case LSO stalls to 15s, meaning individual scoring batches must finish well under 15s.

Step 1

Set transaction.timeout.ms=15000 (must be <= transaction.max.timeout.ms on broker).

Step 2

Monitor kafka_consumer_fetch_manager_records_lag_max and LSO - LEO gap per partition.

Alternative 1

Isolating slow ML inference calls to async stages before the transactional Kafka hop.

Failure Mode 1

Setting transaction.timeout.ms lower than max.poll.interval.ms if a batch takes 20s causes the broker to abort active healthy transactions.

props.put(ProducerConfig.TRANSACTION_TIMEOUT_MS_CONFIG, 15000);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 12000);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 30000);
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

No. Kafka transactions only cover Kafka topics and __consumer_offsets; external DB sinks still require an idempotent UNIQUE key in PostgreSQL.

ChoiceWhat it buysWhat it costs
Trade-off 1Combines Kafka-level abort filtering with DB-level at-least-once idempotency for true end-to-end exactly-once state.

Step 1

Explain the dual-write gap between INSERT INTO fraud_alerts and consumer.commitSync().

Step 2

Combine isolation.level=read_committed (to ignore aborted Kafka records) with ON CONFLICT (event_id) DO NOTHING in PostgreSQL.

Alternative 1

Storing the Kafka offset directly inside the same PostgreSQL transaction as the fraud alert.

Failure Mode 1

Assuming Kafka EOS protects external databases or HTTP webhooks leads to duplicates on consumer restarts.

INSERT INTO fraud_alerts (event_id, payment_id, risk_score)
VALUES ($1, $2, $3)
ON CONFLICT (event_id) DO NOTHING;

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Assign a deterministic, partition-bound transactional.id, call initTransactions() to bump producer_epoch and fence zombies, commit consumer offsets via sendOffsetsToTransaction(), and configure downstream consumers with isolation.level=read_committed.
  • 02Trade-off tip: EOS v2 drastically cuts broker memory and socket count; EOS v1 isolates a slow partition from delaying other partitions' commits.
  • 03Watch out for: Setting transaction.timeout.ms lower than max.poll.interval.ms if a batch takes 20s causes the broker to abort active healthy transactions.
Behavioral Progression · Kafka Exactly Once Processing Fencing

Kafka Exactly Once Processing Fencing — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Assign a deterministic, partition-bound transactional.id, call initTransactions() to bump producerepoch and fence zombies, commit consumer offsets via sendOffsetsToTransaction(), and configure downstream consumers with isolation.level=readcommitted.

Synthesis / One connected design

Putting Kafka Exactly-Once Processing & Zombie Fencing Together

Assign a deterministic, partition-bound `transactional.id`, call `initTransactions()` to bump `producer_epoch` and fence zombies, commit consumer offsets via `sendOffsetsToTransaction()`, and configure downstream consumers with `isolation.level=read_committed`.

A candidate architecture, not a universal answer

Putting Kafka Exactly-Once Processing & Zombie Fencing Together

Step 1
Step 2
Step 3
Step 4
  1. Configure deterministic transactional.id and enable.idempotence=true.
  2. Call producer.initTransactions() once before polling.
  3. Inside the poll loop: beginTransaction(), send(), sendOffsetsToTransaction(offsets, consumer.groupMetadata()), commitTransaction().
  4. On ProducerFencedException, close immediately; on KafkaException, abort and reset consumer offsets.
Lower `transaction.timeout.ms` to `15000 ms` and enable static membership (`group.instance.id`) + `session.timeout.ms=10000` so stalled transactions abort quickly.
Interview Readiness Check

Lower transaction.timeout.ms to 15000 ms and enable static membership (group.instance.id) + session.timeout.ms=10000 so stalled transactions abort quickly.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Kafka Exactly-Once Processing & Zombie Fencing

Assign a deterministic, partition-bound transactional.id, call initTransactions() to bump producer_epoch and fence zombies, commit consumer offsets via sendOffsetsToTransaction(), and configure downstream consumers with isolation.level=read_committed.