A real-time fraud-scoring pipeline reads from Kafka `payments.raw`, scores events, and atomically writes results to `payments.scored` while committing consumer offsets using Kafka Transactions (`EOS`). During a Kubernetes pod eviction and rebalance, a stalled pod hangs in a 25-second JVM GC pause and wakes up while its replacement pod is already processing the same partition. Operators observe both `ProducerFencedException` crashes and duplicate records appearing in a downstream microservice. Design a deterministic `transactional.id` mapping, zombie-fencing lifecycle, and consumer isolation configuration that guarantees zero duplicates across rebalances and GC pauses.
From patterns to decisions.
A real-time fraud-scoring pipeline reads from Kafka `payments.raw`, scores events, and atomically writes results to `payments.scored` while committing consumer offsets using Kafka Transactions (`EOS`). During a Kubernetes pod eviction and rebalance, a stalled pod hangs in a 25-second JVM GC pause and wakes up while its replacement pod is already processing the same partition. Operators observe both `ProducerFencedException` crashes and duplicate records appearing in a downstream microservice. Design a deterministic `transactional.id` mapping, zombie-fencing lifecycle, and consumer isolation configuration that guarantees zero duplicates across rebalances and GC pauses.
Implementation & Guardrails (1).
initTransactions() increments producer_epoch for a given transactional.id, fencing older epochs, while read_committed hides aborted records downstream.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `read_committed` adds latency equal to the upstream commit interval (~200 ms) because consumers wait for the LSO to advance. |
Step 1
Explain that transactional.id maps to a persistent (PID, epoch) in __transaction_state.
Step 2
Show how initTransactions() bumps epoch and aborts open transactions from the previous epoch.
Step 3
Explain that read_uncommitted (Kafka's default) reads uncommitted/aborted bytes before markers arrive.
Alternative 1
Using Kafka Streams with processing.guarantee=exactly_once_v2, which manages transactional.id and fencing automatically.
Failure Mode 1
Leaving isolation.level at default read_uncommitted exposes downstream services to aborted zombie writes.
// Mandatory downstream consumer config for EOS
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "fraud-alert-service");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
A Java EOS loop that.
A Java EOS loop that ties output records and consumer offsets into a single Kafka transaction, aborting/seeking on transient errors and closing on ProducerFencedException.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Using `consumer.groupMetadata()` (KIP-447) simplifies single-producer EOS, while per-partition producers isolate LSO stalls. |
Step 1
Configure deterministic transactional.id and enable.idempotence=true.
Step 2
Call producer.initTransactions() once before polling.
Inside the poll loop
beginTransaction(), send(), sendOffsetsToTransaction(offsets, consumer.groupMetadata()), commitTransaction().
Step 4
On ProducerFencedException, close immediately; on KafkaException, abort and reset consumer offsets.
Alternative 1
Apache Flink KafkaSink with DeliveryGuarantee.EXACTLY_ONCE.
Failure Mode 1
Calling abortTransaction() after ProducerFencedException throws another fencing error instead of shutting down.
producer.initTransactions();
while (running) {
ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(200));
if (records.isEmpty()) continue;
try {
producer.beginTransaction();
Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
for (ConsumerRecord<String, byte[]> rec : records) {
byte[] scored = scoreFraud(rec.value());
producer.send(new ProducerRecord<>("payments.scored", rec.key(), scored));
offsets.put(new TopicPartition(rec.topic(), rec.partition()), new OffsetAndMetadata(rec.offset() + 1));
}
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException fatal) {
producer.close(Duration.ZERO);
consumer.close();
throw fatal;
} catch (KafkaException transientEx) {
producer.abortTransaction();
resetToCommittedOffsets(consumer);
}
}
EOS v2 (KIP-447) uses one.
EOS v2 (KIP-447) uses one transactional producer per worker thread instead of per partition, reducing broker connection and memory overhead by 10-50x.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | EOS v2 drastically cuts broker memory and socket count; EOS v1 isolates a slow partition from delaying other partitions' commits. |
Step 1
EOS v1 required N_partitions producers per instance, creating thousands of open sockets and __transaction_state entries.
Step 2
EOS v2 passes consumer.groupMetadata() so the coordinator validates consumer generation IDs during sendOffsetsToTransaction.
Alternative 1
Idempotent consumer deduplication in PostgreSQL when end-to-end EOS includes an external DB.
Failure Mode 1
Using EOS v2 without static group.instance.id or deterministic instance IDs breaks epoch continuity across pod restarts.
// EOS v2: one deterministic transactional.id per static consumer instance
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "fraud-scorer-" + staticInstanceId);
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, staticInstanceId);
Lower transaction.
Lower transaction.timeout.ms to 15000 ms and enable static membership (group.instance.id) + session.timeout.ms=10000 so stalled transactions abort quickly.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 15s `transaction.timeout.ms` bounds worst-case LSO stalls to 15s, meaning individual scoring batches must finish well under 15s. |
Step 1
Set transaction.timeout.ms=15000 (must be <= transaction.max.timeout.ms on broker).
Step 2
Monitor kafka_consumer_fetch_manager_records_lag_max and LSO - LEO gap per partition.
Alternative 1
Isolating slow ML inference calls to async stages before the transactional Kafka hop.
Failure Mode 1
Setting transaction.timeout.ms lower than max.poll.interval.ms if a batch takes 20s causes the broker to abort active healthy transactions.
props.put(ProducerConfig.TRANSACTION_TIMEOUT_MS_CONFIG, 15000);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 12000);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 30000);
Implementation & Guardrails (5).
No. Kafka transactions only cover Kafka topics and __consumer_offsets; external DB sinks still require an idempotent UNIQUE key in PostgreSQL.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Combines Kafka-level abort filtering with DB-level at-least-once idempotency for true end-to-end exactly-once state. |
Step 1
Explain the dual-write gap between INSERT INTO fraud_alerts and consumer.commitSync().
Step 2
Combine isolation.level=read_committed (to ignore aborted Kafka records) with ON CONFLICT (event_id) DO NOTHING in PostgreSQL.
Alternative 1
Storing the Kafka offset directly inside the same PostgreSQL transaction as the fraud alert.
Failure Mode 1
Assuming Kafka EOS protects external databases or HTTP webhooks leads to duplicates on consumer restarts.
INSERT INTO fraud_alerts (event_id, payment_id, risk_score)
VALUES ($1, $2, $3)
ON CONFLICT (event_id) DO NOTHING;
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Assign a deterministic, partition-bound
transactional.id, callinitTransactions()to bumpproducer_epochand fence zombies, commit consumer offsets viasendOffsetsToTransaction(), and configure downstream consumers withisolation.level=read_committed. - 02Trade-off tip: EOS v2 drastically cuts broker memory and socket count; EOS v1 isolates a slow partition from delaying other partitions' commits.
- 03Watch out for: Setting
transaction.timeout.mslower thanmax.poll.interval.msif a batch takes 20s causes the broker to abort active healthy transactions.
Kafka Exactly Once Processing Fencing — Normal → Load / Fault → Recovery
Assign a deterministic, partition-bound transactional.id, call initTransactions() to bump producerepoch and fence zombies, commit consumer offsets via sendOffsetsToTransaction(), and configure downstream consumers with isolation.level=readcommitted.
Putting Kafka Exactly-Once Processing & Zombie Fencing Together
Assign a deterministic, partition-bound `transactional.id`, call `initTransactions()` to bump `producer_epoch` and fence zombies, commit consumer offsets via `sendOffsetsToTransaction()`, and configure downstream consumers with `isolation.level=read_committed`.
Putting Kafka Exactly-Once Processing & Zombie Fencing Together
- Configure deterministic
transactional.idandenable.idempotence=true. - Call
producer.initTransactions()once before polling. - Inside the poll loop:
beginTransaction(),send(),sendOffsetsToTransaction(offsets, consumer.groupMetadata()),commitTransaction(). - On
ProducerFencedException, close immediately; onKafkaException, abort and reset consumer offsets.
Interview Readiness Check
Lower transaction.timeout.ms to 15000 ms and enable static membership (group.instance.id) + session.timeout.ms=10000 so stalled transactions abort quickly.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Kafka Exactly-Once Processing & Zombie Fencing
Assign a deterministic, partition-bound transactional.id, call initTransactions() to bump producer_epoch and fence zombies, commit consumer offsets via sendOffsetsToTransaction(), and configure downstream consumers with isolation.level=read_committed.
