A 24-pod Java payment-ledger consumer group reading 48 partitions at 18,000 msg/s enters a continuous rebalance loop whenever the downstream PostgreSQL ledger experiences a 90s lock spike. Each poll() fetches 500 records, pushing batch processing past max.poll.interval.ms=300000, causing LeaveGroup/rebalance storms and zero net commit progress.
From patterns to decisions.
A 24-pod Java payment-ledger consumer group reading 48 partitions at 18,000 msg/s enters a continuous rebalance loop whenever the downstream PostgreSQL ledger experiences a 90s lock spike. Each poll() fetches 500 records, pushing batch processing past max.poll.interval.ms=300000, causing LeaveGroup/rebalance storms and zero net commit progress.
The background heartbeat thread proves.
The background heartbeat thread proves TCP liveness, while max.poll.interval.ms measures application-thread progress between poll() calls.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Raising max.poll.interval.ms hides real deadlocks for 30+ minutes. |
Step 1
Compare session.timeout.ms (45s) vs max.poll.interval.ms (300s) in broker GroupCoordinator logs.
Step 2
Measure time between consecutive poll() invocations during PostgreSQL lock spikes.
Alternative 1
Use pause()/resume() to keep poll() spinning every 200 ms.
Failure Mode 1
Cascading LeaveGroup storms across all 24 pods.
max.poll.records: 500
max.poll.interval.ms: 300000
session.timeout.ms: 45000
Bound each partition queue to.
Bound each partition queue to 200 records, pause() at 80% capacity, commitSync() only contiguous completed offsets, and resume() below 40%.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Slightly higher code complexity for offset tracking vs zero rebalance storms. |
Step 1
Set max.poll.records=50 and enable-auto-commit=false.
Step 2
Switch partition.assignment.strategy to CooperativeStickyAssignor with static group.instance.id.
Alternative 1
Confluent Parallel Consumer library for key-level async execution.
Failure Mode 1
Committing out-of-order offsets if worker pool processes a single partition concurrently.
if (queue.size() >= 160) consumer.pause(List.of(tp));
// worker completes batch -> completedOffsets.put(tp, nextOffset);
if (queue.size() <= 80 && consumer.paused().contains(tp)) consumer.resume(List.of(tp));
Increasing max.
Increasing max.poll.interval.ms delays dead-pod detection to 30 minutes, whereas pause() + CooperativeStickyAssignor preserves fast failure detection and incremental rebalances.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requires StatefulSet stable pod names for group.instance.id. |
Step 1
Evaluate MTTR impact of 1800s max.poll.interval.ms on genuine OOM/deadlock.
Quantify partition movement
RangeAssignor moves ~100% vs CooperativeStickyAssignor moving only 2/48 partitions.
Alternative 1
Dynamic membership with CooperativeStickyAssignor only.
Failure Mode 1
Duplicate group.instance.id across two live pods causes fencing.
partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id: ${HOSTNAME}
Implement ConsumerRebalanceListener.
Implement ConsumerRebalanceListener.onPartitionsRevoked to await in-flight DB writes, commit completed offsets, and drop unstarted queued records.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 10s drain adds bounded latency to graceful shutdowns. |
Step 1
Cancel queued unstarted tasks for revoked TopicPartitions.
Step 2
Await currently executing DB batch up to 10s and call commitSync() for those partitions.
Alternative 1
Abort immediately and rely on idempotent DB UPSERTs on replay.
Failure Mode 1
Failing to clear queued records causes zombie writes after another pod owns the partition.
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
Map<TopicPartition, OffsetAndMetadata> toCommit = workers.drainAndCancel(partitions, Duration.ofSeconds(10));
if (!toCommit.isEmpty()) consumer.commitSync(toCommit);
}
Route non-retryable or 3x-failed records.
Route non-retryable or 3x-failed records to a Dead Letter Queue (DLQ) with original partition/offset metadata so contiguous offset commits can advance.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | DLQ breaks strict global ordering for that specific key. |
Step 1
Classify exceptions into transient (SQLTransientException) vs permanent (DeserializationException, ConstraintViolation).
Step 2
Publish permanent failures to <topic>.dlq and mark offset complete.
Alternative 1
Per-key retry topic with exponential delay.
Failure Mode 1
DLQ producer failure must block offset commit to avoid silent drop.
dlqProducer.send(new ProducerRecord<>(topic + ".dlq", record.key(), record.value())).get();
markOffsetCompleted(tp, record.offset() + 1);
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.
- 02Trade-off tip: Requires StatefulSet stable pod names for group.instance.id.
- 03Watch out for: Failing to clear queued records causes zombie writes after another pod owns the partition.
Preventing Kafka Consumer Rebalance Storms — Normal → Load / Fault → Recovery
Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.
Putting Preventing Kafka Consumer Rebalance Storms Together
Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.
Putting Preventing Kafka Consumer Rebalance Storms Together
- Set max.poll.records=50 and enable-auto-commit=false.
- Switch partition.assignment.strategy to CooperativeStickyAssignor with static group.instance.id.
Interview Readiness Check
Implement ConsumerRebalanceListener.onPartitionsRevoked to await in-flight DB writes, commit completed offsets, and drop unstarted queued records.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Preventing Kafka Consumer Rebalance Storms
Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.
