Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Preventing Kafka Consumer
Rebalance Storms.

A 24-pod Java payment-ledger consumer group reading 48 partitions at 18,000 msg/s enters a continuous rebalance loop whenever the downstream PostgreSQL ledger experiences a 90s lock spike. Each poll() fetches 500 records, pushing batch processing past max.poll.interval.ms=300000, causing LeaveGroup/rebalance storms and zero net commit progress.
Start the journey5 stages
Loading diagram…
Preventing Kafka Consumer Rebalance Storms architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A 24-pod Java payment-ledger consumer group reading 48 partitions at 18,000 msg/s enters a continuous rebalance loop whenever the downstream PostgreSQL ledger experiences a 90s lock spike. Each poll() fetches 500 records, pushing batch processing past max.poll.interval.ms=300000, causing LeaveGroup/rebalance storms and zero net commit progress.

From patterns to decisions.

A 24-pod Java payment-ledger consumer group reading 48 partitions at 18,000 msg/s enters a continuous rebalance loop whenever the downstream PostgreSQL ledger experiences a 90s lock spike. Each poll() fetches 500 records, pushing batch processing past max.poll.interval.ms=300000, causing LeaveGroup/rebalance storms and zero net commit progress.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

The background heartbeat thread proves.

The background heartbeat thread proves TCP liveness, while max.poll.interval.ms measures application-thread progress between poll() calls.

ChoiceWhat it buysWhat it costs
Trade-off 1Raising max.poll.interval.ms hides real deadlocks for 30+ minutes.

Step 1

Compare session.timeout.ms (45s) vs max.poll.interval.ms (300s) in broker GroupCoordinator logs.

Step 2

Measure time between consecutive poll() invocations during PostgreSQL lock spikes.

Alternative 1

Use pause()/resume() to keep poll() spinning every 200 ms.

Failure Mode 1

Cascading LeaveGroup storms across all 24 pods.

max.poll.records: 500
max.poll.interval.ms: 300000
session.timeout.ms: 45000
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Bound each partition queue to.

Bound each partition queue to 200 records, pause() at 80% capacity, commitSync() only contiguous completed offsets, and resume() below 40%.

ChoiceWhat it buysWhat it costs
Trade-off 1Slightly higher code complexity for offset tracking vs zero rebalance storms.

Step 1

Set max.poll.records=50 and enable-auto-commit=false.

Step 2

Switch partition.assignment.strategy to CooperativeStickyAssignor with static group.instance.id.

Alternative 1

Confluent Parallel Consumer library for key-level async execution.

Failure Mode 1

Committing out-of-order offsets if worker pool processes a single partition concurrently.

if (queue.size() >= 160) consumer.pause(List.of(tp));
// worker completes batch -> completedOffsets.put(tp, nextOffset);
if (queue.size() <= 80 && consumer.paused().contains(tp)) consumer.resume(List.of(tp));
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Increasing max.

Increasing max.poll.interval.ms delays dead-pod detection to 30 minutes, whereas pause() + CooperativeStickyAssignor preserves fast failure detection and incremental rebalances.

ChoiceWhat it buysWhat it costs
Trade-off 1Requires StatefulSet stable pod names for group.instance.id.

Step 1

Evaluate MTTR impact of 1800s max.poll.interval.ms on genuine OOM/deadlock.

Quantify partition movement

RangeAssignor moves ~100% vs CooperativeStickyAssignor moving only 2/48 partitions.

Alternative 1

Dynamic membership with CooperativeStickyAssignor only.

Failure Mode 1

Duplicate group.instance.id across two live pods causes fencing.

partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id: ${HOSTNAME}
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Implement ConsumerRebalanceListener.

Implement ConsumerRebalanceListener.onPartitionsRevoked to await in-flight DB writes, commit completed offsets, and drop unstarted queued records.

ChoiceWhat it buysWhat it costs
Trade-off 110s drain adds bounded latency to graceful shutdowns.

Step 1

Cancel queued unstarted tasks for revoked TopicPartitions.

Step 2

Await currently executing DB batch up to 10s and call commitSync() for those partitions.

Alternative 1

Abort immediately and rely on idempotent DB UPSERTs on replay.

Failure Mode 1

Failing to clear queued records causes zombie writes after another pod owns the partition.

public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
    Map<TopicPartition, OffsetAndMetadata> toCommit = workers.drainAndCancel(partitions, Duration.ofSeconds(10));
    if (!toCommit.isEmpty()) consumer.commitSync(toCommit);
}
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Route non-retryable or 3x-failed records.

Route non-retryable or 3x-failed records to a Dead Letter Queue (DLQ) with original partition/offset metadata so contiguous offset commits can advance.

ChoiceWhat it buysWhat it costs
Trade-off 1DLQ breaks strict global ordering for that specific key.

Step 1

Classify exceptions into transient (SQLTransientException) vs permanent (DeserializationException, ConstraintViolation).

Step 2

Publish permanent failures to <topic>.dlq and mark offset complete.

Alternative 1

Per-key retry topic with exponential delay.

Failure Mode 1

DLQ producer failure must block offset commit to avoid silent drop.

dlqProducer.send(new ProducerRecord<>(topic + ".dlq", record.key(), record.value())).get();
markOffsetCompleted(tp, record.offset() + 1);

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.
  • 02Trade-off tip: Requires StatefulSet stable pod names for group.instance.id.
  • 03Watch out for: Failing to clear queued records causes zombie writes after another pod owns the partition.
Behavioral Progression · Preventing Kafka Consumer Rebalance Storms

Preventing Kafka Consumer Rebalance Storms — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.

Synthesis / One connected design

Putting Preventing Kafka Consumer Rebalance Storms Together

Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.

A candidate architecture, not a universal answer

Putting Preventing Kafka Consumer Rebalance Storms Together

Step 1
Step 2
  1. Set max.poll.records=50 and enable-auto-commit=false.
  2. Switch partition.assignment.strategy to CooperativeStickyAssignor with static group.instance.id.
Implement ConsumerRebalanceListener.onPartitionsRevoked to await in-flight DB writes, commit completed offsets, and drop unstarted queued records.
Interview Readiness Check

Implement ConsumerRebalanceListener.onPartitionsRevoked to await in-flight DB writes, commit completed offsets, and drop unstarted queued records.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Preventing Kafka Consumer Rebalance Storms

Decouple poll() liveness from slow processing via small max.poll.records, bounded worker queues with pause()/resume(), and CooperativeStickyAssignor.