Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Fixing Kafka
Hot-Partition Skew.

A multi-tenant analytics pipeline produces 25,000 events/sec into a 32-partition Kafka topic keyed by tenant_id. One enterprise tenant ('acme-corp') generates 68% of all traffic (17,000 events/sec, ~17 MB/s). Because Murmur2(tenant_id) % 32 deterministically maps 'acme-corp' to partition 14, partition 14 receives 17,000 msg/s while the other 31 partitions average 258 msg/s each. Consumer pod #14 is 100% CPU-saturated with 4.5M lag, broker #3 disk I/O is pinned at 98%, and the other 31 consumer threads sit at 3% CPU. Redesign the partitioning and consumer aggregation strategy to balance load across partitions without breaking per-tenant analytics.
Start the journey5 stages
Loading diagram…
Fixing Kafka Hot-Partition Skew architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A multi-tenant analytics pipeline produces 25,000 events/sec into a 32-partition Kafka topic keyed by tenant_id. One enterprise tenant ('acme-corp') generates 68% of all traffic (17,000 events/sec, ~17 MB/s). Because Murmur2(tenant_id) % 32 deterministically maps 'acme-corp' to partition 14, partition 14 receives 17,000 msg/s while the other 31 partitions average 258 msg/s each. Consumer pod #14 is 100% CPU-saturated with 4.5M lag, broker #3 disk I/O is pinned at 98%, and the other 31 consumer threads sit at 3% CPU. Redesign the partitioning and consumer aggregation strategy to balance load across partitions without breaking per-tenant analytics.

From patterns to decisions.

A multi-tenant analytics pipeline produces 25,000 events/sec into a 32-partition Kafka topic keyed by tenant_id. One enterprise tenant ('acme-corp') generates 68% of all traffic (17,000 events/sec, ~17 MB/s). Because Murmur2(tenant_id) % 32 deterministically maps 'acme-corp' to partition 14, partition 14 receives 17,000 msg/s while the other 31 partitions average 258 msg/s each. Consumer pod #14 is 100% CPU-saturated with 4.5M lag, broker #3 disk I/O is pinned at 98%, and the other 31 consumer threads sit at 3% CPU. Redesign the partitioning and consumer aggregation strategy to balance load across partitions without breaking per-tenant analytics.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Kafka's default Murmur2 partitioner maps.

Kafka's default Murmur2 partitioner maps a single key ('acme-corp') to a single partition regardless of total partition count, and a partition can only be consumed by one consumer thread in a group.

ChoiceWhat it buysWhat it costs
Trade-off 1Increasing partitions only helps if the key itself is changed to distribute across partitions.

Step 1

Verify Murmur2('acme-corp') % N always produces a single target partition.

Step 2

Confirm Kafka consumer group protocol assigns at most 1 consumer thread per partition.

Step 3

Observe 32 extra pods sitting idle with 0 assigned partitions.

Alternative 1

Remove the key (null key) if ordering is not needed, allowing StickyPartitioner to balance evenly.

Failure Mode 1

Adding partitions without changing keys wastes broker file descriptors while partition 14 remains 100% saturated.

// DefaultPartitioner maps all 'acme-corp' events to ONE partition regardless of numPartitions
int p = Utils.toPositive(Utils.murmur2("acme-corp".getBytes())) % numPartitions;
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Implement org.

Implement org.apache.kafka.clients.producer.Partitioner to check a volatile/atomic hot-tenant set and spread matching tenants across K=16 partitions using a sub-key or random salt.

ChoiceWhat it buysWhat it costs
Using tenant_iduser_id preserves per-user FIFO while balancing 17,000 msg/s across 16 partitions.

Step 1

Check if key (tenant_id) is in hotTenants map (with spread factor K=16).

Step 2

Extract user_id from headers or composite key 'tenant_id:user_id'.

Step 3

Compute (basePartition + Murmur2(user_id) % K) % numPartitions.

Alternative 1

Route whale tenants to a dedicated topic (tenant.analytics.acme-corp) with 16 partitions.

Failure Mode 1

Changing spread K while consumers rely on stateful per-partition joins without a 2-stage reducer.

public class HotTenantAwarePartitioner implements Partitioner {
    private final ConcurrentHashMap<String, Integer> hotTenantSpread = new ConcurrentHashMap<>(Map.of("acme-corp", 16));

    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        int numPartitions = cluster.partitionCountForTopic(topic);
        String rawKey = new String(keyBytes, StandardCharsets.UTF_8);
        String[] parts = rawKey.split(":", 2); // format: tenant_id:user_id
        String tenantId = parts[0];
        int base = Utils.toPositive(Utils.murmur2(tenantId.getBytes(StandardCharsets.UTF_8))) % numPartitions;
        int spread = hotTenantSpread.getOrDefault(tenantId, 1);
        if (spread <= 1) return base;
        int subHash = parts.length > 1
            ? Utils.toPositive(Utils.murmur2(parts[1].getBytes(StandardCharsets.UTF_8)))
            : ThreadLocalRandom.current().nextInt(spread);
        return (base + (subHash % spread)) % numPartitions;
    }
    @Override public void close() {}
    @Override public void configure(Map<String, ?> configs) {}
}
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Key salting reuses existing consumer.

Key salting reuses existing consumer pods and infrastructure uniformly, whereas dedicated topics provide stronger blast-radius isolation at the cost of per-tenant topic/deployment sprawl.

ChoiceWhat it buysWhat it costs
Trade-off 1Dedicated topics prevent a whale burst from filling shared partition queues, but complicate consumer autoscaling.

Evaluate ops overhead

1 topic + dynamic map vs N custom topics + N consumer deployments.

Evaluate noisy-neighbor isolation

dedicated topic protects SLA-bound small tenants from whale bursts.

Step 3

Choose salting for 5-50 whale tenants; choose dedicated tier topics for massive enterprise isolation.

Alternative 1

Two-tier topology: 'standard' shared topic + 'enterprise' shared topic with higher partition count.

Failure Mode 1

Creating thousands of per-tenant topics exhausts ZooKeeper/KRaft controller metadata limits.

# Stage 2 Reducer (Python) merging salted 1-minute partials by tenant_id
def merge_partials(partials):
    totals = {}
    for p in partials:
        key = (p["tenant_id"], p["window_start"])
        totals[key] = totals.get(key, 0) + p["count"]
    return totals
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Push hot-tenant spread updates dynamically.

Push hot-tenant spread updates dynamically via a lightweight config topic or feature-flag SDK so producers expand K from 16 to 32 in seconds without restarts.

ChoiceWhat it buysWhat it costs
Trade-off 1Live spread updates adapt in seconds without rolling restarts, though Stage 2 must tolerate partials arriving from new partitions.

Publish updated {tenant_id

spread_factor} to a compacted config topic or Consul/etcd.

Step 2

CustomPartitioner updates hotTenantSpread ConcurrentHashMap live.

Step 3

Scale consumer pods/partitions if total cluster rate exceeds 32 * 2,200 = 70,400 msg/s.

Alternative 1

Always salt all keys by user_id (tenant_id:user_id) by default if 2-stage aggregation is already in place.

Failure Mode 1

Forgetting to cap spread <= numPartitions wastes modulo math.

// Dynamic update hook in CustomPartitioner
public void updateSpread(String tenantId, int newSpread) {
    hotTenantSpread.put(tenantId, Math.max(1, newSpread));
}
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Compare max partition lag against.

Compare max partition lag against median partition lag and track broker-level BytesInPerSec per partition.

ChoiceWhat it buysWhat it costs
Trade-off 1Ratio-based alerting avoids false positives during global downstream outages where all 32 partitions lag equally.

Step 1

Export per-partition lag via Burrow or kafka_exporter.

Step 2

Alert when max(lag) > 50,000 AND max(lag) > 5 * avg(lag).

Step 3

Alert on per-partition BytesInPerSec > 3x cluster partition mean.

Alternative 1

Use Kafka Cruise Control's built-in partition-load anomaly detector.

Failure Mode 1

Averaging lag across 32 partitions hides a single 4.5M-lag partition.

- alert: KafkaHotPartitionSkew
  expr: |
    max by (topic) (kafka_consumergroup_lag) > 50000
    and
    max by (topic) (kafka_consumergroup_lag) / clamp_min(avg by (topic) (kafka_consumergroup_lag), 1) > 5
  for: 2m

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Salt hot keys (tenant_id + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.
  • 02Trade-off tip: Dedicated topics prevent a whale burst from filling shared partition queues, but complicate consumer autoscaling.
  • 03Watch out for: Forgetting to cap spread <= numPartitions wastes modulo math.
Behavioral Progression · Fixing Kafka Hot Partition Skew

Fixing Kafka Hot Partition Skew — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Salt hot keys (tenantid + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.

Synthesis / One connected design

Putting Fixing Kafka Hot-Partition Skew Together

Salt hot keys (tenant_id + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.

A candidate architecture, not a universal answer

Putting Fixing Kafka Hot-Partition Skew Together

Step 1
Step 2
Step 3
  1. Check if key (tenant_id) is in hotTenants map (with spread factor K=16).
  2. Extract user_id from headers or composite key 'tenant_id:user_id'.
  3. Compute (basePartition + Murmur2(user_id) % K) % numPartitions.
Push hot-tenant spread updates dynamically via a lightweight config topic or feature-flag SDK so producers expand K from 16 to 32 in seconds without restarts.
Interview Readiness Check

Push hot-tenant spread updates dynamically via a lightweight config topic or feature-flag SDK so producers expand K from 16 to 32 in seconds without restarts.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Fixing Kafka Hot-Partition Skew

Salt hot keys (tenant_id + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.