A multi-tenant analytics pipeline produces 25,000 events/sec into a 32-partition Kafka topic keyed by tenant_id. One enterprise tenant ('acme-corp') generates 68% of all traffic (17,000 events/sec, ~17 MB/s). Because Murmur2(tenant_id) % 32 deterministically maps 'acme-corp' to partition 14, partition 14 receives 17,000 msg/s while the other 31 partitions average 258 msg/s each. Consumer pod #14 is 100% CPU-saturated with 4.5M lag, broker #3 disk I/O is pinned at 98%, and the other 31 consumer threads sit at 3% CPU. Redesign the partitioning and consumer aggregation strategy to balance load across partitions without breaking per-tenant analytics.
From patterns to decisions.
A multi-tenant analytics pipeline produces 25,000 events/sec into a 32-partition Kafka topic keyed by tenant_id. One enterprise tenant ('acme-corp') generates 68% of all traffic (17,000 events/sec, ~17 MB/s). Because Murmur2(tenant_id) % 32 deterministically maps 'acme-corp' to partition 14, partition 14 receives 17,000 msg/s while the other 31 partitions average 258 msg/s each. Consumer pod #14 is 100% CPU-saturated with 4.5M lag, broker #3 disk I/O is pinned at 98%, and the other 31 consumer threads sit at 3% CPU. Redesign the partitioning and consumer aggregation strategy to balance load across partitions without breaking per-tenant analytics.
Kafka's default Murmur2 partitioner maps.
Kafka's default Murmur2 partitioner maps a single key ('acme-corp') to a single partition regardless of total partition count, and a partition can only be consumed by one consumer thread in a group.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Increasing partitions only helps if the key itself is changed to distribute across partitions. |
Step 1
Verify Murmur2('acme-corp') % N always produces a single target partition.
Step 2
Confirm Kafka consumer group protocol assigns at most 1 consumer thread per partition.
Step 3
Observe 32 extra pods sitting idle with 0 assigned partitions.
Alternative 1
Remove the key (null key) if ordering is not needed, allowing StickyPartitioner to balance evenly.
Failure Mode 1
Adding partitions without changing keys wastes broker file descriptors while partition 14 remains 100% saturated.
// DefaultPartitioner maps all 'acme-corp' events to ONE partition regardless of numPartitions
int p = Utils.toPositive(Utils.murmur2("acme-corp".getBytes())) % numPartitions;
Implement org.
Implement org.apache.kafka.clients.producer.Partitioner to check a volatile/atomic hot-tenant set and spread matching tenants across K=16 partitions using a sub-key or random salt.
| Choice | What it buys | What it costs |
|---|---|---|
| Using tenant_id | user_id preserves per-user FIFO while balancing 17,000 msg/s across 16 partitions. |
Step 1
Check if key (tenant_id) is in hotTenants map (with spread factor K=16).
Step 2
Extract user_id from headers or composite key 'tenant_id:user_id'.
Step 3
Compute (basePartition + Murmur2(user_id) % K) % numPartitions.
Alternative 1
Route whale tenants to a dedicated topic (tenant.analytics.acme-corp) with 16 partitions.
Failure Mode 1
Changing spread K while consumers rely on stateful per-partition joins without a 2-stage reducer.
public class HotTenantAwarePartitioner implements Partitioner {
private final ConcurrentHashMap<String, Integer> hotTenantSpread = new ConcurrentHashMap<>(Map.of("acme-corp", 16));
@Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
int numPartitions = cluster.partitionCountForTopic(topic);
String rawKey = new String(keyBytes, StandardCharsets.UTF_8);
String[] parts = rawKey.split(":", 2); // format: tenant_id:user_id
String tenantId = parts[0];
int base = Utils.toPositive(Utils.murmur2(tenantId.getBytes(StandardCharsets.UTF_8))) % numPartitions;
int spread = hotTenantSpread.getOrDefault(tenantId, 1);
if (spread <= 1) return base;
int subHash = parts.length > 1
? Utils.toPositive(Utils.murmur2(parts[1].getBytes(StandardCharsets.UTF_8)))
: ThreadLocalRandom.current().nextInt(spread);
return (base + (subHash % spread)) % numPartitions;
}
@Override public void close() {}
@Override public void configure(Map<String, ?> configs) {}
}
Key salting reuses existing consumer.
Key salting reuses existing consumer pods and infrastructure uniformly, whereas dedicated topics provide stronger blast-radius isolation at the cost of per-tenant topic/deployment sprawl.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Dedicated topics prevent a whale burst from filling shared partition queues, but complicate consumer autoscaling. |
Evaluate ops overhead
1 topic + dynamic map vs N custom topics + N consumer deployments.
Evaluate noisy-neighbor isolation
dedicated topic protects SLA-bound small tenants from whale bursts.
Step 3
Choose salting for 5-50 whale tenants; choose dedicated tier topics for massive enterprise isolation.
Alternative 1
Two-tier topology: 'standard' shared topic + 'enterprise' shared topic with higher partition count.
Failure Mode 1
Creating thousands of per-tenant topics exhausts ZooKeeper/KRaft controller metadata limits.
# Stage 2 Reducer (Python) merging salted 1-minute partials by tenant_id
def merge_partials(partials):
totals = {}
for p in partials:
key = (p["tenant_id"], p["window_start"])
totals[key] = totals.get(key, 0) + p["count"]
return totals
Push hot-tenant spread updates dynamically.
Push hot-tenant spread updates dynamically via a lightweight config topic or feature-flag SDK so producers expand K from 16 to 32 in seconds without restarts.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Live spread updates adapt in seconds without rolling restarts, though Stage 2 must tolerate partials arriving from new partitions. |
Publish updated {tenant_id
spread_factor} to a compacted config topic or Consul/etcd.
Step 2
CustomPartitioner updates hotTenantSpread ConcurrentHashMap live.
Step 3
Scale consumer pods/partitions if total cluster rate exceeds 32 * 2,200 = 70,400 msg/s.
Alternative 1
Always salt all keys by user_id (tenant_id:user_id) by default if 2-stage aggregation is already in place.
Failure Mode 1
Forgetting to cap spread <= numPartitions wastes modulo math.
// Dynamic update hook in CustomPartitioner
public void updateSpread(String tenantId, int newSpread) {
hotTenantSpread.put(tenantId, Math.max(1, newSpread));
}
Compare max partition lag against.
Compare max partition lag against median partition lag and track broker-level BytesInPerSec per partition.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Ratio-based alerting avoids false positives during global downstream outages where all 32 partitions lag equally. |
Step 1
Export per-partition lag via Burrow or kafka_exporter.
Step 2
Alert when max(lag) > 50,000 AND max(lag) > 5 * avg(lag).
Step 3
Alert on per-partition BytesInPerSec > 3x cluster partition mean.
Alternative 1
Use Kafka Cruise Control's built-in partition-load anomaly detector.
Failure Mode 1
Averaging lag across 32 partitions hides a single 4.5M-lag partition.
- alert: KafkaHotPartitionSkew
expr: |
max by (topic) (kafka_consumergroup_lag) > 50000
and
max by (topic) (kafka_consumergroup_lag) / clamp_min(avg by (topic) (kafka_consumergroup_lag), 1) > 5
for: 2m
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Salt hot keys (tenant_id + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.
- 02Trade-off tip: Dedicated topics prevent a whale burst from filling shared partition queues, but complicate consumer autoscaling.
- 03Watch out for: Forgetting to cap spread <= numPartitions wastes modulo math.
Fixing Kafka Hot Partition Skew — Normal → Load / Fault → Recovery
Salt hot keys (tenantid + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.
Putting Fixing Kafka Hot-Partition Skew Together
Salt hot keys (tenant_id + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.
Putting Fixing Kafka Hot-Partition Skew Together
- Check if key (tenant_id) is in hotTenants map (with spread factor K=16).
- Extract user_id from headers or composite key 'tenant_id:user_id'.
- Compute (basePartition + Murmur2(user_id) % K) % numPartitions.
Interview Readiness Check
Push hot-tenant spread updates dynamically via a lightweight config topic or feature-flag SDK so producers expand K from 16 to 32 in seconds without restarts.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Fixing Kafka Hot-Partition Skew
Salt hot keys (tenant_id + hash % N) or route VIP tenants to dedicated partitions/topics, and pair salted partitioning with two-stage local+global aggregation if stateful stream processing is required.
