Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Avoiding Partition Hotspots
on Monotonic Keys.

An IoT telemetry platform ingests 250,000 events/sec into a 16-node distributed wide-column/KV cluster (ScyllaDB/Cassandra or DynamoDB) using event_date_hour or created_at_ms as the partition key and device_id as the clustering key. During peak ingestion, a single storage node hits 98% CPU, 100% NVMe write saturation, and p99 write latency of 1,400 ms, while the other 15 nodes sit at 6% CPU. Redesign the partition key schema and query path to distribute writes evenly across all 16 nodes while still supporting time-range queries.
Start the journey5 stages
Avoiding Partition Hotspots on Monotonic Keys architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

An IoT telemetry platform ingests 250,000 events/sec into a 16-node distributed wide-column/KV cluster (ScyllaDB/Cassandra or DynamoDB) using `event_date_hour` or `created_at_ms` as the partition key and `device_id` as the clustering key. During peak ingestion, a single storage node hits 98% CPU, 100% NVMe write saturation, and p99 write latency of 1,400 ms, while the other 15 nodes sit at 6% CPU. Redesign the partition key schema and query path to distribute writes evenly across all 16 nodes while still supporting time-range queries.

From patterns to decisions.

An IoT telemetry platform ingests 250,000 events/sec into a 16-node distributed wide-column/KV cluster (ScyllaDB/Cassandra or DynamoDB) using `event_date_hour` or `created_at_ms` as the partition key and `device_id` as the clustering key. During peak ingestion, a single storage node hits 98% CPU, 100% NVMe write saturation, and p99 write latency of 1,400 ms, while the other 15 nodes sit at 6% CPU. Redesign the partition key schema and query path to distribute writes evenly across all 16 nodes while still supporting time-range queries.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Implementation & Guardrails (1).

Consistent hashing maps identical partition keys to the exact same token and replica set. Because all 250k events/sec share the current hour string, 100% of writes hit one replica set.

ChoiceWhat it buysWhat it costs
Trade-off 1Coarse time buckets simplify global queries to a single partition scan, but completely destroy horizontal write scaling.

Step 1

Inspect nodetool tablestats and nodetool tpstats to confirm one node has 16x higher write count and pending compaction tasks.

Step 2

Check partition key cardinality in the current time window.

Alternative 1

Use a naturally high-cardinality key like device_id if global time-range scans are not required.

Failure Mode 1

The hot node runs out of commitlog/memtable bandwidth, causing cluster-wide coordinator timeouts.

-- ANTI-PATTERN: All writes for the entire hour hash to 1 partition
CREATE TABLE telemetry_hot (
    event_hour text,
    event_ts timestamp,
    device_id uuid,
    payload blob,
    PRIMARY KEY (event_hour, event_ts, device_id)
);
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Use (shard_id, minute_bucket) for global.

Use (shard_id, minute_bucket) for global ingest and (device_id, day_bucket) for entity lookups, with asyncio.gather and heapq.merge for global reads.

ChoiceWhat it buysWhat it costs
Trade-off 1Dual-writing to two query-optimized tables doubles write bandwidth, which is acceptable because writes are now spread across 16 nodes.

Step 1

Compute shard_id = zlib.crc32(device_id.encode()) % 32.

Step 2

Write to both telemetry_by_device and telemetry_global_recent.

Step 3

Query all 32 shards concurrently and merge with heapq.merge.

Alternative 1

Use Kafka + Flink to maintain the global recent view in Redis Sorted Sets instead of ScyllaDB scatter-gather.

Failure Mode 1

Fetching unbounded rows from all 32 shards without per-shard LIMIT can OOM the coordinator.

import asyncio
import heapq
import zlib

NUM_SHARDS = 32

def get_shard_id(device_id: str) -> int:
    return zlib.crc32(device_id.encode("utf-8")) % NUM_SHARDS

async def query_global_minute(session, minute_bucket: str, limit: int = 100):
    tasks = [
        session.execute_async(
            "SELECT event_ts, device_id, payload FROM telemetry_global_recent "
            "WHERE shard_id = %s AND minute_bucket = %s LIMIT %s",
            (shard, minute_bucket, limit),
        )
        for shard in range(NUM_SHARDS)
    ]
    results = await asyncio.gather(*tasks)
    merged = heapq.merge(*results, key=lambda r: r.event_ts, reverse=True)
    return [row for _, row in zip(range(limit), merged)]
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Size N to 2x–4x node.

Size N to 2x–4x node count (32 for 16 nodes) so every node owns ~2 shards without excessive read fan-out; use deterministic hashing unless a single tenant dominates > 10% of writes.

ChoiceWhat it buysWhat it costs
Trade-off 1Deterministic salting keeps single-device reads to 1 shard, whereas random salting forces N-way reads for everything.

Step 1

Calculate per-shard write throughput (250k / N) and partition byte size.

Step 2

Benchmark 32-way vs 256-way async read latency under p99 load.

Alternative 1

Use dynamic key splitting in DynamoDB/Spanner adaptive splitting where supported.

Failure Mode 1

Too few shards (e.g., 4 shards on 16 nodes) leaves 12 nodes completely idle.

# Hybrid salt: deterministic for normal devices, 4-way sub-salted for VIP devices
def get_hybrid_shard(device_id: str, seq: int, is_high_volume: bool) -> int:
    base = zlib.crc32(device_id.encode()) % 32
    return (base + (seq % 4)) % 32 if is_high_volume else base
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

With RF=3 and LOCAL_QUORUM, losing.

With RF=3 and LOCAL_QUORUM, losing 1 node is transparent; if an entire shard's quorum is lost, per-shard timeouts and partial-result degradation prevent cascading thread exhaustion.

ChoiceWhat it buysWhat it costs
Trade-off 1Returning partial results preserves dashboard availability at the cost of missing ~3% of devices during a multi-node outage.

Step 1

Set per-shard read timeout to 30 ms with speculative execution.

Step 2

Catch TimeoutError per shard in asyncio.gather(return_exceptions=True) and return partial results with a flag.

Alternative 1

Fall back to CL=ONE for non-critical telemetry dashboards.

Failure Mode 1

Unbounded retries on a failed shard multiply read traffic by 3x and stall all 32-way queries.

results = await asyncio.gather(*tasks, return_exceptions=True)
valid_streams = [r for r in results if not isinstance(r, Exception)]
partial = len(valid_streams) < NUM_SHARDS
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Range-partitioned databases split contiguous key ranges into tablets/ranges. Leading with a hash prefix, bit-reversed sequence, or UUIDv4 spreads inserts across all range leaders.

ChoiceWhat it buysWhat it costs
Trade-off 1Hash-prefixing eliminates range-leader hotspots at the cost of losing global chronological range scans without scatter-gather.

Step 1

Replace AUTO_INCREMENT / timestamp PK with (shard_id, timestamp, device_id) or bit-reversed sequence.

Step 2

Enable automatic load-based range splitting in CockroachDB/Spanner.

Alternative 1

Use bit-reversed sequential IDs in Spanner for single-column primary keys.

Failure Mode 1

Using COMMIT_TIMESTAMP as the leading key in Spanner pins all inserts to the rightmost split.

CREATE TABLE telemetry_spanner (
    shard_id INT64 NOT NULL,
    event_ts TIMESTAMP NOT NULL,
    device_id STRING(36) NOT NULL,
    payload BYTES(MAX)
) PRIMARY KEY (shard_id, event_ts DESC, device_id);

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Prefix or salt partition keys with a deterministic shard bucket (hash(entity_id) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.
  • 02Trade-off tip: Deterministic salting keeps single-device reads to 1 shard, whereas random salting forces N-way reads for everything.
  • 03Watch out for: Unbounded retries on a failed shard multiply read traffic by 3x and stall all 32-way queries.
Behavioral Progression · Avoiding Monotonic Key Partition Hotspots

Avoiding Monotonic Key Partition Hotspots — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Prefix or salt partition keys with a deterministic shard bucket (hash(entityid) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.

Synthesis / One connected design

Putting Avoiding Partition Hotspots on Monotonic Keys Together

Prefix or salt partition keys with a deterministic shard bucket (hash(entity_id) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.

A candidate architecture, not a universal answer

Putting Avoiding Partition Hotspots on Monotonic Keys Together

Step 1
Step 2
Step 3
  1. Compute shard_id = zlib.crc32(device_id.encode()) % 32.
  2. Write to both telemetry_by_device and telemetry_global_recent.
  3. Query all 32 shards concurrently and merge with heapq.merge.
With RF=3 and LOCAL_QUORUM, losing 1 node is transparent; if an entire shard's quorum is lost, per-shard timeouts and partial-result degradation prevent cascading thread exhaustion.
Interview Readiness Check

With RF=3 and LOCAL_QUORUM, losing 1 node is transparent; if an entire shard's quorum is lost, per-shard timeouts and partial-result degradation prevent cascading thread exhaustion.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Avoiding Partition Hotspots on Monotonic Keys

Prefix or salt partition keys with a deterministic shard bucket (hash(entity_id) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.