An IoT telemetry platform ingests 250,000 events/sec into a 16-node distributed wide-column/KV cluster (ScyllaDB/Cassandra or DynamoDB) using `event_date_hour` or `created_at_ms` as the partition key and `device_id` as the clustering key. During peak ingestion, a single storage node hits 98% CPU, 100% NVMe write saturation, and p99 write latency of 1,400 ms, while the other 15 nodes sit at 6% CPU. Redesign the partition key schema and query path to distribute writes evenly across all 16 nodes while still supporting time-range queries.
From patterns to decisions.
An IoT telemetry platform ingests 250,000 events/sec into a 16-node distributed wide-column/KV cluster (ScyllaDB/Cassandra or DynamoDB) using `event_date_hour` or `created_at_ms` as the partition key and `device_id` as the clustering key. During peak ingestion, a single storage node hits 98% CPU, 100% NVMe write saturation, and p99 write latency of 1,400 ms, while the other 15 nodes sit at 6% CPU. Redesign the partition key schema and query path to distribute writes evenly across all 16 nodes while still supporting time-range queries.
Implementation & Guardrails (1).
Consistent hashing maps identical partition keys to the exact same token and replica set. Because all 250k events/sec share the current hour string, 100% of writes hit one replica set.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Coarse time buckets simplify global queries to a single partition scan, but completely destroy horizontal write scaling. |
Step 1
Inspect nodetool tablestats and nodetool tpstats to confirm one node has 16x higher write count and pending compaction tasks.
Step 2
Check partition key cardinality in the current time window.
Alternative 1
Use a naturally high-cardinality key like device_id if global time-range scans are not required.
Failure Mode 1
The hot node runs out of commitlog/memtable bandwidth, causing cluster-wide coordinator timeouts.
-- ANTI-PATTERN: All writes for the entire hour hash to 1 partition
CREATE TABLE telemetry_hot (
event_hour text,
event_ts timestamp,
device_id uuid,
payload blob,
PRIMARY KEY (event_hour, event_ts, device_id)
);
Use (shard_id, minute_bucket) for global.
Use (shard_id, minute_bucket) for global ingest and (device_id, day_bucket) for entity lookups, with asyncio.gather and heapq.merge for global reads.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Dual-writing to two query-optimized tables doubles write bandwidth, which is acceptable because writes are now spread across 16 nodes. |
Step 1
Compute shard_id = zlib.crc32(device_id.encode()) % 32.
Step 2
Write to both telemetry_by_device and telemetry_global_recent.
Step 3
Query all 32 shards concurrently and merge with heapq.merge.
Alternative 1
Use Kafka + Flink to maintain the global recent view in Redis Sorted Sets instead of ScyllaDB scatter-gather.
Failure Mode 1
Fetching unbounded rows from all 32 shards without per-shard LIMIT can OOM the coordinator.
import asyncio
import heapq
import zlib
NUM_SHARDS = 32
def get_shard_id(device_id: str) -> int:
return zlib.crc32(device_id.encode("utf-8")) % NUM_SHARDS
async def query_global_minute(session, minute_bucket: str, limit: int = 100):
tasks = [
session.execute_async(
"SELECT event_ts, device_id, payload FROM telemetry_global_recent "
"WHERE shard_id = %s AND minute_bucket = %s LIMIT %s",
(shard, minute_bucket, limit),
)
for shard in range(NUM_SHARDS)
]
results = await asyncio.gather(*tasks)
merged = heapq.merge(*results, key=lambda r: r.event_ts, reverse=True)
return [row for _, row in zip(range(limit), merged)]
Size N to 2x–4x node.
Size N to 2x–4x node count (32 for 16 nodes) so every node owns ~2 shards without excessive read fan-out; use deterministic hashing unless a single tenant dominates > 10% of writes.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Deterministic salting keeps single-device reads to 1 shard, whereas random salting forces N-way reads for everything. |
Step 1
Calculate per-shard write throughput (250k / N) and partition byte size.
Step 2
Benchmark 32-way vs 256-way async read latency under p99 load.
Alternative 1
Use dynamic key splitting in DynamoDB/Spanner adaptive splitting where supported.
Failure Mode 1
Too few shards (e.g., 4 shards on 16 nodes) leaves 12 nodes completely idle.
# Hybrid salt: deterministic for normal devices, 4-way sub-salted for VIP devices
def get_hybrid_shard(device_id: str, seq: int, is_high_volume: bool) -> int:
base = zlib.crc32(device_id.encode()) % 32
return (base + (seq % 4)) % 32 if is_high_volume else base
With RF=3 and LOCAL_QUORUM, losing.
With RF=3 and LOCAL_QUORUM, losing 1 node is transparent; if an entire shard's quorum is lost, per-shard timeouts and partial-result degradation prevent cascading thread exhaustion.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Returning partial results preserves dashboard availability at the cost of missing ~3% of devices during a multi-node outage. |
Step 1
Set per-shard read timeout to 30 ms with speculative execution.
Step 2
Catch TimeoutError per shard in asyncio.gather(return_exceptions=True) and return partial results with a flag.
Alternative 1
Fall back to CL=ONE for non-critical telemetry dashboards.
Failure Mode 1
Unbounded retries on a failed shard multiply read traffic by 3x and stall all 32-way queries.
results = await asyncio.gather(*tasks, return_exceptions=True)
valid_streams = [r for r in results if not isinstance(r, Exception)]
partial = len(valid_streams) < NUM_SHARDS
Implementation & Guardrails (5).
Range-partitioned databases split contiguous key ranges into tablets/ranges. Leading with a hash prefix, bit-reversed sequence, or UUIDv4 spreads inserts across all range leaders.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Hash-prefixing eliminates range-leader hotspots at the cost of losing global chronological range scans without scatter-gather. |
Step 1
Replace AUTO_INCREMENT / timestamp PK with (shard_id, timestamp, device_id) or bit-reversed sequence.
Step 2
Enable automatic load-based range splitting in CockroachDB/Spanner.
Alternative 1
Use bit-reversed sequential IDs in Spanner for single-column primary keys.
Failure Mode 1
Using COMMIT_TIMESTAMP as the leading key in Spanner pins all inserts to the rightmost split.
CREATE TABLE telemetry_spanner (
shard_id INT64 NOT NULL,
event_ts TIMESTAMP NOT NULL,
device_id STRING(36) NOT NULL,
payload BYTES(MAX)
) PRIMARY KEY (shard_id, event_ts DESC, device_id);
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Prefix or salt partition keys with a deterministic shard bucket (hash(entity_id) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.
- 02Trade-off tip: Deterministic salting keeps single-device reads to 1 shard, whereas random salting forces N-way reads for everything.
- 03Watch out for: Unbounded retries on a failed shard multiply read traffic by 3x and stall all 32-way queries.
Avoiding Monotonic Key Partition Hotspots — Normal → Load / Fault → Recovery
Prefix or salt partition keys with a deterministic shard bucket (hash(entityid) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.
Putting Avoiding Partition Hotspots on Monotonic Keys Together
Prefix or salt partition keys with a deterministic shard bucket (hash(entity_id) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.
Putting Avoiding Partition Hotspots on Monotonic Keys Together
- Compute shard_id = zlib.crc32(device_id.encode()) % 32.
- Write to both telemetry_by_device and telemetry_global_recent.
- Query all 32 shards concurrently and merge with heapq.merge.
Interview Readiness Check
With RF=3 and LOCAL_QUORUM, losing 1 node is transparent; if an entire shard's quorum is lost, per-shard timeouts and partial-result degradation prevent cascading thread exhaustion.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Avoiding Partition Hotspots on Monotonic Keys
Prefix or salt partition keys with a deterministic shard bucket (hash(entity_id) % N) combined with a coarse time bucket (yyyy-mm-dd-hh), and execute bounded scatter-gather reads across the N buckets.
