A real-time payment ledger table in DynamoDB is provisioned at 10,000 WCU and 20,000 RCU (`PK = MERCHANT#<merchant_id>`, `SK = TX#<timestamp>#<tx_id>`). During a flash sale, the table throws `ProvisionedThroughputExceededException` (HTTP 400) even though CloudWatch shows total `ConsumedWriteCapacityUnits` at only 1,200 WCU (12% of provisioned capacity). Investigation reveals one celebrity merchant is generating 3,500 writes/sec and 12,000 reads/sec. Redesign the key schema, client write/read path, and caching layer to sustain 5,000 writes/sec and 25,000 reads/sec for a single merchant.
From patterns to decisions.
A real-time payment ledger table in DynamoDB is provisioned at 10,000 WCU and 20,000 RCU (`PK = MERCHANT#<merchant_id>`, `SK = TX#<timestamp>#<tx_id>`). During a flash sale, the table throws `ProvisionedThroughputExceededException` (HTTP 400) even though CloudWatch shows total `ConsumedWriteCapacityUnits` at only 1,200 WCU (12% of provisioned capacity). Investigation reveals one celebrity merchant is generating 3,500 writes/sec and 12,000 reads/sec. Redesign the key schema, client write/read path, and caching layer to sustain 5,000 writes/sec and 25,000 reads/sec for a single merchant.
Table-level metrics average across all.
Table-level metrics average across all physical partitions (e.g., 10 partitions), masking that 1 partition is pegged at its 1,000 WCU/s physical limit.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | DynamoDB Contributor Insights is required to see per-key consumption instead of aggregate table averages. |
Step 1
10,000 WCU across ~10 physical partitions = 1,000 WCU per partition.
Step 2
A single PK (MERCHANT#42) hashes to exactly 1 partition and cannot borrow WCU from idle partitions past 1,000 WCU/s.
Alternative 1
None at the table level—On-Demand mode has the exact same 1,000 WCU/partition cap.
Failure Mode 1
Throwing more provisioned WCU at the table without fixing the partition key.
table_wcu = 10_000
partition_limit = 1_000
# Even with 9 idle partitions, 1 hot PK caps at 1,000 WCU/s (~10-12% of table total)
print(f'Max single-PK throughput: {partition_limit} WCU/s ({partition_limit/table_wcu:.0%} of table)')
Hash tx_id modulo shard_count for.
Hash tx_id modulo shard_count for idempotent sharded writes, and fan out Query calls with heapq.nlargest for sorted top-K reads.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Using `hash(tx_id)` instead of `random()` preserves write idempotency across client retries. |
Step 1
Compute shard = blake2b(tx_id) % shard_count.
Step 2
Write with ConditionExpression='attribute_not_exists(sk)'.
Step 3
Query all shard_count partitions concurrently and merge-sort top 50 by sk.
Alternative 1
Asyncio (aioboto3) for lower thread overhead on high-concurrency scatter-gather.
Failure Mode 1
Fetching Limit=50 from only 1 shard instead of all shards misses newer transactions on other shards.
import hashlib, heapq
from concurrent.futures import ThreadPoolExecutor
def write_tx(table, merchant_id: str, tx_id: str, ts_iso: str, payload: dict, shard_count: int = 8):
h = int.from_bytes(hashlib.blake2b(tx_id.encode(), digest_size=4).digest(), 'big')
shard = h % shard_count
table.put_item(
Item={'pk': f'MERCHANT#{merchant_id}#{shard}', 'sk': f'TX#{ts_iso}#{tx_id}', **payload},
ConditionExpression='attribute_not_exists(sk)',
)
def read_latest_tx(table, merchant_id: str, shard_count: int = 8, limit: int = 50) -> list:
def q(s):
return table.query(
KeyConditionExpression='pk = :pk',
ExpressionAttributeValues={':pk': f'MERCHANT#{merchant_id}#{s}'},
ScanIndexForward=False,
Limit=limit,
)['Items']
with ThreadPoolExecutor(max_workers=shard_count) as pool:
batches = list(pool.map(q, range(shard_count)))
return heapq.nlargest(limit, (item for b in batches for item in b), key=lambda x: x['sk'])
Global 16x sharding multiplies read.
Global 16x sharding multiplies read requests by 16x for normal merchants; use tiered sharding and store a shard_read_set during upgrades.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Using `0`-indexed shards means upgrading `1 -> 8` requires zero data migration to keep old items readable. |
Step 1
Default shard_count = 1 for 99% of merchants.
Step 2
When a merchant approaches 700 WCU/s, bump write_shards = 8 while keeping read_shards = 8 (since 0..7 includes shard 0, old items in MERCHANT#id#0 remain visible!).
Alternative 1
Background backfill job migrating #0 items across #0..#7.
Failure Mode 1
Dropping shard 0 from the read set after upgrading breaks historical lookups.
# Notice: shard 0 is a subset of shards 0..7!
# Upgrading from shard_count=1 (always #0) to shard_count=8 (#0..#7)
# automatically includes historical #0 items in the 8-way scatter-gather.
Place a short-TTL Redis cache.
Place a short-TTL Redis cache or DynamoDB Accelerator (DAX) in front of read_latest_tx to absorb 95%+ of repeated queries.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | A 2-second TTL trades up to 2s of read staleness for a 99% reduction in RCU. |
Step 1
Cache the merged top-50 list in Redis under latest:MERCHANT#42 with a 2-second TTL and singleflight/request-coalescing on miss.
Step 2
Even at 25,000 req/sec, DynamoDB sees only 1 scatter-gather every 2 seconds (< 200 RCU).
Alternative 1
Amazon DAX (transparent read-through cache, though requires VPC peering and doesn't cache scatter-gather merges).
Failure Mode 1
Cache stampede when the 2s key expires under 25K req/s (mitigate with singleflight or probabilistic early refresh).
def get_latest_cached(redis_client, table, merchant_id: str, shard_count: int):
key = f'latest:{merchant_id}'
cached = redis_client.get(key)
if cached:
return json.loads(cached)
items = read_latest_tx(table, merchant_id, shard_count=shard_count)
redis_client.setex(key, 2, json.dumps(items))
return items
Enable adaptive retry mode with.
Enable adaptive retry mode with bounded max_attempts (e.g., 4) and client-side token-bucket pacing.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `adaptive` mode throttles client send rate proactively on 400/503s instead of blindly hammering the partition. |
Configure `Config(retries={'max_attempts'
4, 'mode': 'adaptive'}) in boto3`.
Step 2
Monitor ThrottledRequests and ProvisionedThroughputExceededException rates.
Alternative 1
Legacy standard mode (lacks client-side rate pacing).
Failure Mode 1
Setting max_attempts=10 with synchronous threads exhausts worker pools during a partition hot-spot.
import boto3
from botocore.config import Config
ddb_config = Config(
retries={'max_attempts': 4, 'mode': 'adaptive'},
connect_timeout=1,
read_timeout=2,
)
dynamodb = boto3.resource('dynamodb', config=ddb_config)
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (
PK#0..N-1) and offload hot reads to DAX/Redis. - 02Trade-off tip: Using
0-indexed shards means upgrading1 -> 8requires zero data migration to keep old items readable. - 03Watch out for: Cache stampede when the 2s key expires under 25K req/s (mitigate with singleflight or probabilistic early refresh).
Dynamodb Write Sharding Hot Partitions — Normal → Load / Fault → Recovery
Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (PK0..N-1) and offload hot reads to DAX/Redis.
Putting Eliminating DynamoDB Hot-Partition Throttling Together
Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (`PK#0..N-1`) and offload hot reads to DAX/Redis.
Putting Eliminating DynamoDB Hot-Partition Throttling Together
- Compute
shard = blake2b(tx_id) % shard_count. - Write with
ConditionExpression='attribute_not_exists(sk)'. - Query all
shard_countpartitions concurrently and merge-sort top 50 bysk.
Interview Readiness Check
Place a short-TTL Redis cache or DynamoDB Accelerator (DAX) in front of read_latest_tx to absorb 95%+ of repeated queries.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Eliminating DynamoDB Hot-Partition Throttling
Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (PK#0..N-1) and offload hot reads to DAX/Redis.
