Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Eliminating DynamoDB
Hot-Partition Throttling.

A real-time payment ledger table in DynamoDB is provisioned at 10,000 WCU and 20,000 RCU (PK = MERCHANT#<merchant_id>, SK = TX#<timestamp>#<tx_id>). During a flash sale, the table throws ProvisionedThroughputExceededException (HTTP 400) even though CloudWatch shows total ConsumedWriteCapacityUnits at only 1,200 WCU (12% of provisioned capacity). Investigation reveals one celebrity merchant is generating 3,500 writes/sec and 12,000 reads/sec. Redesign the key schema, client write/read path, and caching layer to sustain 5,000 writes/sec and 25,000 reads/sec for a single merchant.
Start the journey5 stages
Loading diagram…
Eliminating DynamoDB Hot-Partition Throttling architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A real-time payment ledger table in DynamoDB is provisioned at 10,000 WCU and 20,000 RCU (`PK = MERCHANT#<merchant_id>`, `SK = TX#<timestamp>#<tx_id>`). During a flash sale, the table throws `ProvisionedThroughputExceededException` (HTTP 400) even though CloudWatch shows total `ConsumedWriteCapacityUnits` at only 1,200 WCU (12% of provisioned capacity). Investigation reveals one celebrity merchant is generating 3,500 writes/sec and 12,000 reads/sec. Redesign the key schema, client write/read path, and caching layer to sustain 5,000 writes/sec and 25,000 reads/sec for a single merchant.

From patterns to decisions.

A real-time payment ledger table in DynamoDB is provisioned at 10,000 WCU and 20,000 RCU (`PK = MERCHANT#<merchant_id>`, `SK = TX#<timestamp>#<tx_id>`). During a flash sale, the table throws `ProvisionedThroughputExceededException` (HTTP 400) even though CloudWatch shows total `ConsumedWriteCapacityUnits` at only 1,200 WCU (12% of provisioned capacity). Investigation reveals one celebrity merchant is generating 3,500 writes/sec and 12,000 reads/sec. Redesign the key schema, client write/read path, and caching layer to sustain 5,000 writes/sec and 25,000 reads/sec for a single merchant.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Table-level metrics average across all.

Table-level metrics average across all physical partitions (e.g., 10 partitions), masking that 1 partition is pegged at its 1,000 WCU/s physical limit.

ChoiceWhat it buysWhat it costs
Trade-off 1DynamoDB Contributor Insights is required to see per-key consumption instead of aggregate table averages.

Step 1

10,000 WCU across ~10 physical partitions = 1,000 WCU per partition.

Step 2

A single PK (MERCHANT#42) hashes to exactly 1 partition and cannot borrow WCU from idle partitions past 1,000 WCU/s.

Alternative 1

None at the table level—On-Demand mode has the exact same 1,000 WCU/partition cap.

Failure Mode 1

Throwing more provisioned WCU at the table without fixing the partition key.

table_wcu = 10_000
partition_limit = 1_000
# Even with 9 idle partitions, 1 hot PK caps at 1,000 WCU/s (~10-12% of table total)
print(f'Max single-PK throughput: {partition_limit} WCU/s ({partition_limit/table_wcu:.0%} of table)')
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Hash tx_id modulo shard_count for.

Hash tx_id modulo shard_count for idempotent sharded writes, and fan out Query calls with heapq.nlargest for sorted top-K reads.

ChoiceWhat it buysWhat it costs
Trade-off 1Using `hash(tx_id)` instead of `random()` preserves write idempotency across client retries.

Step 1

Compute shard = blake2b(tx_id) % shard_count.

Step 2

Write with ConditionExpression='attribute_not_exists(sk)'.

Step 3

Query all shard_count partitions concurrently and merge-sort top 50 by sk.

Alternative 1

Asyncio (aioboto3) for lower thread overhead on high-concurrency scatter-gather.

Failure Mode 1

Fetching Limit=50 from only 1 shard instead of all shards misses newer transactions on other shards.

import hashlib, heapq
from concurrent.futures import ThreadPoolExecutor

def write_tx(table, merchant_id: str, tx_id: str, ts_iso: str, payload: dict, shard_count: int = 8):
    h = int.from_bytes(hashlib.blake2b(tx_id.encode(), digest_size=4).digest(), 'big')
    shard = h % shard_count
    table.put_item(
        Item={'pk': f'MERCHANT#{merchant_id}#{shard}', 'sk': f'TX#{ts_iso}#{tx_id}', **payload},
        ConditionExpression='attribute_not_exists(sk)',
    )

def read_latest_tx(table, merchant_id: str, shard_count: int = 8, limit: int = 50) -> list:
    def q(s):
        return table.query(
            KeyConditionExpression='pk = :pk',
            ExpressionAttributeValues={':pk': f'MERCHANT#{merchant_id}#{s}'},
            ScanIndexForward=False,
            Limit=limit,
        )['Items']
    with ThreadPoolExecutor(max_workers=shard_count) as pool:
        batches = list(pool.map(q, range(shard_count)))
    return heapq.nlargest(limit, (item for b in batches for item in b), key=lambda x: x['sk'])
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Global 16x sharding multiplies read.

Global 16x sharding multiplies read requests by 16x for normal merchants; use tiered sharding and store a shard_read_set during upgrades.

ChoiceWhat it buysWhat it costs
Trade-off 1Using `0`-indexed shards means upgrading `1 -> 8` requires zero data migration to keep old items readable.

Step 1

Default shard_count = 1 for 99% of merchants.

Step 2

When a merchant approaches 700 WCU/s, bump write_shards = 8 while keeping read_shards = 8 (since 0..7 includes shard 0, old items in MERCHANT#id#0 remain visible!).

Alternative 1

Background backfill job migrating #0 items across #0..#7.

Failure Mode 1

Dropping shard 0 from the read set after upgrading breaks historical lookups.

# Notice: shard 0 is a subset of shards 0..7!
# Upgrading from shard_count=1 (always #0) to shard_count=8 (#0..#7)
# automatically includes historical #0 items in the 8-way scatter-gather.
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Place a short-TTL Redis cache.

Place a short-TTL Redis cache or DynamoDB Accelerator (DAX) in front of read_latest_tx to absorb 95%+ of repeated queries.

ChoiceWhat it buysWhat it costs
Trade-off 1A 2-second TTL trades up to 2s of read staleness for a 99% reduction in RCU.

Step 1

Cache the merged top-50 list in Redis under latest:MERCHANT#42 with a 2-second TTL and singleflight/request-coalescing on miss.

Step 2

Even at 25,000 req/sec, DynamoDB sees only 1 scatter-gather every 2 seconds (< 200 RCU).

Alternative 1

Amazon DAX (transparent read-through cache, though requires VPC peering and doesn't cache scatter-gather merges).

Failure Mode 1

Cache stampede when the 2s key expires under 25K req/s (mitigate with singleflight or probabilistic early refresh).

def get_latest_cached(redis_client, table, merchant_id: str, shard_count: int):
    key = f'latest:{merchant_id}'
    cached = redis_client.get(key)
    if cached:
        return json.loads(cached)
    items = read_latest_tx(table, merchant_id, shard_count=shard_count)
    redis_client.setex(key, 2, json.dumps(items))
    return items
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Enable adaptive retry mode with.

Enable adaptive retry mode with bounded max_attempts (e.g., 4) and client-side token-bucket pacing.

ChoiceWhat it buysWhat it costs
Trade-off 1`adaptive` mode throttles client send rate proactively on 400/503s instead of blindly hammering the partition.

Configure `Config(retries={'max_attempts'

4, 'mode': 'adaptive'}) in boto3`.

Step 2

Monitor ThrottledRequests and ProvisionedThroughputExceededException rates.

Alternative 1

Legacy standard mode (lacks client-side rate pacing).

Failure Mode 1

Setting max_attempts=10 with synchronous threads exhausts worker pools during a partition hot-spot.

import boto3
from botocore.config import Config

ddb_config = Config(
    retries={'max_attempts': 4, 'mode': 'adaptive'},
    connect_timeout=1,
    read_timeout=2,
)
dynamodb = boto3.resource('dynamodb', config=ddb_config)

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (PK#0..N-1) and offload hot reads to DAX/Redis.
  • 02Trade-off tip: Using 0-indexed shards means upgrading 1 -> 8 requires zero data migration to keep old items readable.
  • 03Watch out for: Cache stampede when the 2s key expires under 25K req/s (mitigate with singleflight or probabilistic early refresh).
Behavioral Progression · Dynamodb Write Sharding Hot Partitions

Dynamodb Write Sharding Hot Partitions — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (PK0..N-1) and offload hot reads to DAX/Redis.

Synthesis / One connected design

Putting Eliminating DynamoDB Hot-Partition Throttling Together

Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (`PK#0..N-1`) and offload hot reads to DAX/Redis.

A candidate architecture, not a universal answer

Putting Eliminating DynamoDB Hot-Partition Throttling Together

Step 1
Step 2
Step 3
  1. Compute shard = blake2b(tx_id) % shard_count.
  2. Write with ConditionExpression='attribute_not_exists(sk)'.
  3. Query all shard_count partitions concurrently and merge-sort top 50 by sk.
Place a short-TTL Redis cache or DynamoDB Accelerator (DAX) in front of `read_latest_tx` to absorb 95%+ of repeated queries.
Interview Readiness Check

Place a short-TTL Redis cache or DynamoDB Accelerator (DAX) in front of read_latest_tx to absorb 95%+ of repeated queries.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Eliminating DynamoDB Hot-Partition Throttling

Adaptive Capacity and On-Demand mode cannot bypass DynamoDB's hard 1,000 WCU / 3,000 RCU per-partition ceiling; break hot keys with deterministic write sharding (PK#0..N-1) and offload hot reads to DAX/Redis.