Your public product/user lookup endpoint (`GET /v1/items/{item_id}`) serves 25,000 QPS normally from Redis 7 (98% hit rate) backed by PostgreSQL 16 (3,000 QPS max capacity). A credential-stuffing/scraping botnet begins sending 18,000 QPS of random non-existent `item_id`s (both invalid formats like `-1`/`abc` and valid-looking UUIDv4s that do not exist in DB). Because these keys are absent in Redis and absent in PostgreSQL, nothing is ever cached in Redis—all 18,000 QPS punch straight through to PostgreSQL, driving DB CPU to 100% and taking down legitimate traffic. Redesign the read path to absorb 50,000+ QPS of non-existent key probes with < 2% DB impact.
From patterns to decisions.
Your public product/user lookup endpoint (`GET /v1/items/{item_id}`) serves 25,000 QPS normally from Redis 7 (98% hit rate) backed by PostgreSQL 16 (3,000 QPS max capacity). A credential-stuffing/scraping botnet begins sending 18,000 QPS of random non-existent `item_id`s (both invalid formats like `-1`/`abc` and valid-looking UUIDv4s that do not exist in DB). Because these keys are absent in Redis and absent in PostgreSQL, nothing is ever cached in Redis—all 18,000 QPS punch straight through to PostgreSQL, driving DB CPU to 100% and taking down legitimate traffic. Redesign the read path to absorb 50,000+ QPS of non-existent key probes with < 2% DB impact.
Every unique random UUID still.
Every unique random UUID still misses Redis and hits PostgreSQL once, while storing millions of __NULL__ keys in Redis exhausts memory and evicts real hot items.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Negative caching is great for repeated probes of the *same* missing ID, but useless against high-cardinality random IDs. |
Step 1
Observe 50,000 unique random keys/sec * 60s TTL = 3,000,000 useless __NULL__ keys in Redis.
Step 2
Note that first-time misses for unique random IDs still hit PostgreSQL at 50,000 QPS.
Step 3
Conclude that a compact membership filter (Bloom Filter) is required in front of Redis.
Alternative 1
Bloom Filter or Cuckoo Filter.
Failure Mode 1
Redis LRU eviction storm knocking real hot items out of cache.
# Negative caching alone still lets 50k unique random UUIDs/sec hit DB once!
# 50,000 * 60s = 3M junk keys evicting real cached products in Redis.
Validate UUID format at the.
Validate UUID format at the edge, check BF.EXISTS items:bf item_id, read Redis (handling __NULL__), and on DB miss cache __NULL__ for 60s.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Adds 1 fast RedisBloom check (~0.2ms, or 0ms if using an in-pod Bloom filter) to eliminate 99.9% of penetration load. |
Initialize `BF.RESERVE items
bf 0.001 40000000` (~72 MB).
Step 2
Reject non-UUID inputs with HTTP 400.
Step 3
On BF.EXISTS == 0, return 404 immediately.
On DB `None`, `SET item
{id} 'NULL' EX 60`.
Alternative 1
Guava BloomFilter / Redis Cuckoo Filter (CF.RESERVE) if deletions must be reflected immediately.
Failure Mode 1
Forgetting to call BF.ADD and redis.delete(cache_key) when creating a new item causes temporary 404s.
import json, re
UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
NULL_SENTINEL = "__NULL__"
def get_item(redis, db, item_id: str):
if not UUID_RE.match(item_id):
raise BadRequestError("Invalid item_id format")
if not redis.execute_command("BF.EXISTS", "items:bf", item_id):
return None # 99.9% of non-existent probes stop here
cache_key = f"item:{item_id}"
cached = redis.get(cache_key)
if cached is not None:
return None if cached == NULL_SENTINEL else json.loads(cached)
row = db.fetch_item(item_id)
if row is None:
redis.set(cache_key, NULL_SENTINEL, ex=60) # Absorb 0.1% FP repeats
return None
redis.set(cache_key, json.dumps(row), ex=3600)
return row
Implementation & Guardrails (3).
RedisBloom gives instant global consistency for newly created items (BF.ADD), while per-pod in-memory Bloom Filters cut Redis network RTT to zero during massive DDoS floods.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | In-pod Bloom handles >500k QPS with zero Redis traffic, but requires CDC/PubSub delta streaming to avoid false negatives on new inserts. |
Evaluate RedisBloom
1 shared 72 MB filter, instant BF.ADD visibility across all 40 pods.
Evaluate In-Pod Filter
40 * 72 MB = 2.88 GB total pod RAM, zero Redis network I/O during 50k+ QPS attack, but needs 5s delta sync.
Step 3
Choose RedisBloom for most workloads; add L1 in-pod Bloom only if network bandwidth to Redis saturates.
Alternative 1
Edge CDN / Cloudflare Workers KV Bloom Filter.
Failure Mode 1
Stale in-pod Bloom Filter returns 404 for newly created items until the next sync tick.
// Java Guava L1 BloomFilter (zero network hop during DDoS)
BloomFilter<CharSequence> l1Filter = BloomFilter.create(
Funnels.stringFunnel(StandardCharsets.UTF_8),
40_000_000,
0.001
);
On item creation, synchronously execute.
On item creation, synchronously execute BF.ADD and DEL item:{id} (to clear any prior __NULL__ sentinel) right after DB COMMIT, backed by a CDC listener.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Minimal 0.3ms write-path overhead guarantees read-your-own-writes consistency. |
Step 1
Commit new item to PostgreSQL.
Execute `BF.ADD items
bf {item_id} and DEL item:{item_id}` in a Redis pipeline.
Step 3
Also stream inserts via Debezium CDC to heal any crash between DB commit and Redis update.
Alternative 1
Use UUIDv7 and bypass Bloom check if uuid_ts > now - 30s.
Failure Mode 1
If DEL is omitted, a pre-probed __NULL__ key blocks reads of the newly created item for 60s.
def create_item(db, redis, item):
db.insert_item(item)
pipe = redis.pipeline()
pipe.execute_command("BF.ADD", "items:bf", item["id"])
pipe.delete(f"item:{item['id']}") # Clear any lingering __NULL__
pipe.execute()
Layer per-IP / per-ASN rate.
Layer per-IP / per-ASN rate limiting on 404 responses at the API Gateway, and use singleflight on Redis misses so repeated probes for the same ID coalesce into 1 DB query.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Protects origin against targeted false-positive probing without affecting legitimate users. |
Step 1
Apply Nginx/Envoy sliding-window rate limits per IP/subnet.
Step 2
Penalize IPs exceeding 20 404-responses/min with a 429 Too Many Requests.
Step 3
Combine with singleflight + 60s __NULL__ caching for any key that reaches DB.
Alternative 1
WAF bot challenge / Captcha on high-404 ASNs.
Failure Mode 1
Overly aggressive global 404 limits can block legitimate corporate NAT proxies.
# Envoy / Gateway 404 abuse rate-limit policy
rate_limit:
unit: MINUTE
requests_per_unit: 20
match_status: 404
action: return_429
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (
__NULL__sentinels) for keys that pass the filter. - 02Trade-off tip: In-pod Bloom handles >500k QPS with zero Redis traffic, but requires CDC/PubSub delta streaming to avoid false negatives on new inserts.
- 03Watch out for: If
DELis omitted, a pre-probed__NULL__key blocks reads of the newly created item for 60s.
Preventing Cache Penetration Bloom Filters — Normal → Load / Fault → Recovery
Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (NULL sentinels) for keys that pass the filter.
Putting Preventing Cache Penetration with Bloom Filters Together
Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (`__NULL__` sentinels) for keys that pass the filter.
Putting Preventing Cache Penetration with Bloom Filters Together
- Initialize
BF.RESERVE items:bf 0.001 40000000(~72 MB). - Reject non-UUID inputs with HTTP 400.
- On
BF.EXISTS == 0, return 404 immediately. - On DB
None,SET item:{id} '__NULL__' EX 60.
Interview Readiness Check
On item creation, synchronously execute BF.ADD and DEL item:{id} (to clear any prior __NULL__ sentinel) right after DB COMMIT, backed by a CDC listener.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Preventing Cache Penetration with Bloom Filters
Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (__NULL__ sentinels) for keys that pass the filter.
