Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Preventing Cache Penetration
with Bloom Filters.

Your public product/user lookup endpoint (GET /v1/items/{item_id}) serves 25,000 QPS normally from Redis 7 (98% hit rate) backed by PostgreSQL 16 (3,000 QPS max capacity). A credential-stuffing/scraping botnet begins sending 18,000 QPS of random non-existent item_ids (both invalid formats like -1/abc and valid-looking UUIDv4s that do not exist in DB). Because these keys are absent in Redis and absent in PostgreSQL, nothing is ever cached in Redis—all 18,000 QPS punch straight through to PostgreSQL, driving DB CPU to 100% and taking down legitimate traffic. Redesign the read path to absorb 50,000+ QPS of non-existent key probes with < 2% DB impact.
Start the journey5 stages
Loading diagram…
Preventing Cache Penetration with Bloom Filters architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your public product/user lookup endpoint (`GET /v1/items/{item_id}`) serves 25,000 QPS normally from Redis 7 (98% hit rate) backed by PostgreSQL 16 (3,000 QPS max capacity). A credential-stuffing/scraping botnet begins sending 18,000 QPS of random non-existent `item_id`s (both invalid formats like `-1`/`abc` and valid-looking UUIDv4s that do not exist in DB). Because these keys are absent in Redis and absent in PostgreSQL, nothing is ever cached in Redis—all 18,000 QPS punch straight through to PostgreSQL, driving DB CPU to 100% and taking down legitimate traffic. Redesign the read path to absorb 50,000+ QPS of non-existent key probes with < 2% DB impact.

From patterns to decisions.

Your public product/user lookup endpoint (`GET /v1/items/{item_id}`) serves 25,000 QPS normally from Redis 7 (98% hit rate) backed by PostgreSQL 16 (3,000 QPS max capacity). A credential-stuffing/scraping botnet begins sending 18,000 QPS of random non-existent `item_id`s (both invalid formats like `-1`/`abc` and valid-looking UUIDv4s that do not exist in DB). Because these keys are absent in Redis and absent in PostgreSQL, nothing is ever cached in Redis—all 18,000 QPS punch straight through to PostgreSQL, driving DB CPU to 100% and taking down legitimate traffic. Redesign the read path to absorb 50,000+ QPS of non-existent key probes with < 2% DB impact.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Every unique random UUID still.

Every unique random UUID still misses Redis and hits PostgreSQL once, while storing millions of __NULL__ keys in Redis exhausts memory and evicts real hot items.

ChoiceWhat it buysWhat it costs
Trade-off 1Negative caching is great for repeated probes of the *same* missing ID, but useless against high-cardinality random IDs.

Step 1

Observe 50,000 unique random keys/sec * 60s TTL = 3,000,000 useless __NULL__ keys in Redis.

Step 2

Note that first-time misses for unique random IDs still hit PostgreSQL at 50,000 QPS.

Step 3

Conclude that a compact membership filter (Bloom Filter) is required in front of Redis.

Alternative 1

Bloom Filter or Cuckoo Filter.

Failure Mode 1

Redis LRU eviction storm knocking real hot items out of cache.

# Negative caching alone still lets 50k unique random UUIDs/sec hit DB once!
# 50,000 * 60s = 3M junk keys evicting real cached products in Redis.
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Validate UUID format at the.

Validate UUID format at the edge, check BF.EXISTS items:bf item_id, read Redis (handling __NULL__), and on DB miss cache __NULL__ for 60s.

ChoiceWhat it buysWhat it costs
Trade-off 1Adds 1 fast RedisBloom check (~0.2ms, or 0ms if using an in-pod Bloom filter) to eliminate 99.9% of penetration load.

Initialize `BF.RESERVE items

bf 0.001 40000000` (~72 MB).

Step 2

Reject non-UUID inputs with HTTP 400.

Step 3

On BF.EXISTS == 0, return 404 immediately.

On DB `None`, `SET item

{id} 'NULL' EX 60`.

Alternative 1

Guava BloomFilter / Redis Cuckoo Filter (CF.RESERVE) if deletions must be reflected immediately.

Failure Mode 1

Forgetting to call BF.ADD and redis.delete(cache_key) when creating a new item causes temporary 404s.

import json, re

UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
NULL_SENTINEL = "__NULL__"

def get_item(redis, db, item_id: str):
    if not UUID_RE.match(item_id):
        raise BadRequestError("Invalid item_id format")

    if not redis.execute_command("BF.EXISTS", "items:bf", item_id):
        return None  # 99.9% of non-existent probes stop here

    cache_key = f"item:{item_id}"
    cached = redis.get(cache_key)
    if cached is not None:
        return None if cached == NULL_SENTINEL else json.loads(cached)

    row = db.fetch_item(item_id)
    if row is None:
        redis.set(cache_key, NULL_SENTINEL, ex=60)  # Absorb 0.1% FP repeats
        return None

    redis.set(cache_key, json.dumps(row), ex=3600)
    return row
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

RedisBloom gives instant global consistency for newly created items (BF.ADD), while per-pod in-memory Bloom Filters cut Redis network RTT to zero during massive DDoS floods.

ChoiceWhat it buysWhat it costs
Trade-off 1In-pod Bloom handles >500k QPS with zero Redis traffic, but requires CDC/PubSub delta streaming to avoid false negatives on new inserts.

Evaluate RedisBloom

1 shared 72 MB filter, instant BF.ADD visibility across all 40 pods.

Evaluate In-Pod Filter

40 * 72 MB = 2.88 GB total pod RAM, zero Redis network I/O during 50k+ QPS attack, but needs 5s delta sync.

Step 3

Choose RedisBloom for most workloads; add L1 in-pod Bloom only if network bandwidth to Redis saturates.

Alternative 1

Edge CDN / Cloudflare Workers KV Bloom Filter.

Failure Mode 1

Stale in-pod Bloom Filter returns 404 for newly created items until the next sync tick.

// Java Guava L1 BloomFilter (zero network hop during DDoS)
BloomFilter<CharSequence> l1Filter = BloomFilter.create(
    Funnels.stringFunnel(StandardCharsets.UTF_8),
    40_000_000,
    0.001
);
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

On item creation, synchronously execute.

On item creation, synchronously execute BF.ADD and DEL item:{id} (to clear any prior __NULL__ sentinel) right after DB COMMIT, backed by a CDC listener.

ChoiceWhat it buysWhat it costs
Trade-off 1Minimal 0.3ms write-path overhead guarantees read-your-own-writes consistency.

Step 1

Commit new item to PostgreSQL.

Execute `BF.ADD items

bf {item_id} and DEL item:{item_id}` in a Redis pipeline.

Step 3

Also stream inserts via Debezium CDC to heal any crash between DB commit and Redis update.

Alternative 1

Use UUIDv7 and bypass Bloom check if uuid_ts > now - 30s.

Failure Mode 1

If DEL is omitted, a pre-probed __NULL__ key blocks reads of the newly created item for 60s.

def create_item(db, redis, item):
    db.insert_item(item)
    pipe = redis.pipeline()
    pipe.execute_command("BF.ADD", "items:bf", item["id"])
    pipe.delete(f"item:{item['id']}")  # Clear any lingering __NULL__
    pipe.execute()
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Layer per-IP / per-ASN rate.

Layer per-IP / per-ASN rate limiting on 404 responses at the API Gateway, and use singleflight on Redis misses so repeated probes for the same ID coalesce into 1 DB query.

ChoiceWhat it buysWhat it costs
Trade-off 1Protects origin against targeted false-positive probing without affecting legitimate users.

Step 1

Apply Nginx/Envoy sliding-window rate limits per IP/subnet.

Step 2

Penalize IPs exceeding 20 404-responses/min with a 429 Too Many Requests.

Step 3

Combine with singleflight + 60s __NULL__ caching for any key that reaches DB.

Alternative 1

WAF bot challenge / Captcha on high-404 ASNs.

Failure Mode 1

Overly aggressive global 404 limits can block legitimate corporate NAT proxies.

# Envoy / Gateway 404 abuse rate-limit policy
rate_limit:
  unit: MINUTE
  requests_per_unit: 20
  match_status: 404
  action: return_429

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (__NULL__ sentinels) for keys that pass the filter.
  • 02Trade-off tip: In-pod Bloom handles >500k QPS with zero Redis traffic, but requires CDC/PubSub delta streaming to avoid false negatives on new inserts.
  • 03Watch out for: If DEL is omitted, a pre-probed __NULL__ key blocks reads of the newly created item for 60s.
Behavioral Progression · Preventing Cache Penetration Bloom Filters

Preventing Cache Penetration Bloom Filters — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (NULL sentinels) for keys that pass the filter.

Synthesis / One connected design

Putting Preventing Cache Penetration with Bloom Filters Together

Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (`__NULL__` sentinels) for keys that pass the filter.

A candidate architecture, not a universal answer

Putting Preventing Cache Penetration with Bloom Filters Together

Step 1
Step 2
Step 3
Step 4
  1. Initialize BF.RESERVE items:bf 0.001 40000000 (~72 MB).
  2. Reject non-UUID inputs with HTTP 400.
  3. On BF.EXISTS == 0, return 404 immediately.
  4. On DB None, SET item:{id} '__NULL__' EX 60.
On item creation, synchronously execute `BF.ADD` and `DEL item:{id}` (to clear any prior `__NULL__` sentinel) right after DB `COMMIT`, backed by a CDC listener.
Interview Readiness Check

On item creation, synchronously execute BF.ADD and DEL item:{id} (to clear any prior __NULL__ sentinel) right after DB COMMIT, backed by a CDC listener.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Preventing Cache Penetration with Bloom Filters

Block non-existent key floods using a multi-layer shield: strict edge format validation, a fast in-memory/Redis Bloom Filter (or Cuckoo Filter) pre-check, and short-TTL Negative Caching (__NULL__ sentinels) for keys that pass the filter.