Design a distributed web crawler indexing 10B web pages/month (~4,000 pages/sec sustained, 10K+ peak) with strict per-host politeness, `robots.txt` compliance, < 2% duplicate content storage, and resilience against infinite-URL crawler traps.
From patterns to decisions.
Shard the Mercator URL Frontier by `hash(host)`, pair with async DNS caching, RocksDB/Bloom URL deduplication, 64-bit SimHash content dedup, and URL-depth/pattern trap guards.
Scale Envelope & Latency Budget.
Design a distributed web crawler indexing 10B web pages/month (~4,000 pages/sec sustained, 10K+ peak) with strict per-host politeness, robots.txt compliance, < 2% duplicate content storage, and resilience against infinite-URL crawler traps.
Scale Envelope & Latency Budget
1.Stage Objective: Size throughput, storage, and p99 latency targets.
2.Level (Foundation): Design a distributed web crawler indexing 10B web pages/month (~4,000 pages/sec sustained, 10K+ peak) with strict per-host politeness, `robo
3.Operational Invariant: Size throughput, storage, and p99 latency targets.
Caption: Size throughput, storage, and p99 latency targets.
Core Architecture & Data Flow.
Sharding the Mercator Frontier by hash(hostname) guarantees every domain's Back Queue, robots.txt cache (24h TTL), and politeness timer reside on a single owner shard. Fetchers use non-blocking async DNS resolvers (avoiding glibc getaddrinfo thread blocking) and stream parsed pages through a 64-bit SimHash (Hamming distance <= 3) near-duplicate detector before indexing.
Core Architecture & Data Flow
1.Stage Objective: Assemble the primary data path and storage layout.
2.Level (Foundation): Sharding the Mercator Frontier by `hash(hostname)` guarantees every domain's Back Queue, `robots.txt` cache (24h TTL), and politeness timer
3.Operational Invariant: Assemble the primary data path and storage layout.
Caption: Assemble the primary data path and storage layout.
Crawler Trap & Robots Guard.
from urllib.parse import urlparse
def is_safe_to_enqueue(url: str, max_depth: int = 8, max_len: int = 512) -> bool:
if not url or len(url) > max_len:
return False
parsed = urlparse(url)
if parsed.scheme not in ('http', 'https') or not parsed.netloc:
return False
segments = [s for s in parsed.path.split('/') if s]
if len(segments) > max_depth or len(set(segments)) < len(segments) - 2:
return False # Reject repeating path-loop traps (/a/b/a/b/a/b)
return TrueCrawler Trap & Robots Guard
1.Stage Objective: Reject infinite URL traps and disallowed paths before enqueueing discovered links.
2.Level (Intermediate): from urllib.parse import urlparse
3.Operational Invariant: Reject infinite URL traps and disallowed paths before enqueueing discovered links.
Caption: Reject infinite URL traps and disallowed paths before enqueueing discovered links.
Sharding & Hot-Spot Mitigation.
Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
Sharding & Hot-Spot Mitigation
1.Stage Objective: Prevent single-partition bottlenecks at peak load.
2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
3.Operational Invariant: Prevent single-partition bottlenecks at peak load.
Caption: Prevent single-partition bottlenecks at peak load.
Failure Modes & Graceful Degradation.
Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
Failure Modes & Graceful Degradation
1.Stage Objective: Keep critical paths healthy during partial outages.
2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
3.Operational Invariant: Keep critical paths healthy during partial outages.
Caption: Keep critical paths healthy during partial outages.
Consistency, Idempotency & Recovery.
Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
Consistency, Idempotency & Recovery
1.Stage Objective: Guarantee correctness across retries and node crashes.
2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
3.Operational Invariant: Guarantee correctness across retries and node crashes.
Caption: Guarantee correctness across retries and node crashes.
Telemetry, Circuit Breakers & Failure Modes.
Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.
Production Guardrails
1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.
2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de
3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Practice: Make the trade-off yourself.
Why must the crawler frontier shard by `hash(hostname)` rather than `hash(full_url)`?