Skip to main content
← PROBLEMS / SYSTEM-DESIGN
An end-to-end domain learning journey

Design a Distributed Web Crawler
& Indexing Frontier (Googlebot Scale).

Shard the Mercator URL Frontier by hash(host), pair with async DNS caching, RocksDB/Bloom URL deduplication, 64-bit SimHash content dedup, and URL-depth/pattern trap guards.
Start the journey7 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a distributed web crawler indexing 10B web pages/month (~4,000 pages/sec sustained, 10K+ peak) with strict per-host politeness, `robots.txt` compliance, < 2% duplicate content storage, and resilience against infinite-URL crawler traps.

From patterns to decisions.

Shard the Mercator URL Frontier by `hash(host)`, pair with async DNS caching, RocksDB/Bloom URL deduplication, 64-bit SimHash content dedup, and URL-depth/pattern trap guards.

Full walkthrough · 7 stages
01
STAGE 01 · CORE INTUITIONfoundation

Scale Envelope & Latency Budget.

Design a distributed web crawler indexing 10B web pages/month (~4,000 pages/sec sustained, 10K+ peak) with strict per-host politeness, robots.txt compliance, < 2% duplicate content storage, and resilience against infinite-URL crawler traps.

STAGE 01 · CORE INTUITION

Scale Envelope & Latency Budget

Scale EnvelopeIngress / BaselineLatency BudgetScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Size throughput, storage, and p99 latency targets.

2.Level (Foundation): Design a distributed web crawler indexing 10B web pages/month (~4,000 pages/sec sustained, 10K+ peak) with strict per-host politeness, `robo

3.Operational Invariant: Size throughput, storage, and p99 latency targets.

Caption: Size throughput, storage, and p99 latency targets.

02
STAGE 02 · CORE MECHANICSfoundation

Core Architecture & Data Flow.

Sharding the Mercator Frontier by hash(hostname) guarantees every domain's Back Queue, robots.txt cache (24h TTL), and politeness timer reside on a single owner shard. Fetchers use non-blocking async DNS resolvers (avoiding glibc getaddrinfo thread blocking) and stream parsed pages through a 64-bit SimHash (Hamming distance <= 3) near-duplicate detector before indexing.

STAGE 02 · CORE MECHANICS

Core Architecture & Data Flow

Core ArchitectureInput StateData FlowCore MechanismValidation GateInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Assemble the primary data path and storage layout.

2.Level (Foundation): Sharding the Mercator Frontier by `hash(hostname)` guarantees every domain's Back Queue, `robots.txt` cache (24h TTL), and politeness timer

3.Operational Invariant: Assemble the primary data path and storage layout.

Caption: Assemble the primary data path and storage layout.

03
STAGE 03 · SAFE IMPLEMENTATIONintermediate

Crawler Trap & Robots Guard.

from urllib.parse import urlparse

def is_safe_to_enqueue(url: str, max_depth: int = 8, max_len: int = 512) -> bool:
    if not url or len(url) > max_len:
        return False
    parsed = urlparse(url)
    if parsed.scheme not in ('http', 'https') or not parsed.netloc:
        return False
    segments = [s for s in parsed.path.split('/') if s]
    if len(segments) > max_depth or len(set(segments)) < len(segments) - 2:
        return False  # Reject repeating path-loop traps (/a/b/a/b/a/b)
    return True
STAGE 03 · SAFE IMPLEMENTATION

Crawler Trap & Robots Guard

Crawler TrapValidated InputRobots GuardAtomic StepValidation GateError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Reject infinite URL traps and disallowed paths before enqueueing discovered links.

2.Level (Intermediate): from urllib.parse import urlparse

3.Operational Invariant: Reject infinite URL traps and disallowed paths before enqueueing discovered links.

Caption: Reject infinite URL traps and disallowed paths before enqueueing discovered links.

04
STAGE 04 · SCALING & TRADE-OFFSintermediate

Sharding & Hot-Spot Mitigation.

Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

STAGE 04 · SCALING & TRADE-OFFS

Sharding & Hot-Spot Mitigation

ShardingHot PathHot-Spot MitigationPartition / IndexValidation GateLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Prevent single-partition bottlenecks at peak load.

2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

3.Operational Invariant: Prevent single-partition bottlenecks at peak load.

Caption: Prevent single-partition bottlenecks at peak load.

05
STAGE 05 · FAULT ISOLATIONadvanced

Failure Modes & Graceful Degradation.

Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

STAGE 05 · FAULT ISOLATION

Failure Modes & Graceful Degradation

Failure ModesFault TriggerGraceful DegradationIsolation BoundaryValidation GateFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Keep critical paths healthy during partial outages.

2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

3.Operational Invariant: Keep critical paths healthy during partial outages.

Caption: Keep critical paths healthy during partial outages.

06
STAGE 06 · RECOVERY & CONSISTENCYadvanced

Consistency, Idempotency & Recovery.

Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

STAGE 06 · RECOVERY & CONSISTENCY

Consistency, Idempotency & Recovery

ConsistencyIdempotencyCheckpoint / LogRecoveryReplay / SyncValidation GateConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Guarantee correctness across retries and node crashes.

2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

3.Operational Invariant: Guarantee correctness across retries and node crashes.

Caption: Guarantee correctness across retries and node crashes.

07
STAGE 07 · OBSERVABILITY & GUARDRAILSadvanced

Telemetry, Circuit Breakers & Failure Modes.

Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.

STAGE 07 · OBSERVABILITY & GUARDRAILS

Production Guardrails

ProductionGuardrailsTelemetry SignalCore EngineSLO ThresholdGuard & VerifyCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.

2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de

3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

Why must the crawler frontier shard by `hash(hostname)` rather than `hash(full_url)`?

Recommended Learning Flow