System Design HLD Example: Web Crawler

Scenario context

A distributed web crawler must balance global throughput with per-domain politeness. The architectural crux is the URL Frontier, which manages priority and rate-limiting across a distributed fetcher pool. By combining Bloom Filters for URL deduplication and SimHash for content near-duplicate detection, the system can crawl billions of pages while maintaining a minimal storage footprint and respecting the…

This walkthrough focuses on How Googlebot, Bingbot, and CommonCrawl Deploy Crawler Architecture at Scale.

Before making a design choice, clarify:

• Googlebot indexes over 130 trillion web pages and is the world's most sophisticated crawler.
• Google's published patents and engineering papers describe a URL Frontier nearly identical to the two-queue architecture above, augmented with a "freshness" layer that dynamically adjusts re-crawl frequency based on obs…
• News sites are re-crawled hourly, product listing pages daily, and static documentation monthly.

Concepts in play: system-design-hld-architectures, stack-and-queue-data-structures, caching, hash-tables-and-bloom-filters

Stage 1 — URL Deduplication Strategy Decision Framework

Observable outcome: You can explain how “URL Deduplication Strategy Decision Framework” changes the architecture or implementation path.

Stage 2 — Presenting the Web Crawler HLD in a System Design Interview

• Start every web crawler answer by drawing the core loop on the whiteboard: Frontier → Fetcher → Content Store → Link Extractor → Frontier.
• This loop is the entire system.
• Every other component — Bloom Filter, SimHash Engine, Robots.txt cache, DNS cache — is a correctness or performance guard on one of the loop's stages.
• Interviewers reliably probe three specific areas in every web crawler question.

Observable outcome: Interviewers reliably probe three specific areas in every web crawler question.

Stage 3 — What Building a Web Crawler Teaches About Adversarial Distributed Systems

• A web crawler is a distributed system that must cooperate with millions of external systems — the web's servers — over which it has zero control.
• This forces architectural decisions that most internal distributed systems never encounter: adversarial environments, unpredictable response times, resource-constrained counterparties, and legal constraints from robots.…
• Every design choice in a web crawler is a negotiation between the crawler's appetite for throughput and the web's need for stability.
• The politeness constraint is not merely an ethical guideline — it is a self-preservation mechanism.

Observable outcome: The politeness constraint is not merely an ethical guideline — it is a self-preservation mechanism.

End-to-end outcome

Googlebot indexes over 130 trillion web pages and is the world's most sophisticated crawler.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

The Core Trade-offs in Web Crawler Architecture

• Detection: Breadth-First Traversal: A depth-first crawler follows the first link on every page as deep as possible before backtracking.
• User or operator impact: Breadth-First Traversal: A depth-first crawler follows the first link on every page as deep as possible before backtracking.