Design a Web Crawler
On this page
- Requirements
- Functional
- Non-functional
- Back-of-the-Envelope Estimates
- High-Level Architecture
- URL Frontier and Politeness
- URL Normalization and Dedup
- Fetch Workers and Parsing Pipeline
- Robots.txt and Ethics
- Scaling and Failure Handling
- What Breaks When…
- Global queue, no per-host limits
- Dedup only in process memory
- No normalization
- Infinite calendar / faceted navigation
- Parser runs inline in fetcher
- Common Mistakes I Made
- Treating robots.txt as optional
- Same priority for seeds and long tail
- Storing full HTML in Postgres rows
- No cap on redirects
- Key Takeaways
- Test Your Understanding
Design a Web Crawler
I wanted to index blog posts for a personal search side project, so I wrote a
script: fetch a page, regex out <a href>, fetch those pages, repeat. Within an
hour I had duplicated URLs in the queue, infinite loops through calendar widgets,
and an angry email from a host because I'd opened 200 concurrent connections
to their /tag/ pagination.
The crawler wasn't wrong about HTML — it was wrong about systems. Real crawlers ( Googlebot, Bingbot, archive.org) solve politeness, deduplication, frontier scheduling, and distributed fetch as first-class problems. This post designs a crawler that respects sites, avoids redundant work, and scales horizontally — using the same framework as the URL shortener case study.
Intended audience: developers designing search indexing, SEO tools, or data pipelines who need the crawler-shaped version of queues, dedup, and rate limits — not a BeautifulSoup tutorial.
Prerequisites:
- How to Approach Any System Design Problem
- Asynchronous Processing and Messaging — queues, workers
- Load Balancing Explained — distributing work
- Helpful: Caching Strategies (Bloom filter for dedup), Cross-Cutting Concerns (rate limiting)
Requirements
Functional
| Must have | Nice to have (v2) |
|---|---|
| Seed URLs → discover and fetch linked pages | JavaScript rendering (headless browser farm) |
| Extract links; enqueue unvisited URLs | Per-domain crawl policies UI |
| Store page content or metadata for indexer | Change detection / recrawl scheduling |
Respect robots.txt | Sitemap.xml seeding |
| Same URL not fetched twice (dedup) | Priority by PageRank / freshness |
Non-functional
| Category | Target |
|---|---|
| Scale | 1B URLs discovered; 100M pages stored |
| Politeness | Max 1–2 req/sec per host (configurable); backoff on errors |
| Freshness | Recrawl popular pages daily; long tail weekly |
| Availability | Crawler can pause/resume; no single worker is SPOF |
| Storage | Raw HTML + parsed fields; dedup set must fit in memory-efficient structure |
Out of scope v1: full-text search index (downstream), login-walled content, CAPTCHA solving.
Back-of-the-Envelope Estimates
Assume:
- 500M pages crawled over months
- Average HTML 50 KB → ~25 TB raw storage (compress ~3× → ~8 TB)
- 10,000 fetch workers (logical); 1 req/sec/host cap
If average page has 50 outbound links and 30% are new after dedup:
Steady state: millions of URLs in frontier; dedup set dominates RAM unless Bloom + disk
Dedup memory (rough):
- Naive: 500M URLs × 100 bytes = 50 GB just for URL strings in a hash set
- Bloom filter for "probably seen" + disk/KV for confirmation → much smaller RAM footprint
Queue throughput: 10k workers × ~2 sec/page (network + parse) ≈ 5k pages/sec cluster-wide if not host-limited. Per-host politeness is usually the real throttle.
High-Level Architecture
Loading diagram…
Components:
- Frontier — prioritized queues of URLs to fetch (often one logical queue per host)
- Fetcher workers — HTTP GET, timeouts, retries
- Dedup — have we scheduled or fetched this URL before?
- Parser — extract links, enqueue new URLs
- Store — blob storage (S3) or DB for HTML + metadata
- Robots cache — cached
robots.txtrules per host
Async messaging connects stages so fetch isn't blocked on parse or storage.
URL Frontier and Politeness
Frontier = the set of URLs waiting to be fetched. Naive global FIFO starves small sites and hammers big ones. Standard pattern: priority queue per host (or per domain).
Loading diagram…
Politeness rules:
- Max N concurrent requests per host (often 1)
- Minimum delay between requests to same host (e.g. 500 ms–2 s)
- Exponential backoff on 429/503
- Honor
Crawl-delayin robots.txt when present
Implementation sketch:
// Per-host token bucket or last-fetch timestamp in Redis
async function mayFetch(host) {
const key = `crawl:host:${host}`;
const last = await redis.get(key);
const delayMs = await getPolitenessDelay(host); // from config or robots
if (last && Date.now() - last < delayMs) return false;
await redis.set(key, Date.now(), { PX: delayMs });
return true;
}
This is a distributed rate limiter problem scoped per host.
URL Normalization and Dedup
Same page, many strings:
https://Example.com/path/
http://example.com/path
https://example.com/path?utm=1
https://example.com/path#section
Normalize before dedup:
- Lowercase scheme and host
- Default port removed
- Resolve
.and..in path - Strip fragment (
#...) - Policy on query params (drop tracking params like
utm_*) - Optional: trailing slash rules
Then hash normalized URL (SHA-256) for fixed-length keys.
Dedup layers:
| Layer | Purpose |
|---|---|
| In-memory / Bloom filter | Fast "definitely new" vs "maybe seen" |
| Durable set (Redis, RocksDB, DB) | Confirm and survive restarts |
| "Already fetched" store | Skip re-fetch unless recrawl policy says refresh |
Bloom filter: no false negatives for "not seen" if built correctly — safe to skip enqueue when filter says absent. False positive → one extra lookup, not duplicate crawl storm.
Fetch Workers and Parsing Pipeline
Fetcher:
- Timeouts (connect + read)
- Redirect limit (max 5)
- User-Agent identifies bot + contact URL
- Stream body to storage; don't hold 50 MB in RAM
- Content-Type filter (skip
video/*unless scope includes)
Parser:
- HTML link extraction (
<a href>,<link>for sitemaps) - Resolve relative URLs against base URL
- Filter by domain scope (same-site vs whole web)
- Enqueue normalized URLs to frontier via dedup gate
Pipeline decoupling:
fetch → raw blob to S3 → queue "page_ready" → parser workers → enqueue links
Failures in parse don't lose the raw page; replay parser from blob.
Robots.txt and Ethics
Before fetching https://example.com/page:
- Fetch/cache
https://example.com/robots.txt(TTL hours) - Parse rules for your User-Agent (or
*) - If disallowed, do not fetch — mark URL skipped
Also respect:
- Terms of service and legal constraints (not just technical robots)
- No credential stuffing or paywall bypass
- Contact site owners for high-volume crawls
Ethics isn't a nice-to-have — IP bans and legal risk are system requirements.
Scaling and Failure Handling
| Concern | Approach |
|---|---|
| Worker scale | Horizontal pods; partition frontier by host hash |
| Hot host | One viral domain doesn't block entire crawl — per-host queues |
| Duplicate enqueue storm | Dedup before frontier insert |
| Worker crash mid-fetch | At-least-once queue delivery; idempotent store by URL hash |
| Poison URL | Max retries then dead-letter; alert |
| Recrawl | Separate scheduler pushes stale URLs based on last_fetched_at |
Idempotency: storing page
content_hash or url_hash as primary key makes retries safe.
Monitor: pages/sec, queue depth per host, 429 rate, robots block rate, dedup false-positive rate, fetch latency p99.
What Breaks When…
Global queue, no per-host limits
One domain gets hammered; you get blocked; other hosts idle. Fix: per-host frontier + politeness.
Dedup only in process memory
Restart re-crawls everything. Fix: durable dedup + Bloom rebuilt from snapshot.
No normalization
/Page and /page duplicate storage and anger indexers. Fix: canonical URL rules.
Infinite calendar / faceted navigation
Crawler trapped in ?page=999999. Fix: max depth, URL pattern blocklist, duplicate content detection.
Parser runs inline in fetcher
Slow parse blocks fetch throughput. Fix: async pipeline with blob store.
Common Mistakes I Made
Treating robots.txt as optional
"We'll fix it later" → IP block later. Fetch robots first class.
Same priority for seeds and long tail
Important recrawls starved. Fix: priority tiers (freshness, PageRank proxy).
Storing full HTML in Postgres rows
DB bloat and vacuum pain. Fix: object storage for bodies, DB for metadata.
No cap on redirects
Redirect loops burn workers. Fix: redirect counter per fetch.
Key Takeaways
- Crawlers are schedulers + fetchers + dedup — correctness is not fetching HTML once; it's never wasting a host's patience twice.
- Per-host politeness (rate limits, concurrency caps) is mandatory — the frontier must be host-aware.
- Normalize URLs before dedup or you'll store and fetch the same page under dozens of keys.
- Bloom filter + durable dedup scales the "have we seen this URL?" problem beyond RAM.
- Async pipeline (fetch → store blob → parse → enqueue) decouples network from CPU-bound extraction.
- robots.txt and ToS are requirements, not polish — plan for blocks and backoff.
- Scale workers horizontally; throttle per host vertically — the bottleneck is almost always politeness, not CPU.
Test Your Understanding
Next in series: Fan-Out on Write vs Fan-Out on Read