Web crawler
A program that follows links from page to page to discover and fetch content, the traversal engine underneath large-scale scraping.
A crawler discovers; a scraper extracts. A crawler starts from some seed URLs and follows the links it finds, page to page, mapping out a site or a slice of the web. Scraping is what you do with the pages it fetches. The two usually run together, which is why the words blur, but the crawler is specifically the part that decides where to go next.
Crawling at any scale runs into the same wall as scraping: many requests from one address, discovered fast, get throttled or blocked. A crawler that follows thousands of links needs those requests spread across a pool, or it strangles itself on the first busy domain.
Well-behaved crawlers also respect the rules a site publishes for them, which is a practical matter as much as a courtesy: ignoring them is one of the behaviours that gets an address flagged.
The hard problems in crawling are not fetching pages, they are deciding what to fetch and knowing when to stop. A naive crawler that follows every link it sees will revisit the same content through different URLs, wander into infinite spaces such as calendars that generate a new page for every future date, follow filter and sort parameters that multiply one listing into thousands of near-identical URLs, and drift off the site entirely through outbound links.
So the machinery that matters is bookkeeping. A normalised record of what has already been seen, so a URL with reordered parameters is recognised as the same page. Explicit scope rules about which domains and paths are in bounds. A depth limit. And politeness: a delay between requests to one host, and a cap on how many you make of it at once, because a crawler that hammers a single domain is indistinguishable from an attack regardless of intent.
Honeypots are the specific trap crawlers walk into, and they exploit exactly the thoroughness that makes a crawler good. A link hidden with CSS or a path disallowed in robots.txt is something no human ever requests, so anything that requests it has identified itself. Following every link you find is the behaviour being watched for, which is why rendering the page and following only what is genuinely reachable beats parsing raw markup and following everything in it.
The distinction from a scraper is worth keeping straight because it changes what you optimise. A crawler's job is coverage: finding everything without duplicating or looping. A scraper's job is extraction: getting the fields out of a page reliably as layouts change. Most projects need both, and they fail in completely different ways.
Frequently asked questions
What is the difference between a crawler and a scraper?
A crawler discovers and a scraper extracts. The crawler decides which URLs to fetch next by following links and managing what it has already seen; the scraper pulls the fields you want out of whatever was fetched. Most projects run both, and they fail differently: crawlers fail by looping or missing coverage, scrapers fail when a layout changes.
How do I stop my crawler getting stuck in loops?
Normalise URLs before recording them, so reordered query parameters or trailing slashes do not read as new pages, and check every candidate against what you have already seen. Add a depth limit and explicit scope rules for which domains and paths are in bounds. Watch for infinite spaces such as calendars and faceted filters, which generate unlimited valid URLs from finite content.
How fast should a crawler go?
Slower than it can. Delay between requests to the same host and cap concurrency per domain, because a crawler saturating one site is indistinguishable from an attack whatever your intent. If robots.txt specifies a crawl delay, honour it. Spreading load across many hosts rather than exhausting one is both faster overall and far less likely to get you blocked.
Why did my crawler get blocked when it was being polite?
Politeness in pacing does not protect you from tripping a honeypot. Following every link in the markup, including ones hidden with CSS or disallowed in robots.txt, is exactly the behaviour that identifies automation, and the request rate was never the issue. Follow what is genuinely reachable on a rendered page rather than everything the HTML contains.
Back to the full glossary.