Glossary · Scraping & automation
Web crawler
A program that follows links from page to page to discover and fetch content, the traversal engine underneath large-scale scraping.
A crawler discovers; a scraper extracts. A crawler starts from some seed URLs and follows the links it finds, page to page, mapping out a site or a slice of the web. Scraping is what you do with the pages it fetches. The two usually run together, which is why the words blur, but the crawler is specifically the part that decides where to go next.
Crawling at any scale runs into the same wall as scraping: many requests from one address, discovered fast, get throttled or blocked. A crawler that follows thousands of links needs those requests spread across a pool, or it strangles itself on the first busy domain.
Well-behaved crawlers also respect the rules a site publishes for them, which is a practical matter as much as a courtesy: ignoring them is one of the behaviours that gets an address flagged.
Back to the full glossary.