Glossary · Scraping & automation
Rate limiting
A site's cap on how many requests one source may make in a window, the most common obstacle a scraper meets.
Rate limiting is a site drawing a line at how fast a single source may ask for things. Exceed the allowance, so many requests per minute from one address or key, and the server starts refusing, usually with a 429 response, until you slow down. It is the internet's basic defence against being overwhelmed, malice or not.
It is deliberately blind to intent. The limit does not care whether you are abusive or simply busy, it counts requests per source, which is why legitimate high-volume work runs into it just as readily as anything else.
The structural answer is to reduce requests per source, and that is exactly what a rotating pool does: spread the same total volume across many addresses and no single one crosses the line. Rate limiting is, more than anything, the reason rotating proxies exist.
Back to the full glossary.