Rate limiting
A site's cap on how many requests one source may make in a window, the most common obstacle a scraper meets.
Rate limiting is a cap on how fast one source may ask for things. Cross it and the server stops serving you, most often with a 429 Too Many Requests response, until the window resets. It exists to stop any single client from consuming capacity everyone else needs, and it is applied by servers that have no opinion about you whatsoever.
Its defining property is that it is blind to intent, and this is the part worth internalising. The counter does not know whether you are hostile or simply busy. It counts requests per source and acts on the number, which is why perfectly legitimate high-volume work trips it exactly as readily as abuse does. Being well-behaved does not exempt you from arithmetic.
The counter is almost always keyed to something identifying: an IP address, an API key, an account, or a session. Which key it uses decides what your options are, and it is worth finding out rather than assuming. If a site limits per address, spreading requests across a pool works. If it limits per API key, more addresses change nothing at all and you need a different key or a slower pace.
That is why a rotating proxy exists, and it is the honest description of what one does: it does not raise the limit, it stops you concentrating traffic against a single counter. The same total volume divided across a proxy pool leaves each address comfortably under a threshold it never approaches. Nothing was evaded; the arithmetic was rearranged.
The response you get tells you a lot, and reading it properly saves days. A 429 usually carries a Retry-After header naming exactly how long to wait, which is a site telling you its rules in plain terms. Honouring it is both the polite behaviour and the effective one, because ignoring it and retrying immediately is what escalates a temporary throttle into a longer block.
Not every limit announces itself, though, and the silent kind is the dangerous one. An anti-bot system with options will often degrade rather than refuse: slower responses, truncated result sets, stale cached data, or quietly omitted records. Your requests keep returning 200 and your dataset is wrong. Anyone doing web scraping at volume should be comparing results against a known-good baseline rather than trusting status codes.
The most underrated fix is simply going slower with variance. A fixed delay is itself a detectable pattern, and genuine variance in pacing both stays under limits and looks less mechanical. It costs time, and time is usually cheaper than the escalating cost of being throttled, challenged and eventually blocked.
Frequently asked questions
What does a 429 error mean?
Too Many Requests: you have exceeded the allowance for whatever the server counts by, and it will serve you again once the window resets. Check for a Retry-After header, which states exactly how long to wait. It is a temporary refusal rather than a ban, and treating it as one by backing off is usually all that is required.
Do more proxies always solve rate limiting?
Only when the limit is counted per IP address. If it is keyed to an API key, an account or a session cookie, adding addresses changes nothing because the counter never looked at your address. Establish what the limit is keyed to before buying capacity to solve it, or you will scale the wrong dimension.
How do I find a site's rate limit?
Sometimes it is published, in API documentation or in response headers naming the limit, the remaining allowance and the reset time. Where it is not, find it empirically and carefully: start well below any plausible threshold, increase gradually, and watch for the first 429 or the first sign of degraded content. Deliberately hammering a site to discover the limit is both rude and a good way to earn a longer block.
Is it better to rotate IPs or slow down?
Both, and they solve different halves. Rotation stops volume concentrating against one counter; slowing down with genuine variance addresses the pattern of the traffic itself, which rotation does nothing about. Rotating fast enough to keep a machine-regular cadence still looks like a machine, just one wearing many addresses.
Can rate limiting be silent?
Yes, and it is the failure most worth guarding against. Some sites degrade rather than refuse: slower responses, fewer results, stale cached values or quietly missing records, all returned with a perfectly normal 200. Nothing in your logs will flag it. The only reliable defence is periodically comparing your collected data against a request you know is clean.
Back to the full glossary.