Retry logic
How a scraper handles failed requests, which done badly wastes bandwidth and escalates blocks, and done well quietly rescues most transient failures.
Retry logic is what your scraper does when a request fails, and it is the difference between a pipeline that survives a flaky network and one that turns a temporary hiccup into a bill and a block. Failures are inevitable at scale, a dropped connection, a slow origin, a momentary rate limit, and most of them are transient, so retrying is right. Retrying badly is one of the most common self-inflicted wounds in scraping.
The core discipline is backoff: waiting longer between each attempt rather than hammering immediately. Exponential backoff, doubling the delay each time, with a little randomness added so many workers do not retry in lockstep, spreads the load and gives a struggling target room to recover. Retrying instantly does the opposite, piling more requests onto whatever was already failing, which is exactly how a soft rate limit becomes a hard block. The failure was telling you to slow down, and an immediate retry answers by speeding up.
Reading the failure is what separates a smart retry from a dumb one, and different failures deserve different responses. A 429 with a Retry-After header is a precise instruction: wait that long, no guessing. A 503 is worth a backed-off retry. A connection timeout may just need a different exit. But a 403, a CAPTCHA, or degraded content is not a transient error, it is a decision about you, and retrying it unchanged, same address, same fingerprint, just spends money confirming the block. Those call for changing something, a fresh address or a different approach, not for trying again.
There is a cost dimension that is easy to forget on per-gigabyte billing. A request that failed after transferring most of its response still cost you that traffic, so an aggressive retry policy against a flaky target can multiply a bandwidth bill while collecting nothing extra. A cap on attempts, backoff between them, and a rule that stops retrying non-transient failures is not just politeness; it is the difference between a retry policy that rescues requests and one that quietly doubles your costs.
Frequently asked questions
What is exponential backoff?
A retry strategy where the wait between attempts grows each time, typically doubling, usually with a little randomness added so many workers do not retry in unison. It spreads load and gives a struggling target room to recover, which is why it succeeds where instant retries fail. It is the standard sensible default for handling transient request failures.
Should I retry every failed request?
No, and this is the common mistake. Retry transient failures, timeouts, 503s, a 429 after its Retry-After delay, because those often succeed on a second try. Do not retry a 403, a CAPTCHA or degraded content unchanged: those are decisions about you, not glitches, so retrying with the same address and fingerprint just pays to confirm the block. Change something or stop.
Can bad retry logic increase my proxy costs?
Significantly, on per-gigabyte billing. A request that failed after transferring most of its response already cost you that bandwidth, so aggressive retries against a flaky target multiply the bill while collecting nothing. Cap the attempts, back off between them, and stop retrying non-transient failures, which turns retries from a cost multiplier into a genuine rescue.
Back to the full glossary.