Most IP bans start as a rate limit. A server counts your requests, refuses the ones over its limit, and a second program turns repeated refusals into a ban. The nginx web server and fail2ban, a tool that bans addresses from log files, ship exactly that pair. It even comes with a ready-made jail for nginx's refusals.
So on October 23, 2026, we rebuilt the pair on our own machine, with nginx's rate-limit rules and fail2ban's default settings. Then we sent it 60 requests in five different ways. One address at full speed was banned at request 12. Four addresses at full speed got four bans. The same four at a polite pace fetched every page, four times faster than one address could. And a rule that counted by browser header banned that polite pool anyway. This guide walks through each step, the numbers, the pace that sites publish, and where residential proxies help.
How does an IP ban work?
A server keeps a count of requests per key, usually your IP address. Over its limit, it refuses requests, often with status 429. Keep sending, and a ban rule blocks the key for a while, from seconds to a week. The ban shows as a 403 page or a refused connection.
Step one: the rate limit
nginx's rate limiter exists "to limit the request processing rate per a defined key". Its documentation names the usual key: "requests coming from a single IP address". It works like a leaky bucket. Its own example allows "not more than 1 request per second at an average, with bursts not exceeding 5 requests". Requests past the burst are refused.
One detail surprises people. nginx answers a refused request with 503 unless the site sets limit_req_status to another code. So a rate limit can look like an outage, and our 503 guide shows how to tell the two apart. Sites that pick 429 follow RFC 6585, where the status means the user "has sent too many requests in a given amount of time". The server "MAY include a Retry-After header" saying how long to wait, so many refusals arrive without one.
Step two: the ban
fail2ban watches log files and bans the addresses that keep failing. Its nginx-limit-req jail reads nginx's error log, where every refusal is one line, and every line counts as one failure. The defaults in fail2ban's own jail.conf are maxretry = 5, findtime = 10m and bantime = 10m. Five failures within ten minutes ban the address for ten minutes.
The ban itself happens below HTTP. fail2ban's iptables action rejects the address with an "icmp-port-unreachable" message before any HTTP is spoken. The scraper gets a refused connection and no status code at all. curl reports that as exit 7, the same code as a dead server (curl errors guide). A web application firewall usually shows a 403 page instead.
How long do IP bans last?
The automated ones expire. fail2ban's default ban lasts ten minutes. Its sample configuration adds a recidive jail "for more extended banning of persistent abusers": one week, for addresses banned again within a day. Stack Exchange's API drops any address making "more than 30 requests a second". The ban, its docs say, "will be on the order of 30 seconds to a few minutes typically". A ban that a site's staff set by hand has no timer at all.
What we measured
Our stub site ran on 127.0.0.1 and copied both steps. The limit followed nginx's rules at 2 requests a second with a burst of 5, refusing with 429. On top of it sat fail2ban's defaults: five refusals in ten minutes, then a ten-minute ban. Our stub answered a ban with a 403 page, because a refused connection prints nothing a reader could compare. One copy counted requests by client IP, the other by the User-Agent header.
Each run sent 60 requests from loopback addresses between 127.0.0.2 and 127.0.0.15, so the stub saw up to four different addresses at once. We ran everything in Ubuntu 24.04 and Debian under WSL and got the same counts in both. Nothing left the machine.
- Banned (403)
- Refused (429)
- Served (200)
The five runs, one by one:
- One address, as fast as possible. Six pages came through, the burst plus one. The next five were refused, and from request 12 on the address was banned. The whole run took 0.05 seconds.
- Four addresses, as fast as possible. Rotation bought 24 pages, six per address. Then each address collected its own five refusals and its own ban, so the run ended with four banned addresses instead of one.
- One address at 1.6 requests a second, just under the limit of 2. All 60 pages came back, in 36.9 seconds, with no refusal.
- Four addresses at 1.6 a second each, 6.4 a second in total. All 60 pages, in 9.2 seconds. This is what a pool is for: more polite traffic at once, with every address under the limit.
- The same polite pool, counted by User-Agent. Ten pages, five refusals, then a ban for all four addresses. The site saw one client at 6.4 requests a second, because every address sent the same header.
What a site counts you by
The address is only the default key. "By default," the AWS WAF documentation says, "a rate-based rule aggregates and rate limits requests based on the request IP address". It can also count by the JA3 or JA4 fingerprint of the TLS handshake. Cloudflare's rate rules can count by IP or by "IP with NAT support", which tells apart visitors who share one address. They can also count by a header value, a cookie, the AS number, the country, or the JA3 and JA4 fingerprints.
Rotation changes only the first of those counts:
| What the rule counts | What rotating addresses does to the count |
|---|---|
| IP address | Splits it across your addresses |
| AS number, the network | Splits it only if your addresses sit in different networks |
| A header value or a cookie | Nothing, when every address sends the same value |
| TLS fingerprint, JA3 or JA4 | Nothing, when every request comes from the same client software |
| Country | Nothing, within one country |
A scraper that rotates addresses but sends one User-Agent, one cookie and one TLS handshake still looks like one client to such a rule. Run 5 showed exactly that. Our JA3 and JA4 explainer shows how the handshake gives a client away. Sites behind Cloudflare and DataDome read it. The original JA3 method came from Salesforce.
The pace sites publish
Many public data sources state their limit in plain words. Turned into the gap between two requests, they span a factor of 600:
Two of these sources forbid what proxy rotation does. arXiv says its limits "apply to all of the machines under your control as a whole". You "should not attempt to overcome these limits by increasing the number of machines", it adds. Nominatim allows bulk jobs on "1 machine only, no distributed scripts". Against sources like these, a pool of addresses breaks the terms, and the polite answer is their limit or their bulk download.
Identifying yourself often buys more than any proxy. GitHub allows 60 requests an hour per address without a token and 5,000 with one, 83 times as many. The SEC asks bots to declare a User-Agent with a company name and a contact address. Nominatim wants a User-Agent naming the application, because "stock User-Agents as set by http libraries will not do".
Where a site publishes nothing, those numbers give a range. Start at the slow end, with one connection a second or more apart, and speed up only while every answer stays 200. Read its robots.txt too. RFC 9309 standardized the Robots Exclusion Protocol in 2022 with the User-Agent, Allow and Disallow rules. Crawl-delay is a common extension outside the standard, so treat it as the site's stated wish and follow it.
A polite fetcher, in code
This Python sketch paces requests to one site, honors Retry-After, and stops on a ban instead of retrying into it. It sends the requests through a rotating residential gateway, which spreads the pace over addresses without raising it:
import random
import time
import requests
# Residential Premium: port 9000 gives a new address per request, ports 10000 and up hold one
PROXY = "http://YOUR-LOGIN:YOUR-PASSWORD@premium.hproxy.com:9000"
MIN_GAP = 1.0 # seconds between requests to this site; use its published limit when it has one
session = requests.Session()
session.proxies = {"http": PROXY, "https": PROXY}
last = 0.0
def fetch(url, tries=5):
global last
for attempt in range(tries):
wait = last + MIN_GAP * random.uniform(1.0, 1.5) - time.monotonic()
if wait > 0:
time.sleep(wait)
last = time.monotonic()
r = session.get(url, timeout=30)
if r.status_code in (429, 503):
retry = r.headers.get("Retry-After", "")
time.sleep(int(retry) if retry.isdigit() else 5 * 2 ** attempt)
continue
if r.status_code == 403:
raise SystemExit(f"403 on {url}: stop and read the page; a ban does not end by retrying")
return r
raise SystemExit(f"{url} still refused after {tries} tries: lower the pace for the whole run")
Without a Retry-After header, the waits double with each refusal, as GitHub's docs advise. They ask for "an exponentially increasing amount of time between retries". Requests sent while rate limited, they warn, "may result in the banning of your integration". Our 503 guide measured which clients honor Retry-After on their own: curl's --retry and Python's urllib3 did, Wget did not.
Headers matter as much as pace. A site that asks bots to name themselves, as the SEC and Nominatim do, should get a named bot. Where a site sets no such rule, send a complete and consistent header set. A thin or contradictory one is a common reason for a 403.
Where residential addresses help, and where they do not
Residential addresses belong to ordinary home connections, spread across many consumer networks. That helps against two of the counts above. A rule keyed on the IP sees each address make only its share of a polite load. A rule keyed on the AS number cannot sweep the whole pool in one line, as it can when every address sits in one hosting network. Our guide to how websites detect proxies covers the other checks, from reputation lists to timing. Our measurement of 7,082 working free proxies shows how few public proxies pass them.
Some flows need one address to stay put. A login, a cart or a search with several steps follows you by cookie, and a new country between two clicks reads as fraud. Sticky ports hold one address for the whole flow and change it between flows. Rotating vs static residential proxies covers when to pin an address.
Residential addresses do not help in three cases from our runs and reading. They cannot save a scraper that is too fast, as run 2 showed. They cannot split a count kept by header, cookie or TLS fingerprint, as run 5 showed. And they should not be used against a source that forbids distributed requests, or against a ban set on purpose.
On our side, Residential Premium gives a new address per request on port 9000. Ports 10000 and up hold one address, from 3 minutes to 24 hours. It costs $1.49/GB for one GB, falling to $1.00/GB at 1,000 GB+. Test any address before a real run. Our proxy checker shows whether it is alive and where it exits, and the free proxy list is a cheap place to rehearse the pipeline.
When you get banned anyway
Stop the run first. Coming back the moment a ban lifts only collects the next one. The week-long recidive jail in fail2ban exists for addresses banned again within a day. Then read what came back:
- 429, with or without Retry-After: the limit, not yet a ban. Wait as long as Retry-After says, then lower the pace for the whole run. The 429 guide goes deeper.
- 503 right after a burst: possibly nginx's default answer to a rate limit, the same fix.
- A 403 page after a run of refusals: the ban. Wait it out, and treat it as the site's answer.
- Refused connections from one site only: possibly a firewall ban such as fail2ban's, ten minutes by default.
If your address was blocked before you sent anything, the cause is elsewhere: a shared address, a server range, or a reputation list. Why is my IP blocked walks through those. Where your target is a platform we already cover, our scraper APIs return parsed results from one call. Pacing and rotation then happen on our side. For everything else, proxies for web scraping pulls the whole workflow together.
Sources
- nginx: ngx_http_limit_req_module, read October 23, 2026: the leaky bucket per key, burst and nodelay, and "limit_req_status 503" as the default.
- fail2ban: config/jail.conf, read October 23, 2026: bantime 10m, findtime 10m and maxretry 5 by default, the nginx-limit-req jail and the recidive jail. Also its nginx-limit-req filter and its iptables action with "blocktype = REJECT --reject-with icmp-port-unreachable".
- RFC 6585, Additional HTTP Status Codes, April 2012: 429 Too Many Requests and the optional Retry-After.
- Cloudflare: rate limiting rule parameters, read October 23, 2026: the characteristics a rule can count by.
- AWS WAF: rate-based rule aggregation options and high-level settings, read October 23, 2026: IP as the default key, JA3 and JA4 keys, evaluation windows of 60 to 600 seconds and a lowest limit of 10.
- GitHub Docs: rate limits for the REST API, read October 23, 2026: 60 requests an hour per IP without a token, 5,000 with one, and the advice on retries.
- Stack Exchange API: throttles, read October 23, 2026.
- OSMF: Nominatim usage policy, arXiv: terms of use for the APIs and SEC: accessing EDGAR data, read October 23, 2026.
- RFC 9309: Robots Exclusion Protocol, September 2022.
- Our own run on October 23, 2026: a stub site on 127.0.0.1 copying nginx's limit_req and fail2ban's defaults, sent 60 requests five ways with Python 3.12.3 in Ubuntu 24.04 and Python 3.13.5 in Debian under WSL. Nothing outside the machine was contacted.
