Proxies for Scrapy: The Right Type, Setup, and Avoiding Bans

Proxies for Scrapy do one job, and they do it well: they spread your spider's requests across many IP addresses so the target site never sees a single address firing dozens of requests a second. Point the right proxy type at Scrapy's concurrency (rotating residential for defended sites, cheap datacenter for easy ones) and the crawl that used to stall on 429s and CAPTCHAs keeps running instead.

We run a proxy network, so we see Scrapy jobs constantly, from hobby spiders pulling a few hundred pages to production crawlers working through retail catalogs. This is the practical version of how proxies fit into Scrapy specifically: why the framework's own speed is what gets it blocked, which proxy type matches which target, the honest free-versus-paid picture, the exact settings and middleware to wire it up, and the fingerprint problem that proxies alone will never solve. If you want the general mechanics first, our web scraping guide covers the fundamentals this builds on.

Why Scrapy needs proxies

Scrapy is fast by design, and that speed is the tell. Out of the box it runs 16 concurrent requests (CONCURRENT_REQUESTS) and up to 8 per domain (CONCURRENT_REQUESTS_PER_DOMAIN), all from your one IP. No human browser behaves like that, so a site watching request rates flags the address quickly and the defenses escalate the usual way: rate limiting first (429 Too Many Requests), then temporary IP blocks, then a hard ban, sometimes with CAPTCHAs and junk data mixed in. Scrapy even ships with 429 in its default RETRY_HTTP_CODES, which tells you how routine this collision is.

Proxies fix the IP half of that problem by giving each request a different exit address, so no single IP piles up enough activity to trip a limit. What they do not fix is behavior, and Scrapy has behavioral tells beyond raw speed (its default user-agent, its TLS handshake, the fact that it never runs JavaScript). We will get to those, because pretending a proxy makes them disappear is how people end up blocked behind expensive residential IPs and blaming the proxy.

Which proxy type fits your target

There is no single best proxy for Scrapy. The right type is a function of how hard the target defends itself, and the money-saving rule is to use the cheapest tier the site tolerates and escalate only when block rates prove you must.

Datacenter proxies come from hosting providers: fast, cheap, plentiful, and honest about what they are. A site can tell the IP belongs to a datacenter, so anti-bot systems distrust them by default. They are correct for open data, APIs, and small sites that do not scrutinize IP reputation, and Scrapy's concurrency makes them scream on those targets for very little money.

Rotating residential proxies sit on real home connections, drawn from a large pool through a single gateway that changes the exit per request. To a website they look like ordinary consumers, so they clear the reputation checks that reject datacenter ranges. This is the default for defended targets (major retail, search, classifieds, travel), and it maps onto Scrapy perfectly: you point every request at one gateway endpoint and the provider does the rotation, so you never manage a proxy list at all. If you are fuzzy on how these work, we broke it down in what a residential proxy is and compared the two families in datacenter vs residential.

Static residential and ISP proxies are residential-looking but stable and fast. Their Scrapy niche is authenticated crawling: any spider that logs in and has to stay logged in, where rotating the IP mid-session would snap the login. You pin one exit per session instead of rotating per request.

Mobile proxies come from cellular carriers, which share one IP across many real subscribers, so sites are extremely reluctant to block them. That makes them the heavyweight for the most bot-hostile targets, at the highest price. Overkill for ordinary Scrapy work, occasionally the only thing that clears the worst offenders.

One protocol note specific to Scrapy: its built-in proxy support is for HTTP and HTTPS proxies (it tunnels HTTPS targets through them with CONNECT). SOCKS proxies need an extra handler, so for native Scrapy work you want the HTTP/HTTPS entries from a list, even though our pool also carries SOCKS4 and SOCKS5 for other tools.

The honest free-versus-paid reality

For learning Scrapy's proxy plumbing, free proxies are genuinely fine. For production crawling, they are not, and the reason is Scrapy's own concurrency. Most free proxies are datacenter IPs that die within minutes, and only a small fraction of any public list works at once. Now point a framework that fires 16 requests in parallel at a pool like that: most of those requests hit a dead exit, the retry middleware thrashes, and your effective crawl rate collapses while the logs fill with connection errors. The engineering time lost babysitting a free pool almost always costs more than a proxy plan would have.

That said, the free tier has a real job. Our free proxy list re-checks and refreshes every few minutes across 100+ countries and HTTP, HTTPS, SOCKS4 and SOCKS5, which is exactly what you want to prove your middleware, auth and rotation logic work before spending a cent. Pull a handful, wire them into the spider, watch the requests egress from different IPs, then swap in a paid gateway for the real run. You can sanity-check any single proxy first with our free checker, and how to check if a proxy is working walks through what "working" actually means. For the deeper safety picture, are free proxies safe is the honest answer.

When you move to paid, our residential pool starts at $0.99/GB, pay-as-you-go, no KYC, and the balance does not expire, so a spider you pause between runs never burns prepaid credit while it sits idle.

How to set up proxies in Scrapy

Scrapy reads the proxy for a request from request.meta["proxy"], and the built-in HttpProxyMiddleware (enabled by default) applies it and turns any embedded user:pass@ into the Proxy-Authorization header. Everything below is just a different way to set that one meta key.

The one-liner (single rotating gateway). Scrapy honors the standard proxy environment variables, so for a rotating residential gateway you can skip code entirely:

export http_proxy="http://USER:[email protected]:PORT"
export https_proxy="http://USER:[email protected]:PORT"

Every request now leaves through the gateway, which rotates the exit IP for you. This is the whole setup for most residential crawls.

Per request, in the spider. When only some requests need the proxy, set it on the Request:

def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(
            url,
            meta={"proxy": "http://USER:[email protected]:PORT"},
        )

A middleware, applied to every request. The clean, project-wide version. This also carries a fixed gateway, but as a middleware it covers requests you did not author (pagination, followed links):

# middlewares.py
class ProxyMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings["PROXY_ENDPOINT"])

    def __init__(self, endpoint):
        self.endpoint = endpoint

    def process_request(self, request, spider):
        request.meta.setdefault("proxy", self.endpoint)

# settings.py
PROXY_ENDPOINT = "http://USER:[email protected]:PORT"
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
}

The 350 matters: it runs before the built-in HttpProxyMiddleware at 750, so by the time the built-in one looks for meta["proxy"], yours has already set it. setdefault means any per-request override still wins.

Rotating from an explicit list (the free-list case). With a gateway you never need this, because the provider rotates for you. When you hold a raw list instead (say, exits pulled from our free list), rotate per request yourself:

import random

class RotatingProxyMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_POOL"))

    def __init__(self, pool):
        self.pool = pool

    def process_request(self, request, spider):
        request.meta["proxy"] = random.choice(self.pool)

This is deliberately minimal. A raw list also needs ban tracking (drop an exit that keeps failing and retry on a fresh one), which is exactly what the scrapy-rotating-proxies package adds if you would rather not build it. It is one more reason a rotating gateway is less work: the ban logic lives on the provider's side.

The settings that actually prevent bans

Wiring the proxy in is half the job. The other half is Scrapy's throttle and retry settings, because a proxy pool paired with reckless pacing just distributes your bad behavior across more IPs. Sensible values for a defended target:

# settings.py

# Stop announcing yourself as Scrapy
USER_AGENT = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
              "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36")

# Pace like a visitor, and let Scrapy self-tune to the site's latency
DOWNLOAD_DELAY = 1.0
RANDOMIZE_DOWNLOAD_DELAY = True          # on by default, keep it
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

# Retry the codes that mean "slow down" or "that exit is burned"
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]

# Keep cookies within the crawl so you look like a returning human
COOKIES_ENABLED = True

Two things people miss. First, Scrapy's default USER_AGENT is literally Scrapy/VERSION (+https://scrapy.org), which announces the bot before the proxy gets a chance to help. Change it, and ideally rotate it (the scrapy-fake-useragent package does this per request). Second, the scrapy startproject template sets ROBOTSTXT_OBEY = True. That is an ethical default, not a bug, so decide it deliberately for your target rather than flipping it off out of habit. The complete prevention checklist lives in avoiding IP bans while scraping.

On sticky versus rotating, match rotation to state, not to a default. Stateless page pulls (most Scrapy crawls) want a fresh IP per request, which a rotating gateway gives you automatically. Anything that holds state (a login, a paginated search that sets cookies, a multi-step flow) wants one sticky exit held for that session, so the site sees a coherent visit instead of one that teleports between countries mid-task. As for how many IPs, size it from your per-domain concurrency and the site's per-IP tolerance, not a round number, and remember a rotating gateway removes the counting entirely by drawing from a deep pool.

The part proxies can't fix: Scrapy's fingerprint

Here is where good proxies get wasted. A spotless residential IP still gets blocked if the request behind it screams Scrapy, and the framework has two fingerprints that give it away.

The first is its TLS handshake. Scrapy negotiates TLS through its own stack, not a browser's, so its JA3/JA4 signature does not match Chrome or Firefox. Anti-bot systems that fingerprint TLS can flag the traffic as automated no matter how clean the exit IP is. The fix is to impersonate a real browser's handshake, which the scrapy-impersonate download handler does (it is built on curl_cffi and mimics browser TLS).

The second is JavaScript, or rather the total absence of it. Scrapy fetches HTML and never executes a line of JS, so any site that mounts a managed challenge (Cloudflare's interstitial, DataDome, and the like) sees a client that cannot solve it and holds the door shut. The answer there is to render with a real browser: scrapy-playwright plugs Playwright in as a download handler, so the challenge runs in an actual browser engine, behind your residential proxy. That combination (residential IP plus real browser) is what clears JS walls, and it is the same detection layer we cover in how websites detect proxies.

The mental model is simple: the proxy makes your IP believable, and your headers, TLS and rendering make your behavior believable. Scrapy needs both, and no proxy tier substitutes for the second half.

Start free, scale by evidence

Wire your proxy plumbing against our free proxy list until the requests visibly leave from different IPs and your retry and rotation logic behave, then move the real crawl to a pool that will not die under Scrapy's concurrency. For defended targets that means rotating residential through a single gateway, which starts at $0.99/GB pay-as-you-go with no KYC and a balance that does not expire. Point the cheapest tier the target tolerates at Scrapy's speed, keep your throttle and retry settings honest, and fix the fingerprint before you blame the IP. Do that and the crawl stops fighting proxies and goes back to being a data problem, which is the one you actually wanted to solve.