How to Use Proxies With Scrapy (Rotating, Retries, Auth)

Scrapy ships with proxy support already switched on, which is why the first surprise is that it does not rotate. The built-in middleware applies exactly one proxy, and the moment you scrape at any real scale you need a pool, retries that move to a fresh exit, and authentication that does not send last request's password to this request's proxy. This guide covers proxies in Scrapy end to end: the built-in HttpProxyMiddleware, per-request proxies through meta, a rotating middleware with priorities and retries, the auth header trap that quietly breaks rotation, and the add-ons that handle the fingerprint a proxy cannot.

We run a proxy network and a live proxy checker, so we see the same Scrapy pipelines fail the same way: a rotation loop that reuses the one dead exit on every retry, or a Proxy-Authorization header that sticks to the wrong proxy after a swap. Every example below is runnable. Where one needs a live pool, pull fresh endpoints from our free proxy API, which returns real, recently checked proxies as plain text with no key.

How do you set a proxy in Scrapy?

Set request.meta['proxy'] to a proxy URL on the request, or export the http_proxy and https_proxy environment variables. The built-in HttpProxyMiddleware is on by default and applies whichever it finds, with the meta value taking precedence. Put user:pass in the URL and it becomes a Proxy-Authorization header automatically.

import scrapy

class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": "http://user:[email protected]:8080"},
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info("exit IP: %s", response.json()["origin"])

The built-in HttpProxyMiddleware

HttpProxyMiddleware is enabled in every project without any configuration. It reads a proxy from two places, in this order of precedence:

• request.meta['proxy'], set per request to a value like http://host:port or http://user:pass@host:port. This is the one you control from a spider or a middleware.
• Environment variables http_proxy, https_proxy and no_proxy, honored the same way Python's standard library does. Handy for a global default, easy to forget you set in a shell or a CI job.

What it does not do is rotate. Point it at one proxy and every request in the crawl goes out through that single exit, which is fine for a small job on a stable gateway and useless the moment you need a pool. Rotation is your code's job, and the built-in middleware is the piece that turns whatever proxy you choose into an outgoing connection.

One detail matters for everything below: HttpProxyMiddleware runs at order 750 in the downloader middleware chain. Anything that assigns request.meta['proxy'] has to run before it, so your rotating middleware needs an order below 750.

Proxy authentication and the stale-header trap

When HttpProxyMiddleware sees credentials in a proxy URL, it does not pass them along in the URL. It strips the user:pass, base64-encodes it into a Proxy-Authorization: Basic ... header, and remembers which proxy that header belongs to by caching the credential-free URL in request.meta['_auth_proxy']. On the next request it compares the new proxy against that cached value, and if the proxy changed it deletes the stale header. That bookkeeping is the whole reason rotation with per-proxy credentials works at all.

It also tells you the one thing not to do. If you set the Proxy-Authorization header by hand and then rotate request.meta['proxy'] to a different exit, nothing clears your manual header, so this request's proxy receives the previous proxy's credentials and answers 407. The fix is to never hand-set the header while rotating. Put the full user:pass@host:port in meta['proxy'] every time and let the middleware manage the header for you.

# GOOD: credentials ride in the meta URL, so HttpProxyMiddleware tracks and refreshes the header
request.meta["proxy"] = "http://user:[email protected]:8080"

# TRAP: a hand-set header does not follow a proxy change and gets sent to the wrong exit
request.headers["Proxy-Authorization"] = b"Basic dXNlcjpwYXNz"   # do not pair this with rotation

A rotating-proxy middleware

To rotate, write a downloader middleware that assigns a proxy in process_request and moves off a dead one in process_exception. Drop this in your project's middlewares.py.

import random
import logging

logger = logging.getLogger(__name__)


class RotatingProxyMiddleware:
    """Pick an exit per request, and rotate away from one that fails."""

    def __init__(self, proxies):
        if not proxies:
            raise ValueError("PROXY_POOL is empty. Set it in settings.py.")
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_POOL"))

    def _pick(self, exclude=None):
        pool = [p for p in self.proxies if p != exclude] or self.proxies
        return random.choice(pool)

    def process_request(self, request, spider):
        # Assign an exit on the first attempt, and force a new one on every retry.
        # A retried request keeps its old meta['proxy'], so without the retry_times
        # check the crawl would hammer the same dead exit on each retry.
        if "proxy" not in request.meta or request.meta.get("retry_times"):
            request.meta["proxy"] = self._pick(exclude=request.meta.get("proxy"))

    def process_exception(self, request, exception, spider):
        # Transport-level failure (refused connection, timeout, reset): the exit is
        # dead, so reschedule on a different proxy, ahead of the normal queue.
        dead = request.meta.get("proxy")
        logger.warning("proxy %s failed (%s), rotating", dead, exception.__class__.__name__)
        retry = request.copy()
        retry.meta["proxy"] = self._pick(exclude=dead)
        retry.dont_filter = True
        retry.priority = request.priority + 1
        return retry

Wire it in below HttpProxyMiddleware so the exit it picks is still turned into a connection:

# settings.py
PROXY_POOL = [
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
]

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 610,   # before HttpProxyMiddleware (750)
}

Three parts of this are doing real work. The retry_times check in process_request is the one most rotation snippets get wrong: Scrapy's retry logic copies the request and keeps its meta, so a naive if "proxy" not in meta guard would reuse the exit that just failed on every single retry. The exclude argument makes sure a rotation never lands back on the exit that just died. And retry.priority = request.priority + 1 gives the rescheduled request a nudge up the queue, so a failed fetch is retried on a fresh IP soon instead of drifting to the back behind thousands of pending requests.

Let RetryMiddleware handle bad responses

Your rotator handles transport failures, a dead socket, a timeout, a reset. Bad HTTP responses, a 503 or a 429, are a different case, and Scrapy already retries those. RetryMiddleware is enabled by default at order 550, and by default it retries the codes in RETRY_HTTP_CODES ([500, 502, 503, 504, 522, 524, 408, 429]) up to RETRY_TIMES (2) times. The important behavior is what happens on a retry: the request is copied and rescheduled, so it walks the middleware chain again from the top, which means your process_request runs again and picks a new exit. Retries and rotation compose for free.

For scraping you usually want to lean on it harder:

# settings.py
RETRY_TIMES = 5
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429, 403]
RETRY_PRIORITY_ADJUST = 0   # keep retries at the same priority instead of the default -1

Adding 403 is a judgment call worth understanding. A 403 on a proxied scrape usually means that exit is blocked, so retrying only helps because the retry lands on a different IP, which your rotator guarantees. Without rotation, retrying a 403 on the same exit just burns your retry budget on a wall.

Per-request proxies with meta

Not every request wants the pool. Sometimes one spider, or one type of request, needs a specific exit, for example a login that must stay on a single sticky IP while the rest of the crawl rotates. Because meta['proxy'] wins over everything and your rotator only assigns when nothing is set, pinning one request is a one-liner:

yield scrapy.Request(
    "https://example.com/login",
    meta={"proxy": "http://user:[email protected]:8080"},
    callback=self.after_login,
)

The same rule from our DataDome guide applies here: hold one exit for the length of a stateful flow, and let the pool rotate everything else. Pinning through meta is how you carve a sticky session out of an otherwise rotating crawl.

Batteries included: scrapy-rotating-proxies

If you would rather not maintain the middleware, scrapy-rotating-proxies does the same job with liveness tracking and ban detection built in. It probes which exits are alive, backs off from dead ones, and can pause a spider when the whole pool is banned.

# settings.py
ROTATING_PROXY_LIST = [
    "user:[email protected]:8080",
    "user:[email protected]:8080",
]
DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}

You do not have to keep that list by hand. Our free proxy API returns a fresh pool as plain text you can load straight into the setting:

import requests

ROTATING_PROXY_LIST = requests.get(
    "https://hproxy.com/api/proxy-list",
    params={"format": "txt", "protocol": "http", "recent": "true", "limit": 50},
    timeout=15,
).text.split()

A static list is the right tool for a fixed set of exits. For a maintained residential pool, a single rotating gateway is simpler still: one entry in the list, and the pool changes the IP for you on each connection.

When the IP is not enough

A proxy fixes your IP reputation. It does nothing about Scrapy's fingerprint, and on a defended target that gap is where you get caught. Scrapy's default HTTP client has a Twisted TLS handshake and a header order that no real browser produces, so a web application firewall (a WAF, the filter in front of the site) can flag it on a spotless residential IP. Two add-ons close that gap, and both take a proxy.

scrapy-impersonate swaps Scrapy's download handler for one built on curl_cffi, which replays a real browser's TLS and HTTP/2 fingerprint. You keep writing a normal spider, and each request carries a Chrome-shaped handshake instead of a Python-shaped one.

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# per request: choose a browser target the package supports, and keep your proxy
yield scrapy.Request(
    "https://example.com",
    meta={"impersonate": "chrome124", "proxy": "http://user:[email protected]:8080"},
)

scrapy-playwright goes further and renders pages in a real browser, which produces a genuine fingerprint as a side effect and runs the JavaScript that some sites require. The proxy goes in PLAYWRIGHT_LAUNCH_OPTIONS, and it follows the exact credentials rule from our Playwright proxy guide: the username and password are separate fields, never in the server URL.

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {"server": "http://gateway.example.com:8080", "username": "user", "password": "pass"},
}

Which one you reach for depends on the wall. The full breakdown of what these vendors actually read lives in how websites detect proxies, with vendor-specific playbooks in scraping past Cloudflare and getting around DataDome.

Verify the exit IP

Before you scale a crawl, confirm the pool is actually routing traffic and not silently falling back to your own address. A one-off spider against an echo endpoint tells you in seconds:

import scrapy

class VerifySpider(scrapy.Spider):
    name = "verify"
    custom_settings = {"PROXY_POOL": ["http://user:[email protected]:8080",
                                      "http://user:[email protected]:8080"]}

    def start_requests(self):
        for _ in range(5):
            yield scrapy.Request("https://httpbin.org/ip", dont_filter=True, callback=self.parse)

    def parse(self, response):
        self.logger.info("exit IP: %s via %s",
                         response.json()["origin"], response.meta.get("proxy"))

Five requests through the rotator should print several different exit IPs. If they all show your own address, the rotator is not ordered before HttpProxyMiddleware, or PROXY_POOL is empty. It is also worth checking each exit's real geography and anonymity grade rather than only that the IP changed. Our proxy checker runs exit IP, anonymity, real geolocation and latency in one paste.

The honest bottom line

Scrapy gives you proxy plumbing for free and leaves the two hard parts to you: rotating a pool without reusing dead exits, and keeping proxy credentials attached to the right proxy through retries. Get the middleware order right (your rotator below 750), let the credentials ride in the meta URL so the header never goes stale, and let RetryMiddleware re-enter the chain so retries land on fresh IPs. That covers the IP layer completely, and the IP layer is where most blocks start.

What it does not cover is the fingerprint, which is why scrapy-impersonate and scrapy-playwright exist. For the exit itself, our residential proxies rotate from a single gateway at $0.65/GB, pay as you go, with a balance that never expires, so the pool management above collapses to one line in PROXY_POOL. Prove one exit end to end with the proxy checker before you point a real crawl at a real target, and if the block survives a clean IP, the avoiding IP bans while scraping guide is the request-hygiene checklist for everything a proxy was never able to fix.

Sources

• Scrapy: HttpProxyMiddleware: the request.meta['proxy'] key, the http_proxy / https_proxy / no_proxy environment variables, and Basic proxy authentication.
• Scrapy: RetryMiddleware and RETRY_* settings: the default order 550, RETRY_TIMES, RETRY_HTTP_CODES and RETRY_PRIORITY_ADJUST.
• Scrapy: DOWNLOADER_MIDDLEWARES_BASE: the default order numbers, including HttpProxyMiddleware at 750.
• scrapy-rotating-proxies on PyPI: the ROTATING_PROXY_LIST setting and ban detection.
• scrapy-impersonate and scrapy-playwright: the download handlers that add a browser TLS fingerprint and full browser rendering.