How to Use Proxies With Scrapy (Rotating, Retries, Auth)

Scrapy takes a proxy per request, through the request's meta or the environment, and one built-in middleware applies it. It does not rotate proxies, and the download handler you pick decides whether HTTPS and SOCKS5 proxies work at all. This guide follows Scrapy 2.19.0, released on September 10, 2026, and its documentation, source and release notes, all read on October 13, 2026. The code comes from those docs, and none of it was run for this page.

Our page on proxies for Scrapy covers the basics and the settings that set a crawl's pace. This one goes deeper into four things: what happens to credentials, which handler carries which proxy, rotation and retries, and the errors a failing proxy produces.

How do you set a proxy in Scrapy?

The built-in HttpProxyMiddleware "sets the HTTP proxy to use for requests, by setting the proxy meta value". You can set that value per request, "to a value like" these two from the docs.

yield scrapy.Request(url, meta={"proxy": "http://some_proxy_server:port"})
yield scrapy.Request(url, meta={"proxy": "http://username:password@some_proxy_server:port"})

Or set a proxy for the whole process. "Like the Python standard library module urllib.request", the middleware obeys http_proxy, https_proxy and no_proxy.

export http_proxy="http://some_proxy_server:port"
export https_proxy="http://some_proxy_server:port"

The meta value "will take precedence over" the environment variables, and with it Scrapy will "also ignore no_proxy". The source adds one more case. A meta proxy of None turns the proxy off for that request, even when the environment sets one. The middleware sits at position 750 in DOWNLOADER_MIDDLEWARES_BASE, and it is on by default; HTTPPROXY_ENABLED switches it off.

Meta belongs to one request. A request your callback yields gets its own meta, empty unless you fill it, so it leaves without the proxy unless you set the key again or a middleware does. Pass the proxy key on its own. Scrapy's docs call copying all the metadata of a previous request into a follow-up request "a bad practice", since Scrapy's own components keep their state there. Their example is retry_times, which "can lower the amount of retries allowed" for the new request.

What Scrapy does with credentials

When the proxy URL carries a login, the middleware strips it out and sends it as a Proxy-Authorization: Basic header. Two details in the source matter. The username and password are percent-decoded first, so a password with @, : or / in it must be percent-encoded in the URL. And the bytes are encoded with HTTPPROXY_AUTH_ENCODING, whose default is "latin-1".

The rules for that header changed in 2022, after a security bug. Scrapy 2.6.2, released on July 25, 2022, explains it plainly. "These third-party proxy-rotation downloader middlewares could change the" proxy of a request and leave the old header behind, "causing the credentials of one proxy to be sent" to another. Since then the middleware follows three rules.

1. With credentials in the meta URL, the header is always "updated to feature those credentials".
2. With a meta URL without credentials, the header is removed unless "it was originally defined for the same proxy URL".
3. With no proxy in meta, or None, the header is removed.

Scrapy 2.7.1, on November 2, 2022, relaxed the rule a little. The header "can again be set explicitly", but only "for as long as that proxy URL remains the same". So a header you set by hand fits a single fixed proxy. Rotate, and the header is dropped. A next proxy that needs a login then answers with 407 Proxy Authentication Required. Put the credentials into every proxy URL you set. The middleware keeps track of them in a meta key called _auth_proxy. The docs warn that such underscore keys "are internal, and may change or disappear at any time".

Which download handler carries which proxy

Scrapy's download handlers do the actual connecting, and the meta key "needs to be implemented inside the download handler". Scrapy 2.19 has four handlers for web pages, and they differ for proxies.

HTTP11 is the default for http and https pages. Proxy URLs usually start with http://, the docs say, and that handler "supports HTTPS proxies only for HTTP destinations". Since 2.16.0 it raises an exception when an https:// page meets an https:// proxy; before, it quietly "tried to connect to the proxy via HTTP". So write the proxy as http:// even for HTTPS pages. The H2 handler, for HTTP/2, has no proxy support.

HttpxDownloadHandler arrived in 2.15.0, gained SOCKS5 in 2.17.0 and moved to the httpx2 library in 2.18.0. It needs the httpx extra, installed with pip install 'scrapy[httpx]', and this setting from the docs.

DOWNLOAD_HANDLERS = {
    "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
    "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
}

With it, a socks5://username:password@some_proxy_server:port URL works in meta. Two warnings come with it. The docs call it "experimental, and not yet recommended for production environments". And it "creates a separate connection pool for each proxy URL", which "may lead to higher resource usage when using proxy rotation". The aiohttp handler, new in 2.19.0, takes HTTP proxies, no SOCKS, and no HTTPS proxy for HTTPS pages on Python older than 3.11.

Scrapy can also run without a Twisted reactor, an experimental mode since 2.15.0. There the default for http and https is the aiohttp handler since 2.19.0, and before it was the httpx handler. So in that mode a SOCKS5 proxy needs the httpx handler set by hand, as above.

SOCKS took a long road. The request in Scrapy's issue 747, "Support for socks5 proxy", was opened on June 14, 2014. It closed on June 8, 2026, when the pull request that added SOCKS to the httpx handler was merged.

How to rotate proxies

None of Scrapy's default middlewares rotates proxies. There are two clean ways to spread a crawl over several addresses that you are allowed to use.

The first is a rotating gateway: one proxy URL, with the outgoing address changing behind it. Scrapy sees a single fixed proxy, so the credentials rule above never comes into play.

The second is your own downloader middleware. Its process_request() writes a proxy URL from your list into request.meta['proxy']. Scrapy calls process_request() "in increasing middleware order", so register yours below HttpProxyMiddleware's 750. The docs show the registration with this example.

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CustomDownloaderMiddleware": 543,
}

Remember that a retry is a copy. The retry code builds the new request with request.copy(), and the docs note that copies "shallow-copy request metadata", proxy included. A middleware that only fills an empty meta['proxy'] therefore sends every retry back to the proxy that just failed. Change the proxy on retried requests too; the meta key retry_times tells your middleware a request is a retry.

Ready-made rotation packages exist, and the best known, scrapy-rotating-proxies, last shipped version 0.6.2 on May 25, 2019. Rotation also changes nothing about the site's rules. Its robots.txt still applies, which Scrapy respects when ROBOTSTXT_OBEY is on, as it is in projects made with scrapy startproject. And DOWNLOAD_DELAY exists, in the docs' words, "to avoid hitting servers too hard".

Retries when a proxy fails

RetryMiddleware sits at 550 and is on by default. The docs list these defaults.

RETRY_TIMES = 2
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]

RETRY_TIMES is the "Maximum number of times to retry, in addition to the first download". The meta key max_retry_times overrides it for one request. Errors are a separate list, RETRY_EXCEPTIONS. It holds connection refusals, timeouts and lost connections, and also TunnelError, the error for a proxy that will not open a tunnel. So a dead or refusing proxy is retried twice before Scrapy logs "Gave up retrying".

Two status codes deserve care. A 407 from the proxy means its login is wrong or missing, and retrying the same request will not fix it. A 403 is the site's own refusal: RFC 9110 says the server understood "the request but refuses to fulfill it". Respect that answer, and leave both codes out of RETRY_HTTP_CODES.

Reading the errors of a failing proxy

How a proxy's refusal shows up depends on the page's scheme.

• https:// pages. The default handler first asks the proxy for a tunnel with an HTTP CONNECT request, which carries the Proxy-Authorization header. Any answer other than 200 raises TunnelError, "An HTTP CONNECT tunnel could not be established by the proxy". The message reads "Could not open CONNECT tunnel with proxy" followed by the proxy's host, port, status and reason.
• http:// pages. The request goes straight to the proxy, so the proxy's own answer, such as a 407, comes back as the response.
• Timeouts and refusals. A proxy that does not answer ends in a timeout or a refused connection, both in RETRY_EXCEPTIONS.

A 407 means what RFC 9110 says: the client needs to "authenticate itself in order to use a proxy for this request". Check the credentials in the proxy URL, their encoding, and the rule from 2.6.2 above. Our guide to 407 Proxy Authentication Required covers the login side. For a proxy that does not answer at all, see unable to connect to the proxy server.

Proxies with scrapy-playwright

scrapy-playwright hands pages to a real browser, and it takes proxies its own way. "Proxies are supported at the Browser level by specifying the" proxy key in PLAYWRIGHT_LAUNCH_OPTIONS, its README says, with the login in separate fields.

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": "http://myproxy.com:3128",
        "username": "user",
        "password": "pass",
    },
}

"Proxies can also be set at the context level with the" PLAYWRIGHT_CONTEXTS setting, one proxy per browser context. There is no proxy per request. A proxy in the "Request meta key is not supported", the README says.

Check the exit before a long crawl

Before a long crawl, send one request through the proxy to a page that shows your IP address. Compare it with the address you see without the proxy. Scrapy's log shows the proxy's errors as described above. Our proxy checker tests a proxy from your own connection. Our free proxy API gives "The whole live pool in one call, no key", which is fine for a quick test and too unstable for a real crawl.

Where HProxy fits

A rotating gateway keeps Scrapy's side simple: one proxy URL with the login in it, and the address changes behind it. Our residential proxies work that way, with a fresh IP per request, and start at $0.44 per GB. Premium adds city and ISP targeting and keeps one address for up to 24 hours from $1.00 per GB, which suits a crawl that has to stay on one exit. Datacenter IPv4 proxies from $1.58 per IP speak HTTP, HTTPS and SOCKS5, the last one for the httpx handler. Our API docs cover the proxy endpoints.

What we could not find out

• How the httpx handler behaves under heavy rotation. The docs warn about one connection pool per proxy URL; we found no numbers, and we ran no crawl.
• Which SOCKS versions the httpx handler takes besides SOCKS5. The docs name SOCKS5 only.
• Whether _auth_proxy will stay. The docs call underscore meta keys internal.
• When the experimental handlers become recommended. The docs give no date.

Sources

Read on October 13, 2026.

• Scrapy documentation, Downloader Middleware: HttpProxyMiddleware, RetryMiddleware and their settings.
• Scrapy documentation, Settings: DOWNLOADER_MIDDLEWARES_BASE, DOWNLOAD_HANDLERS_BASE, ROBOTSTXT_OBEY, DOWNLOAD_DELAY.
• Scrapy documentation, Download handlers, the FAQ and Requests and Responses.
• Scrapy source on GitHub: scrapy/downloadermiddlewares/httpproxy.py, retry.py and core/downloader/handlers/http11.py.
• Scrapy release notes, docs/news.rst and docs/news/2.x.rst; release dates from PyPI; issue 747.
• scrapy-playwright README, Proxy support.
• RFC 9110, HTTP Semantics, section 15.5.8.
• Google Keyword Planner volumes, and this page's Search Console totals from July 15 to October 12, 2026.