The traceback ends with a line that tells you everything, if you know how to read it: scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy 203.0.113.7:8080 [{'status': 407, 'reason': b'Proxy Authentication Required'}]. The address is the proxy Scrapy tried. The dictionary is the proxy's answer to Scrapy's request for a tunnel. And the status inside it is the whole diagnosis, because a tunnel request has only a few ways to fail and each one has its own number.
Scrapy raises this for https:// URLs, and only for those. For a plain http:// page, Scrapy sends the request to the proxy and the proxy fetches it. For an HTTPS page, Scrapy first sends CONNECT target:443 to the proxy, asking it to open a raw connection it can then encrypt end to end. If the proxy answers anything but 200, the tunnel never opens, the target is never reached, and Scrapy has nothing to hand to the spider except this exception. That is why the error is about the proxy and not the site, why the site's anti-bot systems are not involved, and why retrying with the same proxy and the same credentials produces the same status every time.
Reading the status
| Status in the TunnelError | What the proxy said | Usual cause | Right response |
|---|---|---|---|
| 407 | Authenticate first | No credentials sent, wrong ones, or your address is not on an IP-authenticated proxy's allowlist | Fix credentials or the allowlist; do not retry |
| 403 | Tunnel forbidden | Port or destination not permitted, or a gateway refusing the country or session | Change the port, target, or proxy; do not retry |
| 400 | Malformed tunnel request | A bad proxy URL or a target with an odd hostname or port | Fix the request |
| 429 | The proxy is rate-limiting you | Too many concurrent connections through one proxy or account | Lower concurrency per proxy |
| 502 | Could not reach the target | Target unreachable from the proxy, or no exit matched the request | Retry on a different exit |
| 503 | No capacity, or no exit available | Overloaded proxy, or a gateway with nothing free for that country or session | Retry later or on another exit |
| 504 | The proxy timed out reaching the target | Slow or dropping target | Retry, with a longer timeout on the proxy side if configurable |
Two families, in other words. 4xx statuses are the proxy refusing you, and they repeat on retry, so the fix is configuration. 5xx statuses are the proxy failing to reach the world on your behalf, and they are worth retrying, ideally somewhere else.
Where Scrapy gets the proxy, and the credentials
Scrapy's HttpProxyMiddleware reads the proxy from two places: request.meta['proxy'], which a rotator or the spider sets per request, and the http_proxy and https_proxy environment variables, which apply to everything. The proxy value is a URL, and credentials belong inside it:
request.meta["proxy"] = "http://user:p%40ss@gateway.example:8080"
The middleware strips the credentials out of the URL and sends them as a Proxy-Authorization header on the request. That is the only way credentials should reach the proxy; setting the header by hand invites the trap below. Special characters in the password are URL-encoded, so p@ss is written p%40ss, or the URL parses at the wrong @ and the proxy sees a mangled username.
For a gateway that selects country or session from the username, the username carries that too, and a typo there produces a 407 or a 403 just as a wrong password does. Our Scrapy proxy guide covers the middleware order and the rotator that sets meta['proxy'] per request.
The Proxy-Authorization trap
Because the header is derived from the URL and stored on the request, a rotator that changes request.meta['proxy'] on a retry can leave the previous proxy's header in place, and the new proxy receives credentials meant for the old one. The symptom is a 407 that appears only on retried requests, or only after the pool rotates, while a fresh request through the same proxy works. Recent Scrapy releases clear the header when the proxy URL changes without credentials; on older releases, and as a habit, a rotator that changes the proxy should delete the header:
request.headers.pop(b"Proxy-Authorization", None)
request.meta["proxy"] = next_proxy_url_with_credentials
Never set Proxy-Authorization yourself for a pool whose proxies have different credentials. Put the credentials in each proxy's URL and let the middleware do the rest.
Retrying the right statuses
Scrapy's RetryMiddleware treats TunnelError as a retryable exception, so a 407 gets retried RETRY_TIMES times with the same result, which wastes attempts and hides the real problem in the log. A downloader middleware that reads the status from the exception and decides per status is a few lines:
import re
from scrapy.core.downloader.handlers.http11 import TunnelError
class TunnelErrorMiddleware:
STATUS = re.compile(r"'status': (\d{3})")
def process_exception(self, request, exception, spider):
if not isinstance(exception, TunnelError):
return None
m = self.STATUS.search(str(exception))
status = int(m.group(1)) if m else 0
proxy = request.meta.get("proxy")
if status in (407, 403, 400):
spider.logger.error("proxy %s refused the tunnel with %s; fix configuration", proxy, status)
return None # let it fail; do not burn retries on a repeatable refusal
# 5xx and 429: rotate and retry
request.headers.pop(b"Proxy-Authorization", None)
request.meta["proxy"] = spider.next_proxy() # your rotator
request.dont_filter = True
return request
Place it in DOWNLOADER_MIDDLEWARES after the proxy middleware's order so that it sees the exception, and give the spider a next_proxy() that returns a proxy URL with credentials. The effect is that a 407 fails fast and loudly, a 502 moves to another exit, and the log tells you which happened.
When it is not the proxy's fault
A 502 or 503 in the tunnel from every proxy in the pool against one target means the target is refusing or dropping connections from the proxy network as a whole, or is down. Rotation cannot fix that; a different kind of address can, if the target is dropping datacenter ranges, and our guide on how websites detect proxies covers the address side. A 502 from one proxy against everything means that proxy's upstream is broken, and it belongs out of the pool. Free proxies produce both constantly, which is why a spider on a public list spends most of its time in this exception.
For the equivalent error in a browser, where Chrome reports the same failed CONNECT as ERR_TUNNEL_CONNECTION_FAILED, our tunnel error guide shows how to read the proxy's answer with curl, which is also the quickest way to test a proxy URL before a spider uses it:
curl -v -x http://user:pass@gateway.example:8080 https://example.com/ -o /dev/null
The status after CONNECT in curl's output is the status Scrapy would put in the TunnelError.
Sort by status
Read the number in the brackets. 407 and 403 are configuration: credentials in the proxy URL, encoded, the address on the allowlist, and a port or target the proxy permits; retrying them is waste. 502, 503, and 504 are the proxy failing to reach the target: rotate and retry. Clear Proxy-Authorization when the proxy changes. And if every exit gets a 502 from one site, the site is dropping the network, not the spider. A residential pool with sticky sessions, credentials in the URL, and a middleware that sorts the statuses is the setup that keeps this exception rare; the Scrapy use-case page covers sizing it.