Free Proxies for Scrapy: What Actually Works

Free proxies for Scrapy are genuinely useful for one job: proving your middleware, rotation, and auth logic work before you spend a cent. For the real crawl they fail, and the reason is Scrapy's own speed. The framework fires 16 requests in parallel by default, and pointed at a free pool where most exits are dead, most of those requests hit a corpse, the retry middleware thrashes, and your effective crawl rate collapses while the logs fill with connection errors. Wire your plumbing against a free list, watch it egress from different IPs, then move the real run to a pool that will not die under that concurrency.

We build and run proxy pools, so we see Scrapy jobs constantly, from hobby spiders to production crawlers working through retail catalogs. This is the free-first version: what a free list is good for, the exact middleware and settings to plug one in, why an unattended crawl chews through free exits, and how to verify rotation is actually happening. If you want the paid-tier decisions (which proxy type per target, how many IPs, sticky versus rotating), those live in proxies for Scrapy.

What free proxies are good for in Scrapy

The valuable thing a free list gives you is a real, moving target to build against. Rotation logic that looks right on paper behaves differently once some exits are dead, some are slow, and some leak, and a free pool has all three. Testing against one forces your middleware to handle failure before a paid pool ever sees it.

Our free proxy list re-checks and refreshes every entry every few minutes across HTTP, HTTPS, SOCKS4, and SOCKS5, which is exactly what you want to confirm that requests egress from different IPs, that your Proxy-Authorization handling is correct, and that a dead exit triggers a retry on a fresh one instead of killing the request. Prove all of that on free, then swap in the paid gateway and the code does not change.

A free list also flushes out the mistakes that only surface under load. Entries that need a login prove your embedded user:pass@ is reaching the Proxy-Authorization header. Slow entries prove your timeout is actually bounding the request. A pool that keeps dying proves your process_exception retries on a new exit instead of dropping the request on the floor. None of that shows up when you test against a single hand-picked proxy that happens to stay alive, which is why the churn of a free pool is the feature here, not the bug.

Wiring a free list into a spider

Scrapy reads the proxy for a request from request.meta["proxy"], and the built-in HttpProxyMiddleware (on by default) applies it and turns any embedded user:pass@ into the Proxy-Authorization header. To rotate a raw free list, add a small downloader middleware that sets that meta key and, crucially, drops an exit that fails so the request retries on a live one:

# middlewares.py
import random

class FreeListProxyMiddleware:
    """Rotate a raw free-proxy list and forget exits that fail."""

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_POOL"))

    def __init__(self, pool):
        self.pool = list(pool)

    def process_request(self, request, spider):
        if "proxy" not in request.meta and self.pool:
            request.meta["proxy"] = f"http://{random.choice(self.pool)}"

    def process_exception(self, request, exception, spider):
        # a connection-level failure means a dead exit: drop it and retry fresh
        dead = request.meta.get("proxy", "").replace("http://", "")
        if dead in self.pool:
            self.pool.remove(dead)
        retry = request.copy()
        retry.meta.pop("proxy", None)   # process_request picks a new one
        retry.dont_filter = True
        return retry

The process_exception half is the part that matters for free lists. It fires on connection errors and timeouts (a dead or refused exit), not on HTTP status codes, which the built-in retry middleware already handles. Register it ahead of the built-in proxy middleware, and turn down the settings that assume a healthy pool:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.FreeListProxyMiddleware": 350,  # before HttpProxyMiddleware (750)
}

# dead free exits must fail fast, not hold a slot for the 180s default
DOWNLOAD_TIMEOUT = 15

# a free pool cannot take Scrapy's default 16-wide firehose
CONCURRENT_REQUESTS = 8
AUTOTHROTTLE_ENABLED = True

# retry the codes that mean "slow down" or "that exit is burned"
RETRY_ENABLED = True
RETRY_TIMES = 4
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 408, 522, 524]

One protocol detail decides which entries you can feed this. Scrapy's built-in proxy support covers HTTP and HTTPS proxies only, and it tunnels HTTPS targets through them with CONNECT. SOCKS proxies need an extra download handler, so for a plain spider pull the HTTP or HTTPS rows from a free list rather than the SOCKS4 and SOCKS5 ones. Our list carries all four for other tools, but only the HTTP entries drop straight into the middleware above.

You do not have to maintain the list by hand. Our free proxy API returns a fresh pool as plain text you can load straight into the setting at startup:

# settings.py
import requests

PROXY_POOL = requests.get(
    "https://hproxy.com/api/proxy-list",
    params={"format": "txt", "protocol": "http", "recent": "true", "limit": 50},
    timeout=15,
).text.split()

For anything past a test, the scrapy-rotating-proxies package wraps this pattern with the ban tracking a raw list needs (it reads ROTATING_PROXY_LIST, retires exits that keep failing, and re-checks them later), which is one more reason a real crawl belongs on a gateway rather than a hand-rolled loop.

Why a free pool collapses an unattended crawl

The failure is not subtle, and it is worth seeing why so you do not fight it. Scrapy is built to run many requests at once, and a free pool is built to disappear.

The default DOWNLOAD_TIMEOUT is 180 seconds, so a request that draws a dead exit can hold one of your concurrency slots for three full minutes before it gives up. Multiply that across a pool where only a small fraction is alive and the crawl spends most of its time blocked. Lowering the timeout to 15 seconds and dropping concurrency (both shown above) makes the thrash survivable for a test, but neither makes a dead pool fast. Free exits also carry no anonymity guarantee, so on top of the churn some are transparent and defeat the point entirely. The safety picture is in are free proxies safe.

Verify rotation is actually happening

Never assume the middleware works because the spider runs. Point it at an IP echo endpoint and log what each request sees:

import scrapy

class IpCheckSpider(scrapy.Spider):
    name = "ipcheck"
    start_urls = ["https://httpbin.org/ip"] * 20

    def parse(self, response):
        self.logger.info("exit IP: %s", response.json()["origin"])

If the logged addresses vary across requests and none is your own, rotation is working. Before you blame the middleware for a bad run, confirm a single entry is genuinely alive and elite in our free proxy checker, and read how to check if a proxy is working for what "working" actually means. A middleware that is correct will still look broken if every entry you fed it was already dead.

When to switch to a gateway

Free proxies end where the real crawl begins. For a defended target, the answer is rotating residential through a single gateway: you point every request at one endpoint and the provider rotates the exit per request, so the whole FreeListProxyMiddleware disappears and the ban tracking lives on their side. That maps onto Scrapy perfectly, because Scrapy's per-request model is exactly what a rotating gateway is built for.

Our residential pool starts at $0.44 per GB, pay as you go, with no KYC and a balance that does not expire, so a spider you pause between runs never burns prepaid credit while it sits idle. If your automation is a browser rather than a crawler, the same free-versus-paid split applies in free proxies for Selenium.

Wire your plumbing against our free proxy list until the requests visibly leave from different IPs, pull the test pool from the free proxy API, and vet any single entry in the proxy checker. Then move the real crawl to residential at $0.44/GB, keep your throttle and retry settings conservative, and fix the fingerprint before you blame the IP (the parts a proxy will never fix, from Scrapy's TLS handshake to its missing JavaScript, are in how to use proxies with Scrapy). Do that and the crawl stops fighting proxies and goes back to being a data problem.