Integrations · Automation & scraping

Use HProxy with Scrapy.

Send every Scrapy request through an HProxy residential exit.

Scrapy is Python's dedicated scraping framework, built for crawling at volume with its own scheduler, pipelines and middleware.

Scrapy is happiest running many concurrent requests, which is exactly what trips a per-IP rate limit fastest. A rotating pool behind it is what keeps a high concurrency setting from getting the whole crawl blocked.

Setting up HProxy in Scrapy

1.Set the proxy on each request via its meta dict, or globally in a downloader middleware.
2.Put your login in the proxy URL, host and port from the dashboard.
3.Raise CONCURRENT_REQUESTS once the pool is carrying the load, not before.

# in your spider
def start_requests(self):
    yield scrapy.Request(
        "https://ipinfo.io/json",
        meta={"proxy": "http://user-country-us:pass@res.hproxy.com:8080"},
    )

Worth knowing

For a whole project, set the proxy in a custom HttpProxyMiddleware instead of per request.

Frequently asked questions

Where should the proxy go in a Scrapy project?

Per request is fine for a quick job, and for a whole project a custom middleware based on HttpProxyMiddleware is the right place. Setting it centrally means every spider inherits it and you change the endpoint in one file rather than in every request you build.

How do I rotate proxies in Scrapy?

With a gateway proxy you do not have to: the pool assigns a different exit per request behind one unchanged endpoint. If you are managing a list of individual addresses yourself, a middleware that assigns one per request is the usual approach, though it is more moving parts for the same outcome.

How does Scrapy concurrency interact with proxy limits?

Directly, and it is worth setting deliberately. Scrapy's concurrent request settings decide how much traffic you push at once, and pushing high concurrency through too few addresses trips per-address rate limiting faster than running sequentially would. Concurrency and pool size have to move together.

Should I still respect robots.txt in Scrapy?

Scrapy honours it by default and that default is worth keeping. Disallowed paths are exactly what nothing legitimate requests, so ignoring the file is one of the most reliable ways to have an address flagged, quite apart from it being poor practice.

The host, port and login shown here are the format; your exact values are on your dashboard, ready to copy.