Integrations · Automation & scraping
Use HProxy with Scrapy.
Send every Scrapy request through an HProxy residential exit.
Scrapy is Python's dedicated scraping framework, built for crawling at volume with its own scheduler, pipelines and middleware.
Scrapy is happiest running many concurrent requests, which is exactly what trips a per-IP rate limit fastest. A rotating pool behind it is what keeps a high concurrency setting from getting the whole crawl blocked.
Setting up HProxy in Scrapy
- 1.Set the proxy on each request via its meta dict, or globally in a downloader middleware.
- 2.Put your login in the proxy URL, host and port from the dashboard.
- 3.Raise CONCURRENT_REQUESTS once the pool is carrying the load, not before.
# in your spider
def start_requests(self):
yield scrapy.Request(
"https://ipinfo.io/json",
meta={"proxy": "http://user-country-us:pass@res.hproxy.com:8080"},
)Worth knowing
For a whole project, set the proxy in a custom HttpProxyMiddleware instead of per request.
The host, port and login shown here are the format; your exact values are on your dashboard, ready to copy.