Integrations · Automation & scraping

Use HProxy with Python Requests.

Route the requests library through an HProxy residential IP in three lines.

Requests is the most widely used HTTP library in Python, the default choice for scripts, scrapers and API clients.

A script that hits a site repeatedly from one address is the pattern rate limits are built to catch. Routing Requests through a rotating residential pool spreads the traffic so each address stays under the radar.

Setting up HProxy in Python Requests

1.Copy your proxy host, port and login from the HProxy dashboard.
2.Build a proxies dict pointing both http and https at the same proxy URL.
3.Pass it to every request, or set it once on a session.

import requests

proxies = {
  "http":  "http://user-country-us:pass@res.hproxy.com:8080",
  "https": "http://user-country-us:pass@res.hproxy.com:8080",
}
r = requests.get("https://ipinfo.io/json", proxies=proxies)
print(r.json())

Worth knowing

The username carries targeting: user-country-de sends you out of Germany, add -city-frankfurt to go finer.

Frequently asked questions

Why does requests still show my real IP?

Usually because the proxies dict only covered one scheme. Set both the http and https keys, since a request to an HTTPS URL uses the https entry and an http-only dict silently sends that request direct. Verify by fetching an IP-echo endpoint through the session rather than assuming.

How do I use one proxy for every request?

Set it once on a Session object and use that session throughout, rather than passing the argument to each call. It is less code, it keeps connections alive between requests, and it removes the class of bug where one call silently goes direct because the argument was forgotten.

Why does my scraper work in a browser but fail from requests?

Most often the TLS fingerprint. The handshake a Python HTTP client sends differs from a browser's regardless of how carefully you copy the headers, and detection systems read it before your request is decrypted. Copying the user agent does not change the handshake underneath it.

Does requests resolve DNS through the proxy?

For an HTTP proxy given a full URL, the proxy generally resolves the hostname at its end. For SOCKS it depends on the scheme you use: the variant that resolves remotely keeps the lookup on the proxy side, while the other resolves locally and produces a DNS leak that shows your own resolver every destination.

The host, port and login shown here are the format; your exact values are on your dashboard, ready to copy.