Integrations · Automation & scraping
Use HProxy with Scrapling.
- 5 steps
- Working code
- HTTP and SOCKS5
- Login in the proxy URL
Why a proxy
Why use a proxy with Scrapling
Scrapling is a Python scraping framework with two fetchers in one package: plain HTTP with a browser's TLS fingerprint, and a stealth browser that solves Cloudflare challenges. Since version 0.4 it also ships its own spider framework. Version 0.5 removed the older Playwright fetcher, DynamicFetcher, so code written for 0.4 that imports it has to move to StealthyFetcher.
Scrapling changes how a request looks, and the address it comes from stays yours. Sites count requests per address, so a crawl from one IP stops after a few hundred pages however good the fingerprint is. A rotating residential pool gives every request, or every session, its own address.
Setup
Setting up HProxy in Scrapling
- 1
Install the fetchers: pip install "scrapling[fetchers]", then run scrapling install once for the browsers.
- 2
Copy your proxy host, port and login from the HProxy dashboard.
- 3
Pass proxy= to a fetcher or a session. The HTTP fetcher also takes proxy_auth=(user, pass), which keeps the password out of the logs.
- 4
For several lines, build a ProxyRotator from the list and pass it as proxy_rotator= to the session.
- 5
Fetch an IP echo page once through the session and confirm the address before a real run.
Worth knowing
The browser fetchers accept http, https, socks4 and socks5 only, and SOCKS5 with a login fails in the browser. Use the HTTP line of the same proxy there.
The host, port and login shown here are the format; your exact values are on your dashboard, ready to copy.
Scrapling, with HProxy
from scrapling.fetchers import Fetcher, StealthyFetcher
# HTTP fetcher: the login stays out of the proxy URL
page = Fetcher.get(
"https://ipinfo.io/json",
proxy="http://example.hproxy.com:8080",
proxy_auth=("user-country-us", "pass"),
)
print(page.json())
# Stealth browser: same proxy, as a URL with the login inside
page = StealthyFetcher.fetch(
"https://ipinfo.io/json",
proxy="http://user-country-us:pass@example.hproxy.com:8080",
block_webrtc=True,
)Scrapling questions
Frequently asked questions
How do I rotate proxies in Scrapling?
With a rotating gateway you do not need to: one unchanged endpoint hands out a different exit per request. For a list of fixed lines, create ProxyRotator([...]) from scrapling.fetchers and pass it as proxy_rotator= to a session. It cycles in order and accepts a custom strategy function. A session takes proxy_rotator= or proxy=, never both.
Why does my Scrapling session ignore the proxy?
From version 0.4 to 0.4.8, FetcherSession(proxy=...) was silently ignored and requests left from the real IP. The fix shipped in 0.4.9. Update Scrapling, then fetch an IP echo page through the session to confirm the address it shows.
Why does my proxy password fail in Scrapling's browser fetcher?
Before version 0.5.1, the browser fetchers handed the login from a proxy URL to the browser still percent-encoded. A password written as p%40ss reached the proxy as p%40ss, and the proxy answered 407. Version 0.5.1 decodes it. On an older version, pass the proxy as a dictionary with server, username and password, which takes the login as typed.
Does Scrapling support SOCKS5 proxies?
The HTTP fetcher does. The browser fetchers accept a socks5:// line only without a login, because the browser underneath refuses SOCKS5 authentication, and they reject socks5h:// with Invalid proxy string. With a username and password, use the HTTP line of the same proxy.
Which fetcher should carry the proxy for a login flow?
A session, with a sticky line. StealthySession keeps one browser open across requests, and a login only survives when the address stays the same for its whole length. A rotating line fits page-by-page reads with Fetcher, where no state carries over.
How do I keep residential traffic low in Scrapling's browser fetchers?
Turn on disable_resources=True and block_ads=True. A browser loads images, fonts and trackers that a scraper never reads, and residential traffic is billed per gigabyte. Both options are off by default.
Ready when you are.
Your dashboard is ten seconds away. No sales call, no subscription, no minimum.