Integrations · Automation & scraping
Use HProxy with Selenium.
Drive a real browser through an authenticated HProxy proxy with Selenium.
Selenium automates a real browser, the tool of choice when a site only yields to something that renders JavaScript and behaves like a person.
A headful browser is convincing until every session comes from the same IP. A residential exit per session is what lets Selenium scale past a handful of runs before a site starts challenging it.
Setting up HProxy in Selenium
- 1.Install Selenium Wire (pip install selenium-wire), which adds the authenticated-proxy support vanilla Selenium lacks.
- 2.Put your host, port and login in the seleniumwire_options proxy config.
- 3.Launch the driver as usual; every request the browser makes now exits through the proxy.
from seleniumwire import webdriver
opts = {"proxy": {
"http": "http://user-country-us:pass@res.hproxy.com:8080",
"https": "http://user-country-us:pass@res.hproxy.com:8080",
}}
driver = webdriver.Chrome(seleniumwire_options=opts)
driver.get("https://ipinfo.io/json")Worth knowing
Vanilla Selenium cannot pass a proxy username and password; Selenium Wire is the standard way around it.
The host, port and login shown here are the format; your exact values are on your dashboard, ready to copy.