How to Use Proxies With Python Requests

Routing a Python script through a proxy is one line of code and about five things that will surprise you. This guide covers proxies with Python requests end to end: the proxies dict and why its keys are not what most people assume, authentication, reusing a connection with a Session, SOCKS5, rotating across a pool, and the timeout and error handling that separate a scraper that runs overnight from one that hangs on the first dead proxy.

We run a proxy network and a live proxy checker, so we see the same python requests proxy mistakes constantly: a run that silently does nothing because the wrong dict key was set, or one that stalls forever because no timeout was passed. Every example below is copy-paste runnable. Where one needs a live proxy, pull a fresh one from our free proxy API, which returns real, recently checked endpoints without a key.

How do you use a proxy with Python requests?

Build a dictionary mapping the schemes http and https to your proxy URL, then pass it as the proxies argument: requests.get(url, proxies=proxies). The dict keys are the scheme of the target URL, not of the proxy, and both usually point at the same proxy. Always add a timeout so a dead proxy cannot hang the call.

import requests

proxies = {
    "http": "http://203.0.113.7:8080",
    "https": "http://203.0.113.7:8080",
}
r = requests.get("https://example.com", proxies=proxies, timeout=(5, 15))

The proxies dict, and the key that trips everyone

The proxies argument is a dict, and its keys are the source of most confusion. The key is the scheme of the URL you are requesting, not the scheme of the proxy. So the https key means "route my requests to https:// URLs through this proxy." The value's own scheme (http://) describes the proxy's protocol.

A plain HTTP proxy carries your HTTPS traffic through a CONNECT tunnel, so "https": "http://..." is not a mistake, it is the normal, correct combination that looks wrong the first time you see it.

Set only the http key and every https:// request quietly bypasses the proxy and goes out on your real IP. That silent no-op is the single most common python requests proxy bug we see. Set both keys, always.

You can also target a specific host with a scheme-plus-host key, which wins over the plain scheme key:

proxies = {
    "https": "http://203.0.113.7:8080",                      # default for all https
    "https://api.example.com": "http://198.51.100.14:3128",  # override for one host
}

Environment variables. requests also honors the standard proxy environment variables (HTTP_PROXY, HTTPS_PROXY, NO_PROXY) when trust_env is on, which is the default. That is convenient until a proxy set in your shell or CI proxies you without your knowledge. Pass proxies= explicitly to override it, or set session.trust_env = False to ignore the environment entirely.

Proxy authentication

Paid proxies require a username and password. Put them in the userinfo part of the proxy URL, user:pass@host:port:

proxies = {
    "http": "http://user:[email protected]:8080",
    "https": "http://user:[email protected]:8080",
}
r = requests.get("https://example.com", proxies=proxies, timeout=(5, 15))

If the password contains characters that are special in a URL (@, :, /, #), percent-encode it first or the URL parses wrong:

from urllib.parse import quote

user = "myuser"
password = quote("p@ss:word/!", safe="")   # -> p%40ss%3Aword%2F%21
proxies = {s: f"http://{user}:{password}@203.0.113.7:8080" for s in ("http", "https")}

Wrong or missing credentials come back as HTTP 407 Proxy Authentication Required. A 407 is good news in one sense: it proves the proxy is alive and reachable, and only the credentials are wrong.

Reuse the connection with a Session

Calling requests.get fresh every time re-opens a TCP connection, and with a proxy that means re-doing the handshake to the proxy on every single call. A requests.Session keeps a connection pool alive, persists cookies, and lets you set the proxy once:

session = requests.Session()
session.proxies = {
    "http": "http://user:[email protected]:8080",
    "https": "http://user:[email protected]:8080",
}
session.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"})

r1 = session.get("https://example.com/one", timeout=(5, 15))
r2 = session.get("https://example.com/two", timeout=(5, 15))   # reuses the tunnel

For any script that makes more than a couple of requests through the same proxy, a Session is a real speed-up, because the expensive part of a proxied request is the connection setup, not the transfer.

SOCKS5 proxies

requests speaks HTTP proxies out of the box. For SOCKS you need one extra dependency:

pip install requests[socks]

That pulls in PySocks. Now use a socks5 or socks5h scheme in the proxy value:

proxies = {
    "http": "socks5h://user:[email protected]:1080",
    "https": "socks5h://user:[email protected]:1080",
}
r = requests.get("https://example.com", proxies=proxies, timeout=(5, 15))

The h matters. With socks5://, your machine resolves the hostname and hands the proxy an IP, so your local resolver sees every site you visit and any name that only resolves on the proxy's network fails. With socks5h://, the proxy does the DNS lookup. For privacy or scraping, socks5h:// is almost always the one you want. We go deep on the difference in what is a SOCKS5 proxy.

Rotating across a pool

One IP sending every request is exactly the pattern rate limiters look for. Spread the load across a list and pick a fresh exit per attempt:

import random
import requests

POOL = [
    "203.0.113.7:8080",
    "203.0.113.24:3128",
    "198.51.100.14:8080",
    "198.51.100.66:8000",
]

def get_rotating(url, tries=4):
    for _ in range(tries):
        proxy = random.choice(POOL)
        proxies = {"http": f"http://{proxy}", "https": f"http://{proxy}"}
        try:
            r = requests.get(url, proxies=proxies, timeout=(5, 15))
            r.raise_for_status()
            return r
        except requests.RequestException:
            continue   # dead or blocked exit, try the next one
    raise RuntimeError(f"all {tries} proxies failed for {url}")

html = get_rotating("https://example.com").text

Random choice is fine for most jobs. If you want strict round-robin instead, itertools.cycle(POOL) hands out proxies in order. Either way, the load-bearing part is retrying with a different proxy on failure, because on any pool some exits are always down.

You do not have to maintain that list by hand. Our free proxy API returns a fresh pool as plain text you can split straight into a list:

POOL = requests.get(
    "https://hproxy.com/api/proxy-list",
    params={"format": "txt", "protocol": "http", "recent": "true", "limit": 50},
    timeout=15,
).text.split()

For production, a rotating gateway that gives you one endpoint and a fresh residential IP per request removes list management entirely, but rotating a list by hand is the best way to understand what that gateway is doing for you.

Timeouts, retries and error handling

This is the section that decides whether an unattended run survives contact with real proxies.

Always pass a timeout. Without one, a proxy that accepts your connection and then goes silent hangs your program forever. Use the tuple form to bound connect and read separately:

# (connect timeout, read timeout) in seconds
requests.get(url, proxies=proxies, timeout=(5, 30))

The 5 caps how long to wait for the proxy to answer; the 30 caps how long to wait for data once connected. A slow or overloaded proxy trips the read timeout; an unreachable one trips the connect timeout.

Catch the exceptions that proxies actually raise. requests wraps them in a clean hierarchy, all under RequestException:

from requests.exceptions import ConnectTimeout, ReadTimeout, ProxyError, RequestException

try:
    r = requests.get(url, proxies=proxies, timeout=(5, 30))
    r.raise_for_status()
except ConnectTimeout:
    print("could not reach the proxy in time")
except ProxyError:
    print("proxy refused or broke the tunnel")   # dead proxy, bad port, refused CONNECT
except ReadTimeout:
    print("proxy connected then stalled")
except RequestException as e:
    print(f"other request failure: {e}")

ConnectTimeout means the proxy never answered, ReadTimeout means it answered then went quiet, and ProxyError is a proxy-level failure such as a refused tunnel or a broken CONNECT. Catching RequestException last sweeps up everything else, including ConnectionError and SSLError.

Automatic retries with backoff. For transient failures against a stable proxy, mount a urllib3 Retry on a Session so requests re-attempts with exponential backoff instead of giving up on the first blip:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(
    total=3,
    backoff_factor=0.5,                       # exponential: ~0.5s, 1s, 2s between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods={"GET", "HEAD"},
)
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

One caveat worth knowing: this Retry re-tries the same proxy, because the proxy is fixed on the session. It is the right tool for a stable gateway that occasionally returns a 503. To retry with a different IP, use the application-level rotation loop from the previous section. Real pipelines often use both: rotation to change exits, Retry to smooth over blips on whichever exit is in play.

Verify the exit IP

Never assume a proxy is working; confirm the target sees the proxy's address and not yours:

import requests

proxies = {"http": "http://203.0.113.7:8080", "https": "http://203.0.113.7:8080"}

real = requests.get("https://httpbin.org/ip", timeout=10).json()["origin"]
via_proxy = requests.get(
    "https://httpbin.org/ip", proxies=proxies, timeout=(5, 15)
).json()["origin"]

print("real IP: ", real)
print("proxy IP:", via_proxy)
assert real != via_proxy, "proxy is not changing your IP"

If the two IPs match, the proxy is not routing your traffic, which is usually the https key problem from earlier. It is also worth checking what headers arrive at the target, because a transparent proxy can forward your real IP in X-Forwarded-For even while origin looks changed:

h = requests.get("https://httpbin.org/headers", proxies=proxies, timeout=(5, 15)).json()
print(h["headers"].get("X-Forwarded-For"))   # your real IP here means the proxy leaks

Where to go from here

Two things separate proxy code that works in a demo from proxy code that works at 3am unattended: verifying every exit before you trust it, and pulling from a pool that is actually alive.

For the first, our proxy checker runs the whole battery in one paste (exit IP, anonymity grade, real exit geolocation, latency), which is faster than scripting those checks by hand. For the second, the free proxy API hands you recently verified endpoints with no key, ideal for testing the code in this guide before you wire in a paid pool.

From here, the cURL guide is the shell-side mirror of everything above, proxies for web scraping covers choosing the right proxy type and the request hygiene that proxies alone cannot fix, and how to avoid IP bans while scraping is the prevention checklist for when rotation is not enough.