Headers tell a server who is asking and what the answer may look like. In requests, you set them with one argument, but what arrives is not always what the code says.
We sent requests 2.34.2 to a small server of our own on 12 October 2026. It answers with the request line and every header it received, in order. That showed what the defaults are, what None really does, when a token disappears, and what a proxy sees.
The headers requests sends by itself
import requests
URL = "http://127.0.0.1:31292/echo" # our test server: it answers with every header it received
r = requests.get(URL, timeout=10)
for name, value in r.json()["headers"]:
print(f"{name}: {value}")
Host: 127.0.0.1:31292
User-Agent: python-requests/2.34.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Five headers, and the User-Agent names the library and its version, so every site can tell that a script is calling. That is why most scrapers set their own.
Set your own headers
headers = {
"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/python-requests-headers)",
"Accept-Language": "de-DE,de;q=0.9",
}
r = requests.get(URL, headers=headers, timeout=10)
print([name for name, _ in r.json()["headers"]])
['Host', 'User-Agent', 'Accept-Encoding', 'Accept', 'Connection', 'Accept-Language']
Your User-Agent replaced the default in its place, and Accept-Language came last. Header names are case-insensitive in HTTP, so user-agent also replaced the default, and it arrived written in lower case.
Headers on a Session
with requests.Session() as s:
s.headers.update({"X-Token": "from the session"})
r = s.get(URL, headers={"X-Request": "from the call", "X-Token": "the call wins"}, timeout=10)
print(dict(r.json()["headers"])["X-Token"])
the call wins
Every request on the Session carries its headers, and the call's own headers are added on top. When both set the same name, the call wins, as the requests documentation says: "The method-level parameters override session parameters".
None does not remove a default header
r1 = requests.get(URL, headers={"User-Agent": None}, timeout=10)
r2 = requests.get(URL, headers={"Accept-Encoding": None}, timeout=10)
print(dict(r1.json()["headers"])["User-Agent"], "|", dict(r2.json()["headers"])["Accept-Encoding"])
python-urllib3/2.8.0 | identity
The requests documentation says a key set to None will "automatically be omitted", and requests does leave it out. But the libraries under it fill the gap. urllib3 adds its own User-Agent when none is set, python-urllib3/2.8.0, and Python's http.client adds Accept-Encoding: identity. To send a different value, set one; to look like no library at all, None will not do it.
Tokens and redirects
We sent Authorization: Bearer test-token to an address that answered with a redirect. After a redirect to the same host, the token was sent again. After a redirect to another host name, requests dropped it. The documentation says the same: "Authorization headers will be removed if you get redirected off-host". That protects your token from a site you never meant to send it to.
Values outside ASCII, and line breaks
Header values travel as Latin-1. "Zoë" arrived intact, but a Japanese name raised UnicodeEncodeError, so encode such values first or keep them out of headers. A value with a line break in it raised InvalidHeader. That is good, because a line break would start a header of its own.
Read the response headers
r = requests.get(URL, timeout=10)
print(r.headers["content-type"], r.headers["Content-Type"])
application/json application/json
r.headers is a case-insensitive dictionary, so any spelling of the name works.
Rotating headers
A scraper that sends many requests often rotates its headers. Rotate whole sets that belong together, not the User-Agent alone:
import itertools
HEADER_SETS = [
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/143.0.0.0 Safari/537.36", "Accept-Language": "en-US,en;q=0.9"},
{"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/143.0.0.0 Safari/537.36", "Accept-Language": "en-GB,en;q=0.8"},
]
rotation = itertools.cycle(HEADER_SETS)
for page in range(1, 4):
r = requests.get(URL, params={"page": page}, headers=next(rotation), timeout=10)
print(page, dict(r.json()["headers"])["Accept-Language"])
1 en-US,en;q=0.9
2 en-GB,en;q=0.8
3 en-US,en;q=0.9
Even a full set is still not a browser. We opened the same test server in headless Chrome 143 and counted what arrived:
Chrome sent sec-ch-ua, the Sec-Fetch headers and more, and its own User-Agent said HeadlessChrome. A changed User-Agent alone leaves nine of those fourteen lines missing.
Copying a browser's headers into requests has a trap of its own. Browsers send Accept-Encoding: gzip, deflate, br. We sent that to our own site: it answered in brotli (br), and r.text came out as unreadable bytes. urllib3 only decodes br when the brotli package is installed. Leave br out, or install brotli first.
What a proxy sees of your headers
We pointed requests at our test server as if it were a proxy, with an Authorization header, and recorded what it received:
Over plain HTTP, the proxy received the whole request, every header included, the token too. Over HTTPS, it received only CONNECT example.test:443 with a Host header: everything else travels encrypted. So send tokens and cookies over HTTPS when a proxy sits in between. Our guide to proxies with Python requests covers the setup.
What this page could not check
- One version of requests and urllib3, against our own test server.
- One browser, headless Chrome 143, whose headers differ from a normal Chrome window.
- We did not test how real sites score headers, HTTP/2, or the TLS fingerprint, which a site also sees.
- We will rerun these tests by 12 January 2027.
Sources
- Requests documentation, Quickstart (custom headers): requests.readthedocs.io.
- Requests documentation, Advanced Usage (Session objects): requests.readthedocs.io.
- urllib3 2.8.0 source, connection.py and response.py: github.com.
- Python documentation, http.client: docs.python.org.
- RFC 9110 (field names) and RFC 9112 (requests to a proxy), June 2022: rfc-editor.org.
- Our own tests on our server, 12 October 2026: headers_tests.py at 00:51 UTC and the code on this page at 00:49 UTC, against our own test server; one request to hproxy.com for the br test.
