User agent
The string a client sends to identify its browser and operating system, one of the simplest signals a site checks against your IP.
A user agent is a line of text sent with every request in which your client states what it is. A browser's version is a long, cluttered string naming a rendering engine, a platform and several compatibility tokens that exist only because of decades of sites sniffing for them. A scripting library's version is usually short and honest, something like python-requests or curl followed by a version number.
Sites read it for entirely ordinary reasons most of the time: choosing a mobile layout, warning an outdated browser, serving a format the client supports. It became a signal for proxy detection by accident, because automated clients announce themselves in it by default and a great many never think to change that. Anyone doing web scraping with an untouched library default is identifying themselves in the first line of the request.
It is also the single easiest thing in the entire request to falsify, which is exactly why it proves nothing on its own. Any client can send any string, and every site knows that. Nobody serious treats a convincing user agent as evidence of a real browser, so setting one is table stakes rather than a technique.
Its value to a defender is as a consistency check, and this is the part that actually matters. Claiming to be Chrome on Windows commits you to a great deal else: a matching TLS fingerprint, a matching set of client hints, a plausible screen size, the header ORDER a real Chrome sends, and everything browser fingerprinting reads about the machine underneath. Contradict any of it and the string has not helped you, it has given the site something specific to catch you disagreeing with.
Header order deserves its own mention because it is routinely overlooked. Real browsers emit headers in a consistent, characteristic sequence, and HTTP libraries emit them in whatever order the code added them. A perfect user-agent string arriving in an order no browser produces is a contradiction that costs nothing to detect.
Which makes the practical advice narrower than the usual advice. Do set a realistic, current user agent, because a default library string is a free giveaway. Do not rotate it randomly per request, which is a common instinct and an actively bad one: real visitors do not change browser between page loads, and a session whose client identity keeps changing is more conspicuous than one that never changes at all. Pick a plausible identity per session and hold it.
Frequently asked questions
Should I rotate user agents?
Not per request, and this is where a lot of advice goes wrong. A real visitor does not switch browser between page loads, so a session whose user agent keeps changing is more suspicious than one that stays put. Choose a plausible, current string per session or per identity and hold it for that session's lifetime.
Does changing the user agent stop me being detected?
No. It removes one obvious giveaway, the default library string, and nothing more. Detection systems check whether the claim is consistent with your TLS fingerprint, your header order, your client hints and your behaviour, and a browser user agent on a Python handshake is a contradiction rather than a disguise. It is necessary and nowhere near sufficient.
What is the best user agent for web scraping?
A current, common one that matches everything else you present. There is no magic string, and copying an old or unusual one is worse than useless because rarity itself is identifying. Pick a widely-used current browser version and then make sure the rest of your request could plausibly have come from it.
Can a site see my real user agent if I change it?
It only receives what you send, so no. What it can do is notice that what you sent disagrees with everything else about the connection: the TLS handshake belongs to a library rather than that browser, the header order is wrong, the client hints are missing, or the reported screen size is implausible. It does not need your real string to know the one you sent is not true.
Back to the full glossary.