Web scraping
The automated collection of data from websites, the single most common reason people use proxies at scale.
Web scraping is collecting information from websites by program instead of by hand. Rather than a person copying prices into a spreadsheet, a script fetches pages and parses the values out, at a volume and frequency no manual process could sustain. Price monitoring, market research, SEO rank tracking, lead research, travel fare comparison and training-dataset assembly all rest on it.
Proxies enter for a reason that is arithmetic rather than adversarial. Rate limits are almost always counted per address, so a single IP requesting ten thousand pages trips a threshold that ten thousand requests spread across many addresses never approaches. You have not outwitted anything; you have stopped concentrating traffic in one place, which is also what a population of real visitors looks like.
Which proxy type you need follows from what the target screens on, and that is worth establishing empirically rather than by reputation. A site that only counts requests per address may be perfectly happy with cheap datacenter proxies. A site that classifies by network type will reject every datacenter address regardless of pacing. Test the actual URLs you need before buying for a difficulty you assumed.
The harder targets are not defeated by addresses at all, which is the part people learn expensively. Beyond network reputation sit TLS fingerprinting, browser fingerprinting, behavioural scoring and honeypots, and every one of those operates on layers a proxy does not touch. A scraper that works in a browser and fails from a script with the same IP is usually announcing itself at the handshake, not at the address.
There is a large practical difference between collecting HTML and collecting rendered pages, and it decides most of your cost. Fetching documents is fast and cheap. Driving a real browser renders JavaScript and looks far more human, and it also pulls every image, font and script, which on per-gigabyte residential billing is money spent on assets you discard. Many sites expose a JSON endpoint that returns the same data at a fraction of the size, and finding it is usually the highest-value hour of the project.
On conduct, our position is plain and we apply it to ourselves. Collect public, non-personal data. Respect what a site publishes about automated access. Pace requests so you are not degrading a service other people are using. Do not treat a block as an invitation to escalate. Beyond good manners, what is lawful depends on jurisdiction, on the site's terms and on the nature of the data, particularly where personal information is involved, and that is a question for a lawyer rather than a glossary.
The honest closing note is that scraping is maintenance, not a build. Layouts change, defences update, endpoints move, and a scraper is a thing you keep working rather than a thing you finish. Budgeting for that from the start is the difference between a pipeline and a series of surprises.
Frequently asked questions
Do I need residential proxies for web scraping?
Only if the target screens by network type, which is far from universal. Plenty of sites simply count requests per address and are perfectly happy with datacenter proxies, which are faster and much cheaper. Test your actual target URLs with the cheaper option first; paying residential rates for a site that never checks is spending money on trust nobody asked you for.
How many proxies do I need to scrape a site?
Work back from the limit rather than picking a number. If a target tolerates roughly one request every few seconds per address and you need a fixed volume in a fixed window, the arithmetic gives you the pool size. Then add headroom, because addresses drop out and some will already be flagged. Starting slow with a handful and measuring where errors begin beats guessing large.
Why does my scraper get blocked even with good proxies?
Because the address is one layer of several. TLS fingerprinting identifies your HTTP client before anything is decrypted, browser fingerprinting catches contradictions between what you claim and what you report, and pacing that is too regular is visible on its own. A request that works in a real browser and fails from a script on the same IP is being caught above the network.
Is web scraping legal?
It depends on jurisdiction, on the site's terms of service, and heavily on what the data is, with personal information carrying much stricter obligations. That makes it a question for a lawyer rather than for us. Our own operating rules, which we would suggest as a floor rather than as legal advice: public non-personal data, respect published rules on automated access, and pace requests so you are not degrading the service for anyone else.
Should I use a headless browser or plain HTTP requests?
Requests where you can, a browser where you must. Plain fetching is dramatically faster and cheaper, and on per-gigabyte billing it avoids paying for images and fonts you discard. A headless browser is warranted when the data genuinely only exists after JavaScript runs. Before reaching for one, check whether the page is fed by a JSON endpoint you can call directly, which is common and usually far smaller.
Back to the full glossary.