Proxies for Web Scraping: Which Type to Use, How to Rotate, What It Costs

For web scraping, use the cheapest proxy the target tolerates. That means datacenter proxies for sites that do not check IP reputation, and rotating residential proxies for defended ones. Defended sites do look at the network. Cloudflare documents a scraping detection that watches request patterns per network (ASN). A job that sends all its traffic from one hosting provider stands out as one pattern. The address is rarely the whole story, though. On October 22, 2026, the same plain request got the same answer from a home and a datacenter address at 14 of 14 sites. Changing only the client's TLS stack changed two answers (the test). Residential traffic costs more per GB: a median $3.65 on the smallest plan across 64 sellers in our price records, against $0.60 for datacenter.

This guide covers which type fits which target, what each costs and how to rotate. It also covers the request signals no proxy fixes, and a loop you can run today.

What proxies are best for web scraping?

Match the proxy to the target's defenses, and start one step lower than you fear.

The money rule inside that table: use the cheapest type the target accepts, and move up when block rates show you must. Residential traffic spent on a site that would have accepted datacenter only burns budget.

Why does scraping need proxies at all?

A scraper sends far more requests, far faster, from one address than any person would. Sites notice, and the defenses escalate in a predictable order:

1. Rate limiting. Too many requests from one IP and you get 429 Too Many Requests or a slowdown.
2. IP blocks. Keep going and the address is blocked, sometimes for hours, sometimes for good. Our guide on avoiding IP bans while scraping is the full prevention checklist.
3. Bot fingerprinting. Defended sites profile your headers, your TLS fingerprint (JA3, or the newer JA4), your HTTP/2 fingerprint and your behavior, then serve a CAPTCHA or false data to anything that looks automated. The named systems each have their own playbook: Cloudflare, DataDome and reCAPTCHA.

Proxies answer the first two by spreading requests across many addresses, so no single IP trips a limit. The third needs work on your side, which most guides leave out and this one covers below.

What does a scraper meet on a defended site?

Our test of 52 US retail sites on August 24, 2026 shows the walls a scraper actually meets. Each site got one plain request from a home connection and one from a datacenter server. We identified the bot-defense vendor from the response.

Nine sites answered our home and our datacenter request differently. But those two requests also came from different clients, and the client alone can change a Cloudflare answer. Our controlled test shows that, so we no longer count the nine as proof that the address decided. Seventeen of the 52 ran two or more defenses at once. A homepage that loads proves little, too: 17 sites returned a normal homepage to a bare client. Most of those still gate prices behind a sensor cookie or a behavioral score. The retail bot-defense study lists every site.

The proxy types, matched to the job

Each type is right for one tier of target.

Datacenter proxies

Addresses from hosting providers: fast, cheap and plentiful. Sites can tell an address belongs to a data center, so anti-bot systems distrust these by default. They suit sites with light defenses, APIs, and any target that does not score IP reputation. This is the cheapest tier, and the place to start whenever the target allows it.

Rotating residential proxies

Addresses on home internet connections, drawn from a pool through a gateway and changed per request or per session. To a website they look like ordinary consumers, so they pass the reputation checks that turn datacenter addresses away on many sites. The tradeoffs are cost, metered per GB, and variable speed, because a home connection is sometimes slow. They suit the hard targets: major retailers, search results, travel and ticketing, anything with a real bot team. Rotating vs static residential proxies covers the mechanics, and for scraping the rotating side is almost always the right one.

Static residential (ISP) proxies

Residential-looking, stable and fast. For scraping, their niche is collection behind a login, where a mid-session address change would end the session. Most pure collection jobs do not need them, and account-bound ones cannot work without them. What an ISP proxy is explains the product.

Mobile proxies

Addresses from cellular carriers. Carriers put many subscribers behind one address, so blocking a mobile IP can block many real customers at once. Sites hesitate to do it. That makes mobile proxies the heavyweight option for the most aggressively defended targets, at the highest price per GB.

What does each proxy type cost per GB?

Rotating pools bill by the gigabyte, so the per-GB price is the one to compare. Our price records hold the per-GB plans of 64 sellers across the four types.

Datacenter and ISP proxies are mostly sold per IP, so their per-GB samples are smaller; residential and mobile are sold per GB almost everywhere. Our Residential Lite costs $0.50 per GB for one GB, below the residential median on the smallest plan. Residential Premium adds state, city and ISP targeting from $1.49 per GB.

The page you load decides the bill. In the HTTP Archive's October 2024 crawl, the median desktop page weighed 2,652 KB, and its HTML alone 18 KB. Blocking images, fonts and media in your client removes most of the bytes. How to use a proxy charts how far one GB goes.

What are the best residential proxies for web scraping in 2026?

For most scraping, the best value is a rotating residential pool sold by the GB whose traffic does not expire. We read each seller's own pricing page on October 26, 2026:

Rainproxy falls furthest with volume, to $0.24 per GB at 1 TB. The cheap pools differ in what they let you target. Rainproxy sells city and ASN targeting on a separate pool from $2.98 per GB. Our Premium pool adds state, city and ASN targeting from $1.49/GB.

Bright Data lists $8 per GB pay as you go, or $4 with a three-month code. Its documentation opens the residential network only to verified companies. A price per GB also says nothing about success on your target, so test two or three sellers on a small buy. Our best residential proxies comparison sets ten sellers side by side.

How should a scraper rotate?

How you rotate matters more than the size of the pool.

Per-request rotation suits stateless collection: independent pages with no login and no cart. Every request gets a fresh exit, so no single address builds up enough activity to look suspicious.

Sticky sessions suit anything with several steps: a search that paginates, a flow that sets cookies, a cart. You hold one exit for a set time, so the site sees one coherent visit. On our residential pools a rotating session gets a new exit on the next request. A Premium sticky session lasts from 3 minutes to 24 hours. Sticky vs rotating sessions compares the two in depth.

Two mistakes show up again and again:

• Rotating too fast on stateful flows. A new IP on every page of a paginated search looks less human than one IP would. Match rotation to the interaction.
• Ignoring geography. A session that hops from Germany to Brazil to Japan across three requests describes a bot in one sentence. Pin one country per session; every residential pool of ours targets by country, and Premium adds state and city.

What a proxy cannot fix: request hygiene

A clean residential IP paired with a careless request still gets blocked, because the address is one signal among several. The rest are on you.

Send believable headers

Default library user agents (python-requests/2.x) give a scraper away at once. Send a real browser user agent and the Accept, Accept-Language and Accept-Encoding headers a browser sends, as one consistent set.

Keep cookies

Browsers keep cookies across a visit, and many scrapers throw them away. Keeping cookies within a session makes you look like one returning visitor.

Pace like a person

No person fires 20 requests a second at one site. Add delays, randomize them, and pause between logical steps. Slower and unblocked beats fast and banned every time.

Match your TLS and JavaScript to your story

Defended systems fingerprint your TLS handshake. One method is JA3, which its authors describe as "a method for creating SSL/TLS client fingerprints". The newer one is the JA4 family. On script-heavy sites they also expect JavaScript to run. If a target demands it, a headless browser such as Playwright behind your residential proxy beats raw HTTP requests. It produces a browser's fingerprint as a side effect. This is the layer that trips scrapers behind Cloudflare and DataDome.

The model to keep: the proxy makes your IP believable, and hygiene makes your behavior believable. You need both, and no proxy tier substitutes for the second.

A minimal loop that behaves

This pulls fresh proxies from our free API, which is fine for learning (use a paid pool for real runs). It scrapes politely: browser headers, a timeout per proxy, capped retries and randomized pacing.

import random, time, requests

# A pool for experiments. Swap for your paid gateway in production.
pool = requests.get(
    "https://hproxy.com/api/proxy-list",
    params={"format": "txt", "protocol": "http", "recent": "true", "limit": 50},
    timeout=15,
).text.split()

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch(url, tries=3):
    for _ in range(tries):
        proxy = random.choice(pool)
        try:
            r = requests.get(
                url, headers=HEADERS,
                proxies={"http": f"http://{proxy}", "https": f"http://{proxy}"},
                timeout=(5, 15),
            )
            if r.status_code == 200:
                return r.text
        except requests.RequestException:
            pass
        time.sleep(random.uniform(1.5, 4.0))   # human-ish pacing + backoff
    return None

It is small on purpose, and every choice carries weight: real headers and a pool sized for a session. Connect and read timeouts mean a dead proxy cannot hang the run. Retries are there because free exits die, and randomized sleeps mean you never hammer a host. Production adds persistent sessions, rate control per domain and real logging, and the shape is already right. Our Python requests guide goes deeper on sessions, pools and retries. The Scrapy proxy guide moves the same rotation into a middleware. The cURL guide has the shell-side checks for testing a pool before you wire it in.

How do you choose a proxy provider for scraping?

Five questions cut through the marketing:

1. Coverage where you scrape. Check the countries and cities you need, and skip the global headline number.
2. A pricing model that fits your shape. Per GB suits many identities at modest traffic; per IP suits heavy traffic through few addresses. Do the arithmetic on your real volume.
3. Rotation control. Per-request and sticky sessions, a stated sticky window, and the targeting you need.
4. Honest success claims. A seller quoting 99.9% for every site is advertising. Test on your actual targets with a small purchase first.
5. No lock-in. Pay as you go and a balance that does not expire, so a paused project keeps its credit. That is how our pricing works.

Start free to learn the moving parts (here are the limits). Prototype on the cheapest tier the target tolerates, and move up when block rates demand it. Keep your request hygiene as sharp as your IP quality. Do that and scraping stops being a fight with proxies. It goes back to being a data problem, which is the one you wanted to solve.

When you do not need proxies at all

This whole guide assumes you are building the scraper. That is right for most targets and wrong for a few. For a set of common platforms, from social networks to search and marketplaces, we already run the scraper and expose it as an endpoint. Our scraper APIs take a handle or a URL and return parsed JSON. Proxies, rotation and anti-bot handling are included in the call. Where an endpoint exists, it usually costs less than building the same thing. Where it does not, everything above applies.

More and more scrapers are now written by an AI coding tool. That makes the code easy to produce and leaves the blocking just as hard. If you built yours with Claude Code, Cursor or Copilot, the proxy still belongs in the code. Proxies and AI coding tools covers how to have the tool write it in from the start.

What this page could not check

• No network tests. We measured no proxy network's speed or success rate, ours included.
• One day's front door. The retail test is a snapshot of August 24, 2026; bot defenses are tuned continuously, and a site that loaded may still be hard to scrape at depth.
• Per-GB samples. Datacenter and ISP proxies are mostly sold per IP, so 17 and 11 sellers stand behind their per-GB medians.
• Sellers as printed. The seller table repeats what each pricing page printed on October 26, 2026; we bought from none of them.
• Check by January 22, 2027. Prices and defenses move; every figure here needs a fresh read by then.

Sources

• HProxy retail bot-defense test, August 24, 2026: 52 US retail and marketplace storefronts, one request each from a home connection and a datacenter server.
• HProxy price records: per-GB plans of 64 sellers across datacenter, ISP, residential and mobile proxies, captured July 20 to October 14, 2026, with headline "from" prices left out.
• HTTP Archive, Web Almanac 2024, Page Weight chapter, read October 22, 2026.
• JA3 (Salesforce) and JA4+ (FoxIO), repository READMEs, read October 22, 2026.
• RFC 9309, Robots Exclusion Protocol, IETF, September 2022: the robots.txt standard to respect while collecting.
• Requests documentation, Advanced Usage, Proxies, read October 22, 2026.
• HProxy residential proxies page, October 22, 2026.
• Rainproxy, Evomi, DataImpulse, Decodo, MarsProxies and Bright Data, residential pricing pages, read October 26, 2026.
• Bright Data documentation, Residential network access, updated September 1, 2026, read October 26, 2026.