Proxies for web scraping. Built to not get blocked.
Scraping fails for one reason above all others: too many requests from one address. Rotating residential IPs spread the load so a crawl looks like many visitors, not one, and finishes instead of getting blocked.
What fits
3 products fit, best first
ResidentialRecommended for Web scraping
Real-device IPs in 200+ countries
- Rotating or sticky sessions
- State, city and ISP targeting on Premium
- Pay per GB, balance never expires
- HTTP and SOCKS5
From
$0.44/ GB
Web Scrapers
Structured JSON from one REST call
- Proxies, retries and parsing handled
- Blocked or errored calls are not billed
- Pay per call from your balance
- No subscription, no monthly minimum
From
$1.49/ 1,000 calls
Datacenter
IPv4 and IPv6
- Dedicated to you for the term
- Unlimited bandwidth
- Up to 1 Gbps port speed
- HTTP, HTTPS and SOCKS5
From
$0.10/ IP
- Residential proxies
- The workhorse of scraping: a rotating pool of real home IPs, billed per GB, that keeps any single address under the limits sites count against.
- Scraper API
- The hands-off option: a URL in, clean data out, with proxy rotation, retries and unblocking handled so you skip the anti-bot arms race entirely.
- Datacenter proxies
- The cheap, fast choice for sites that do not screen by network, load-testing your own service, or scraping sources that simply do not care.
Almost every block traces back to requests per IP
The obstacles a scraper meets, rate limits, CAPTCHAs, sudden bans, are overwhelmingly responses to one address making too many requests too fast. Sites count per IP, so the fix is structural: reduce the requests any single address makes by spreading them across a pool.
That is what a rotating residential proxy does. The same total volume, distributed across thousands of real home IPs, means no single one crosses the threshold that trips a defence, and the crawl that used to die at a few thousand requests runs to completion.
Match the proxy to the target, not the other way round
The right proxy depends on what you are scraping. A site that screens datacenter ranges needs residential; one that only rate-limits may be fine on cheaper datacenter IPs; one behind a serious anti-bot stack is often better handed to the scraper API than fought directly.
The honest rule is to start with the cheapest option that survives the target and move up only when it does not, rather than paying for residential on a source that never needed it. The free proxy list is there for a first probe before you commit.
How to set up proxies for web scraping
- 1
Pick the pool by the target
Datacenter for sites that do not screen by network, residential for the ones that do, and the scraper API for targets behind a heavy anti-bot wall. Try the cheaper option on a small sample first.
- 2
Generate one line per worker
In the dashboard or through the API, choose a country and how many lines you need. A count of 10 gives ten workers ten lines, and every request on a rotating line leaves from a different address.
- 3
- 4
Check one request before the crawl
Send a single request through the line to the proxy checker or the IP lookup. The exit should sit in the country you chose and should not be your own address.
- 5
Hold an address only where the site needs it
For a login or a multi-page flow, add stickyMinutes so one address serves the whole session, up to 1,440 minutes on Premium. Everything else rotates, which keeps each address under the limits sites count.
Web scraping: what goes wrong, and the fix
- HTTP 429, too many requests
- One address is sending too much. Spread the work over more lines, slow each worker down, or rotate on every request.
- A CAPTCHA on every page
- The network or the client gives the scraper away. Move from datacenter to residential, send the headers a real browser sends, or hand that target to the scraper API.
- The page shows another country's prices or language
- The exit country is wrong, or a cookie remembers an earlier visit. Set the country on the line and start each session without old cookies.
- Requests time out
- Residential exits are real home connections, and some are slow. Set a timeout, retry the request on a fresh line, and do not pile many parallel requests onto one line.
- The crawl runs for an hour, then fails
- The site has learned the pattern. Vary the order and the timing of requests, and spread the crawl across more addresses.
Setup guides and tools
FAQ
Web scraping questions, answered
What kind of proxy is best for web scraping?
Rotating residential for defended sites, datacenter for undefended ones, and the scraper API when you would rather not manage proxies at all. The target decides: start cheap and move up only when a source blocks the cheaper option.
Why do I keep getting blocked when scraping?
Almost always because too many requests come from one IP. Sites count per address, so a single proxy pulling volume gets rate-limited or CAPTCHA'd. Spreading the crawl across a rotating pool is the structural fix.
How many proxies do I need for web scraping?
On residential you buy traffic, not addresses: one plan generates as many lines as your workers need, and each request on a rotating line can leave from a different address. Datacenter is sold per address, so there you buy as many as you run at the same time.
Do the residential proxies rotate on every request?
Yes, unless you ask a line to hold its address with stickyMinutes. That is the setting for logins and multi-page flows; plain page fetching should rotate.
How much does scraping with proxies cost?
Residential is billed per gigabyte of traffic, so cost tracks how much you fetch, and the balance never expires. Fetching only what you need, skipping images you will not parse, keeps it low. See the pricing page for the current per-GB rate.
Is scraping with proxies legal?
A proxy changes where a request comes from, not what you are allowed to collect. What you scrape, how often, and what you do with personal data stay your responsibility under the site's terms and the law where you operate.
Related: AI scraping, AI agents.