web scraping

Challenge not detected, timeout after 60000 ms, cookies not valid, IP banned: what each FlareSolverr error means, its fix, and where a proxy is the missing piece.

A measured study: 120 fresh free proxies against 12 travel booking sites (OTAs, metasearch, airlines, hotel chains). How many reached each, why zero got real OTA or hotel-chain content, and what it means.

A measured atlas of 47 consumer sites (delivery, restaurants, grocery, pharmacy, retail, fuel): what each front door does to a bare datacenter client and to 20 fresh free proxies, five door types, which bot vendors sit where, and why 'dead' is a property of the proxy list rather than the site.

A 503 through a proxy can come from the target, a bot-management layer, a rate limiter, or the proxy itself. How to read which one answered, and the fix for each.

A 522 is Cloudflare failing to reach the site's origin, so swapping proxies almost never fixes it. What causes it, why scrapers see it more, and what to do.

How to scrape flight prices: why airline and OTA sites are hard, why point-of-sale pricing forces an in-market exit, how to handle Akamai and JavaScript, and the residential proxy setup that holds.

How to scrape hotel prices: why the rate depends on the guest's country, login and device, how to read Country Rates and member rates, and the residential proxy setup that reads each market's price.

Every AI coding tool, Claude Code, Cursor, Copilot, Cline and the rest, raises the same proxy question, and the answer is always two answers. The tool's own connection to its model rarely needs a proxy, and never to hide you. The code and agents you build with it often do. This explainer draws the line once, so the per-tool guides can skip it.

Proxies for Claude Code split into three questions. The big one is the code it writes for you, scrapers, crawlers, monitors and browser agents, which needs residential proxies to run at scale. The second is routing Claude Code itself through a proxy, which it documents with HTTP(S)_PROXY (no SOCKS) for corporate networks and reachability. The third is everything a proxy never does.

Cline is unusual among AI coding agents: it has a built-in browser tool, so it browses the web itself, and it builds automation that does the same. Both meet the walls a proxy solves. This covers where the proxy goes for the code Cline writes, the reachability angle of its bring-your-own-key model access, and the honest limits.

Cursor is an agent-first IDE, and its agents write and run the scrapers, crawlers and automation that need proxies. This covers the residential-proxy layer for what Cursor builds, Cursor's own VS Code proxy settings for corporate networks, and what an IP never does for the editor itself.

GitHub Copilot's proxy story is mostly the corporate one: it documents proxy settings per IDE, basic and Kerberos authentication, and custom certificates for TLS-inspecting networks. That is your company's proxy, not a residential one you buy. The residential proxy belongs to the scrapers and agents you build with Copilot, and this covers both.

An MCP server that fetches the web, a scraper, a search tool, a price checker, hits the same bot walls and per-IP limits as any client, so it needs residential proxies. One that talks only to a private API does not. This covers which MCP servers need a proxy, where it goes in the server, and how to wire it in the common runtimes.

Proxies for travel data: the complete map of scraping flights, hotels, cruise, rail and car rental, why every travel price depends on the market, and which residential setup reads each one.

Proxys für Web Scraping: warum du sie brauchst, welcher Typ zu welchem Ziel passt, wie viele IPs, rotierend oder statisch, und wie du Sperren vermeidest statt sie mit CAPTCHA-Lösern zu überkleben.

Retail bot-defense study: we sent 52 major US retail sites the same request from a home and a datacenter IP on one day. Which run Akamai, Cloudflare, DataDome, PerimeterX, F5 Shape, Kasada or Imperva, how many block datacenter on sight, and which have no wall at all.

Free proxies for OpenSea fail sustained floor and metadata scraping behind its rate limits and Cloudflare. A logged-out spot check works; real work needs residential.

Free proxies for Scrapy are fine for testing your middleware, not a real crawl. The rotating-middleware config, why dead exits thrash a run, and how to verify.

Free proxies for Selenium work for a throwaway session, not an unattended run. How to set the Chrome proxy arg and Selenium Wire, and why free exits hang.

How to use proxies with aiohttp: the proxy argument, BasicAuth, trust_env, SOCKS5 through aiohttp-socks, rotating a pool across concurrent tasks, timeouts and retries.

How to use proxies with axios: why the proxy option fails on HTTPS, the httpsAgent fix with https-proxy-agent, SOCKS, auth, rotating a pool, timeouts and verifying the exit.

How to use proxies with Colly in Go: SetProxy for one proxy, the RoundRobinProxySwitcher for rotation, SOCKS, authentication, timeouts and retries, and verifying the exit IP.

Native fetch in Node has no proxy option. How to route it through a proxy with undici's ProxyAgent and setGlobalDispatcher, per-request dispatchers, auth, SOCKS, and verifying the exit.

How to route OkHttp through a proxy: the proxy() builder method, proxyAuthenticator for auth, SOCKS, a ProxySelector for rotation, per-client design, timeouts, and verifying the exit.