Skip to content

web scraping

Everything tagged web scraping.

149 articles

Topic web scraping

Data study

We Pointed 120 Free Proxies at 12 Travel Sites. 31 Reached Any of Them

A measured study: 120 fresh free proxies against 12 travel booking sites (OTAs, metasearch, airlines, hotel chains). How many reached each, why zero got real OTA or hotel-chain content, and what it means.

August 24, 2026 · 7 min read

Data study

We Knocked on 47 Front Doors With 20 Free Proxies Each: Who Challenges, Who Blocks, Who Stalls

A measured atlas of 47 consumer sites (delivery, restaurants, grocery, pharmacy, retail, fuel): what each front door does to a bare datacenter client and to 20 fresh free proxies, five door types, which bot vendors sit where, and why 'dead' is a property of the proxy list rather than the site.

August 24, 2026 · 10 min read

Guide

503 Service Unavailable Through a Proxy: Who Sent It, and the Fix for Each

A 503 through a proxy can come from the proxy, a rate limiter, the site's edge or the site. How to tell which, measured with curl, Wget and Python requests.

August 24, 2026 · 10 min read

Guide

Cloudflare Error 522 Connection Timed Out Through a Proxy: What Actually Fixes It

A 522 means Cloudflare could not reach the site's own server in time, so a new proxy rarely helps. Cloudflare's timers, the causes, and what to do.

August 24, 2026 · 13 min read

Guide

How to Scrape Flight Prices in 2026: The APIs, the Rules and the Proxy Setup

How to collect flight prices in 2026: the official APIs and their costs, what 12 fare sites' robots.txt files allow, the proxy setup, its traffic and dated proxy prices.

August 24, 2026 · 12 min read

Guide

How to Scrape Hotel Prices in 2026: The APIs, the Rules and the Rate Layers

How to collect hotel prices in 2026: the Booking.com and Expedia APIs, what the sites' terms and robots files allow, which rate layers follow the guest, the setup and dated proxy prices.

August 24, 2026 · 11 min read

Guide

Proxies for MCP Servers: The Fetch-Heavy Ones Need Residential IPs, and Where the Proxy Goes

An MCP server that fetches the web, a scraper, a search tool, a price checker, hits the same bot walls and per-IP limits as any client, so it needs residential proxies. One that talks only to a private API does not. This covers which MCP servers need a proxy, where it goes in the server, and how to wire it in the common runtimes.

August 24, 2026 · 4 min read

Guide

Proxys für Web Scraping: der praktische Leitfaden

Proxys für Web Scraping: warum du sie brauchst, welcher Typ zu welchem Ziel passt, wie viele IPs, rotierend oder statisch, und wie du Sperren vermeidest statt sie mit CAPTCHA-Lösern zu überkleben.

August 24, 2026 · 3 min read

Data study

Retail Bot Defense Study 2026: Who Runs What at 52 US Retailers

Retail bot-defense study: we sent 52 major US retail sites the same request from a home and a datacenter IP on one day. Which run Akamai, Cloudflare, DataDome, PerimeterX, F5 Shape, Kasada or Imperva, how many block datacenter on sight, and which have no wall at all.

August 24, 2026 · 9 min read

Free proxies

Free Proxies for OpenSea: What Actually Works

Free proxies for OpenSea fail sustained floor and metadata scraping behind its rate limits and Cloudflare. A logged-out spot check works; real work needs residential.

August 23, 2026 · 5 min read

Free proxies

Free Proxies for Scrapy: What a Test of 300 Showed, and the Settings to Use

We sent the first request of a new Scrapy project through 300 free proxies from our list. 83 returned the file. The settings and middleware that follow.

August 23, 2026 · 10 min read

Free proxies

Free Proxies for Selenium: What 300 Page Loads Showed, and the Setup to Use

We loaded a page and its 25 files through 300 free proxies from our list, the way Chrome does. 111 arrived whole. The setup and timeouts that follow.

August 23, 2026 · 9 min read

Guide

How to Use Proxies With aiohttp: HTTPS, SOCKS5 and trust_env

aiohttp's own docs and code on proxies: proxy= per request or per session since 3.11, the login after 3.14, trust_env, HTTPS proxies and SOCKS5.

August 23, 2026 · 11 min read

Guide

How to Use Proxies With Axios (Node.js)

How to use proxies with axios: why the proxy option fails on HTTPS, the httpsAgent fix with https-proxy-agent, SOCKS, auth, rotating a pool, timeouts and verifying the exit.

August 23, 2026 · 6 min read

Guide

How to Use Proxies With Colly (Go)

How to use proxies with Colly in Go: SetProxy for one proxy, the RoundRobinProxySwitcher for rotation, SOCKS, authentication, timeouts and retries, and verifying the exit IP.

August 23, 2026 · 4 min read

Guide

How to Use Proxies With Node fetch (undici ProxyAgent)

Native fetch in Node has no proxy option. How to route it through a proxy with undici's ProxyAgent and setGlobalDispatcher, per-request dispatchers, auth, SOCKS, and verifying the exit.

August 23, 2026 · 4 min read

Guide

How to Use Proxies With OkHttp (Java, Kotlin, Android)

OkHttp's proxy rules from its own source: proxy() and the ProxySelector, the CONNECT tunnel and proxyAuthenticator, SOCKS through Java's socket, and the errors.

August 23, 2026 · 8 min read

Guide

How to Use Proxies With Postman: Settings, SOCKS5 and Newman

Postman's proxy settings in its own words: system or custom proxy, SOCKS since v11.56.4, the proxy login, the bypass list, OS variables, the 407 and Newman.

August 23, 2026 · 15 min read

Guide

How to Use Proxies With Puppeteer

How to use proxies with Puppeteer: the proxy-server launch arg, authenticated proxies with page.authenticate and proxy-chain, per-context proxies, and rotating a pool.

August 23, 2026 · 5 min read

Guide

How to Use a Proxy With wget (and Wget2): Variables, Flags and What Is Sent

How to send wget through a proxy: the lowercase variables, -e and .wgetrc, proxy auth, no_proxy, why there is no SOCKS, and what changes on Wget2, measured against a stub proxy.

August 23, 2026 · 8 min read

Guide

LLM API Rate Limits and Proxies: Why Changing Your IP Does Nothing

Developers hitting 429s on an LLM API often reach for a rotating proxy pool. The quota follows the key in the header, not the address it came from. What actually fixes it, and where proxies do belong in an AI stack.

August 23, 2026 · 9 min read

Guide

requests vs httpx vs aiohttp: Proxies Compared

How the three Python HTTP clients handle proxies: the config each one uses, SOCKS support, sync versus async, connection reuse, and which to pick for a scraper.

August 23, 2026 · 5 min read

Guide

Do Your Proxies Support WebSockets? How to Test It in Two Minutes

WebSockets fail through proxies in a few specific ways, and provider docs rarely mention it. The two-minute test, how to read the failure, and why rotation is the usual cause.

August 23, 2026 · 11 min read

Data study

Scraping API Price Spread: Costs Run 50x Apart for the Same Job

Four scraping APIs, converted to one number: dollars per 1,000 requests. The range runs from $0.0749 to $3.75, and on one vendor both ends of a 25x spread sit inside the same $30 plan.

August 11, 2026 · 10 min read

Showing 73 to 96 of 149

Proxies that don't die mid-job

Residential, ISP, datacenter and mobile, verified by the same engine that runs tens of millions of checks. They read as a real device and hold up under load. Pay as you go, and your balance never expires. $0.23/GB is the 10,000 GB+ rate; a single gigabyte is $0.50/GB, with no minimum order.

129M+ proxy checks run · 100+ countries · HTTP / HTTPS / SOCKS · re-checked every few minutes · no signup

HProxy.

Honest guides and comparisons on proxies, scraping and staying unblocked, from the team that runs the network.

RSS feed