Skip to content

web scraping

Everything tagged web scraping.

149 articles

Topic web scraping

Tutorial

Ruby web scraping: HTTParty, Nokogiri and Ferrum, tested

How to scrape websites with Ruby: HTTParty and Nokogiri for pages, threads for many pages, Ferrum for JavaScript, and the defaults we measured on Ruby 4.0.

October 12, 2026 · 14 min read

Tutorial

Selenium find element: locators, waits and the errors you will meet

How to find elements in Selenium with Python, tested on Chrome: the eight locators, find_element or find_elements, waits in seconds, and each error explained.

October 12, 2026 · 14 min read

Tutorial

Web scraping with BeautifulSoup: parse HTML in Python, with examples

How to scrape with BeautifulSoup, tested: find and select, three parsers against Chrome, the tbody trap, r.content over r.text, and what makes it fast.

October 12, 2026 · 12 min read

Tutorial

Web scraping with Python: requests, BeautifulSoup and proxies, step by step

A Python scraper built step by step with requests and BeautifulSoup and run on 1,000 books: the encoding bug, timeouts, a CSV file and two proxy traps.

October 12, 2026 · 9 min read

Tutorial

Web scraping with Scrapy: spiders, pipelines and proxies

A Scrapy tutorial run end to end on a practice shop: a project, a spider, pipelines that drop bad rows, retries, speed settings, the HTTP cache and a proxy.

October 12, 2026 · 8 min read

Tutorial

Web scraping with Selenium in Python: setup, waits, proxies and blocks

How to scrape a JavaScript page with Selenium in Python, tested on a practice shop: waits, reading in one call, every page to CSV, and a proxy with a login.

October 12, 2026 · 10 min read

Tutorial

XPath contains(): match text and attributes, with examples

XPath contains() tested in lxml, parsel and Chrome: why contains(text()) misses split text, why class matching overshoots, not(), case, spaces and empty values.

October 12, 2026 · 6 min read

Review

CaptchaSonic.com Review: Prices, Agent Tools and Claims Checked

CaptchaSonic prices 27 captcha types and ships an MCP server for AI agents. We set its prices against six other solvers and checked its claims.

October 11, 2026 · 12 min read

Explainer

HTTP cookies explained: how they work and why they matter when scraping

What an HTTP cookie is, what each Set-Cookie attribute does, which cookies 100 top sites set on a first visit, and why one cookie links requests across proxies.

October 11, 2026 · 7 min read

Explainer

Is web scraping legal? What courts, laws and site terms actually say

Is web scraping legal? What US courts held in hiQ and Van Buren, what the GDPR and EU copyright law say, and what 20 big sites allow in their terms and robots.txt.

October 11, 2026 · 8 min read

Tutorial

XPath vs CSS selector: which to use for scraping

XPath vs CSS selectors, tested on one page in lxml, parsel, BeautifulSoup and Chrome: what each can find, the class trap that drops matches, and which is faster.

October 11, 2026 · 6 min read

Explainer

Concurrency vs parallelism in web scraping: requests, threads and proxies

Concurrency overlaps waiting, parallelism runs work on several cores. We timed both on 100 pages in Python: threads and asyncio for fetching, processes for parsing.

October 10, 2026 · 7 min read

Free proxies

Free Rotating Proxies: What Exists, and How to Rotate a Free List

No free rotating proxy gateway holds up. What exists: trials, free quotas, Tor, and rotating a free list yourself, with tested Python code and measured decay.

September 26, 2026 · 8 min read

Use case

Proxies for Aider: installing the browser turns your proxy off

Aider's web command has two implementations. One reads your proxy variables, the other passes nothing, and an optional install decides which you get.

September 22, 2026 · 5 min read

Use case

Proxies for Camoufox: the proxy dict, geoip and one IP per browser

Camoufox matches timezone and locale to the proxy's IP with geoip=True, and gives each context its own proxy with NewContext. Which proxy lines keep it right, in 0.5.8.

September 19, 2026 · 10 min read

Use case

Proxies for CrewAI: the scrape tool refuses them on purpose

CrewAI's ScrapeWebsiteTool ignores your proxy variables and raises if you pass one. Why that is deliberate, and the two routes that still work.

September 19, 2026 · 6 min read

Use case

Proxies for self-hosted Firecrawl: PROXY_SERVER and what it reaches

A self-hosted Firecrawl has no stealth proxies. Where PROXY_SERVER goes, which parts use it, and the checks that break it, read in v2.11.0 and main.

September 19, 2026 · 10 min read

Use case

Proxies for google-maps-scraper: what the list really does

The open source Google Maps scraper takes a proxy list in one flag. How v1.18.0 spends it: one address per browser, a login on every line, and the traps.

September 19, 2026 · 7 min read

Use case

Proxies for Lightpanda: the browser that takes the login

Lightpanda runs on libcurl, not Chrome, so its proxy takes a login in the address and SOCKS5 works. What the CDP proxy parameter really changes.

September 19, 2026 · 6 min read

Use case

Proxies for n8n: the node's Proxy option, HTTP_PROXY and NO_PROXY

n8n ignores a socks5:// proxy in the HTTP Request node, and HTTP_PROXY moves every node's traffic. Which proxy settings work, read in n8n 2.39.8.

September 19, 2026 · 8 min read

Use case

Proxies for nodriver: browser_args, create_context and the login

nodriver ignores a login in browser_args. Where a proxy with a username and password goes (create_context), and what its local forwarder does, in 0.50.3.

September 19, 2026 · 6 min read

Use case

Proxies for Playwright MCP: the proxy flag, the login and localhost

Playwright MCP drops a user:pass written into --proxy-server and sends localhost through the proxy. Where the proxy and its login go, read in the code.

September 19, 2026 · 8 min read

Use case

Proxies for Scrapling: proxy=, ProxyRotator and the three fetchers

Scrapling takes a proxy per fetcher, per session or through ProxyRotator. The formats, the rotation and where the password shows up, read in 0.4.15.

September 19, 2026 · 8 min read

Use case

Proxies for SeleniumBase: the proxy string, the login and Chrome 142

SeleniumBase takes a proxy as one string with the login inside. How each mode answers the login, what Chrome 142 changed, and how to run several at once.

September 19, 2026 · 7 min read

Showing 25 to 48 of 149

Proxies that don't die mid-job

Residential, ISP, datacenter and mobile, verified by the same engine that runs tens of millions of checks. They read as a real device and hold up under load. Pay as you go, and your balance never expires. $0.23/GB is the 10,000 GB+ rate; a single gigabyte is $0.50/GB, with no minimum order.

129M+ proxy checks run · 100+ countries · HTTP / HTTPS / SOCKS · re-checked every few minutes · no signup

HProxy.

Honest guides and comparisons on proxies, scraping and staying unblocked, from the team that runs the network.

RSS feed