web scraping
Everything tagged web scraping.
149 articles
Topic web scraping

Explainer
The history of Beautiful Soup: what bs4 is, how it works and how 22 years of releases made it
The history of Beautiful Soup, the Python library behind bs4, first released in April 2004: what version 4.15.0 contains, read from all 85 releases.
November 17, 2026 · 14 min read

Explainer
The history of Crawl4AI: what it is, how it works and how 30 months of commits made it
The history of Crawl4AI, an open-source Python crawler for AI pipelines, first committed in May 2024: what version 0.9.5 contains, read from 1,792 commits.
November 16, 2026 · 15 min read

Explainer
The history of Scrapy: what it is, how it works and how 18 years of commits made it
The history of Scrapy, a Python framework for crawling sites, first committed in June 2008: what version 2.20.0 contains, read from 11,758 commits.
November 14, 2026 · 13 min read

Explainer
The history of Scrapling: what it is, how it works and how it changed in 55 releases
The history of Scrapling, a Python web scraping framework with 101,070 GitHub stars: what version 0.5.1 contains, read from all 1,825 commits.
November 12, 2026 · 14 min read

Tutorial
How to Set Up CloakBrowser With Residential Proxies
CloakBrowser takes the proxy you pass at launch. Connect an HProxy residential line in Python or JavaScript, match timezone and locale to the exit IP with geoip, and keep one profile and one sticky session per account.
November 4, 2026 · 10 min read

Tutorial
How to Use NoneCap With Residential Proxies to Solve hCaptcha
NoneCap solves hCaptcha through a proxy you pass with each solve. Give it a sticky residential line, submit the token from that same line within about two minutes, and enterprise sitekeys accept it.
October 22, 2026 · 8 min read

Tutorial
Best language for web scraping: Python, JavaScript, Go and the rest
We ran the same scraper in Python, JavaScript, Go and Java on our server: speed, memory, lines of code, broken HTML, timeouts and proxies, measured.
October 12, 2026 · 15 min read

Tutorial
Best web scraping tools: what each does well and where it gets blocked
The scraping tools you can run yourself, sorted by job, with what our tests measured, and what 16 tool settings got back from eight sites behind bot defenses.
October 12, 2026 · 7 min read

Tutorial
C# web scraping: HttpClient, HtmlAgilityPack, AngleSharp and Playwright, tested
How to scrape websites in C#: HttpClient with HtmlAgilityPack or AngleSharp, Parallel.ForEachAsync for many pages, and Playwright for JavaScript, tested on .NET 10.
October 12, 2026 · 13 min read

Tutorial
Golang web scraping: net/http, goquery, Colly and chromedp, tested
How to scrape websites in Go: net/http and goquery for pages, Colly for whole sites, chromedp for JavaScript, and the defaults we measured on Go 1.27.2.
October 12, 2026 · 15 min read

Use case
Image scraping: how to download images from a website, and what is allowed
How to download a website's images with Python, tested: what img src misses, srcset and lazy loading, duplicates, robots.txt read the right way, and copyright.
October 12, 2026 · 7 min read

Tutorial
Java web scraping: jsoup, HttpClient and HtmlUnit, tested
How to scrape websites in Java: jsoup for pages, virtual threads for many pages, HtmlUnit for JavaScript, and the defaults we measured on Java 25.
October 12, 2026 · 12 min read

Tutorial
JavaScript web scraping: fetch, Cheerio and Puppeteer in Node.js
How to scrape with JavaScript in Node.js, tested: fetch and Cheerio for HTML, Puppeteer for script-built pages, garbled text, timeouts and proxies.
October 12, 2026 · 8 min read

Use case
Job scraping: how to collect job postings and which sites allow it
How to collect job postings without breaking a site's rules: what 15 job sites' robots.txt files allow, what Indeed and LinkedIn's terms say, and the public job APIs.
October 12, 2026 · 10 min read

Tutorial
Open source web scrapers: what each project is good for
Which open-source web scraper fits which job: 18 projects with their licenses and releases, and 7 crawlers run with their defaults on a test site of our own.
October 12, 2026 · 12 min read

Tutorial
pandas read_html: scrape HTML tables into DataFrames
How to read HTML tables into pandas DataFrames with read_html, tested on pandas 3.0: picking a table, numbers, links, headers, and the errors you will meet.
October 12, 2026 · 6 min read

Tutorial
Parse HTML with regex: when it works and when to use a parser
Can you parse HTML with regex? We ran four regular expressions and two parsers on 13 small HTML inputs and checked every answer against Chrome's own parser.
October 12, 2026 · 7 min read

Tutorial
Parse JSON in Python: json.loads, files and API responses
How to parse JSON in Python, tested: strings, files and API answers, the 'Expecting value' error, NaN and big numbers in other languages, dates and orjson.
October 12, 2026 · 5 min read

Tutorial
Parse XML in Python: ElementTree, lxml and sitemaps
How to parse XML in Python, tested on our own sitemaps: the namespace trap that empties findall(), large files with iterparse, five parsers timed, and safety.
October 12, 2026 · 6 min read

Tutorial
PDF scraping: extract tables and text from PDFs with Python
PDF scraping in Python, tested on a 28-page IRS tax table: which library keeps rows in order, why the table finders failed, what fixed it, and scanned pages.
October 12, 2026 · 5 min read

Tutorial
PHP web scraping: cURL, Guzzle, the DOM parsers and chrome-php, tested
How to scrape websites with PHP: the cURL extension and PHP 8.4's HTML5 parser for pages, Guzzle for many pages, chrome-php for JavaScript, tested on PHP 8.5.
October 12, 2026 · 15 min read

Tutorial
Playwright wait for page load: load states, selectors and network idle
Which Playwright wait to use when a page loads its content with JavaScript: load states, networkidle, locator waits and responses, each timed on test pages of our own.
October 12, 2026 · 8 min read

Tutorial
Python HTTP clients compared: requests, httpx, aiohttp and urllib3
Six Python HTTP clients tested on one server: what each one sends, which follow redirects, which give up on a silent server, how fast they are, and what they install.
October 12, 2026 · 10 min read

Tutorial
Python web scraping libraries compared: what each one is for
The Python web scraping libraries that matter, tested on one job: requests and httpx, BeautifulSoup, lxml, parsel, selectolax, Scrapy, Playwright and Selenium.
October 12, 2026 · 11 min read
Showing 1 to 24 of 149
Proxies that don't die mid-job
Residential, ISP, datacenter and mobile, verified by the same engine that runs tens of millions of checks. They read as a real device and hold up under load. Pay as you go, and your balance never expires. $0.23/GB is the 10,000 GB+ rate; a single gigabyte is $0.50/GB, with no minimum order.
129M+ proxy checks run · 100+ countries · HTTP / HTTPS / SOCKS · re-checked every few minutes · no signup
HProxy.
Honest guides and comparisons on proxies, scraping and staying unblocked, from the team that runs the network.