A web scraping tool fetches pages and pulls data out of them. It can be a library, a framework, a browser driver or a paid service. Which one is best depends on the job, and on how hard the site pushes back.
This page sorts the tools you can run yourself by job, with numbers from our own tests. On 12 October 2026 we also sent 16 tool settings to eight sites behind different bot defenses, to see where each one gets stopped.
The short answer
- Plain pages in Python: requests with selectolax or lxml. In our test that matched Go's speed, while BeautifulSoup in eight threads was the slowest setup.
- Whole sites: Scrapy. A new project is polite by default and fast when a site allows it.
- Pages built by JavaScript: Playwright, and wait for the element you need.
- Text for AI tools: Crawl4AI for Markdown, trafilatura for the main text of articles.
- Sites behind bot defenses: no tool got everywhere. curl_cffi, which copies Chrome's connection, loaded five of our eight shops, and three shops stopped all 16 settings.
Where the tools got blocked
We picked eight shops whose bot defenses our August study of 52 retailers had named. Each of 16 tool settings asked each home page once, in two passes, from our server's datacenter address. We counted a shop only when the answer showed it, with its title and its name and no block message.
| Shop | Defense found in August | Settings that got the page, of 16 |
|---|---|---|
| Target | HUMAN | 16 |
| TCGplayer | none found | 16 |
| Best Buy | Akamai | 11 |
| Whatnot | Kasada, Cloudflare and others | 3 |
| GameStop | Cloudflare | 1 |
| Pokémon Center | Imperva, DataDome | 0 |
| AutoZone | DataDome | 0 |
| Nordstrom | F5 Shape | 0 |
Every setting gave the same answer in both passes. Five results stand out.
- Three shops stopped all 16 settings. Pokémon Center, AutoZone and Nordstrom showed a block page or a page of script to every tool.
- curl_cffi got the furthest. It loaded five shops, and it was the only tool past GameStop's Cloudflare wall. Its README says it copies browsers' "TLS/JA3 and HTTP/2 fingerprints".
- The User-Agent decided two shops. At Best Buy, curl, requests, httpx and Go got through only with Chrome's User-Agent. At Whatnot, Playwright got through only after we replaced the HeadlessChrome in its User-Agent.
- A real browser opened no extra doors. Playwright's headless shell and Chrome's new headless mode each loaded three shops, as many as requests with a Chrome User-Agent.
- Two results we cannot explain. Scrapy with Chrome's User-Agent got no answer from Best Buy, and with its own it got the page. Go with Chrome's User-Agent passed Whatnot's check, and the other plain clients did not.
What each tool tells the site
Every request carries a User-Agent header, and most tools fill it with their own name. We sent each tool to a test server of our own that echoes back what it receives.
| Tool | User-Agent it sent |
|---|---|
| curl 8.5.0 | curl/8.5.0 |
| requests 2.34.2 | python-requests/2.34.2 |
| httpx 0.28.1 | python-httpx/0.28.1 |
| Scrapy 2.19.0 | Scrapy/2.19.0 (+https://scrapy.org) |
| Node's fetch | node |
| Go's net/http | Go-http-client/1.1 |
| Playwright's Chromium | a Chrome string with HeadlessChrome/153.0.8010.12 |
| curl_cffi as Chrome | a Chrome 150 string, with Chrome's full set of headers |
The first six announce a program, and a site can refuse them on that header alone. Best Buy did: requests with its own User-Agent got "Access Denied", and with Chrome's it got the shop. Playwright's default browser names itself too. It is "a separate chromium headless shell", and its User-Agent says HeadlessChrome.
The headers are only part of it. A site can also read how a client opens the encrypted connection. Cloudflare's documentation says these fingerprints "identify TLS clients based on how they initiate connections". curl_cffi's README says it "can impersonate browsers' TLS/JA3 and HTTP/2 fingerprints".
The best tool for each job
Each row below comes from a test on one of our own pages, linked in the last column.
| Job | Tool | What our test found | Our test |
|---|---|---|---|
| Pages in Python | requests with selectolax or lxml | 100 pages in 2.62 and 3.06 s. BeautifulSoup in threads: 9.57 s. | best language |
| Whole sites in Python | Scrapy | 2,000 products in 1.80 s. requests with BeautifulSoup: 6.67 s. | Python libraries |
| Polite crawls | Scrapy, a new project | 111 requests in 136.7 s, one at a time. With 8 at once: 4.1 s. | Scrapy |
| Pages built by JavaScript | Playwright | The default goto() came back with 0 of 10 products. | Playwright waits |
| A browser from older code | Selenium | 50 pages in 36.87 s. Element by element: 341.16 s. | Python libraries |
| Scraping in JavaScript | fetch and Cheerio | 50 pages in 0.27 s. Puppeteer: 18.87 s and 785 MB. | JavaScript |
| Markdown for AI tools | Crawl4AI | The only one of seven crawlers that ran the page's script. | open-source scrapers |
| Big crawls on small servers | Go with goquery | 14 MB of memory, the least of all. | best language |
Paid services and point-and-click apps
Most lists of the best scraping tools come from companies that sell one. We read six of them. Four put their own product first, and only one described how it tested.
This page covers tools you run yourself. We did not test paid scraping services or point-and-click apps.
Where proxies come in
Every answer above went to our server's datacenter address. In our August study, nine shops loaded for a home connection and blocked or challenged the same request from a datacenter server. A tool with a browser's fingerprint and a home address are two separate fixes.
Our page on datacenter and residential proxies compares the two kinds of address. Our residential proxies start at $0.44/GB. Before you scrape a shop, read its terms and its robots.txt file. Our page on whether web scraping is legal covers the rules.
What this page could not check
- One server with a datacenter address, and two passes of one request per tool and site. Bot defenses change, and they can react to the earlier requests of the same test.
- Home pages only. Pages deeper in a site can be guarded more closely.
- No paid services, no point-and-click apps and no proxies in the wall test.
- One version of each tool: curl 8.5.0, requests 2.34.2, httpx 0.28.1, Scrapy 2.19.0, Node 20.20.2, Go 1.27.2, curl_cffi 0.16.3 and Playwright 1.63.0.
- We will check this page again by 12 January 2027.
Sources
- curl_cffi README: github.com.
- Cloudflare, JA3/JA4 fingerprint: developers.cloudflare.com.
- Scrapy settings, USER_AGENT: docs.scrapy.org.
- Playwright for Python, browsers and headless mode: playwright.dev.
- Our own tests on our server, 12 October 2026: run_walls.py from 12:31 to 12:56 UTC in two passes, run_ua.py at 12:12 UTC, and the earlier pages linked in the table above.
- Our study of the bot defenses of 52 US retailers, 24 August 2026: hproxy.com.
