Open source web scrapers: what each project is good for

An open-source web scraper is a tool whose code anyone can read, change and run. There are many of them, and they do different jobs. Some crawl whole sites, some drive a browser, some turn pages into text for AI tools, and some copy a site to disk.

On 12 October 2026 we read the records of 18 projects: licence, latest release and GitHub stars. Then we ran seven crawlers with their defaults on a small test site of our own, and logged every request they sent.

The short answer

• Crawl a site in Python: Scrapy. Start with scrapy startproject, because a project obeys robots.txt and a single script does not.
• Crawl in JavaScript or Python, with a browser when needed: Crawlee.
• Turn pages into Markdown for an AI tool: Crawl4AI. It was the only crawler in our test that ran the page's script.
• Pull the main text out of articles: trafilatura, the most careful crawler in our test.
• Pages that only work in a browser: Playwright, Puppeteer or Selenium.
• Copy a site to disk: wget or HTTrack.

18 projects at a glance

The figures come from GitHub, PyPI and npm, read on 12 October 2026 at 06:56 UTC. Stars measure attention, not quality, and they change every day.

GitHub could not name Heritrix's licence, so we read the file itself. It says "Licensed under the Apache License, Version 2.0".

Stars and downloads tell different stories. trafilatura has fewer GitHub stars than Scrapy, Scrapling, Crawl4AI or Crawlee for Python, but more PyPI downloads than all of them. A package that other tools install for you is counted each time.

What seven crawlers do by default

We built a small site on our own server. Its home page links to ten product pages, a staff page under /private/ and a page whose text appears only after its script runs. Each product page takes 300 ms to answer. Its robots.txt forbids /private/ and asks for a Crawl-delay of 2 seconds.

Each crawler started at the home page with its defaults. The site logged every request it received.

Five things stand out.

• Robots.txt. Scrapy run as a single script, both versions of Crawlee, Crawl4AI and Scrapling never asked for robots.txt. They fetched the forbidden page too. A Scrapy project, trafilatura and wget read the file and skipped it.
• Load. Crawlee for Python had 9 requests in flight at once, Crawlee for JavaScript 7, the Scrapy script 6, Crawl4AI 5 and Scrapling 4. The three that obeyed robots.txt sent one request at a time.
• Crawl-delay. Only trafilatura kept to it. After its first two pages, it waited 2.3 to 2.4 seconds before each request. A Scrapy project waited about 1 second, and wget sent 7 page requests within 2 seconds.
• JavaScript. Only Crawl4AI drives a real browser, and only Crawl4AI saw the sentence the page's script writes.
• Names. Scrapy, trafilatura and wget named themselves. The others sent the User-Agent of a browser.

The Scrapy runs show 14 requests for 13 pages because Scrapy asked for the home page twice. Its source creates start requests with dont_filter=True, so its duplicate filter lets the home page through once more.

That is by design. Crawlee for Python's documentation says "The default HTTP client is ImpitHttpClient". The impit README says the library "allows you to impersonate a browser". A site that blocks crawlers by name will not see these as crawlers.

One crawler needed a change before it could run at all. trafilatura sent no request to our first try, because it blocks non-public addresses by default. Its settings file reads "block requests to non-public IP addresses (loopback, private, link-local)" and "set to "off" for intranet use cases". Our test site runs on the server's own loopback address, so we switched that one setting off for its run.

Turning robots.txt on

The five crawlers that ignored robots.txt each have a documented switch. We ran them again with only that switch changed. All five read robots.txt and skipped /private/.

None of them kept to the Crawl-delay of 2 seconds. Scrapling came closest. Its documentation says the switch makes it "Respect Crawl-delay and Request-rate directives". It also says the switch "does not affect your concurrency settings". So each of its four parallel requests waited, and four still arrived within 2 seconds.

That is allowed. Crawl-delay is not part of the robots.txt standard, RFC 9309. The standard says crawlers "MAY interpret other records that are not part of the robots.txt protocol". If a site's delay matters to you, set a delay in the crawler itself.

Scrapy's documentation explains the split between a script and a project. "While the default value is False for historical reasons, this option is enabled by default in settings.py file generated by scrapy startproject command". The same project file sets CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1. That is why the project ran one request at a time.

With its switch on, Crawl4AI fetched robots.txt under a different name than its pages: Python/3.12 aiohttp/3.14.5. It also keeps a copy. A later run reused the cached file and never asked our site again, so we gave every run a fresh home folder.

How big each install is

We installed each Python tool into its own virtual environment and measured the folder, pip included. Crawlee for JavaScript is the size of its node_modules folder. Crawl4AI also downloaded a browser of 658 MB through Playwright.

What each project is good for

Crawling a site in Python

Scrapy calls itself "a fast high-level web crawling & scraping framework for Python". Its behaviour depends on how you start it. A single script ignores robots.txt, and a project from scrapy startproject obeys it and waits a second between requests.

Scrapling calls itself "An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl". Its documentation says "If you've used Scrapy before, the patterns will feel familiar". By default its spiders run four requests at once and ignore robots.txt, which the documentation calls "turned off by default to avoid surprising behavior".

Crawlee for Python describes "A web scraping and browser automation library for Python to build reliable crawlers". In its own words, it "Works with Parsel, BeautifulSoup, Playwright, and raw HTTP".

Crawling in JavaScript or Go

Crawlee began in JavaScript, and that version is still the larger one by stars. Colly is an "Elegant Scraper and Crawler Framework for Golang". We did not run it, but its source sets c.IgnoreRobotsTxt = true and the User-Agent colly - https://github.com/gocolly/colly when a collector starts. Katana, also in Go, calls itself "A next-generation crawling and spidering framework".

Pages for AI tools

Crawl4AI promises "any website into clean, LLM-ready Markdown". It drove a real browser in our test, and it was the slowest crawler without a delay. It also marked 12 of our 13 pages "Blocked by anti-bot protection", although our server answered every one. Its message gave the reason as "minimal_text on small page", and the text was still in its results.

Firecrawl does similar work under the AGPL-3.0. We did not run it. Its README says "By default, Firecrawl respects robots.txt directives". Its self-hosting guide lists what the free stack lacks. Fire-engine, with "its advanced anti-bot behavior", is a separate service, and "it is not included". Screenshots and page actions are "Not available in the default stack".

ScrapeGraphAI, a "Python scraper based on AI", lets a language model pick the data. trafilatura is a "Python & Command-line tool to gather text and metadata on the Web". Its documentation says its spider "implements politeness rules as defined by the Robots Exclusion Standard", and our test bore that out.

Pages that need a browser

Playwright, Puppeteer and Selenium drive a real browser. Playwright calls itself "a framework for Web Testing and Automation", and Selenium "A browser automation framework and ecosystem". They do not follow links on their own. You write that part yourself, or let Crawlee do it: its JavaScript version "Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP".

Archives and search-engine crawls

Heritrix is "the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project". Apache Nutch calls itself "an extensible and scalable web crawler", and StormCrawler "A scalable, mature and versatile web crawler based on Apache Storm". All three are written in Java for large crawls, and we did not run them.

Copying a site

GNU Wget's manual says "Wget honors RES when downloading recursively", where RES is the robot exclusion standard. In our test it read robots.txt and skipped /private/, but it did not wait between files. HTTrack calls itself a "Website Copier" to "copy websites to your computer". We did not run it.

Forms and logins without a browser

MechanicalSoup is "A Python library for automating interaction with websites". Its README says it "can follow links and submit forms", and adds "It doesn't do JavaScript".

Licences: read them before you build on one

Most of the 18 projects use permissive licences: Apache-2.0, MIT or BSD-3-Clause. Two do not. HTTrack is under the GPL-3.0. Firecrawl's README says the project "is primarily licensed under the GNU Affero General Public License v3.0 (AGPL-3.0)". It adds that "The SDKs and some UI components are licensed under the MIT License".

The AGPL matters if you change Firecrawl and run it as a service for others. Its section 13 covers that case. The modified version "must prominently offer all users interacting with it remotely through a computer network" a way to get its source code. If a licence matters to your plans, read it in full or ask a lawyer.

Through a proxy

Most of these tools take a proxy in a setting or a launch option. Our guides show the setup for Scrapy, Crawlee, Colly and Firecrawl. Others cover Playwright, Puppeteer, Selenium and wget.

Check the defaults above before you add one. A crawler that ignores robots.txt and keeps ten requests in flight puts more load on a site than one that asks first and waits.

What this page could not check

• One small test site of 13 pages, on our own server. Real sites are larger and slower.
• One run per setting. The number of requests in flight can differ on another machine.
• We ran 7 of the 18 projects. The others were read from their own records and documentation, not run.
• trafilatura blocks non-public addresses by default. We switched that one setting off to reach our test site.
• Install sizes are for one Linux server with Python 3.12.3 and Node 20.
• Stars, downloads and releases change every day. We will check this page again by 12 January 2027.

Sources

• The 18 repositories, read through GitHub's REST API: Scrapy, Crawlee for Python, Crawlee, Crawl4AI, Firecrawl, trafilatura, Scrapling, ScrapeGraphAI, MechanicalSoup, Colly, Katana, Playwright, Puppeteer, Selenium, Heritrix, Apache Nutch, StormCrawler, HTTrack.
• PyPI and pypistats for Python packages, the npm registry for JavaScript packages: pypi.org, pypistats.org, npmjs.com.
• Scrapy documentation, settings: docs.scrapy.org.
• Crawl4AI documentation, browser and crawler parameters: docs.crawl4ai.com.
• Scrapling documentation, spiders: scrapling.readthedocs.io.
• trafilatura documentation, crawls: trafilatura.readthedocs.io.
• Crawlee for Python documentation, HTTP clients, on crawlee.dev, and the impit README: github.com.
• Colly source, colly.go: github.com.
• Firecrawl README (licence) and its self-hosting guide: github.com.
• GNU Wget manual: gnu.org.
• RFC 9309, Robots Exclusion Protocol: rfc-editor.org.
• GNU Affero General Public License v3: gnu.org.
• Our own tests on our server, 12 October 2026: project_facts.py at 06:56 UTC, the installs at 07:03 UTC and crawl_runs.py at 07:34 UTC, on a test site of our own.