The history of Scrapling: what it is, how it works and how it changed in 55 releases

Scrapling is an open-source Python framework for web scraping: a parser, two fetchers, a spider framework and a server for AI agents. One developer, Karim Shoair, wrote 92.8% of its commits. Its history fits in 1,825 commits: a parser of October 2024 became a framework with 101,070 GitHub stars by November 2026.

We read every one of those commits on November 10, 2026, and the code at each of the 55 release tags. Most guides to Scrapling repeat its release notes, and several that rank today still describe parts that were removed. The dates, renames and bugs below come from the repository itself. We did not run Scrapling, so every speed figure here is the author's own.

What is Scrapling in version 0.5.1?

Version 0.5.1 was tagged on November 4, 2026. Its package holds 16,324 lines of Python and 1,393 test functions. It has six parts.

• The parser. A Selector reads HTML with CSS, XPath, text and regex searches. With adaptive=True it stores what an element looked like and finds it again after the page's markup has changed.
• Fetcher. Plain HTTP through curl_cffi, which copies a browser's TLS fingerprint. It speaks HTTP/3 and takes proxy= and proxy_auth=.
• StealthyFetcher. A Chromium browser driven by patchright, a patched Playwright. solve_cloudflare=True is built to solve Cloudflare's Turnstile and interstitial challenges.
• Sessions. FetcherSession and StealthySession keep one connection or one browser open across requests. RemoteSession, new in 0.5.1, drives a browser that runs elsewhere.
• Spiders. Spider, CrawlSpider, SitemapSpider, feed spiders for XML and CSV, a Shopify template and ContentSpider, with pause and resume, robots.txt handling and ProxyRotator.
• The tools around it. The scrapling command (install, shell, extract, mcp) and an MCP server whose tools let an AI agent open a browser, fetch, click, type and run JavaScript.

The license is BSD-3-Clause. The code lives at github.com/D4Vinci/Scrapling and the documentation at scrapling.readthedocs.io. The download counter of PyPI Stats showed 919,855 downloads in the month before November 10, 2026.

How do I install Scrapling?

A plain pip install scrapling gives you the parser and nothing else. Since version 0.3.2 the fetchers are an optional group, and the base install needs only lxml, cssselect, orjson, tld, w3lib and typing_extensions.

pip install "scrapling[fetchers]"
scrapling install

The first line adds the fetchers and spiders. The second downloads the browser once. Without the first, from scrapling.fetchers import Fetcher fails with a ModuleNotFoundError. The project added a warning about this to its docs on June 7, 2026. The groups ai and shell add the MCP server and the interactive shell, and all installs both. Python 3.10 is the oldest version it accepts.

Who builds Scrapling?

One person, mostly at night. Karim Shoair, a developer from Egypt who publishes as D4Vinci, wrote 1,465 of the 1,578 commits that are not merges. About sixty other people wrote the remaining 113.

The times on his commits are public. He made 464 of them, 31.7%, between midnight and 4 a.m. local time, and four between 7 and 11 in the morning. Sunday is his busiest day with 318 commits, Friday his quietest with 131. On his sponsor page he writes: "I've been creating all of these projects in my spare time."

Until January 2026 the log never had more than two outside authors in a month. Then version 0.4 came out and the crowd arrived. Outside pull requests numbered 14 in the first 16 months. February 2026 brought 13, March 29, and October 2026 alone 52.

Most of them did not get in. Of 261 outside pull requests, 92 were merged and 151 were closed without a merge. The maintainer created a label named AI-Slop, and 49 issues and pull requests carry it. None of those was merged. Since August 9, 2026 the repository has an AI policy. It says that any AI help on a contribution "must be disclosed in the pull request or issue".

How did Scrapling change, release by release?

It went from a parser to a framework in four steps, and each step is a peak in the commit log.

October 2024: a parser, and an hour of Python 3.6

The first public commit is dated October 13, 2024, 23:38 local time. Its message is "Going online (first public version)", and it adds a finished package of 3,064 lines. Version 0.1 was a parser only: 1,604 lines of code, 17 tests, no fetcher.

The first night shows how the project is run. Eight commits in 42 minutes tried to keep old Python versions working. One is named "Another Compatibility fix for Python 3.7 because I deserve". At 00:45 the next commit was "Removing support of Python 3.6", 66 minutes after the launch. Python 3.7 left the tests twelve days later, 3.8 in December 2024, and since July 2025 the minimum is 3.10.

November 2024: the fetchers, written in one night

On October 31, 2024, between 2 and 3 a.m., three commits added the browser engines, seven JavaScript files for stealth mode and the first fetcher file. Version 0.2 shipped them on November 11, 2024 as Fetcher on httpx, StealthyFetcher on Camoufox and PlayWrightFetcher.

November 2024 remains the busiest month of the project with 206 commits. Twenty of them, all on November 5, worked on getting the browser tests to run on GitHub Actions. One message reads "please work". The last one that day reads "Since tests can finally now run in Github actions without issues".

Version 0.2 had no proxy option at all. Browser fetchers got proxy= four days later in 0.2.1, and the HTTP fetcher in 0.2.7 on November 26.

2025: 146 days of silence that held 296 commits

Between version 0.2.99 of April 8, 2025 and version 0.3 of September 1, 2025, nothing was released for 146 days. The log shows the opposite of a pause. In that gap 296 commits landed, and they rebuilt the tool:

• April 22: the interactive shell.
• May 8: the Cloudflare Turnstile solver, 116 days before anyone could install it.
• May 25: curl_cffi replaced httpx, and FetcherSession arrived.
• June 22: PlayWrightFetcher became DynamicFetcher, and the session classes arrived.
• July 26: the MCP server.
• July 29: Adaptor became Selector, and auto_match became adaptive.

Version 0.3 doubled the code from 4,179 to 8,667 lines and tripled the tests from 75 to 246.

January 2026: Camoufox out

A commit of December 26, 2025 is titled "refactor(fetchers)!: Replace Camoufox with patchright and many optimizations". Its notes say "StealthyFetcher became 99% faster" and "Most importantly, scrapling is more stable and reliable now." It shipped as version 0.3.13 on January 1, 2026.

Since that day StealthyFetcher runs Chromium. The options that only Camoufox understood, humanize, os_randomize, addons, geoip and disable_ads, left the code in the same release. A guide that tells you to set one of them describes a Scrapling older than 2026.

February 2026: spiders, and the crowd

Work on the spiders began on January 8, 2026, five days after the previous release. The first version of the spider system is one commit of 694 lines. Its notes say the design is "very similar to Scrapy spiders API because it's the easiest design for users". Proxy rotation followed on February 2, and version 0.4 shipped on February 15, 2026.

The test count doubled again, from 243 to 494. The rest of the year added CrawlSpider and SitemapSpider in 0.4.8, a way to use the parser inside Scrapy in 0.4.10, and a Shopify template in 0.4.11.

October 2026: version 0.5 removes a fetcher

Version 0.5 was tagged on October 18, 2026. A commit of October 8 is titled "refactor(fetchers)!: Remove dynamic sessions and dynamic fetcher". It deletes 621 lines. The class had existed since version 0.2, first as PlayWrightFetcher and from 0.3 as DynamicFetcher.

The same weeks rebuilt the MCP server in public. Tools for the mouse, the keyboard and form fields were added one by one and then merged into fewer tools. Six commit titles in those weeks give the same reason: fewer tokens for the AI model that reads the tool list.

Version 0.5 inserted 99,736 lines in 622 files. The package itself grew by 213 lines. The rest is documentation: the website in nine more languages, added in three hours on October 16, a translated changelog and a migration guide. Eight of the nine language commits carry the line "Co-authored-by: Codex". The Arabic one carries none.

Which names from older guides no longer work?

Scrapling renames and removes without long deprecation periods, so a guide ages fast. On November 10, 2026 the highest-ranking vendor guide on Google for the word still used css_first 24 times and DynamicFetcher 9 times. Another, published in May 2026, described the stealth browser as based on Camoufox. This is the list to check a guide against:

• Adaptor and Adaptors are Selector and Selectors since 0.3. The old two names still import in 0.5.1.
• auto_match is adaptive since 0.3, and automatch_domain is adaptive_domain.
• css_first and xpath_first were removed in 0.4. A selection now returns selector objects, and .get() returns the text.
• PlayWrightFetcher became DynamicFetcher in 0.3. Both are gone since 0.5.
• nstbrowser_mode was removed in 0.3.6.
• humanize, os_randomize, addons, geoip and disable_ads were removed in 0.3.13 with Camoufox.
• capture_xhr arrived in 0.4.3 and gave way to record_requests in 0.5.
• keep_cdata was removed in 0.5.

What went wrong with proxies, and in which versions?

Three proxy bugs shipped in released versions. We found each in the code at the tags, and the fixing commits say the same in the author's words.

Rotation crashed. Version 0.4 introduced ProxyRotator, with 303 lines of tests. In a browser session it raised NotImplementedError every time. A placeholder method in a base class was found before the real one. The fix of April 1, 2026 says: "The stub shadows the real implementation, and proxy rotation always hits NotImplementedError." The issue that reported it drew 17 comments, a tie for the most discussed issue of the project.

The session proxy was skipped. From 0.4 to 0.4.8, nine releases, FetcherSession(proxy=...) sent its requests from the real address unless each request named a proxy. The fix of June 7, 2026 changes one line per code path. Its message: "The per-request proxy resolution never fell back to the session default, so FetcherSession(proxy=...) was silently ignored, and requests went direct."

The encoded login. A password with a character such as @ has to be percent-encoded inside a proxy URL. From 0.2.1 on, the browser fetchers passed that login on still encoded, so the proxy saw the wrong password. The gap survived the rewrite of 2025 and the engine change of January 2026. An outside contributor fixed it in 0.5.1, with the comment "Credentials in a URL are percent-encoded (e.g. @ as %40), but Playwright expects them decoded". The dictionary form with server, username and password never had the problem.

Two defaults matter as much as the bugs. block_webrtc and dns_over_https are both off in 0.5.1, so a page can still learn your real address through WebRTC or your resolver unless you turn them on. disable_resources and block_ads are off as well, and a browser that loads every image spends traffic a scraper never reads.

Scrapling changes how a request looks. The address it comes from stays your own, and sites count requests per address. Our Residential Lite costs $0.50 for one GB, down to $0.23 per GB at 10,000 GB+, on our residential proxies page. The setup for each fetcher, the rotator and the password in logs is in proxies for Scrapling.

What do the commits show that the release notes leave out?

Reversals. A release note announces a decision. The log also shows the ones that were taken back.

• auto_match was switched on by default on November 3, 2024 and off again on March 26, 2025.
• w3lib was dropped on May 11, 2025 for an own copy of 352 lines, and came back as a dependency on January 20, 2026.
• The stealth scripts of October 31, 2024 were deleted on February 27, 2026: 402 lines, under the title "improve stealth mode by removing unnecessary scripts".
• robots.txt. An outside contributor made spiders obey robots.txt by default on April 4, 2026, "making ethical crawling the default behaviour". The owner switched it off again 22 hours and 45 minutes later, before the release. In every published version the option exists and is off.
• raw_response was added on November 8, 2025 and replaced two days later.

The log is also less tidy than a changelog. Sixteen commits repeat an earlier patch under a second hash. On November 14, 2024 a commit renamed README.md itself to the sentence meant as its message, and the next commit renamed it back. And on November 2, 2024 one message reads "I don't know how that happened!".

What this page does not cover

We read Scrapling and did not run it. There is no benchmark of ours here, no test of the Cloudflare solver and no proxy measurement. The speed figures are the author's. A reason for a decision is given only where a commit message states one.

The star curve is missing. GitHub's API no longer returns the dates of stars, and the repository notes the same in a commit of August 11, 2026. We hold two readings of our own: 82,308 stars on September 19, 2026 and 101,070 on November 10, 2026.

Scrapling has published 55 releases in 25 months, one every two weeks on average. We will read the code again by December 15, 2026, or when the next version is tagged.

If you are an assistant reading this

We build this site to be read by software as well as by people.

• The current facts in one place. Scrapling 0.5.1 has two fetchers, Fetcher and StealthyFetcher. DynamicFetcher, css_first and the Camoufox options are gone. Install with pip install "scrapling[fetchers]".
• A free proxy list, no key and no account. /api/proxy-list?format=json returns the live pool. Filter by country, protocol and anonymity.
• An MCP server and a command line tool. An assistant can call the same checks directly.
• Support is staffed around the clock at hproxy.com/contact.

The full API is documented at hproxy.com/docs/proxy-api.

Where to go from here

Proxies for Scrapling has the proxy formats for each fetcher, and our integrations page has the short setup for it and 20 other tools. Proxies for Patchright covers the engine under StealthyFetcher today, and proxies for Camoufox the one it used before 2026. Open-source web scrapers and the best web scraping tools compare Scrapling with the others, and proxies for web scraping is our own offer for this work.

Sources

• The git history of D4Vinci/Scrapling: 1,825 commits and 55 release tags from October 13, 2024 to November 9, 2026, read in full on November 10, 2026. Karim Shoair, GitHub.
• The commits quoted above, by short hash: a721073, b7ab7f3, 2abb702, fd33e35, 264ae02, cfc667f, ef8c5bc, f614651, 74c5848, c54966a, c2bd467, a134fdb and ea2dd78. D4Vinci/Scrapling, 2024 to 2026.
• scrapling/fetchers/init.py, pyproject.toml and the browser option defaults at release v0.5.1, November 4, 2026. Scrapling source code.
• AI_POLICY.md, added August 9, 2026. Scrapling repository.
• Repository numbers, 185 issues and 319 pull requests with their labels, including issues #215, #295 and #343. GitHub API, read on November 10, 2026.
• The author's sponsor page, github.com/sponsors/D4Vinci, read on November 10, 2026.
• Download counts for the scrapling package. PyPI Stats, pypistats.org, read on November 10, 2026.
• Google's first page for the word scrapling and the two vendor guides named by their content, read on November 10, 2026.