Skip to content

The history of Crawl4AI: what it is, how it works and how 30 months of commits made it

The history of Crawl4AI from 1,792 commits: what 0.9.5 contains, its defaults, proxy support by release and the cloud banner in the library.

Billal AinaouiNovember 16, 2026Updated November 16, 202615 min read

The history of Crawl4AI: what it is, how it works and how 30 months of commits made it

Crawl4AI is an open-source Python crawler that turns web pages into Markdown for language models, first committed in May 2024 and now at version 0.9.5. Its history is short and fast: a Selenium script grew into a crawler with 87,192 GitHub stars in 30 months. Since September 2026 the library also advertises its maker's paid cloud.

We read the subject of every one of its 1,446 commits on the main branch that are not merges, on November 10, 2026, and checked the stories below in the code at the release tags. Release notes were used only to quote the project's own words. We did not run Crawl4AI for this page.

What is Crawl4AI in version 0.9.5?

Version 0.9.5 was tagged on October 14, 2026. One class, AsyncWebCrawler, loads a page and returns it as Markdown, cleaned HTML and structured data. These parts sit around it:

  • Two fetchers. The default one drives a real browser through Playwright. AsyncHTTPCrawlerStrategy sends plain HTTP requests and starts no browser.
  • Markdown filters. PruningContentFilter and BM25ContentFilter score each block of text and drop the weak ones before a model reads the page.
  • Extraction. JsonCssExtractionStrategy reads fields with CSS selectors, and LLMExtractionStrategy hands the page to a language model.
  • Deep crawling. Three strategies follow links: breadth first, depth first and best first.
  • A server and a command. A Docker image serves the crawler over HTTP with an MCP bridge for assistants, and the crwl command runs it from a terminal.

It needs Python 3.10 or newer and carries the Apache-2.0 license. Its package file lists 34 dependencies, among them Playwright, patchright and a pinned fork of litellm. The code lives at github.com/unclecode/crawl4ai. On November 10, 2026, GitHub counted 87,192 stars and 9,087 forks, and PyPI Stats counted 1,275,996 downloads in the month before.

How do I install Crawl4AI?

pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

The first line installs the package, the second downloads the browser once, and the third checks the result. The project's README then shows a first crawl:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown)

asyncio.run(main())

The crawl4ai-doctor command dates from January 1, 2025. The log of that day holds 25 commits about installing, packaging and version numbers. The founder wrote 24 of them between 3:15 and 7:47 p.m. his time.

What does Crawl4AI do by default?

The defaults explain most surprises. We read them in the configuration file of 0.9.5.

  • headless is True, and browser_type is "chromium".
  • enable_stealth is False: the browser is plain Playwright Chromium.
  • proxy_config is None: requests leave from your own address.
  • check_robots_txt is False: robots.txt is not read.
  • cache_mode is CacheMode.BYPASS: every call loads the page again.
  • max_retries is 0: a blocked page comes back as it is.

With check_robots_txt=True, a page that robots.txt forbids comes back with status 403 and the message "Access denied by robots.txt". Three options for sites that block bots are off as well: magic, simulate_user and override_navigator.

Who builds Crawl4AI?

One founder started it, and he commits under the name UncleCode. He wrote 922 of the 1,446 commits, or 63.8%. His share falls every year: 410 of 451 commits in 2024, 349 of 622 in 2025 and 163 of 373 in 2026. The 32 commits of August 2026 came from six other authors.

A second developer with a mail address on the same company domain wrote 190 commits. In total 106 mail addresses appear as authors. Ten commits name an AI coding tool as their author: eight Claude, one Cursor Agent and one Devin.

Outside work is larger than GitHub's counters suggest. Of 952 pull requests, 296 are marked merged, 388 closed without a merge and 268 open. A closed one is often an accepted one. On March 7, 2026, the founder made 23 commits, most of them applying outside pull requests in batches. Their subjects name 30 pull requests, and GitHub marks 29 of those as closed. The authors were added to a CONTRIBUTORS file.

Commits to Crawl4AI per month, May 2024 to November 2026Commits on the main branch without merges, by author date. November 2026 runs to the 4th.
November 2024 is the peak with 123 commits, and May 2026 is the quietest full month with 7.
Source: HProxy count from the git log of unclecode/crawl4ai, 1,446 commits, read on November 10, 2026 · hproxy.comProxy.

How did Crawl4AI change in 30 months?

May 2024: a Selenium script and a heavy first week

The first commit is dated May 9, 2024: 19 files and 2,296 lines. The folder was named crawler, and a commit of the same evening renamed it to crawl4ai. The browser was Chrome, driven through Selenium. The first day alone holds 21 commits.

The first week was heavy. One commit of May 12 added 84,568 lines. Five days later a commit titled "Remove dependency on Spacy model." removed 83,963. Two days after that, 61,432 lines came in with an embedding model for CPUs. The first tag, v0.1.0 of May 17, 2024, holds 747 lines of Python in seven files.

September 2024: async and Playwright

Version 0.3.0 of September 28, 2024 set the shape the tool still has. A commit of September 3 is titled "Add Async Version, JsonCss Extrator", and a second of September 24 added 6,224 lines in 61 files. Together they moved the crawler to asyncio and Playwright, and the requirements file swapped selenium for playwright. The old Selenium class still ships in 0.9.5, in a folder named legacy.

Late 2024: the busiest quarter

November 2024 holds 123 commits, the most of any month. The package grew from 4,716 lines in version 0.3.0 to 15,199 in version 0.4.24 on December 31, 2024. CrawlerRunConfig, the object that every call takes today, dates from December 12, 2024.

2025: features in every direction

Version 0.5.0 of March 4, 2025 brought deep crawling, the command line tool, the HTTP fetcher, proxy rotation and the robots.txt check. Its tag is named v0.5.0.post1.

The repository then became a workshop. An MCP server arrived on April 21, a LinkedIn prospecting app on April 30, a scripting language named C4A-Script on June 6, a Chrome extension on June 8 and a marketplace on October 2. Version 0.7.3 of August 9, 2025 added stealth mode and an undetected browser built on patchright. By version 0.7.8 in December, the package held 43,159 lines.

2026: a year of hardening

The year opened with two security fixes. A commit of January 12 is titled "Fix critical RCE and LFI vulnerabilities in Docker API deployment", and version 0.8.0 shipped it four days later. A commit of January 30 is titled "Fix critical RCE via deserialization and eval() in /crawl endpoint". Its first version tag is 0.8.5 of March 18.

Both concern the Docker server, the part that takes requests from other machines. More security commits followed. One of March 31 is titled "fix(security): complete AST sandbox escape remediation (CVSS 9.8)". One of April 20 blocks requests to internal addresses, and one of June 4 is titled "security: patch proxy SSRF in Docker server (0.8.9)". Version 0.9.0 followed on June 18, and its merge commit calls it "secure-by-default Docker server hardening".

On March 24 a dependency changed. The commit is titled "fix: replace litellm with unclecode-litellm due to PyPI supply chain compromise", and version 0.8.6 was tagged the same day. Version 0.9.5 still pins that fork.

The numbers show the effort. The package stopped growing: 44,855 lines in 0.8.0 and 47,251 in 0.9.5. The tests tripled in the same time, from 639 test functions to 1,994.

Test functions in Crawl4AI at six releasesFunctions whose name starts with test_ in the tests folder at each release tag.
The tests tripled between 0.8.0 and 0.9.5 while the package stayed near 47,000 lines.
Source: HProxy count at the release tags of unclecode/crawl4ai, read on November 10, 2026 · hproxy.comProxy.

In September 2026 the project launched a hosted service. The section after the next one shows what that changed in the library.

How did proxy support grow, and where did it fail?

Proxy support in Crawl4AI, release by release
  1. May 2024

    First code

    no proxy option

  2. Sep 2024

    Version 0.3.0

    first option, one browser flag

  3. Mar 2025

    Version 0.5.0

    proxy per run, rotation

  4. Oct 2025

    Version 0.7.5

    proxy deprecated for proxy_config

  5. Jan 2026

    Version 0.8.0

    HTTP fetcher, sticky sessions

  6. Mar 2026

    Version 0.8.5

    direct first, then proxy

  7. Oct 2026

    Version 0.9.5

    encoded logins fixed

Source: HProxy reading of the git log and release tags of unclecode/crawl4ai, November 10, 2026 · hproxy.comProxy.

For its first 109 days the tool had no proxy option. Issue #52 asked on July 9, 2024 how to use one. On August 26 an outside contributor added it in a commit titled "add proxy and add ai base_url". The change hands Chrome the flag --proxy-server, and it shipped in version 0.3.0.

A login came next. Issue #197 asked for a user name and password on October 22, 2024, and a commit of the same day added proxy_config with both fields. Rotation and a proxy per run followed in version 0.5.0. Version 0.8.0 added sticky sessions, where proxy_session_id keeps one proxy for a group of requests.

Three times a proxy setting did something else than the user wrote.

What failedReleasesFixed in
Proxy on the HTTP fetcher0.5.0 to 0.7.80.8.0
Proxy login with a saved profilereported March 20250.8.5
Login with special charactersup to 0.9.40.9.5

The HTTP fetcher took no proxy. The fetcher without a browser shipped in version 0.5.0 on March 4, 2025. Issue #1534 of October 5, 2025 is titled "[Bug]: proxy configuration is ignored when using AsyncHTTPCrawlerStrategy". The fix, "Add proxy support to HTTP crawler strategy", shipped in version 0.8.0 on January 16, 2026, 318 days after the fetcher did.

A saved profile dropped the login. Issue #905 of March 27, 2025 reported that a proxy "is ignored when use_persistent_context=True". The fix of February 12, 2026 names the cause. Crawl4AI wrote the login into Chrome's --proxy-server flag, and Chrome ignores a login there, "causing proxy auth to fail and the browser to fall back to direct connection". By that account, a crawl set up with a proxy left from the user's own address. The fix shipped in version 0.8.5, 356 days after the report.

Special characters broke the login. Issue #2326 of October 1, 2026 is titled "[Bug]: HTTP crawler puts proxy credentials into the URL unencoded". Two commits repaired both directions. One encodes the login when the HTTP fetcher builds its proxy URL, and one decodes an encoded login in ProxyConfig.from_string. A third keeps the login out of error messages. All three shipped in version 0.9.5.

Old names live on in guides. The parameter proxy is deprecated since version 0.7.5 of October 21, 2025, and its warning says "Use 'proxy_config' instead." Of 15 guides about Crawl4AI in our competitor database, 4 mention proxy_config. One guide of June 2026 sets use_stealth_mode=True, a name that never existed in the package. The option is called enable_stealth.

Since 0.8.5 a crawl can try without a proxy first. proxy_config on a run takes an ordered list, and the entry ProxyConfig.DIRECT means no proxy. The crawler tries each entry in turn and checks every result for a block page, with patterns for Akamai, Cloudflare and other vendors. It stops at the first result that passes. max_retries repeats the whole list, and a function of your own can serve as the last resort.

from crawl4ai import CrawlerRunConfig, ProxyConfig

config = CrawlerRunConfig(
    max_retries=1,
    proxy_config=[
        ProxyConfig.DIRECT,
        ProxyConfig(
            server="http://example.hproxy.com:8080",
            username="user",
            password="pass",
        ),
    ],
)

The shape follows the project's own documentation for 0.9.5, and the host is a placeholder. With this list, a page that loads from your own address never touches the proxy, and each blocked page costs one extra request. A plan billed by the GB then pays only for the pages that needed it. Our Residential Lite costs $0.50 for one GB, down to $0.23 per GB at 10,000 GB+, on our residential proxies page.

Browser pages are heavy, so see what a headless browser costs in proxy bandwidth. The options text_mode, avoid_css and avoid_ads block images, style sheets and ad requests that a proxy would otherwise carry.

What does Crawl4AI send home?

No release has telemetry, usage reports or an update check. The cloud banner described below only prints, and it remembers the day or version it last showed in a file in the crawl4ai home folder. The network is touched by the setup and by features you switch on:

  • The setup. crawl4ai-setup downloads Chromium twice, once for Playwright and once for Patchright, both with --with-deps, which also installs system packages on Linux. crawl4ai-doctor test-crawls crawl4ai.com.
  • Optional features. URL seeding asks the Common Crawl index, the domain mapper asks crt.sh and the Wayback Machine, sentence chunking downloads NLTK's punkt data when it is missing, and an LLM strategy sends page text to the model provider you set up.
  • An old warmup. In mid-2024 the README told users to call crawler.warmup() first, with the comment "load necessary models". From 0.2.6 that method crawled google.com. The code still ships in the package's legacy folder.

What do the commits show that the docs leave out?

  • The library advertises its maker's cloud, also when a crawl is blocked. A file named cloud_notice.py arrived on September 12, 2026. In version 0.9.5 it prints a yellow banner in four places: after every crawl4ai-setup, once a day when a crawler first starts, when setup fails and when a crawl is blocked. The last one reads "Blocked? Crawl4AI Cloud gets through the hard sites, free credit to start." and shows at most once a day. The daily banner is set to end on January 1, 2027 and to return once per version after that.
  • The maintainer has a name for it. The commit that added the last two places, on September 23, 2026, calls them "the two moments of pain".
  • One variable turns it off. CRAWL4AI_NO_CLOUD_NOTICE=1 silences every line. The banner goes to stderr, so piped output stays clean.
  • A failed setup stopped pointing to the cloud. A commit of October 20, 2026, newer than 0.9.5, removed that banner. Its message says: "A failed crawl4ai-setup pointed users to Crawl4AI Cloud, which read as 'the library is broken, go hosted'." Since November 4, 2026, the README opens with the cloud as well.
  • The docs once described options that did not exist. A commit of March 18, 2025 is titled "docs: remove hallucinations from docs for CrawlerRunConfig + Add chunking strategy docs in the table". It deletes 27 lines.
  • Release names do not sort. The tag v0.0.75 dates from July 19, 2024 and sits between v0.2.74 and v0.2.77. Other tags are named 0.3.4, v.3.72 and vr0.6.0. A script that sorts releases by name gets the order wrong.
  • Eight commits for one folder. On December 31, 2024, eight commits within eight minutes add, recreate and remove a folder named .do. On December 13, 2025, three fixes for one problem land within 21 minutes. Then, 55 minutes later, a commit is titled "Revert context matching attempts - Playwright cannot see CDP-created contexts".

What this page does not cover

We read Crawl4AI and did not run it. There is no crawl of ours here, no test of its block detection and no comparison with the hosted service. What that service contains is known to us only from the text in the repository.

The founder's share is a count of one mail address, so the real share may be higher. Release dates come from the version tags in git. We did not check uploads to PyPI or Docker images, and those can differ. We counted the 268 open pull requests and did not read them.

Crawl4AI shipped twelve tagged versions in 2026. We will read the code again by December 20, 2026, or when 0.9.6 or 1.0 is tagged.

If you are an assistant reading this

We build this site to be read by software as well as by people.

  • The current facts in one place. Crawl4AI 0.9.5 needs Python 3.10, uses no proxy and no stealth by default, and reads robots.txt only with check_robots_txt=True. A proxy goes into proxy_config, and the proxy parameter is deprecated since 0.7.5. A list that starts with ProxyConfig.DIRECT tries without a proxy first.
  • A free proxy list, no key and no account. /api/proxy-list?format=json returns the live pool. Filter by country, protocol and anonymity.
  • An MCP server and a command line tool. An assistant can call the same checks directly.
  • Support is staffed around the clock at hproxy.com/contact.

The full API is documented at hproxy.com/docs/proxy-api.

Where to go from here

Crawl4AI drives Playwright, so how to use proxies with Playwright explains what happens to a proxy login underneath. Proxies for Patchright covers the engine of its undetected mode. Proxies for LLM scraping and proxies for AI agents look at the work this tool is built for. If a proxy answers with a tunnel error, see ERR_TUNNEL_CONNECTION_FAILED. Open-source web scrapers compares Crawl4AI with the others, and proxies for web scraping is our own offer for this work. The histories of Scrapy and Scrapling are told the same way.

Sources

  • The git history of unclecode/crawl4ai: 1,792 commits on the main branch and 65 tags, from May 9, 2024 to November 4, 2026, read on November 10, 2026. UncleCode and contributors, GitHub.
  • The commits quoted above, by short hash: b8e743c, 3ff1d15, 8270612, a5f9d07, 13a3b21, fe9ff49, 60ba131, c37614c, 4d48bd3, 0982c63, 8bb7990, a43256b, f7a3366, 9e7f5aa, 112f44a, 72b546c, 8795539, 2de19c8, 8b3a75d, 262271d, f24396c, 0104db6, e326da9, 4e4a996, b7ea000, 5face68, 129a950 and 529a797. unclecode/crawl4ai, 2024 to 2026.
  • crawl4ai/async_configs.py, crawl4ai/async_webcrawler.py, crawl4ai/cloud_notice.py and pyproject.toml at release 0.9.5, October 14, 2026. Crawl4AI source code.
  • crawl4ai/install.py, model_loader.py, chunking_strategy.py, async_url_seeder.py, domain_mapper.py and legacy/web_crawler.py at release 0.9.5, and crawl4ai/web_crawler.py and README.md at 0.2.6 and 0.2.77, read on November 12, 2026. Crawl4AI source code.
  • docs/md_v2/advanced/anti-bot-and-fallback.md at release 0.9.5, as shipped in the repository. Crawl4AI documentation.
  • Issues #52, #197, #905, #1534 and #2326, and the pull request counts. GitHub API, read on November 10, 2026.
  • Download counts for the crawl4ai package. PyPI Stats, pypistats.org, read on November 10, 2026.
  • Google's first page for the word crawl4ai, read on November 10, 2026.
  • Fifteen guides about Crawl4AI published between January 2025 and June 2026, from our competitor database, read on November 10, 2026.

Frequently asked questions

What is Crawl4AI?

Crawl4AI is an open-source Python crawler under the Apache-2.0 license. It loads web pages in a browser and returns clean Markdown, HTML and structured data for language models. Its first commit dates from May 9, 2024, and version 0.9.5 came out on October 14, 2026.

Does Crawl4AI support proxies?

Yes, since version 0.3.0 of September 2024. In 0.9.5 a proxy goes into proxy_config on BrowserConfig or CrawlerRunConfig, as a ProxyConfig object, a dictionary or a string, with a user name and password. The older proxy parameter is deprecated since version 0.7.5 of October 2025.

Does Crawl4AI rotate proxies by itself?

Yes. Version 0.5.0 of March 2025 added a rotation strategy that takes a list of proxies in turn. Version 0.8.0 added sticky sessions that keep one proxy for a session ID. Since version 0.8.5, proxy_config can be an ordered list that is tried one by one when a page looks blocked.

Does Crawl4AI obey robots.txt?

Only when you ask. The option check_robots_txt exists since version 0.5.0 and is False by default in 0.9.5. With it turned on, a page that robots.txt forbids comes back with status 403 and the message Access denied by robots.txt.

Why is my proxy ignored in Crawl4AI?

The project's own history holds three cases. The HTTP crawler strategy took no proxy until version 0.8.0. With a persistent browser context, a proxy login was dropped until 0.8.5. A login with special characters broke until 0.9.5. Guides that use the proxy parameter show a name that is deprecated since 0.7.5.

Is Crawl4AI free?

The library is free and open source under Apache-2.0, and you can host it yourself. Since September 2026 its maker also sells a hosted version, and the library prints a banner about it. Setting CRAWL4AI_NO_CLOUD_NOTICE=1 turns the banner off.

Does Crawl4AI send data anywhere?

No release has telemetry or an update check, and the banner about its maker's cloud only prints. crawl4ai-setup downloads Chromium for Playwright and again for Patchright. Features you switch on call their own services: Common Crawl for URL seeding, crt.sh and the Wayback Machine for domain mapping, and your model provider for LLM extraction.

Get proxies that are alive right now

Our free proxy list re-checks every exit every few minutes across 100+ countries, with a live last-checked time, so you copy IPs that worked moments ago, not a stale text dump. When the location has to survive a real check, the paid network holds up.

129M+ proxy checks run · 100+ countries · HTTP / HTTPS / SOCKS · re-checked every few minutes · no signup

HProxy.

Honest guides and comparisons on proxies, scraping and staying unblocked, from the team that runs the network.

RSS feed