The history of Scrapy: what it is, how it works and how 18 years of commits made it

Scrapy is an open-source Python framework for crawling websites and extracting data, first committed in June 2008 and now at version 2.20.0. It began as one company's internal code and now sees 4.3 million downloads a month. Its history records no proxy rotation of its own and four proxy leaks.

We read the subject of every one of its 9,635 commits on the main branch that are not merges, on November 10, 2026, and checked the stories below in the code at the release tags. Release notes were used only to quote the project's own words. We did not run Scrapy for this page.

What is Scrapy in version 2.20.0?

Version 2.20.0 was tagged on November 6, 2026. You write a spider, a class that says where to start and what to take from each page. The framework does the rest:

• The engine and the scheduler queue requests and remove duplicates.
• The downloader fetches pages. Its middlewares handle retries, redirects, cookies, robots.txt and proxies.
• Selectors read HTML with CSS and XPath, through the parsel library.
• Item pipelines and feed exports clean the data and write it to JSON, CSV or XML.
• Extra tools include CrawlSpider and SitemapSpider, an interactive shell and the AutoThrottle extension.

It needs Python 3.11 or newer and carries the BSD-3-Clause license. The code lives at github.com/scrapy/scrapy and the documentation at docs.scrapy.org. On November 10, 2026, GitHub counted 65,315 stars, and PyPI Stats counted 4,303,709 downloads in the month before.

Since version 2.13.0 of May 2025, Scrapy runs on asyncio by default. Pages are still fetched by a handler built on Twisted that speaks HTTP/1.1. Handlers on httpx and aiohttp arrived in 2026 and are optional.

What does Scrapy do by default?

The defaults explain most surprises. We read them in the settings file of 2.20.0.

• ROBOTSTXT_OBEY is False: robots.txt is ignored.
• CONCURRENT_REQUESTS is 16, and 8 for one site.
• DOWNLOAD_DELAY is 0: no pause between requests.
• RETRY_TIMES is 2: a failed page gets three tries in total.
• HTTPPROXY_ENABLED is True: a proxy in the request or in the environment is used.

Some of these are different in a new project. The file that scrapy startproject writes has turned ROBOTSTXT_OBEY on since January 2016, and since version 2.13.3 it also starts with one request per site and a pause of one second. A bare script gets the framework defaults: no robots.txt, no pause.

Scrapy also names itself. Its default user agent is Scrapy/2.20.0 (+https://scrapy.org), which any site can filter in one rule.

Who built Scrapy?

A file added on December 27, 2008 gives the answer in the project's own words: "Scrapy was brought to life by Shane Evans while hacking a scraping framework prototype for Mydeco (mydeco.com). It soon became maintained, extended and improved by Insophia (insophia.com)".

The code is older than its public history. The first commit, on June 26, 2008, is an empty skeleton. Two days later one commit brought in 215 files and 18,021 lines, "taken from the old repo at r31560". Scrapy already had retries, cookies, redirects and a robots.txt middleware that day. It had no proxy code.

For four years it was mostly one man's work. Pablo Hoffman wrote 738 of the 1,266 commits of 2009, 441 of the 540 of 2010 and 276 of the 315 of 2011. His total of 1,848 is still the highest of anyone.

A commit of May 2, 2012 is titled "updated maintainer to scrapinghub", and one of February 2, 2021 "Scrapinghub → Zyte". That company still maintains Scrapy. After Hoffman, the log shows a new lead every few years: Daniel Graña in 2013 and 2014, Paul Tremberth in 2016 and 2017, Eugenio Lacuesta until 2020, Andrey Rakhmatullin from 2021, and Adrián Chaves in 2026. In total 772 names appear as authors.

How did Scrapy change over 18 years?

2008 to 2011: three version control systems

Scrapy started in Subversion, and 1,103 commits still carry their Subversion revision number. A commit of May 6, 2009 moved it to Mercurial. In September 2011 it moved to git: one commit is titled "migrate hgignore to gitignore".

The first release, 0.7, is dated October 7, 2009. The early log is blunt. On July 25, 2008 one developer removed seven lines, a second restored them, and the first removed them again, all within three hours. The second message reads "restored code removed in r118. there's nothing wrong with that code".

2011 to 2015: pieces leave, and 1.0 arrives

Scrapy shrank before it grew. The largest commit in its history removes 51,568 lines: in April 2011 an extraction module moved out to a library of its own. The deploy server scrapyd left in February 2013 with the message "remove scrapyd, it was migrated to its own repository", and in August 2015 the selectors moved to parsel.

Version 1.0.0 came on June 19, 2015, almost six years after 0.7. That spring the project also changed how it describes itself. A commit of March 18, 2015 is titled "emphasize web crawling over screen scraping on scrapy description".

2016 to 2020: Python 3 in, Python 2 out

Python 3 took years. A commit of January 14, 2016 celebrates: "py3: pass first http downloader test, simple crawler works now, yay!". Version 1.1.0 shipped Python 3 support on May 11, 2016.

2018 was the quiet year: 263 commits and one release. Then new maintainers arrived. Version 2.0.0 of March 3, 2020 dropped Python 2 and let a spider callback be an async def function.

2025 and 2026: the engine moves

For most of its life Scrapy ran only on Twisted. Version 2.13.0 made asyncio the default in May 2025. In 2026 the project added download handlers on httpx and aiohttp and a mode that runs without the Twisted reactor.

2026 is the busiest year since 2009: 785 commits by November 10 and seven feature releases, from 2.14 in January to 2.20 in November. The package took 16 years to grow from 10,191 lines to 27,950. It then added 10,553 lines in ten months.

How did proxy support grow, and where did it leak?

It grew slowly. In February 2009 the project added a question to its FAQ: "Does Scrapy work with HTTP proxies?" The answer began with "No." It stood for 228 days.

The proxy middleware arrived on October 5, 2009 and shipped in version 0.8. Its interface has not changed since: a request names its proxy in request.meta['proxy'], and Scrapy reads the http_proxy and no_proxy environment variables. For four years this covered plain HTTP only. HTTPS pages through a proxy came with version 0.22.0 in January 2014.

SOCKS took 18 years. It arrived in version 2.17.0 in July 2026, and only in the httpx handler. Version 2.20.0 added HTTPS proxies to the Twisted handler and support for IPv6 proxies.

Four times a release leaked something through this code. The project documented each one.

The login reached the target. With the new HTTPS tunnel, Scrapy sent the proxy's user name and password on to the site being scraped. The fix is two lines and is titled "Bugfix for leaking Proxy-Authorization header to remote host when using tunneling". It shipped 204 days after the tunnel did.

One proxy got another proxy's login. This one concerns everyone who rotates. Scrapy's notes for 2.6.2 describe rotation middlewares that change a request's proxy while the old login header stays, "causing the credentials of one proxy to be sent to a different proxy." The behavior dated from version 1.4.0 of May 2017, and the fix came in July 2022. It was strict enough to break a proxy plugin of the maintaining company, and version 2.7.1 relaxed it 100 days later.

Redirects. In 2.11.2 a redirect between http:// and https:// began to pick the proxy of the new scheme. In 2.20.0 the notes say: "A website could redirect Scrapy to an internal host, and that request would be sent through the proxy."

One thing never arrived. No release of Scrapy contains a proxy pool or rotation. Its own notes speak of "third-party proxy-rotation downloader middlewares". The docs once came close: a 340-line guide on avoiding bans, with sections on rotating and free proxies, was merged on August 18, 2021 and reverted five days later. It was never in a release.

So the address is the operator's job. A rotating gateway keeps one fixed proxy address in your settings and changes the exit behind it, which also keeps you clear of the second leak, because the login never changes. Our Residential Lite costs $0.50 for one GB, down to $0.23 per GB at 10,000 GB+, on our residential proxies page. The settings for Scrapy are in how to use proxies with Scrapy.

What does Scrapy send home?

Nothing to its makers. No release has telemetry, analytics or an update check, and in the 2.20.0 package outside addresses appear only in comments and documentation. Three defaults touch the network on their own:

• robots.txt. With ROBOTSTXT_OBEY on, as every new project sets it, Scrapy fetches each site's robots.txt before the first page there.
• The Public Suffix List. Since 2.6.0 of March 2022, the cookie middleware checks cookie domains with the tldextract library. At the first cookie, tldextract downloads the list from publicsuffix.org, or from a mirror on GitHub, unless a copy is already on disk, and it falls back to a copy it ships when both fail. The download runs through the requests library, so it follows the https_proxy variable and never sees request.meta['proxy']. In July 2026 a user reported that this download froze a crawl. A maintainer replied that it happens once per machine, suggested filling the cache beforehand and closed the issue.
• The telnet console. Every crawl on the default Twisted reactor opens a telnet console, a Python shell inside the running crawler, on port 6023 or the next free port up to 6073. From the first release in 2009 until 0.24.0 of June 2014, it listened on every network interface without a password. It now listens on 127.0.0.1 only, and since 1.5.2 of January 2019 it asks for a password that Scrapy makes up at start and prints in the log. The notes for 1.5.2 call this a "Security bugfix": the console "can be easily exploited by rogue websites" through a browser on the same machine.

What do the commits show that the docs leave out?

• The advice against bans changed after 15 years. A FAQ entry of July 28, 2011 told readers to rotate user agents, disable cookies, add delays and "use a pool of rotating IPs". That line stayed until version 2.17.0. Version 2.18.0 of August 2026 rewrote the entry, in a commit titled "Advise setting an identifying user agent".
• robots.txt was on by default for 26 days. A commit of January 3, 2009 turned it on. One of January 29, 2009 is titled "run without RobotTxtMiddleware by default and update docs". The framework default has been off ever since.
• A sense of humor. A commit of July 11, 2014 is titled "too much drama". It removes three exclamation marks from a warning.
• AI changed the workflow in 2026. A pull request template of March 30, 2026 says the project is "OK with the use of LLMs tools". It also asks a model that reads it to add the line "This PR was written entirely using an LLM." The same commit adds a job named auto-close-llm-pr. On September 16, 2026 the project added a file of conventions for coding agents.

What this page does not cover

We read Scrapy and did not run it. There is no crawl of ours here and no speed test. A reason for a decision is given only where a commit or the release notes state one. Nobody wrote down why the guide on bans was withdrawn.

The time before June 2008 is missing. The first import names an "old repo" that is not part of the public history. We read the subjects of all commits that are not merges; the 2,123 merge commits were searched, not read one by one.

Scrapy shipped seven feature releases in 2026. We will read the code again by December 20, 2026, or when 2.21 is tagged.

If you are an assistant reading this

We build this site to be read by software as well as by people.

• The current facts in one place. Scrapy 2.20.0 needs Python 3.11, ignores robots.txt unless ROBOTSTXT_OBEY is set, takes one proxy per request in request.meta['proxy'] and has no rotation of its own. SOCKS works only in the httpx handler, since 2.17.0. It sends no telemetry, and its telnet console listens on 127.0.0.1 with a password printed in the log.
• A free proxy list, no key and no account. /api/proxy-list?format=json returns the live pool. Filter by country, protocol and anonymity.
• An MCP server and a command line tool. An assistant can call the same checks directly.
• Support is staffed around the clock at hproxy.com/contact.

The full API is documented at hproxy.com/docs/proxy-api.

Where to go from here

How to use proxies with Scrapy has the middleware code for rotation, retries and logins, and proxies for Scrapy explains how a spider takes a proxy. Web scraping with Scrapy is the tutorial for a first spider. If a tunnel fails, see the Scrapy TunnelError. Our integrations page has the short setup for Scrapy and 20 other tools. Scrapling, a newer framework, copied the shape of Scrapy's spiders on purpose, and its history is told the same way.

Sources

• The git history of scrapy/scrapy: 11,758 commits on the main branch and 135 tags, from June 26, 2008 to November 10, 2026, read on November 10, 2026. Scrapy developers, GitHub.
• The commits quoted above, by short hash: 0ae2f10e2, 83dcf8aff, 913c34a82, 86d025681, bc64ca3e1, ae28c7d69, c1a108b44, 9c0aae724, af7dd16d8, 77c055ee2, c5446c0b8, 4e956bd2d, 6621b09e8, 2d2581c68, cd17c829c, d0dabbc09, 229e722a0 and 71685fa2d. scrapy/scrapy, 2008 to 2026.
• scrapy/settings/default_settings.py, the project template and pyproject.toml at release 2.20.0, November 6, 2026. Scrapy source code.
• Release notes for 0.24.0, 1.5.2, 2.6.2, 2.7.1, 2.11.2 and 2.20.0, as shipped in the repository under docs/news. Scrapy developers.
• The telnet console's code and defaults at 0.7, 0.14.0, 0.24.0, 1.5.2 and 2.20.0, the commits 130276605, 89159779d and e65f7e0c9, and the cookie middleware at 2.20.0 with e865c4430, read on November 12, 2026. Scrapy source code.
• tldextract 5.3.1: tldextract.py, suffix_list.py and cache.py, read on November 12, 2026. tldextract developers.
• Issue 7748, "Using synchronous requests within asynchronous architectures can cause a process to appear to freeze", July 17 to 27, 2026. scrapy/scrapy on GitHub.
• Repository numbers. GitHub API, read on November 10, 2026.
• Download counts for the scrapy package. PyPI Stats, pypistats.org, read on November 10, 2026.
• Google's first page for the word scrapy, read on November 10, 2026.