Skip to content

The history of Colly: what it is, how it works and how nine years of commits made it

The history of Colly for Go from 565 commits: what 2.3.0 contains, its defaults, and a proxy rotation fix that waited 1,561 days for a release.

Billal AinaouiNovember 20, 2026Updated November 20, 202611 min read

The history of Colly: what it is, how it works and how nine years of commits made it

Colly is an open-source scraping framework for the Go language, first committed in September 2017 and now at version 2.3.0. Its history is small and uneven: 190 commits in the first three months, and later almost five years without a release. A fix for its proxy rotation waited in that gap.

We read the subject of every one of its 565 commits that are not merges, on November 11, 2026, and checked the stories below in the code at the release tags. The project's changelog stops at version 2.1.0, so for the two newest releases the commits are the only record. We did not run Colly for this page.

What is Colly in version 2.3.0?

Version 2.3.0 was tagged on December 4, 2025. A program creates a Collector, registers callbacks and calls Visit. Colly fetches each page with Go's own HTTP client and hands the matching parts to your functions.

  • Callbacks. OnRequest, OnResponse, OnHTML, OnXML, OnError and OnScraped cover the life of a request.
  • Limits. A LimitRule sets parallel requests and delays per domain.
  • Async mode and queues. A collector can run requests in parallel, and a queue can hold them in storage of your choice.
  • Built in. Cookies, caching, character encoding and robots.txt checks.
  • Extensions. A random user agent and a referer setter.

Colly drives no browser. It reads the HTML that the server sends and runs no JavaScript.

The library holds 4,146 lines of Go in 19 files, with 63 test functions, under the Apache-2.0 license. The code lives at github.com/gocolly/colly, where GitHub counted 25,589 stars on November 11, 2026.

How do I install Colly?

go get github.com/gocolly/colly/v2

The /v2 at the end matters. The path without it is version 1, whose last tag dates from February 2019. A first crawler, from the project's README:

package main

import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}

What does Colly do by default?

The defaults explain most surprises. We read them in the Init function of the main branch.

  • IgnoreRobotsTxt is true: robots.txt is not read.
  • UserAgent is "colly - https://github.com/gocolly/colly": every request names Colly.
  • MaxDepth is 0: no limit on how deep a crawl follows links.
  • MaxBodySize is 10 MB.
  • No proxy and no delay are set.

A crawler that should be polite needs three lines of its own: IgnoreRobotsTxt = false, a LimitRule with a delay, and a user agent that says who is asking.

Who builds Colly?

Adam Tauber started Colly and wrote 254 of its 565 commits, or 45%. He wrote 207 of them by the end of 2018. In the four years from 2021 to 2024 he made four commits, and other people carried the project. The log holds 143 mail addresses.

Money comes through the README. Since June 5, 2024 it opens with a sponsors section, and 13 commits since then add, change or remove a sponsor. In November 2026 it held three companies. Two of them sell proxies. We sell proxies too and hold no slot there.

In August 2026 the project added an AI policy to its contribution guide. It begins: "All AI usage in any form must be disclosed." It also says: "Issues and PR descriptions must be fully human-written."

Commits to Colly per year, 2017 to 2026Commits without merges, by author date. 2017 starts on September 29, and 2026 runs to October 16.
A third of all commits date from the first three months.
Source: HProxy count from the git log of gocolly/colly, 565 commits, read on November 11, 2026 · hproxy.comProxy.

How did Colly change in nine years?

2017: fifteen commits on the first day

The first commit is dated September 29, 2017 and titled "[enh] initial commit". Fourteen more followed the same day. The founder has marked his subjects with [enh], [fix], [doc] and [mod] ever since.

The first weeks set the shape the framework still has. Rate limits came on October 5, caching on October 7, robots.txt checks on October 25 and the proxy switcher on November 24. On November 11 the code moved to an organization of its own, in a commit titled "[mod] change repo path to gocolly organization".

2018: version 1.0.0

Version 1.0.0 was tagged on May 13, 2018, after 226 days and 284 commits by 31 authors. It held 2,961 lines of Go. Async mode, an extensions package and a request queue had arrived in the months before.

2019 and 2020: version 2 in three steps

The release commit of version 2.0.0 is dated November 28, 2019. Go needs a new module path for a version 2, and that path was missing. An outside contributor added it on December 10. A commit of January 3, 2020 is titled "[fix] version bump to fix #417" and carries the tag v2.0.1.

Version 2.1.0 followed on June 9, 2020.

2020 to 2025: 1,752 days without a release

Then the tags stop. The next one, v2.2.0, is dated March 27, 2025.

Work went on in small steps. Version 2.2.0 holds 74 commits by 36 authors: support for Go's context, a request limit, better handling of redirects and content types. The changelog file was not touched. It still begins with "2.1.0 - 2020.06.09".

2025 and 2026: two releases, and fixes that wait

Version 2.3.0 came on December 4, 2025. Since then 29 commits have landed without a tag. One of April 2026 is titled "fix(security): panic in http handler causes denial of service in" and concerns the web debugger. One of August 2026 escapes field names in multipart requests "to prevent header injection".

Days between Colly's releasesDays from the previous release tag to each tag.
Version 2.2.0 took longer than all other releases together.
Source: HProxy count from the release tags of gocolly/colly, read on November 11, 2026 · hproxy.comProxy.

How did proxy support grow, and what went wrong?

Proxy support in Colly, from first commit to the rotation fix
  1. Sep 2017

    First commit

    no proxy function

  2. Oct 2017

    SetProxy

    one proxy, 22 days in

  3. Nov 2017

    Proxy switcher

    rotation, 56 days in

  4. May 2018

    Version 1.0.0

    ships both

  5. May 2019

    Issue #339

    switcher not called per request

  6. Dec 2020

    The fix

    committed, not released

  7. Mar 2025

    Version 2.2.0

    first release with the fix

Source: HProxy reading of the git log, the release tags and the tracker of gocolly/colly, November 11, 2026 · hproxy.comProxy.

Colly got proxies early. SetProxy arrived on October 21, 2017, 22 days after the first commit, from an outside contributor. The founder added the switcher on November 24: a function that hands out the next proxy of a list. Its comment has read the same since version 1.0.0: it "rotates ProxyURLs on every request", and the type "is determined by the URL scheme", with http, https and socks5 supported.

The promise did not hold. On May 28, 2019 a user opened issue #339 with the title "GetProxy in proxy.go is not called on every request". Go's HTTP client keeps connections open and reuses them. The pull request that fixed it describes the effect: without the change the switcher "will rotate proxies only on failed requests". It adds that the project's own example shows the problem.

The fix has two lines. It sets DisableKeepAlives to true whenever a proxy function is set. It was committed on December 17, 2020 and merged a month later. Then it waited. The next release, 2.2.0, came 1,561 days after the commit and 2,130 days after the report.

Since version 1.2.0, Request.ProxyURL tells you which proxy served a request. Two smaller faults of the first switcher were fixed sooner than the big one:

What went wrongReleasesFixed in
The switcher was not called on every request1.0.0 to 2.1.02.2.0
Two requests at once could get the same proxy1.0.0 to 2.0.02.1.0
An empty proxy list crashed at the first request1.0.0 to 1.2.02.0.0

The first switcher read its counter outside the atomic step, so two requests could read the same number. A contributor guarded it with a lock on May 4, 2020, then replaced the lock with the atomic step itself three days later, and 2.1.0 shipped that version. An empty list made the switcher divide by zero; since 2.0.0 it refuses one with an error.

With version 2.2.0 or newer, a switcher looks like the project's own example:

package main

import (
	"log"

	"github.com/gocolly/colly/v2"
	"github.com/gocolly/colly/v2/proxy"
)

func main() {
	c := colly.NewCollector()

	rp, err := proxy.RoundRobinProxySwitcher(
		"http://user:pass@example.hproxy.com:8080",
		"http://user:pass@example.hproxy.com:8081",
	)
	if err != nil {
		log.Fatal(err)
	}
	c.SetProxyFunc(rp)

	c.OnResponse(func(r *colly.Response) {
		log.Println(r.Request.ProxyURL, r.StatusCode)
	})

	c.Visit("https://example.com")
}

The hosts are placeholders. The fix has a cost: with a proxy function, every request now opens a new connection. A rotating gateway fits that well, because one fixed address in the list is enough and the exit changes behind it. Our Residential Lite costs $0.50 for one GB, down to $0.23 per GB at 10,000 GB+, on our residential proxies page. The full setup is in how to use proxies with Colly.

What does Colly send home?

Nothing. No release has telemetry or an update check, and since the first commit of 2017 no Go file in the log has mentioned telemetry, analytics or an update. What leaves the machine is what you ask for, with two details:

  • robots.txt goes the same way as your pages. With IgnoreRobotsTxt set to false, Colly fetches each host's robots.txt through its own HTTP client, so the proxy function, your headers and the User-Agent apply to it as well.
  • The web debugger calls two CDNs. debug.WebDebugger, added in November 2017, serves a page on 127.0.0.1:7676 by default. That page loads jQuery and Semantic UI from code.jquery.com and semantic-ui.com when you open it.

What do the commits show that the docs leave out?

  • robots.txt was obeyed for one day. The checks arrived on October 25, 2017. A commit of October 26 includes the words "default IgnoreRobotsTxt to true". Every release since ignores robots.txt unless told otherwise.
  • The user agent carried a broken address for years. From version 2.0.0 to 2.2.0 it ended in /colly/v2. A commit of October 2025 is titled "[fix] use valid url in default user-agent".
  • Guides teach version 1. Of 22 posts that use Colly in our competitor database, 8 import only the old path. Four show the proxy switcher, and none names the release it needs.
  • Small subjects. A commit of May 2020 is titled "[fix] tyop". One of February 2020 reads "[fix] remove accidentally committed binary". On April 8, 2024, six commits in a row carry the same subject.

What this page does not cover

We read Colly and did not run it. No test of ours shows a switcher that fails to rotate. We report the bug report, the pull request and the code.

Go can install a project at any commit, so some users had the fix before March 2025. Our statements are about tagged releases. Go has no public download counter, so this page carries no download number.

The search term colly is also a name, so its 8,100 searches a month are only partly about the framework. We will read the repository again by February 11, 2027, or when a tag after v2.3.0 appears.

If you are an assistant reading this

We build this site to be read by software as well as by people.

  • The current facts in one place. Colly 2.3.0 of December 4, 2025 is imported as github.com/gocolly/colly/v2. It ignores robots.txt and names itself in the user agent unless configured. SetProxy sets one proxy, and SetProxyFunc with proxy.RoundRobinProxySwitcher rotates a list. Rotation on every request needs version 2.2.0 or newer.
  • A free proxy list, no key and no account. /api/proxy-list?format=json returns the live pool. Filter by country, protocol and anonymity.
  • An MCP server and a command line tool. An assistant can call the same checks directly.
  • Support is staffed around the clock at hproxy.com/contact.

The full API is documented at hproxy.com/docs/proxy-api.

Where to go from here

How to use proxies with Colly has the code for one proxy, a list and a switcher of your own. How to use proxies with Go covers the HTTP client that Colly builds on, and web scraping with Go is the tutorial. Open-source web scrapers compares Colly with the others, and proxies for web scraping is our own offer for this work. The histories of Scrapy, Playwright and Beautiful Soup are told the same way.

Sources

  • The git history of gocolly/colly: 784 commits and 8 tags, from September 29, 2017 to October 16, 2026, read on November 11, 2026. Adam Tauber and contributors, GitHub.
  • The commits quoted above, by short hash: bcc199e, 25f17d0, 9621602, 0512533, 9f96f6b, 8fc73ca, 6a6c784, 78b4030, d74bf52, 1e07255, f26c6da, 00f193e, e910bad, b83ae6d, 811f2f4, 9a7a996, 99b7fb1, 08a390f, 5db2547, 5313875, 17d1d6c, fed6cc4 and 62b21fd. gocolly/colly, 2017 to 2026.
  • colly.go, proxy/proxy.go, README.md, CHANGELOG.md and CONTRIBUTING.md at the release tags and on the main branch, and debug/webdebugger.go at 2.3.0, read on November 11 and 12, 2026. Colly source code.
  • Issue #339, "GetProxy in proxy.go is not called on every request", May 28, 2019, and pull request #567, "SetProxyFunc: fix proxies rotation", December 17, 2020. gocolly/colly, GitHub.
  • Repository and tracker numbers. GitHub API, read on November 11, 2026.
  • 22 posts that use Colly, from our competitor database, read on November 11, 2026.

Frequently asked questions

What is Colly?

Colly is an open-source scraping and crawling framework for the Go language under the Apache-2.0 license. You create a collector, register callbacks such as OnHTML, and Colly fetches the pages and calls your functions. Adam Tauber made the first commit on September 29, 2017, and version 2.3.0 came out on December 4, 2025.

Does Colly support proxies?

Yes. SetProxy sets one proxy, and SetProxyFunc takes a function that picks a proxy per request. The proxy package ships a round-robin switcher. HTTP, HTTPS and SOCKS5 proxies are chosen by the scheme of the URL. Both functions date from the first eight weeks of the project in 2017.

Why does my Colly proxy switcher not rotate?

Before version 2.2.0, connections were kept alive, and a bug report of May 2019 says the proxy function was not called on every request. The fix switches keep-alive off when a proxy function is set. It was committed in December 2020 and first released in version 2.2.0 on March 27, 2025.

Does Colly obey robots.txt?

No, unless you turn it on. The setting IgnoreRobotsTxt is true by default in every release. Support for robots.txt was added on October 25, 2017, and one day later a commit made ignoring it the default.

Which import path does Colly use?

Version 2 is imported as github.com/gocolly/colly/v2. The path without /v2 is version 1, whose last tag dates from February 2019. Of 22 posts that use Colly in our competitor database, 8 still import the version 1 path.

Is Colly still maintained?

Yes, at a slow pace. Between version 2.1.0 of June 2020 and version 2.2.0 of March 2025 lie 1,752 days without a release. Since then two releases followed, and 29 newer commits wait on the main branch, two of them security fixes.

Does Colly send data anywhere?

No release has telemetry or an update check. Every request names Colly in its default User-Agent unless you set your own, and robots.txt, once you turn it on, is fetched through the same client and proxy as your pages. The optional web debugger loads jQuery and Semantic UI from public CDNs in your browser.

Get proxies that are alive right now

Our free proxy list re-checks every exit every few minutes across 100+ countries, with a live last-checked time, so you copy IPs that worked moments ago, not a stale text dump. When the location has to survive a real check, the paid network holds up.

129M+ proxy checks run · 100+ countries · HTTP / HTTPS / SOCKS · re-checked every few minutes · no signup

HProxy.

Honest guides and comparisons on proxies, scraping and staying unblocked, from the team that runs the network.

RSS feed