Golang web scraping: net/http, goquery, Colly and chromedp, tested

Go's standard library can download a page. The Go team's HTML parser, golang.org/x/net/html, comes as a separate module. On top of it you add goquery to read the HTML, Colly to crawl a whole site, or chromedp to drive Chrome.

We ran every program on this page with Go 1.27.2 on our server on 12 October 2026, against a test site of our own. The libraries worked. Their defaults decided whether the data came out right.

The short answer

• One page, or a list of known pages: net/http with goquery. Give the client a timeout, because Go's default client never gives up.
• A whole site: Colly v2, imported as github.com/gocolly/colly/v2. It ignores robots.txt and cuts every answer at 10 MB until you change two settings.
• Pages built by JavaScript: chromedp. Its API changed between 3 and 11 October 2026, and older chromedp code no longer builds.
• Pages that are not UTF-8: pass the body through charset.NewReader before goquery reads it.
• Speed: 8 goroutines read our 100 pages in 2.62 seconds, Colly with 8 at once in 3.06 seconds.

The libraries

We ran the first three of these six libraries, in the versions shown.

The release dates come from Go's module proxy and the stars from GitHub's API on 12 October 2026. Four of the six use the MIT license, goquery uses BSD-3-Clause and Colly Apache-2.0.

Colly brings more code with it: our Colly programs linked 18 modules and built to 19.7 MB. Our goquery programs linked 3 modules and stayed under 10 MB.

Scrape one page with net/http and goquery

Save this program as main.go in a new folder. It reads one product page of our test site.

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/PuerkitoBio/goquery"
)

func main() {
	client := &http.Client{Timeout: 15 * time.Second}

	req, err := http.NewRequest("GET", "http://127.0.0.1:8261/product/25", nil)
	if err != nil {
		log.Fatal(err)
	}
	req.Header.Set("User-Agent", "lamp-scraper/1.0 (you@example.com)")

	resp, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		log.Fatalf("%s: HTTP %d", req.URL, resp.StatusCode)
	}

	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		log.Fatal(err)
	}
	product := doc.Find("#product")
	fmt.Println(product.Find("h1").Text())
	fmt.Println(product.Find(".price").Text())
	fmt.Println(product.AttrOr("data-sku", ""))
}

Then create a module for it, add goquery and run it.

go mod init lampscraper
go get github.com/PuerkitoBio/goquery@v1.13.0
go run .

It printed the product's name, price and code.

$ go run .
Lampe à poser 25
£34.99
LMP-025

Three lines in it guard against Go's defaults. Each one fixes something we saw go wrong.

• The timeout. http.Get uses Go's default client, and the net/http documentation says "A Timeout of zero means no timeout". Against a server that never answered, our http.Get was still waiting at 120 seconds when we stopped it.
• The User-Agent. Without one, Go announces itself as Go-http-client/1.1. In our test of eight shops, Best Buy turned that name away and served Go with a Chrome User-Agent.
• The status check. A 404 is no error to net/http. Our test site's 404 page came back with err set to nil, and goquery read a product name from it.

Crawl a whole site with Colly

Colly visits a page, runs your callbacks on its HTML and follows the links you hand it. This crawler follows the five list pages of our test shop and saves all 100 products. Most of its lines change one of Colly's defaults.

package main

import (
	"encoding/csv"
	"log"
	"os"
	"sort"
	"sync"
	"time"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(
		colly.AllowedDomains("127.0.0.1"),
		colly.UserAgent("lamp-scraper/1.0 (you@example.com)"),
		colly.Async(),
	)
	c.IgnoreRobotsTxt = false // Colly ignores robots.txt unless told otherwise
	c.SetRequestTimeout(15 * time.Second)
	c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 8})

	var mu sync.Mutex
	var rows [][]string
	var retried sync.Map

	c.OnHTML("a.product, a.next", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})
	c.OnHTML("#product", func(e *colly.HTMLElement) {
		mu.Lock()
		defer mu.Unlock()
		rows = append(rows, []string{e.ChildText("h1"), e.ChildText(".price"), e.Attr("data-sku")})
	})
	c.OnError(func(r *colly.Response, err error) {
		if _, again := retried.LoadOrStore(r.Request.URL.String(), true); !again {
			r.Request.Retry() // c.Visit would refuse: the address already counts as visited
			return
		}
		log.Println(r.Request.URL, err)
	})

	c.Visit("http://127.0.0.1:8261/")
	c.Wait() // with Async, main would otherwise end before the requests do

	sort.Slice(rows, func(i, j int) bool { return rows[i][2] < rows[j][2] })
	f, err := os.Create("products.csv")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()
	w := csv.NewWriter(f)
	w.Write([]string{"name", "price", "sku"})
	w.WriteAll(rows)
	log.Println(len(rows), "products")
}

Save it as main.go in a second folder, then run it the same way.

go mod init lampcrawler
go get github.com/gocolly/colly/v2@v2.3.0
go run .

It wrote 100 rows to products.csv, and the file starts like this.

name,price,sku
Desk lamp 1,£10.99,LMP-001
Desk lamp 2,£11.99,LMP-002

Each of those settings changed a result in our tests.

• c.IgnoreRobotsTxt = false. Colly's own setup code sets IgnoreRobotsTxt = true. With the default, it fetched a page our robots.txt closes. With false, it refused with "URL blocked by robots.txt".
• c.Wait(). With colly.Async(), Visit returns at once. Without Wait, our program reached its next line with 0 products saved. With it, 100.
• r.Request.Retry(). Colly records an address as visited before it sends the request. Calling c.Visit again after a 503 returned "already visited" all 5 times and saved 0 of 5 pages. Retry saved 5 of 5.
• The retried map. Do not count retries in r.Request.Ctx. Request.Visit gives each new page the context of the page that linked to it, so a whole crawl can share one. In our test, product 2 saw the value product 1 had stored.
• c.Limit. Without a LimitRule, async Colly sent up to 19 requests at once to our test site.
• c.SetRequestTimeout. Colly's client gives up after 10 seconds by default. Our silent page failed after 10.1 seconds with "Client.Timeout exceeded while awaiting headers". The line shows where to change it.

The same job, five ways

We timed five versions of the job: 5 list pages and 100 product pages of 80 KB, with every answer 0.1 seconds late. Each version ran five times, taking turns, and saved all 100 products correctly every time.

Waiting for the server took most of the time. Eight requests at once cut about 13.5 seconds to about 3, in plain Go and in Colly alike. Colly with no limit was slower than with a limit of 8 in all five runs, and we did not find out why.

Our language test ran the same 100 pages in Python. Python with selectolax took 2.62 seconds there, and Python with BeautifulSoup in eight threads 9.57.

What the defaults do

Our probes ran both with their defaults.

Colly documents two of these limits. Its documentation says "The default value for MaxBodySize is 10MB", and that it "parses only successful HTTP responses". Set c.ParseHTTPErrorResponse = true if you need the HTML of error pages.

Pages that are not UTF-8

Go's HTML parser assumes UTF-8. goquery's README puts it plainly: "it is the caller's responsibility to ensure that the source document provides UTF-8 encoded HTML". We served one French product page in windows-1252 three ways: with the charset in the header, only in a <meta> tag, and nowhere.

Wrong means "Lampe à poser 25" came out with a broken byte where the "à" should be. The fix for goquery takes a few lines from golang.org/x/net/html/charset. That package looks at the header and at the first 1,024 bytes of the page.

// parseAnyCharset is the fix for pages that are not UTF-8: x/net/html/charset reads the
// Content-Type header and the page's <meta> tag, then decodes the body to UTF-8.
func parseAnyCharset(resp *http.Response) (*goquery.Document, error) {
	r, err := charset.NewReader(resp.Body, resp.Header.Get("Content-Type"))
	if err != nil {
		return nil, err
	}
	return goquery.NewDocumentFromReader(r)
}

In Colly, set c.DetectCharset = true. Colly converts the body when the header names a charset, and guesses with that switch on.

Pages bigger than 10 MB

Our test site served one list page of 12,789,038 bytes with 50,000 items and an end marker. goquery read all of it.

Colly's default collector kept the first 10,485,760 bytes, found 41,003 items and missed the end marker. Visit returned no error, so the loss would go unnoticed. With c.MaxBodySize = 0, Colly read all 50,000 items. A number of your own works too, if you want a cap.

Pages built by JavaScript: chromedp

goquery and Colly only read the HTML the server sends. On our test page, a script adds 10 products half a second after it starts, and both found 0 of them. chromedp starts Chrome, lets the page run and then reads it. This program waits until the first product is visible, then reads the names.

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/chromedp/chromedp"
)

func main() {
	ctx, cancel := chromedp.NewContext(context.Background())
	defer cancel()
	ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
	defer cancel()

	err := chromedp.Do(ctx,
		chromedp.Navigate("http://127.0.0.1:8261/js"),
		chromedp.WaitVisible(".product"),
	)
	if err != nil {
		log.Fatal(err)
	}
	names, err := chromedp.Run(ctx, chromedp.Evaluate[[]string](
		`[...document.querySelectorAll(".product h2")].map(h => h.textContent)`))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(len(names), "products, from", names[0], "to", names[len(names)-1])
}

Save it in a third folder and run it with chromedp's current version.

go mod init lampbrowser
go get github.com/chromedp/chromedp@v0.21.0
go run .

It found all ten.

$ go run .
10 products, from Desk lamp 1 to Desk lamp 10

Each of three runs took 3.9 to 6.5 seconds, with Chrome's start. Chrome ran as 11 processes and used up to 225 MB of private memory. The runs used Chrome for Testing 153.0.8010.12, which chromedp found on the path and started with --no-sandbox.

A browser costs far more than plain HTTP. It took up to 225 MB and several seconds for one page, against 13.9 MB and 2.62 seconds for 100 pages over HTTP. Use it for the pages that need it. Our page on waiting for a page to load in Playwright shows why the wait matters.

Older chromedp code no longer builds

chromedp published eight releases between 3 and 11 October 2026, from v0.17.0 to v0.21.0. Its migration guide lists every change since v0.16.0 of 14 July. Run(ctx, actions...) became Run[T](ctx, action) for one action that returns a value, and Do(ctx, steps...) for the rest.

Code written for the old API passes several actions and pointers to chromedp.Run. We built such a program against v0.21.0, as a reader would after go get today.

$ go run .
# lampold
./main.go:21:3: too many arguments in call to chromedp.Run
	have (context.Context, chromedp.Action[chromedp.Void], chromedp.Action[chromedp.Void], chromedp.Action[T])
	want (context.Context, chromedp.Action[T])
./main.go:22:20: in call to chromedp.Evaluate, cannot infer T (declared at <module cache>/github.com/chromedp/chromedp@v0.21.0/eval.go:37:15)

The same file built and ran with v0.16.0 in a new module. In the module that already had v0.21.0, go get github.com/chromedp/chromedp@v0.16.0 was not enough. It left chromedp's protocol module cdproto at v0.157.8, and the build stopped with "undefined: input.KeyDown". Going back takes both modules in one command.

go get github.com/chromedp/chromedp@v0.16.0 github.com/chromedp/cdproto@v0.0.0-20260714215040-dc233986426f

With that, the old program found its 10 products again. For new code, write to the current API, as above.

Proxies in Go

Go's default client reads the proxy from HTTP_PROXY and HTTPS_PROXY, and so does Colly's default collector. We sent 20 pages through a counting proxy of our own to see what each setup does.

Colly v2's SetProxyFunc sets DisableKeepAlives: true on its transport, and SetProxy goes through it. Colly v1.2.0 did not. Each request then opens its own connection. Through a rotating gateway, that can give every request a new exit address. Through one fixed proxy, every page pays for a new connection.

The last row matters when you test against a local server. Go never sends requests for localhost or a loopback address through a proxy. Our Go proxy guide covers transports, SOCKS5 and NO_PROXY. Our Colly proxy guide covers rotation with RoundRobinProxySwitcher.

All of our tests stayed on our own server. Real sites also judge the address a request comes from. In our test of eight shops behind bot defenses, Go's net/http loaded two with its own User-Agent and four with Chrome's. Our residential proxies start at $0.44/GB. Before you scrape a site, read its terms and its robots.txt, and our page on whether web scraping is legal.

Colly v1 or v2

Import github.com/gocolly/colly/v2. The old path, github.com/gocolly/colly, ends at v1.2.0 from 13 February 2019, and Go's module proxy lists v2.3.0 from 4 December 2025.

Five of the seven Colly tutorials we read still import the old path. The defaults we tested match in v1.2.0: the User-Agent, the 10 MB cap, robots.txt off and the 10-second timeout. The proxy behavior differs, as shown above.

What other guides get wrong

We read eleven guides on Go scraping from companies in our market. Most teach working basics and skip the defaults above.

• None of the seven with Colly code turns robots.txt on, and none of the eleven handles character sets.
• All three with chromedp code use the API that no longer builds against v0.21.0.
• One counts html/template among Go's tools "for parsing HTML". Go's documentation says the package generates "HTML output safe against code injection". The parser is golang.org/x/net/html.
• Four say Go is faster than Python and show no test of their own. Our same-job numbers above suggest the server sets the pace.
• The most careful guide names the Go and Colly versions it tested and sets timeouts. Its chromedp example is among the three above.

What this page could not check

• One server and a test site of our own on 127.0.0.1, with every answer 0.1 seconds late. Real sites add their own delays, limits and bot defenses.
• One version of each library: Go 1.27.2, goquery v1.13.0, Colly v2.3.0, chromedp v0.21.0 and v0.16.0, Chrome for Testing 153.0.8010.12. Colly v1.2.0 was read, not run.
• chromedp published eight releases in nine days, so its newest version may differ again.
• We did not run rod, htmlquery or playwright-go, and our proxy test used a counting proxy of our own.
• We will check this page again by 12 January 2027.

Sources

• Go, net/http: Client.Timeout, DefaultClient and ProxyFromEnvironment: pkg.go.dev.
• Go, golang.org/x/net/html and its charset package: pkg.go.dev.
• goquery README, on UTF-8: github.com.
• Colly v2 source and documentation: Init, MaxBodySize, ParseHTTPErrorResponse, SetProxyFunc, Request.Visit: pkg.go.dev.
• chromedp README and migration guide: github.com.
• Go, html/template: pkg.go.dev.
• Release dates from Go's module proxy and stars from GitHub's API, both read on 12 October 2026.
• Our own tests on our server, 12 October 2026: run_tests.py from 13:53 to 14:01 UTC and reader_flow.sh at 14:06 UTC.
• Our test of the same job in four languages: hproxy.com.
• Our test of 16 tool settings against eight shops behind bot defenses: hproxy.com.
• Our page on the law, with the hiQ ruling: hproxy.com.