How to Use Proxies With Colly (Go)

Colly is the scraping framework most Go projects use, and it ships proxy rotation in the box, which is rare and worth building on correctly. This guide covers proxies with Colly end to end: SetProxy for a single exit, the RoundRobinProxySwitcher for rotation, SOCKS, authentication, the concurrency and timeout settings that keep a proxied crawl from thrashing, and verifying the exit IP so you know the requests actually left through the proxy.

We run a proxy network and a live proxy checker, so the Colly proxy question we answer most is why a crawl that rotates a free pool still gets blocked. The usual answer is concurrency: Colly's async mode fires a wall of requests, and against a small or datacenter pool that pattern is exactly what defenses look for. The settings below fix that. Every example is runnable Go. Where one needs a live proxy, pull a fresh one from our free proxy API, which returns recently checked endpoints with no key.

How do you set a proxy in Colly?

For one proxy, call SetProxy on the collector. Every request that collector makes then goes through it:

package main

import (
	"fmt"
	"github.com/gocolly/colly"
)

func main() {
	c := colly.NewCollector()
	c.SetProxy("http://203.0.113.7:8080")

	c.OnResponse(func(r *colly.Response) {
		fmt.Println(string(r.Body))
	})
	c.Visit("https://httpbin.org/ip")
}

The proxy URL scheme can be http:// or socks5://, because Colly hands the URL to Go's net/http, which understands both.

Rotating with the RoundRobinProxySwitcher

Colly's own proxy package gives you rotation without a third-party library. Build a switcher from a list of proxy URLs and set it with SetProxyFunc:

import (
	"github.com/gocolly/colly"
	"github.com/gocolly/colly/proxy"
)

c := colly.NewCollector()

rp, err := proxy.RoundRobinProxySwitcher(
	"http://203.0.113.7:8080",
	"http://203.0.113.24:3128",
	"socks5://198.51.100.14:1080",
)
if err != nil {
	panic(err)
}
c.SetProxyFunc(rp)

Colly cycles through the list, one exit per request. Note that a socks5:// entry sits happily alongside HTTP ones in the same list, which is convenient when a free pool hands you a mix.

A smarter switcher that drops dead exits

Round-robin is fine until half your list is dead, which on a free pool it will be. SetProxyFunc takes any function of the form func(*http.Request) (*url.URL, error), so you can write one that skips exits known to be down:

import (
	"math/rand"
	"net/http"
	"net/url"
	"sync"
)

type Pool struct {
	mu    sync.Mutex
	alive []string
}

func (p *Pool) Get(_ *http.Request) (*url.URL, error) {
	p.mu.Lock()
	defer p.mu.Unlock()
	return url.Parse(p.alive[rand.Intn(len(p.alive))])
}

// call Drop(proxy) from an OnError handler when an exit fails
func (p *Pool) Drop(dead string) {
	p.mu.Lock()
	defer p.mu.Unlock()
	for i, u := range p.alive {
		if u == dead {
			p.alive = append(p.alive[:i], p.alive[i+1:]...)
			return
		}
	}
}

Wire Drop into an OnError callback so a connection failure retires that exit, and the crawl stops wasting requests on corpses. Fill alive from our free proxy API at startup.

Authentication

For a paid proxy, put the credentials in the URL, user:pass@host:port, the form net/http expects. Both SetProxy and the switcher accept it:

c.SetProxy("http://myuser:mypass@203.0.113.7:8080")

A wrong or missing login returns HTTP 407 Proxy Authentication Required, which confirms the proxy is reachable and only the credentials are off.

Concurrency and timeouts, the part that decides success

This is where a proxied Colly crawl lives or dies. Async mode is fast and is exactly what overwhelms a small or datacenter pool, so bound it and set a timeout so a dead exit fails fast:

c := colly.NewCollector(colly.Async(true))

c.Limit(&colly.LimitRule{
	DomainGlob:  "*",
	Parallelism: 4,                 // not 50: a free pool cannot take it
	RandomDelay: 2 * time.Second,   // space requests so one exit is not hammered
})

c.SetRequestTimeout(15 * time.Second)   // a dead exit fails in 15s, not on a default wait

Parallelism caps simultaneous requests per domain, RandomDelay spaces them so a single exit is not obviously scripted, and SetRequestTimeout stops a stalled proxy from holding a worker. On a free pool these three are the difference between a crawl that makes progress and one that thrashes on dead IPs.

Verify the exit IP

Confirm the proxy took effect rather than trusting it. Visit an IP echo and print what the target saw:

c.OnResponse(func(r *colly.Response) {
	fmt.Println("exit IP:", string(r.Body))   // from https://httpbin.org/ip
})
c.Visit("https://httpbin.org/ip")

If the address is your own, the proxy did not apply, which usually means SetProxy or SetProxyFunc was called on a different collector than the one doing the visit. For a fuller read of the exit, our proxy checker reports the exit IP, country, latency and anonymity grade in one paste.

Where to go from here

Colly gives you rotation for free, and the work is in feeding it a pool that is alive and pacing the crawl so it does not out-run the exits. Once the switcher, the concurrency limit, and the timeout are set, the discipline is the same as any scraper: verify exits, drop dead ones, and match the proxy type to the target.

The Go proxy guide covers routing plain net/http and other Go clients if part of your code is outside Colly, proxies for web scraping covers picking the right proxy type, and how to avoid IP bans while scraping is the prevention checklist for when rotation alone is not enough. When a hand-managed pool becomes the bottleneck, a rotating gateway that returns a fresh residential IP per request at $0.44/GB replaces the whole switcher with one endpoint.