How to Use Proxies With Go (net/http and Colly)

Go's standard library has proxy support built in and it is genuinely clean, which is rare. The catch is that the clean built-in path covers HTTP and HTTPS proxies but stops at SOCKS, the proxy lives on the http.Transport rather than on a request, and a custom Transport silently drops the environment-variable handling you got for free. This guide covers the standard-library way, SOCKS5 through the extended x/net package, authentication, rotating a pool with Colly, and the verification step that proves the proxy is carrying your traffic.

We run a proxy network and write a fair amount of Go against it, so the mistakes below are ones we have made ourselves: a scraper that quietly went direct because a custom Transport forgot its Proxy field, a SOCKS setup that would not compile until the dialer was asserted to the right interface. Every example is a complete, runnable program. Where one needs a live proxy, pull a fresh one from our free proxy API, which returns real, recently checked endpoints without a key.

How do you use a proxy with Go?

Build an http.Transport with its Proxy field set to http.ProxyURL(parsed), attach that Transport to an http.Client, and every request through that client goes via the proxy. For environment-driven proxying use http.ProxyFromEnvironment. For SOCKS5, step outside net/http into golang.org/x/net/proxy. The proxy is a property of the client's Transport, not of any single request.

The Transport.Proxy field

http.Transport has a Proxy field with the signature func(*http.Request) (*url.URL, error). It is called once per request to decide which proxy, if any, to use. The standard library ships two functions that fit that shape: http.ProxyURL, which always returns the same fixed proxy, and http.ProxyFromEnvironment, which reads the environment. For a single known proxy, http.ProxyURL is all you need:

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func main() {
	proxyURL, err := url.Parse("http://user:[email protected]:8080")
	if err != nil {
		panic(err)
	}

	transport := &http.Transport{
		Proxy: http.ProxyURL(proxyURL),
	}
	client := &http.Client{
		Transport: transport,
		Timeout:   15 * time.Second,
	}

	resp, err := client.Get("https://httpbin.org/ip")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	body, _ := io.ReadAll(resp.Body)
	fmt.Println(string(body))
}

Two things are quietly doing the right thing here. For an https:// target, the Transport opens a CONNECT tunnel through the proxy and performs TLS end to end with the origin, so a plain HTTP proxy carries your encrypted traffic without seeing inside it. And the user:pass in the proxy URL is not decorative: net/http reads the userinfo and sets the Proxy-Authorization header for you, so authenticated HTTP proxies need no extra code.

Environment variables, and the custom-Transport trap

http.DefaultTransport, the Transport a plain http.Get uses, already has Proxy: http.ProxyFromEnvironment set. That is why an ordinary Go program respects HTTP_PROXY, HTTPS_PROXY and NO_PROXY with no code at all. Per the net/http documentation, ProxyFromEnvironment matches the request scheme to the right variable, reads either a full URL or a bare host[:port] (assuming http when the scheme is missing), and returns no proxy for localhost.

The trap is what happens the moment you build your own Transport, which you eventually will, for a custom timeout or TLS config. A Transport you construct has an empty Proxy field, so it ignores the environment completely. If you want the environment respected on a custom Transport, say so explicitly:

transport := &http.Transport{
	Proxy: http.ProxyFromEnvironment, // restore env handling on a custom Transport
	// ... your other settings ...
}

This one line is the fix for the surprisingly common "my HTTP_PROXY stopped working after I customized the client" report.

SOCKS5 proxies

net/http does not speak SOCKS, but the extended standard library does, in golang.org/x/net/proxy. Install it and build a dialer:

go get golang.org/x/net/proxy

proxy.SOCKS5(network, address, *Auth, forward) returns a Dialer. To wire it into an http.Transport you assert that dialer to proxy.ContextDialer and hand its DialContext to the Transport:

package main

import (
	"fmt"
	"io"
	"net/http"
	"time"

	"golang.org/x/net/proxy"
)

func main() {
	auth := &proxy.Auth{User: "user", Password: "pass"}
	dialer, err := proxy.SOCKS5("tcp", "203.0.113.7:1080", auth, proxy.Direct)
	if err != nil {
		panic(err)
	}

	// The SOCKS5 dialer implements ContextDialer; assert to use DialContext.
	ctxDialer, ok := dialer.(proxy.ContextDialer)
	if !ok {
		panic("SOCKS5 dialer is not a ContextDialer")
	}

	transport := &http.Transport{DialContext: ctxDialer.DialContext}
	client := &http.Client{Transport: transport, Timeout: 15 * time.Second}

	resp, err := client.Get("https://httpbin.org/ip")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	body, _ := io.ReadAll(resp.Body)
	fmt.Println(string(body))
}

If you would rather parse a URL than pass arguments, proxy.FromURL("socks5://user:[email protected]:1080", proxy.Direct) does the same thing from a *url.URL. Worth knowing on the DNS front: the x/net SOCKS5 dialer forwards the destination hostname to the proxy, so resolution happens on the proxy's side, the remote-DNS behavior curl calls socks5h. You do not opt into it separately in Go, you get it by default, which is the privacy-preserving choice for scraping. We cover why that matters in what is a SOCKS5 proxy.

Rotating a pool with Colly

For scraping, most Go code uses Colly, and Colly has proxy rotation built in. proxy.RoundRobinProxySwitcher takes a list of proxy URLs and returns a function you attach with SetProxyFunc; it hands out a different exit on every request. Per its documentation, it understands http, https and socks5 schemes, and treats a missing scheme as http:

package main

import (
	"fmt"

	"github.com/gocolly/colly/v2"
	"github.com/gocolly/colly/v2/proxy"
)

func main() {
	c := colly.NewCollector()

	rp, err := proxy.RoundRobinProxySwitcher(
		"http://203.0.113.7:8080",
		"http://203.0.113.24:3128",
		"socks5://198.51.100.14:1080",
	)
	if err != nil {
		panic(err)
	}
	c.SetProxyFunc(rp)

	c.OnResponse(func(r *colly.Response) {
		fmt.Println(string(r.Body)) // prints the exit IP that httpbin saw
	})

	c.Visit("https://httpbin.org/ip")
	c.Visit("https://httpbin.org/ip") // a different exit than the first visit
}

For a single fixed proxy, c.SetProxy("http://203.0.113.7:8080") is the shortcut; it wraps a one-entry switcher internally. You do not have to hardcode the list either. Our free proxy API returns a fresh pool you can pull at startup and splice straight into RoundRobinProxySwitcher. For production, a rotating gateway that gives you one endpoint and a fresh residential IP per request removes list management entirely, but rotating a real list first is the best way to understand what that gateway is doing for you.

Verify the exit IP

Never assume the proxy is working. Confirm the target sees the proxy's address and not yours before you trust an exit with real traffic. The clean check compares a direct request against a proxied one:

func exitIP(client *http.Client) string {
	resp, err := client.Get("https://httpbin.org/ip")
	if err != nil {
		return "error: " + err.Error()
	}
	defer resp.Body.Close()
	body, _ := io.ReadAll(resp.Body)
	return string(body)
}

// direct := &http.Client{Timeout: 10 * time.Second}
// proxied := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)}}
// fmt.Println("real: ", exitIP(direct))
// fmt.Println("proxy:", exitIP(proxied))

If the two responses show the same IP, the request never went through the proxy. It is also worth checking https://httpbin.org/headers through the proxy, because a transparent proxy can forward your real address in X-Forwarded-For even while origin looks changed. Our proxy checker runs that whole battery, exit IP, anonymity grade, real geolocation, and latency, in one paste if you would rather not script it.

Where to go from here

Two habits keep Go proxy code alive unattended: set http.Client.Timeout so a silent proxy cannot hang a goroutine forever, and verify every exit before you trust it. For a live pool to test against, the free proxy list re-verifies its entries every few minutes, and the free API hands them to you with no key.

From here, proxies for web scraping covers choosing the right proxy type and the request hygiene no HTTP client can add for you, the cURL guide is the fastest way to sanity-check a proxy from the shell before you wire it into code, and if you hit an auth wall, how to fix 407 Proxy Authentication Required is the focused fix. When a project graduates to production, get IPs nobody else is burning on our paid pools.

Sources

• net/http package documentation: Transport.Proxy, http.ProxyURL, and http.ProxyFromEnvironment, including the HTTP_PROXY/HTTPS_PROXY/NO_PROXY and localhost rules.
• golang.org/x/net/proxy: SOCKS5, FromURL, the Auth struct, and the Dialer/ContextDialer interfaces used with Transport.DialContext.
• Colly proxy package: RoundRobinProxySwitcher and the http/https/socks5 schemes it accepts.