Best language for web scraping: Python, JavaScript, Go and the rest

Any language that can send a request and read HTML can scrape a website. The real differences sit in the libraries, and they can be measured.

We ran the same scraping job in Python, JavaScript, Go and Java on our own server on 12 October 2026. We also checked how each one repairs broken HTML, waits on a silent server and uses a proxy.

The short answer

• Python is the safe default. It has the most help online, the shortest code here (24 lines) and, with the right parser, a run as fast as Go's.
• Choose the Python parser with care. With BeautifulSoup in eight threads, the same job took 9.57 seconds instead of 2.62.
• Go needs the least memory. It used 14 MB against 37 to 158 MB for the others, and builds into one file.
• JavaScript fits when you already write it or drive a browser from Node. It used the most memory here, 158 MB.
• Java suits teams already on the JVM. Its jsoup parser repairs HTML like a browser, but Java ignores the proxy variables.
• Whatever you choose, set a timeout. Three default clients were still waiting on a silent server after 330 seconds.

The test: one job in four languages

Our test shop runs on our server and answers every request after 0.1 seconds. It has five list pages of 20 products and 100 product pages of 80 KB each. Each product page carries a large menu, an inline data script and 24 related products with prices of their own.

Each scraper follows the "Next" links through the list pages, then fetches the product pages 8 at a time. It reads the name, price and SKU of each product and writes a CSV file. Every run had to produce the same 100 rows as the shop's own list.

We ran each scraper 5 times, the languages taking turns, and timed them with GNU time. Python ran four ways, to compare its parsers.

No scraper could finish much faster than 1.8 seconds. The five list pages come one after another, and 100 pages at 8 at a time need 13 rounds of 0.1 seconds. Go and Python with selectolax or lxml came close to that floor.

The slower runs were busy on the processor. Java's median processor time, 5.63 seconds, was longer than its median run, 5.02 seconds. So several of its threads worked at once.

Why BeautifulSoup made Python the slowest

BeautifulSoup builds its tree in Python code. With lxml, it asks the parser to call back into Python for every tag and every piece of text. Python lets only one thread run Python code at a time. Its documentation calls this lock "The mechanism used by the CPython interpreter to assure that only one thread executes Python bytecode at a time".

We parsed one product page 80 times, on one thread and spread over eight.

On one thread, BeautifulSoup with lxml was the faster of the two. On eight, it took five times as long, and its system time rose from 0.05 to 2.19 seconds. lxml alone hardly noticed the threads. Its FAQ warns that when most of the work is done in Python code, "your gain will be close to zero".

The fix is a parser that does its work outside Python code. lxml's own API and selectolax both finished the job in about 3 seconds or less.

The four scrapers

Each scraper takes the same steps in the same order. Each sets a timeout of 10 seconds and stops on an error status. The Python one uses selectolax, the fastest Python setup in our test.

import csv
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urljoin

import requests
from selectolax.lexbor import LexborHTMLParser

START = "http://127.0.0.1:8001/"


def get(url):
    r = requests.get(url, timeout=10)
    r.raise_for_status()
    return LexborHTMLParser(r.content)


def product(url):
    p = get(url).css_first("#product")
    return [p.css_first("h1").text(), p.css_first(".price").text(), p.attributes["data-sku"]]


# 1. the list pages: follow "Next" and collect the product links
links, url = [], START
while url:
    page = get(url)
    links += [urljoin(url, a.attributes["href"]) for a in page.css("a.product")]
    nxt = page.css_first("a.next")
    url = urljoin(url, nxt.attributes["href"]) if nxt else None

# 2. the product pages, 8 at a time
with ThreadPoolExecutor(8) as pool:
    rows = sorted(pool.map(product, links), key=lambda row: row[2])

with open("products.csv", "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows([["name", "price", "sku"], *rows])
print(len(rows), "products")

The JavaScript scraper uses the fetch built into Node and Cheerio.

import * as cheerio from "cheerio";
import { writeFileSync } from "node:fs";

const START = "http://127.0.0.1:8001/";

async function get(url) {
  const r = await fetch(url, { signal: AbortSignal.timeout(10_000) });
  if (!r.ok) throw new Error(`${r.status} ${url}`);
  return cheerio.load(await r.text());
}

async function product(url) {
  const p = (await get(url))("#product");
  return [p.find("h1").text(), p.find(".price").text(), p.attr("data-sku")];
}

// 1. the list pages: follow "Next" and collect the product links
const links = [];
for (let url = START; url; ) {
  const $ = await get(url);
  $("a.product").each((_, a) => links.push(new URL($(a).attr("href"), url).href));
  const next = $("a.next").attr("href");
  url = next ? new URL(next, url).href : null;
}

// 2. the product pages, 8 at a time
const rows = [];
const queue = [...links];
const worker = async () => {
  while (queue.length) rows.push(await product(queue.shift()));
};
await Promise.all(Array.from({ length: 8 }, worker));
rows.sort((a, b) => a[2].localeCompare(b[2]));

writeFileSync("products.csv", [["name", "price", "sku"], ...rows].map((row) => row.join(",")).join("\n") + "\n");
console.log(rows.length, "products");

The Go scraper uses net/http and goquery. Go asks for the most code, mostly error checks and the worker pool.

package main

import (
	"encoding/csv"
	"fmt"
	"net/http"
	"net/url"
	"os"
	"sort"
	"sync"
	"time"

	"github.com/PuerkitoBio/goquery"
)

const start = "http://127.0.0.1:8001/"

var client = &http.Client{Timeout: 10 * time.Second}

func get(u string) *goquery.Document {
	r, err := client.Get(u)
	if err != nil {
		panic(err)
	}
	defer r.Body.Close()
	if r.StatusCode != http.StatusOK {
		panic(fmt.Sprintf("%d %s", r.StatusCode, u))
	}
	doc, err := goquery.NewDocumentFromReader(r.Body)
	if err != nil {
		panic(err)
	}
	return doc
}

func resolve(base, ref string) string {
	b, _ := url.Parse(base)
	r, _ := b.Parse(ref)
	return r.String()
}

func product(u string) []string {
	p := get(u).Find("#product")
	return []string{p.Find("h1").Text(), p.Find(".price").Text(), p.AttrOr("data-sku", "")}
}

func main() {
	// 1. the list pages: follow "Next" and collect the product links
	var links []string
	for page := start; page != ""; {
		doc := get(page)
		doc.Find("a.product").Each(func(_ int, a *goquery.Selection) {
			links = append(links, resolve(page, a.AttrOr("href", "")))
		})
		if next, ok := doc.Find("a.next").Attr("href"); ok {
			page = resolve(page, next)
		} else {
			page = ""
		}
	}

	// 2. the product pages, 8 at a time
	rows := make([][]string, len(links))
	jobs := make(chan int)
	var wg sync.WaitGroup
	for w := 0; w < 8; w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for i := range jobs {
				rows[i] = product(links[i])
			}
		}()
	}
	for i := range links {
		jobs <- i
	}
	close(jobs)
	wg.Wait()
	sort.Slice(rows, func(i, j int) bool { return rows[i][2] < rows[j][2] })

	f, err := os.Create("products.csv")
	if err != nil {
		panic(err)
	}
	defer f.Close()
	w := csv.NewWriter(f)
	w.Write([]string{"name", "price", "sku"})
	w.WriteAll(rows)
	fmt.Println(len(rows), "products")
}

The Java scraper uses the HttpClient built into Java and jsoup.

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

public class Scraper {
    static final String START = "http://127.0.0.1:8001/";
    static final HttpClient client = HttpClient.newHttpClient();

    static Document get(String url) throws Exception {
        var request = HttpRequest.newBuilder(URI.create(url)).timeout(Duration.ofSeconds(10)).build();
        var response = client.send(request, HttpResponse.BodyHandlers.ofString());
        if (response.statusCode() != 200) throw new Exception(response.statusCode() + " " + url);
        return Jsoup.parse(response.body(), url);
    }

    static List<String> product(String url) throws Exception {
        Element p = get(url).selectFirst("#product");
        return List.of(p.selectFirst("h1").text(), p.selectFirst(".price").text(), p.attr("data-sku"));
    }

    public static void main(String[] args) throws Exception {
        // 1. the list pages: follow "Next" and collect the product links
        var links = new ArrayList<String>();
        for (String url = START; url != null; ) {
            Document page = get(url);
            for (Element a : page.select("a.product")) links.add(a.absUrl("href"));
            Element next = page.selectFirst("a.next");
            url = next == null ? null : next.absUrl("href");
        }

        // 2. the product pages, 8 at a time
        var pool = Executors.newFixedThreadPool(8);
        var futures = new ArrayList<Future<List<String>>>();
        for (String url : links) futures.add(pool.submit(() -> product(url)));
        var rows = new ArrayList<List<String>>();
        for (var f : futures) rows.add(f.get());
        pool.shutdown();
        rows.sort(Comparator.comparing(row -> row.get(2)));

        var lines = new ArrayList<String>(List.of("name,price,sku"));
        for (var row : rows) lines.add(String.join(",", row));
        Files.write(Path.of("products.csv"), lines);
        System.out.println(rows.size() + " products");
    }
}

To run them, install each one's libraries first. For Java, put jsoup.jar from Maven Central in the same folder.

pip install requests selectolax
python scraper.py
npm install cheerio
node scraper.mjs
go mod init scraper
go get github.com/PuerkitoBio/goquery
go build -o scraper .
./scraper
javac -cp jsoup.jar Scraper.java
java -cp jsoup.jar:. Scraper

Each one printed the same line, in every run.

100 products

The libraries add up differently. requests, BeautifulSoup and lxml brought 9 Python packages, 16.7 MB. Cheerio brought 23 npm packages, 6.4 MB. Go compiled three libraries into one 9.7 MB file. Java needed one 0.54 MB jar, plus a Java runtime.

That Go file still uses the system's C library. Built with CGO_ENABLED=0, it was 9.6 MB and fully static.

Which parsers see the page like a browser

Real pages are full of broken HTML, and browsers repair it by fixed rules from the HTML standard. A parser that repairs it another way builds another tree. Selectors copied from Chrome can then find nothing.

We gave each parser six pieces of broken HTML and a table written without <tbody>. Chrome's own parser was the referee.

Two paragraphs without end tags show the difference.

<p>first<p>second
   Chrome                         <p>first</p><p>second</p>
   BeautifulSoup + html.parser    <p>first<p>second</p></p>

BeautifulSoup's html.parser and lxml are the only parsers here that disagree with the browser. Cheerio's documentation says its default parser "produces the same tree a browser would". jsoup "parses HTML to the same DOM as modern browsers". Our page on BeautifulSoup shows the same trap in more cases.

Timeouts: when a server never answers

A server that accepts the connection and then sends nothing is a quiet way for a scraper to hang. We pointed each language's usual client at one, with no timeout set.

The documentation of each says so. Requests: "By default, requests do not time out unless a timeout value is set explicitly". Java's HttpRequest: not setting a timeout means "block forever". Go: "A Timeout of zero means no timeout", and http.Get uses a client with none set. Node's fetch waits 300 seconds for headers, and jsoup 30 seconds.

So every scraper on this page sets 10 seconds. Our page on Python HTTP clients measures the same trap in six Python libraries.

Proxies: which clients read HTTPS_PROXY

Setting HTTPS_PROXY is the quickest way to send a program through a proxy. Only some clients read it. The others go straight to the site, with no error.

We set HTTPS_PROXY and its siblings to a test proxy on our server and fetched one HTTPS page. The proxy's log showed which requests passed through it.

For Node 20, our JavaScript scraping page tested undici's ProxyAgent, and it worked. -Djava.net.useSystemProxies=true did not help on our Linux server. Java's documentation limits it to "Windows systems, macOS systems, and Gnome systems".

Node added an opt-in switch in versions 22.21 and 24.0. With NODE_USE_ENV_PROXY=1, "Node.js parses the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables during startup". Our test ran Node 20, which has no such switch.

Our proxy guides cover each language in depth: Python requests, Node.js, Go and Java.

Browsers: which languages can drive one

Some sites show their data only after JavaScript runs in a browser. Then the language needs a browser library. Playwright, Selenium and Puppeteer each support a different set.

Go is not among their official languages. Our page on waiting in Playwright shows the most common browser trap.

How many people can help

When a scraper breaks, you search for someone who had the same problem. We counted questions on Stack Overflow and projects on GitHub, both on 12 October 2026.

Stack Overflow holds 51,300 web-scraping questions, and about 58 percent carry the Python tag. On GitHub, 11,343 of 22,147 web-scraping repositories are mainly Python, about 51 percent.

The languages we did not run

We did not test R, PHP, Ruby, C# or Rust. Of these, R has the most Stack Overflow questions, more than JavaScript. C# is the one both Playwright and Selenium support.

What this page could not check

• One test shop on one server, under that server's normal load. Real sites add blocking, JavaScript and slower answers.
• One version of each: Python 3.12.3, Node 20.20.2, Go 1.27.2 and Java 25.0.4.1.
• Python 3.14 made its build without the lock "officially supported". We tested the standard build.
• Single runs varied. Java took between 3.82 and 5.55 seconds, so we compare medians of 5.
• Lines of code follow our own style. Another style moves them by a few lines.
• The Stack Overflow and GitHub counts cover all years.
• We will check this page again by 12 January 2027.

Sources

• Python documentation, glossary, global interpreter lock: docs.python.org.
• What's new in Python 3.14, PEP 779: docs.python.org.
• lxml FAQ, threads and the GIL: lxml.de.
• Beautiful Soup documentation, differences between parsers: crummy.com.
• Requests documentation, timeouts and proxies: requests.readthedocs.io.
• Go net/http and x/net/html: pkg.go.dev and pkg.go.dev.
• Java 25, HttpClient.Builder, HttpRequest.Builder and networking properties: docs.oracle.com.
• jsoup home page and Connection.timeout: jsoup.org and jsoup.org.
• Node.js command line options, NODE_USE_ENV_PROXY: nodejs.org.
• undici Client, headersTimeout: github.com.
• Cheerio, configuring the parser: cheerio.js.org.
• selectolax: github.com.
• Playwright supported languages, Selenium language bindings and Puppeteer: playwright.dev, selenium.dev and pptr.dev.
• Stack Exchange API and GitHub REST API: api.stackexchange.com and docs.github.com.
• Our own tests on our server, 12 October 2026: run_same_job.py at 10:54 UTC, parse_probe.py at 10:39, run_defaults.py at 10:47, run_trees.py at 10:49 and community.py at 10:51.