Python web scraping libraries compared: what each one is for

Python's scraping libraries do different jobs, so comparing them head to head mixes things up. Some download pages: requests, httpx and aiohttp. Some read the HTML: BeautifulSoup, lxml, parsel and selectolax. Scrapy does both for whole crawls, and Playwright and Selenium drive a real browser.

So we gave all of them the same job on 12 October 2026, on our own server with Python 3.12.3. The job was to collect 2,000 products from a 50-page test catalogue whose pages each took 50 ms to answer, like a real site. We also read PyPI's record of each library, to see which ones are still maintained.

The libraries at a glance

The downloads are pypistats.org's count for the 30 days before 12 October 2026. Sizes come from a fresh install of each library on its own. Playwright needs a browser on top of its 148.2 MB. One release date stands out: httpx has had none since December 2024.

One job, eleven setups

Each setup collected the same 2,000 products, in a fresh process, three times; we kept the middle run. The catalogue ran on the same server. Every page waited 50 ms before it answered, to stand in for the trip to a real site.

Four things stand out.

• One page at a time, the parser hardly mattered. All four setups with requests took 5.69 to 6.67 seconds, because most of the time went to waiting for pages. The parser shows in the processor time: BeautifulSoup used 1.93 seconds of it and lxml 0.72.
• Ten pages at a time cut the job to under 1.5 seconds. aiohttp with lxml took 1.28 seconds and httpx with selectolax 1.47. The waits overlapped instead of adding up.
• Scrapy's speed depends on where the spider lives. As a script it took 1.80 seconds. With the settings that scrapy startproject writes, the same spider took 61.91 seconds.
• Browsers cost seconds, and how you read the page decides the rest. Playwright took 38.37 seconds and Selenium 36.87 when each read a whole page in one call. Selenium reading each element's .text took 341.16 seconds. Chrome used 915 to 1,135 MB.

The sections below show each library on the same small task: the first three section headings of one of our own posts. Every block ran on our server, and its output is pasted under it.

Downloading pages: requests, httpx and aiohttp

requests is the simple default, and the other blocks on this page use it to download. httpx looks like requests and adds an async mode, which is how the job above asked for ten pages at a time.

import asyncio

import httpx
from selectolax.lexbor import LexborHTMLParser

URLS = [f"https://hproxy.com/blog/{slug}" for slug in ("web-scraping-with-python", "python-http-clients", "parse-json-python")]
HEADERS = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/python-web-scraping-libraries)"}


async def title(client, url):
    r = await client.get(url)
    return LexborHTMLParser(r.content).css_first("h1").text(strip=True)


async def main():
    async with httpx.AsyncClient(headers=HEADERS, timeout=30) as client:
        for t in await asyncio.gather(*(title(client, url) for url in URLS)):
            print(t)


asyncio.run(main())

Web scraping with Python: requests, BeautifulSoup and proxies, step by step
Python HTTP clients compared: requests, httpx, aiohttp and urllib3
Parse JSON in Python: json.loads, files and API responses

Our Python HTTP clients comparison covers what these clients do by default. requests has no timeout unless you set one, httpx does not follow redirects, and httpx.get() in a loop is slow.

Reading HTML: BeautifulSoup, lxml, parsel and selectolax

All four read the same HTML; they differ in speed and in how you point at things. We parsed one catalogue page of 40 products 200 times with each, and took the middle time.

selectolax was the fastest, lxml next, and BeautifulSoup about ten times slower than lxml. On a job with network waits that gap shrank to under a second for 50 pages, as the table above shows.

BeautifulSoup is the friendliest, and our BeautifulSoup tests show which of its parsers builds the same tree as a browser.

import requests
from bs4 import BeautifulSoup

URL = "https://hproxy.com/blog/web-scraping-with-python"
HEADERS = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/python-web-scraping-libraries)"}

soup = BeautifulSoup(requests.get(URL, headers=HEADERS, timeout=30).content, "lxml")
print([h2.get_text(strip=True) for h2 in soup.select("article h2")][:3])

['What you need', 'Step 1: download a page, then read the bytes', 'Step 2: find the data with CSS selectors']

lxml takes XPath as well as CSS, and is the engine under several of the others.

import lxml.html
import requests

URL = "https://hproxy.com/blog/web-scraping-with-python"
HEADERS = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/python-web-scraping-libraries)"}

tree = lxml.html.fromstring(requests.get(URL, headers=HEADERS, timeout=30).content)
print([h2.text_content().strip() for h2 in tree.xpath("//article//h2")][:3])

['What you need', 'Step 1: download a page, then read the bytes', 'Step 2: find the data with CSS selectors']

parsel, the selector library Scrapy uses, takes CSS and XPath and adds ::text.

import requests
from parsel import Selector

URL = "https://hproxy.com/blog/web-scraping-with-python"
HEADERS = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/python-web-scraping-libraries)"}

sel = Selector(text=requests.get(URL, headers=HEADERS, timeout=30).text)
print([text.strip() for text in sel.css("article h2::text").getall()][:3])

['What you need', 'Step 1: download a page, then read the bytes', 'Step 2: find the data with CSS selectors']

selectolax is "A fast HTML5 parser with CSS selectors, written in Cython, using the Lexbor engine", in its own description.

import requests
from selectolax.lexbor import LexborHTMLParser

URL = "https://hproxy.com/blog/web-scraping-with-python"
HEADERS = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/python-web-scraping-libraries)"}

tree = LexborHTMLParser(requests.get(URL, headers=HEADERS, timeout=30).content)
print([h2.text(strip=True) for h2 in tree.css("article h2")][:3])

['What you need', 'Step 1: download a page, then read the bytes', 'Step 2: find the data with CSS selectors']

Our page on XPath and CSS selectors compares the two ways of pointing at elements.

Crawling: Scrapy

Scrapy downloads, reads, follows links, retries and exports in one framework. A spider is a class, and scrapy runspider runs a single file.

import scrapy


class OutlineSpider(scrapy.Spider):
    name = "outline"
    start_urls = ["https://hproxy.com/blog/web-scraping-with-python"]
    custom_settings = {"USER_AGENT": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/python-web-scraping-libraries)"}

    def parse(self, response):
        for h2 in response.css("article h2::text").getall()[:3]:
            yield {"heading": h2.strip()}

Save it as outline_spider.py and run it with scrapy runspider.

scrapy runspider outline_spider.py -O headings.jsonl --nolog

{"heading": "What you need"}
{"heading": "Step 1: download a page, then read the bytes"}
{"heading": "Step 2: find the data with CSS selectors"}

How fast Scrapy goes depends on settings that differ by setup. Its documentation lists CONCURRENT_REQUESTS_PER_DOMAIN as "Default: 1 (fallback: 8)" and DOWNLOAD_DELAY as "Default: 1 (fallback: 0)". A project made with scrapy startproject gets the first values: one request at a time, a second apart, with robots.txt obeyed. A spider run as a script, as in our job, gets the fallbacks: eight at a time and no delay. That is the difference between 61.91 and 1.80 seconds above. Set both on purpose, and keep the polite values for sites that are not yours.

Driving a browser: Playwright and Selenium

Some pages build their content with JavaScript, so their HTML holds nothing to read. A browser library runs the page first. Playwright needs one more step after pip install playwright, in its documentation's words: "Install the required browsers: playwright install". It reads the matching elements' text in one call.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://hproxy.com/blog/web-scraping-with-python")
    print(page.locator("article h2").all_inner_texts()[:3])
    browser.close()

['What you need', 'Step 1: download a page, then read the bytes', 'Step 2: find the data with CSS selectors']

Selenium is the older of the two.

from selenium import webdriver
from selenium.webdriver.common.by import By

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
try:
    driver.get("https://hproxy.com/blog/web-scraping-with-python")
    print([h2.text for h2 in driver.find_elements(By.CSS_SELECTOR, "article h2")][:3])
finally:
    driver.quit()

['What you need', 'Step 1: download a page, then read the bytes', 'Step 2: find the data with CSS selectors']

Selenium reads elements one at a time unless you ask otherwise. In our job, reading each product's .text took 341.16 seconds, because every element it touches is a separate round trip to the browser. The same job with one execute_script call per page took 36.87 seconds.

Selenium also looks after its own driver. Its documentation calls Selenium Manager "a command-line tool implemented in Rust that provides automated driver and browser management for Selenium". When it cannot read the version of the browser you point it at, it fetches one. On our first run it downloaded Chrome 155 and used that instead of the Chrome 143 we had set. So check which browser your script really starts.

Libraries to skip for new code

PyPI's record shows two libraries that tutorials still mention but nobody releases any more.

• requests-html: its last release, 0.10.0, came out on 17 February 2019. One of the rival guides we read, updated in October 2026, still lists it among four options and shows how to install it.
• pyppeteer: its README says "This repo is unmaintained and has been outside of minor changes for a long time", and points to Playwright instead.

Which one to use

• A few hundred plain pages: requests with BeautifulSoup, or lxml if processor time matters.
• Thousands of pages: aiohttp or httpx with lxml or selectolax, ten or so at a time. Or Scrapy, with its concurrency and delay set on purpose.
• Pages that need JavaScript: Playwright, reading many elements in one call. Use a browser only for those pages.

Each library takes a proxy in its own way. Our guides cover requests, httpx, aiohttp, Scrapy, Playwright and Selenium.

What this page could not check

• The catalogue ran on the same server as the libraries, with a fixed 50 ms wait. Real sites answer slower and less evenly.
• The server was busy with its normal work, so the times are the middle of three runs and move between runs.
• The browsers ran the Chrome 143 already on our server, not the builds Playwright and Selenium would download.
• No site with bot protection was tested, and neither were libraries made for that, such as curl_cffi.
• We will rerun these tests by 12 January 2027.

Sources

• PyPI's JSON records and pypistats.org download counts for fifteen libraries, read on 12 October 2026: pypi.org.
• Scrapy documentation, settings: docs.scrapy.org.
• Selenium documentation, Selenium Manager: selenium.dev.
• Playwright for Python documentation, installation: playwright.dev.
• selectolax on PyPI: pypi.org.
• parsel documentation: parsel.readthedocs.io.
• lxml documentation: lxml.de.
• pyppeteer README: github.com/pyppeteer/pyppeteer.
• Our own tests on our server, 12 October 2026: libs_test.py at 05:21 UTC and the code on this page at 05:50 UTC, against a test catalogue of our own and hproxy.com.