Concurrency vs parallelism in web scraping: requests, threads and proxies

Concurrency means several tasks are in progress at once and take turns, so the time one spends waiting is used by another. Parallelism means several tasks run at the same moment on different processor cores. A scraper spends most of its time waiting for the network. Concurrency (threads or asyncio) gives it most of its speed. Parallelism (processes) pays off only for heavy parsing.

We timed both on 10 October 2026 in Python 3.12. We fetched 100 pages that each took 200 ms to answer. One at a time took 26.84 seconds, 10 threads took 3.11 seconds and asyncio took 0.75 seconds. Parsing 400 pages took 3.47 seconds on one thread, longer on 4 threads, and 1.05 seconds on 4 processes.

What is the difference?

The Python documentation draws the same line. In CPython, "due to the Global Interpreter Lock, only one thread can execute Python code at once". Even so, "threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously". Processes avoid the lock: the multiprocessing package works by "effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads".

What we measured

Our test site was a stub on the same server. It answered each request after 200 ms with a 26 KB page, so the network wait was the same every time. A second stub acted as a forward proxy. Each method fetched 100 pages five times, and the table shows the median.

Fetching pages: concurrency wins

Fetching is waiting, so overlapping the waits is what counts. Ten threads cut the time from 26.84 to 3.11 seconds, and asyncio was the fastest at 0.75 seconds. Four processes that each fetched one page at a time only divided the time by four, to 6.54 seconds. For fetching, processes on their own are the slowest way to go fast.

With threads, size the connection pool to the number of workers:

from concurrent.futures import ThreadPoolExecutor
import requests
from requests.adapters import HTTPAdapter

WORKERS = 50
session = requests.Session()
adapter = HTTPAdapter(pool_connections=10, pool_maxsize=WORKERS)
session.mount("http://", adapter)
session.mount("https://", adapter)

def fetch(url):
    return session.get(url, timeout=(3.05, 27)).text

with ThreadPoolExecutor(max_workers=WORKERS) as pool:
    pages = list(pool.map(fetch, urls))

With asyncio, the connector sets how many connections run at once. Its default is 100.

import asyncio
import aiohttp

async def main(urls):
    connector = aiohttp.TCPConnector(limit=50)
    async with aiohttp.ClientSession(connector=connector) as session:
        async def fetch(url):
            async with session.get(url) as r:
                return await r.text()
        return await asyncio.gather(*(fetch(u) for u in urls))

pages = asyncio.run(main(urls))

Without a number, ThreadPoolExecutor picks min(32, os.cpu_count() + 4) threads. That suits a laptop, but it is a guess about your workload, not a measurement. Pick the count from the site's limits, covered below.

Parsing pages: parallelism wins

Parsing is computing. We parsed 400 copies of the page with the standard library parser.

Four threads were slower than one, because only one thread could run the parser at a time and switching between them cost extra. Four processes cut the time to about a third. Here threads lose.

from concurrent.futures import ProcessPoolExecutor

def parse(html):
    ...  # your parsing code, it must be a top-level function

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as pool:
        items = list(pool.map(parse, pages, chunksize=25))

One change is on its way. Since Python 3.13 there is a separate free-threaded build, in which "the global interpreter lock (GIL) is disabled". Our numbers come from the standard build, which is what most people run.

The "connection pool is full" warning

A requests Session keeps 10 connections per host unless told otherwise: the source code sets DEFAULT_POOLSIZE = 10. With 50 threads on one session, urllib3 printed "Connection pool is full, discarding connection" 40 times per run in our test. The 40 extra connections were simply thrown away when their request ended.

On our local test that did not cost time, because opening a connection on the same machine is almost free. On a real site it is not. The requests documentation explains that reusing the connection to a host "can result in a significant performance increase". A discarded connection cannot be reused. Setting pool_maxsize to the number of threads removed every warning, both directly and through the proxy. The proxy counts as the host here, so the same limit of 10 applies to the connections you keep open to it.

Concurrency through proxies

More threads do not mean more requests a site will accept. Sites count requests per address. The ones that limit rates answer 429 Too Many Requests and may add a Retry-After header that says how long to wait. Fifty threads through one exit IP look like one very busy visitor.

The answer is to spread the workers over exits. Our gateway gives each generated line its own sticky session, so ten lines give "ten simultaneous stable identities". One line per worker keeps each worker on one address for its whole task, such as a login, while the workers together use many. Our residential proxies work this way. Our guide on how to fix 429 Too Many Requests covers what to do when a site starts refusing.

Which one should you use?

• Fetching: asyncio first, threads second. Both beat processes by a wide margin for network waits. With threads, set pool_maxsize to the thread count.
• Parsing: processes. Threads lost to a single thread in our test.
• Both: fetch with threads or asyncio, then hand the pages to a process pool for parsing.
• Through proxies: size the concurrency to what each site accepts per IP, and give each worker its own exit.

If requests start to hang as you add workers, set timeouts on every call. Python requests timeout shows which error each stall raises.

What this page could not check

Our site and proxy were stubs on one server with a fixed 200 ms delay. The test shows waiting and pooling, not real network effects. We did not measure what a discarded connection costs over the internet, where it means a new TCP and TLS handshake. We ran the standard Python build, not the free-threaded one, and did not test Scrapy, httpx or a real proxy network. The server was busy with other work during the test, so single runs varied, and we report medians of five. Python, requests, urllib3 and aiohttp change between releases, so we will run the benchmark again by 15 January 2027.

Sources

• Python documentation: threading, concurrent.futures, multiprocessing, asyncio, and the free threading HOWTO, read 10 October 2026.
• Requests source code, src/requests/adapters.py, psf/requests on GitHub, read 10 October 2026.
• Requests documentation, Advanced Usage: Session Objects, for Requests 2.34.2, read 10 October 2026.
• urllib3 source code, src/urllib3/connectionpool.py, read 10 October 2026.
• aiohttp documentation, client reference: TCPConnector, read 10 October 2026.
• RFC 6585, Additional HTTP Status Codes, IETF, April 2012: 429 Too Many Requests.
• HProxy documentation: plans and line generation, read 10 October 2026.
• Our own benchmark on 10 October 2026: bench.py with Python 3.12.3, requests 2.34.2, urllib3 2.8.0 and aiohttp 3.14.4.