Concurrency
How many requests you run at the same time, the main lever on scraping throughput and a common hidden limit.
Concurrency is the number of requests in flight at once. It, far more than any single request's speed, sets how much a scraper gets done: a job that fetches a hundred pages in parallel finishes in a fraction of the time of one that fetches them one after another, even if each individual request is no faster.
Proxies interact with concurrency in two ways. A pool lets you run high concurrency without piling requests onto one address, since each parallel request can take a different exit. But the proxy service itself may cap concurrent connections, and a plan that limits threads quietly limits throughput no matter how fast the network is.
The number worth checking on any proxy is therefore not just price per gigabyte but whether concurrent connections are limited. Unlimited concurrency is what lets a pool's real advantage, parallelism, actually show up.
More is not automatically better, though, and this is where people overshoot. Concurrency raises throughput only until something else becomes the bottleneck: your own bandwidth, your CPU if you are parsing as you go, the target's tolerance, or memory if you are running real browsers, each of which consumes far more than a plain HTTP request. Past that point, raising it produces timeouts and retries rather than results, and retries cost you traffic you already paid for.
It also interacts with rate limiting in a way that catches people out. Fifty parallel requests through fifty different exits is fifty separate visitors as far as a per-address counter is concerned. Fifty parallel requests through ONE address is a burst that will trip almost any threshold instantly. Concurrency and pool size have to move together, or the parallelism you added simply gets you blocked faster.
The honest way to find your number is empirically and from below. Start conservative, raise it in steps, and watch the error rate rather than the throughput graph. The right setting is the one just below where failures begin, and it differs per target because it is really a property of the target rather than of your machine.
How HProxy handles it
Our residential plans do not cap concurrent connections or threads, because a thread limit is a throughput limit wearing different words, and it is exactly the constraint that makes a cheap-looking plan slow in practice.
Frequently asked questions
What is a good concurrency level for scraping?
There is no universal number, because the limit belongs to the target rather than to you. Start low, raise it in steps, and watch the error rate: the right setting is just below where failures begin. A tolerant site may accept hundreds of parallel requests, a strict one a handful, and the same code will need different settings for each.
Does higher concurrency mean faster scraping?
Up to a point, then it reverses. Once you pass your own bandwidth, CPU or memory ceiling, or the target's tolerance, extra parallelism produces timeouts and retries rather than results. Since retries pay for traffic that already crossed the wire, over-shooting concurrency on per-gigabyte billing costs money as well as time.
Do proxy providers limit concurrent connections?
Many do, and it is often buried in the plan details rather than advertised. A thread or connection cap is a throughput cap in different words, and it can make a cheap-looking per-gigabyte rate slow in practice. It is worth checking before comparing prices, because two plans at the same rate can differ enormously in what they let you actually run.
How does concurrency relate to the size of my proxy pool?
They have to move together. Parallel requests spread across many exits look like many separate visitors, which is the point. The same parallelism forced through one address is a burst that trips a per-address limit immediately. Raising concurrency without enough distinct addresses behind it just gets you blocked faster than running sequentially would have.
Back to the full glossary.