Crawlee is the rare scraping library that ships a real proxy strategy rather than a setting. It can hold one address per session, and it can climb between tiers of proxies on its own. Both are worth understanding before you buy the expensive tier, because what moves the crawler between them is narrower than most people expect. We read this in Crawlee 3.18.1 for Node and 1.10.1 for Python on 20 September 2026.
Three options, one at a time
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: ['http://USER:PASS@GATEWAY_HOST:GATEWAY_PORT'],
});
There are three ways to supply addresses, and you may use exactly one. A flat list, a list of tiers, or a function you write. Setting two of them throws an error. Any list may also contain a null, which means no proxy for that pick, and the guide suggests it as a first tier: try direct, then pay.
How an address is chosen
| Your setup | What decides the address |
|---|---|
| A list, with a session | the first address picked for that session, kept for its life |
| A list, no session | the next address in the list, in order |
| Tiers, in Node | the next address inside the predicted tier; the session is ignored |
| Tiers, in Python | the address stored for that session, kept even when the tier moves |
| A function | whatever you return |
That third row surprised us. In the Node version the function that handles tiers takes the session identifier as a parameter and never reads it. The Python version stores the chosen address under the session identifier after the tier is picked, so a session keeps its first address. The same program in the two languages does not behave the same way here.
What makes the tier move
One thing only: a retry.
A request that comes back for another attempt carries the tier it used last. That tier is penalised, and only then is a new tier predicted. Nothing else reports anything to the tracker. So a page that answers 200 with a block message, a captcha or an empty result is not a failure to Crawlee, nothing is retried, and the expensive tier is never reached. In our run of its algorithm with nothing ever retried, all twelve requests stayed on the first tier.
The arithmetic is small enough to hold in your head. An error adds ten points against a tier. Every later prediction takes one point back off every tier that is not currently in use. Then the crawler compares its tier with its neighbours and steps one place toward the lower score.
Where 100 retried requests actually go
That last rule has a consequence nobody writes about. The score of the tier you are on is the only score that never decays, so the tracker is always eventually pushed off whatever tier it sits on. It does not settle. We re-implemented the published algorithm and ran it on a domain where every request is retried:
An even split. With two tiers it is fifty and fifty. The first climb is fast, off tier 0 on the second request and touching the top tier on the third, and then it cycles.
This is partly intentional. The project says it "periodically probes lower tier proxies to see if they are unblocked", which is how it saves money. The number is the part worth knowing: on a domain that keeps refusing you, the probing is not occasional. It is a third of your requests.
Two practical readings. If your first tier is null, a direct connection, then a third of your requests to a blocking domain keep leaving from your own server address, indefinitely. And if your top tier is a paid residential line, it carries about a third of the load rather than all of it, which is good for the bill and worth knowing before you size it.
What does not go through the proxy
The robots file. The crawler fetches it with a function that accepts a proxy, and calls that function without one. With the robots setting enabled, that is one request per origin leaving from your own address, before any proxied request happens. A fix has been proposed and is not merged.
If that matters to you, fetch the robots file yourself through the proxy and hand it to the crawler, or leave the setting off and decide the rules another way.
Browsers never see your password
A pleasant surprise in the browser crawlers. A browser cannot take a proxy login on its own, so when your address carries credentials Crawlee starts a small local forwarder, hands the browser that local address, and closes it with the browser. Your login stays in the process. Without credentials no forwarder is started at all.
The opposite hygiene point is worth a line too. The request handler receives the proxy address including the user name and password, which is convenient for logging which address was used, and an easy way to write a password into a log file.
Which proxy type fits it?
This library is built for exactly the split we sell into: something cheap underneath, something residential on top. In our own test a residential address changed 4 of 13 answers where a server address failed, and four sites refused both. That is the size of the prize, and the reason to keep a cheap tier under it rather than routing everything through the expensive one.
Two things to match. First, put a sticky port in the tier you want to hold an address, because Crawlee's session pinning only means anything if the address behind it stays put; write a retry anyway, since a sticky address can still change early when its device leaves the network. Second, remember our lines are priced per gigabyte, and the tier arithmetic above decides what share of your traffic lands on the paid tier.
The residential proxies page lists the plans, and the plan API manages allowed addresses from code if you would rather not put a password in the configuration at all.
What breaks
- The crawler never climbs. Nothing is being retried. Make your handler fail on a block page, not just on a network error.
- A proxy failure is not recognised. In the Python version a refused tunnel stopped being mapped to a proxy error after a client update, so sessions were not rotated. Keep the client library current.
- Tiers behave like one flat list. The tier options need a crawler around them. Used on their own, the tiers are flattened into a single rotation.
- Your own address appears in the logs of a site you proxy. The robots file, or a first tier of
null. - 407 Proxy Authentication Required. The login in the address is wrong or missing. Our 407 guide covers it.
- Node and Python disagree. Sessions and tiers combine differently. Check the language you actually run.
What this page does not cover
We read both libraries as text and did not run a crawl. The tier numbers come from running the project's own published algorithm, not from live traffic, so a real crawl that mixes successes and failures will split differently. The shape is the point: it cycles rather than settling. The robots file claim rests on the call site and the function signature in the shipped code plus the open pull request; we did not capture the request. Version 4.0 is in prerelease and its development branch is already ahead, so we will read this again at the next stable release, by 20 October 2026.
Where to go from here
Proxies for Scrapling covers a scraper that takes one address and no strategy. Sticky versus rotating sessions explains the port choice behind the session pinning above. Datacenter versus residential is the tier decision itself.
Sources
- The three options and their validation, the session pinning, the tiered path and the tier tracker (packages/core/src/proxy_configuration.ts). The local forwarder for browsers (packages/browser-pool/src/anonymize-proxy.ts). The robots call site (packages/basic-crawler/src/internals/basic-crawler.ts) and the function it calls (packages/utils/src/internals/robots.ts). apify/crawlee, release v3.18.1 of 12 August 2026, read 20 September 2026.
- The Python selection and pinning, and its copy of the tier tracker (src/crawlee/proxy_configuration.py). apify/crawlee-python, release v1.10.1 of 16 September 2026.
- The proxy management guide and the project's own post on tiered proxies, both shipped in the repository.
- Pull request 3940 on the robots file, issue 2599 and issue 821 on SOCKS support, issue 2111 on a refused tunnel that was not counted as a proxy error. Crawlee issue trackers, read 20 September 2026.
- Our own run of the published tier algorithm, 20 September 2026. The script and its output are kept in the research folder of this page.
- Our paired address test of 19 September 2026, from our server and through a residential line of our house plan. Raw output is kept in the research folder of our OpenClaw page.
- Plans, allowed addresses and sticky sessions. HProxy documentation, hproxy.com/docs, 20 September 2026.
