A deep research agent searches, fetches, reads and writes all day. When it comes back with thin results, the first instinct is to export a proxy variable. Sometimes that fixes everything and sometimes it changes nothing, and the reason is always the same: an agent does not have one network path. It has several, and your variable reaches only some of them.
We have now read fifty three of these tools at named releases. This page is what they have in common.
One agent, several lanes
The clearest example is a single agent with two tools. In DeerFlow the page fetch tool reads its own configuration entry and passes it on:
proxy = _coerce_proxy(config.model_extra.get("proxy"))
Its search tool, in the same release, is constructed with a timeout and a configuration that has no proxy key at all. One agent, two tools, two answers.
| Lane | Who makes the request | Does a proxy variable reach it |
|---|---|---|
| a plain fetch | the agent's HTTP library | usually yes |
| a search through a keyed vendor | the vendor's servers | no, and your address is not used |
| a keyless search fallback | your machine | yes, and this is where blocks land |
| a browser scraper | a browser the agent launches | often not, see below |
| model calls | the model client | separately, if at all |
Two rules follow. The library decides, not the agent. And a keyed vendor lane never uses your address, so no proxy improves it.
There is a sharp edge on that second rule. In one agent we read, seven search providers fall back to a keyless search when their key is missing. A typo in an environment variable therefore does not produce an error. It quietly moves the search from a vendor's servers to your own address, which is exactly the lane that gets rate limited.
The lane that cannot be told
The pattern that surprised us most: the plain fetch is usually routed for free, and the browser scraper, which is what you reach for when a page refuses you, often cannot be told anything.
In GPT Researcher the Selenium scraper builds its own option list, with a user agent, headless mode, a debugging port and a sandbox flag, and no proxy argument. The newer browser scraper configures two fields: headless, and a connection timeout.
The good news is that the better designed ones let you bring the object. AutoGen's web surfer takes a context you built. One LlamaIndex reader takes a driver you built. A component that accepts the browser does not need a parameter for every browser option, and those are the ones that age well.
Five ways out of one conflict
Here is a conflict nobody warns you about. Several of these projects resolve a hostname, check that it is not a private or internal address, and then pin it so the connection goes to the address they checked. That is a sensible defence. It is also incompatible with a proxy, because behind a proxy the client does not resolve anything at all.
Five projects, five answers:
| Project | What it does when a proxy meets a pinned address |
|---|---|
| nanobot | installs the pinning only when no proxy is set |
| PicoClaw | keeps the guard and allows one hop to your proxy |
| Langflow | keeps the pinning and refuses the proxy with a named error |
| World Monitor | moves the fetch to a second host entirely |
| ZeroClaw | makes it selectable per service and fails closed on the two that cannot move |
Langflow's refusal is written into the code: DNS pinning with proxies is not currently supported. ZeroClaw's is written into the tool's own output: cannot be proxied: selecting this key makes the request fail closed so its validated answer remains pinned.
If your proxy stops working after a security update, this is the first thing to check.
Three walls
The projects themselves are blunt about this, and we will quote them against our own interest.
Agent-Reach's readme states that all Reddit access needs a logged in state, that the anonymous interfaces are comprehensively blocked and that the official API needs manual approval. career-ops tells the assistant that after two failed browser attempts it should mark the entry and ask the user to paste the text, and never treat a login wall as a verified posting.
No address opens a page that requires an account.
And one more, from World Monitor's build file: they install curl in their relay container because the runtime's TLS fingerprint was refused while curl passed. When the handshake is what a site rejects, a different address changes nothing.
What we measured
Three of our own tests sit behind this page.
An address changes some answers. 16 URLs, plain requests, from our server and through a residential line of our own plan: 4 of 13 answered differently. Indeed, Glassdoor, Amazon and Booking refused both.
A block list belongs to one range. World Monitor keeps a list of seventeen sources that refuse its platform, four of them public institutions. We asked ten of them from our own server: eight answered normally with a real feed. Only two refused us, and one source the project fetches directly refused us instead. A block list describes one platform's addresses, not hosting as a category.
A variable can be ignored by the runtime. On Node v22.19.0 we pointed the proxy variables at our own listener and ran a fetch. The listener saw nothing, with and without the opt in flag, because that flag was added in 22.21.0 and 24.5.0. The variable is not the setting. The runtime is.
Which proxy type fits it?
Work in this order, because three of the four steps are free.
Name the lane that failed. Check which library it uses and whether that library reads the variables. Slow down, if what you met was a rate limit. Then, for the lanes that use your own address and meet a block list, residential is the type that changes the answer, and 4 of 13 is the honest size of that effect in our test.
Research traffic is text: feeds, search results and extracted pages. The largest feed in our own measurement was 167,899 bytes, so a per gigabyte line goes a long way, and the browser lane is the one that will actually spend. An allowed address suits an agent particularly well, because several of these projects keep configuration in a file the agent itself can read: on a Residential Premium plan your machine's address can be allowed, up to 150 per plan. The residential proxies page lists the plans, and the plan API manages allowed addresses from code.
What breaks
- The proxy works for one source and not the next. Different lanes, different libraries.
- It stopped working after an update. Check whether a pinned address transport was installed.
- A search that used to be a vendor's is now yours. A missing key can silently move it.
- The browser part ignores everything. Some scrapers have no proxy option. Pass a context or a driver if the component allows it.
- Nothing you buy fixes it. Then it is a login wall, a rate limit or a handshake check, not an address.
What this page does not cover
We read releases as text and ran none of these agents, so no number here describes an agent in flight. Our three measurements are a plain request address test, a feed test from one hosting address and a runtime test on one machine, each on one day. The generalisations come from the agents we read, which are the popular ones, and a different sample might weight the lanes differently. The mechanism, that the library decides, does not depend on the sample. We will re-read the five designs and repeat the feed measurement by 20 November 2026.
Where to go from here
Proxies for DeerFlow is the clearest case of one agent with two different answers. Proxies for GPT Researcher covers the browser scrapers that cannot be told. Proxies for last30days covers the runtime version that decides whether a variable is read. Proxies for Agent-Reach covers an agent that routes rather than fetches. Proxies for World Monitor carries the block list measurement. Proxies for career-ops covers the job boards that refused both of our addresses.
Sources
- The two tools with two answers (bytedance/deer-flow, release v2.0.0). The two browser scrapers with no proxy option (assafelovic/gpt-researcher, release v3.6.1). The environment allowlist and the runtime flag (mvanhorn/last30days-skill v3.25.0, with the Node CLI documentation). The provider fallback to a keyless search (HKUDS/nanobot v0.3.5). The list of sources that refuse one platform (koala73/worldmonitor v2.10.0). The Reddit login statement (Panniantong/Agent-Reach v1.5.0). The throttle and the login wall rule (career-ops-hq/career-ops v1.33.0). All read 20 September 2026.
- The five designs for pinning against proxying: nanobot v0.3.5, sipeed/picoclaw v0.3.1, langflow-ai/langflow v1.12.2, koala73/worldmonitor v2.10.0, zeroclaw-labs/zeroclaw v0.8.5.
- Our own measurements: the paired address test of 19 September 2026, the relay only feed test of 20 September 2026, and the Node runtime test of 20 September 2026. Scripts and raw output are kept in the research folders of the pages that carry them.
- Plans, allowed addresses and per gigabyte pricing. HProxy documentation, hproxy.com/docs, 20 September 2026.
