Set HTTP_PROXY for a CrewAI crew, run it, and the scraping tool still reads pages from your own address. That is not a bug. CrewAI's website scraping tool fetches through a helper that ignores the environment on purpose, and refuses a proxy if you pass one by hand. The reason is worth knowing, because it also tells you which routes are left. We read every setting on this page in release 1.15.22, published on 16 September 2026.
Which part of a crew takes a proxy
| Part of the crew | Do the proxy variables reach it? |
|---|---|
ScrapeWebsiteTool | no, the helper sets trust_env to false and empties its proxies |
| Search and vendor API tools | yes, they call requests directly |
SeleniumScrapingTool | not by itself, but it accepts a driver you build |
| A custom tool you write | whatever your own code does |
Why the scraping tool refuses it
The tool calls safe_get, and the first lines of that module say what it does: it validates every URL and redirect hop, then fetches "through a session that pins TCP to the checked IP and ignores environment proxies". Passing a proxy yourself raises ValueError: Proxies are not allowed for safe_get.
The pull request that added the pinning explains the reasoning in one sentence: "An HTTP proxy becomes the connected peer. Then the destination IP is not checked." The check exists because the validator behind it was shipped as a security fix, and a proxy would quietly defeat it: the tool would verify one address and connect to another.
So this is a deliberate trade. The tool keeps a guarantee about where it connects, and gives up the ability to route.
The three routes that are left
# 1. the Selenium tool, with a browser you build
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from crewai_tools import SeleniumScrapingTool
options = Options()
options.add_argument("--headless")
options.add_argument("--proxy-server=http://GATEWAY_HOST:GATEWAY_PORT")
tool = SeleniumScrapingTool(driver=webdriver.Chrome(options=options))
That is the route we would take. Chrome ignores a login written into its proxy flag, so give it a line that works by address.
The second route is the API tools. A search tool that posts to a vendor endpoint uses requests, which reads the proxy variables by default, so those calls already follow your line without any change.
The third route is a tool of your own: a small class whose _run fetches with your own client and your own proxy settings. You keep the responsibility that safe_get was carrying, which is the point of the next section.
The escape hatch, and what it costs
There is a switch. CREWAI_TOOLS_ALLOW_UNSAFE_PATHS=true lets a proxy argument through. It also turns off the URL validation for every tool that uses the helper, which is what keeps a crew from being talked into fetching a private address or a cloud metadata endpoint. The project labels it as not recommended, and we agree: use the Selenium route instead, which keeps both.
Which proxy type fits a crew?
Residential, for the sites that refuse a hosting address. In our test of 19 September 2026 a residential address changed four of thirteen answers, while four sites refused both. Those requests carried browser headers, much like the ones CrewAI's scraper sends, which is the point: the header set is not the problem, the address is. The table is on our OpenClaw page.
For the browser route, allow the machine's address on a Residential Premium plan, up to 150 per plan, so the Chrome flag carries no password. For the API tools, a generated line works as it stands, because those calls read the variables and requests accepts a login in the URL. Rotating ports change the address, a sticky port holds one, and a sticky address can still change early if its device leaves the network, so let the crew retry. The residential proxies page lists the plans, and the plan API generates lines and manages allowed addresses from code.
What breaks when the proxy is on
- Nothing changes. You set the variables and the scraping tool still uses your own address. That is the behaviour above.
Proxies are not allowed for safe_get. You passed a proxy argument to a tool that refuses one.- The crew suddenly reaches internal addresses. You turned the escape hatch on. Turn it off and route the browser instead.
- 407 Proxy Authentication Required from the browser route. Chrome cannot send the login. Allow the machine's address. Our 407 guide explains the error.
- Only some traffic changes address. That is expected: the API calls follow the variables, the built-in scraper does not.
- The site still refuses. A proxy changes the address, not the browser or the way the agent behaves.
What this page does not cover
We read CrewAI 1.15.22 as text and did not run a crew, so nothing here measures how a proxy performs inside one. We did not test the escape hatch, and we did not trace the model calls, which go through a separate library. The pinning that causes the refusal was merged on 17 August 2026, so an older install behaves differently. CrewAI ships most weeks, so we will read these settings again by 19 October 2026.
Where to go from here
Proxies for Dify covers another builder whose outbound traffic runs through a guard of its own. Proxies for Scrapling covers a scraping library that takes a proxy per fetcher, if you want the tool inside your custom crew tool to do the routing. How websites detect proxies explains what a site checks besides the address.
Sources
- The website scraping tool, the SSRF-safe request helper and its path validation, the Serper search tool and the Selenium scraping tool, plus the tool's documentation page. crewAIInc/crewAI, release 1.15.22, 16 September 2026, with the tools under lib/crewai-tools.
- Pull requests 6981 (merged 17 August 2026) and 6038, and issues 1456, 5421 and 4402. CrewAI issue tracker, 2024 to 2026.
- Credentials in the proxy flag and the implicit bypass rules. The Chromium Authors, net/docs/proxy.md, read 19 September 2026.
- Plans, IP whitelist and sticky sessions; errors; the proxy API. HProxy documentation, hproxy.com/docs, 19 September 2026.
- Our own test of 19 September 2026: plain GET requests to 13 sites and 3 controls, two runs from our server and two through a residential line of our own house plan. Raw output is kept in the research folder of our OpenClaw page.
