LLM scraping means collecting web pages to train a model or to feed a retrieval system. The big AI companies collect in the open: OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User each announce themselves, and sites decide which to let in. Many say no. One audit covered 2023 to 2024. In that year, 28% or more of the most actively maintained sources in the C4 corpus closed to AI use through robots.txt. Common Crawl offers petabytes collected since 2008, free on Amazon. We read the sellers' pricing pages on October 26, 2026. For collection that sites allow, residential traffic cost $0.24 to $2.10 per GB at about 1 TB. Our Lite pool costs $0.50/GB from one GB.
The sections below cover how AI bots identify themselves, what sites now allow, where proxies fit, and which proxies fit. Our page on proxies for AI training data covers the training side in depth.
How do AI companies' own crawlers identify themselves?
By name, by purpose and, for some, by published address. OpenAI's crawler documentation lists three:
| Bot | What it does | How sites choose |
|---|---|---|
| GPTBot | crawls for training | robots.txt rule |
| OAI-SearchBot | finds pages for search | robots.txt rule |
| ChatGPT-User | visits a page a user asks about | published IP list |
OpenAI says it uses the OAI-SearchBot and GPTBot tags "to enable webmasters to manage how their sites and content work with AI". Its page adds that "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models".
ChatGPT-User visits a page "when users ask ChatGPT or a CustomGPT a question". OpenAI publishes its IP addresses.
That openness is the standard a collector is measured against. A site can see who is asking and decide.
How much of the web now restricts AI use?
A fast-growing share. The Data Provenance Initiative's Consent in Crisis audit measured the change over one year:
Networks act on those wishes too. Cloudflare changed its default "to block AI crawlers unless they pay creators for their content" on July 1, 2025. RFC 9309 says robots rules "are not a form of access authorization", so a site's terms and the law can still say more. A proxy changes where a request comes from and grants no consent a site has withheld.
Where do proxies fit in LLM data collection?
In a narrow part of it. Start from what is published, and collect only the gap:
| Source | Proxy type | Why |
|---|---|---|
| Common Crawl, open datasets | None | the data is published |
| Official APIs | None | the API is the route |
| Sites that allow bulk crawls | Datacenter | cheapest per GB |
| Pages that change by place | Residential, by country | the local version |
| Sites that restrict AI use | None | respect the restriction |
Common Crawl "contains petabytes of data, regularly collected since 2008", and "access to the corpus hosted by Amazon is free". A retrieval system that fetches a page when a user asks behaves more like ChatGPT-User. The site sees the visit, and its terms still apply.
Our explainer on datacenter vs residential proxies covers the difference. For the request habits on sites that allow collection, see our web scraping guide and avoiding IP bans.
What are the best residential proxies for LLM data collection in 2026?
Pools priced for your volume, from a pilot of 10 GB to a crawl of about 1 TB. Each figure below comes from the seller's own pricing page, read on October 26, 2026:
| Seller | Price per GB, Oct 26 | Why it fits LLM data |
|---|---|---|
| HProxy Lite | $0.50/GB from 1 GB, $0.44/GB from 2,000 GB+ | balance never expires |
| Rainproxy | $0.60 at 10 GB, $0.24 at 1 TB | traffic never expires |
| DataImpulse | $1.00, $0.80 at 1 TB | traffic never expires |
| Evomi | $0.99 from 15 GB, $0.79 from 1,000 GB | no expiry, pay as you go |
| Decodo | $3.75 at 3 GB, $2.00 at 1 TB | ZIP and ASN targeting |
| MarsProxies | $3.99 at 10 GB, $2.10 at 1,000 GB | non-expiring traffic |
Bright Data opens its residential network only to verified companies, and asks buyers of more than 1 TB to contact sales. For ten sellers compared in full, see our best residential proxies guide.
What does HProxy offer for LLM data work?
Residential Lite targets by country, from $0.50/GB for one GB down to $0.44/GB at 2,000 GB+. It bills by the GB, and the balance never expires. Datacenter proxies and static ISP proxies are priced per IP with traffic unmetered, which suits heavy collection from sites that allow it. Unlimited residential proxies explains why residential traffic is metered.
Our pricing page lists every product. Our scraper APIs return parsed JSON for the platforms they cover, and the free proxy checker tests an address. For agents that browse on a user's behalf, see proxies for AI agents and proxies for Claude Code.
What this page could not check
- No test of any network. We measured no speed or success rate at any seller, ours included, and built no dataset or index.
- One company's bots. OpenAI's page describes its own crawlers only.
- One audit. The consent figures cover the domains behind C4, measured from 2023 to 2024.
- Prices as published. The table shows each seller's own figures of October 26, 2026, without a purchase to confirm them.
- No legal advice. Copyright, data protection and AI laws differ by country.
- Check by January 26, 2027. Bot policies, robots files and prices change, and every figure here needs a fresh read by then.
Sources
- OpenAI, Overview of OpenAI Crawlers, read October 26, 2026.
- Shayne Longpre and others, Data Provenance Initiative, "Consent in Crisis: The Rapid Decline of the AI Data Commons", arXiv 2407.14933, July 2024, read October 26, 2026.
- Cloudflare, "Content Independence Day: no AI crawl without compensation!", July 1, 2025, read October 26, 2026.
- Common Crawl, Overview, read October 26, 2026.
- IETF, RFC 9309, Robots Exclusion Protocol, September 2022.
- Rainproxy, DataImpulse, Evomi, Decodo, MarsProxies and Bright Data, residential pricing pages, read October 26, 2026.
- HProxy, residential, datacenter and ISP product pages and price list, read October 26, 2026.
