AI teams use proxies to collect public web pages for training, fine-tuning, retrieval and evaluation data, and the web's consent signals now come first. The Data Provenance Initiative audited 14,000 domains behind AI corpora. In one year, 2023 to 2024, about 5% or more of all tokens in C4 became fully restricted through robots.txt. Cloudflare has blocked AI crawlers by default since July 1, 2025. Before any collection, Common Crawl offers petabytes of pages collected since 2008, free on Amazon. For collection that sites allow, residential traffic at about 1 TB cost $0.24 to $2.10 per GB. We read those prices on each seller's page on October 26, 2026. Our Lite pool costs $0.44/GB per GB from 2,000 GB+.
The sections below cover what websites now say about AI crawling, the open corpus, which proxy fits, and what each seller charges at volume.
What do websites now say about AI crawling?
More and more, no. The Data Provenance Initiative's Consent in Crisis study audited the web domains behind corpora such as C4. In a single year it found "a rapid crescendo of data restrictions from web sources". About 5% or more of all tokens in C4 became fully restricted from use through robots.txt. So did 28% or more of C4's "most actively maintained, critical sources". Counting terms of service, "a full 45% of C4 is now restricted".
Large networks act on those wishes. In June 2024, Cloudflare found AI bots reaching about 39% of the top one million sites on its network. Only 2.98% of those sites blocked or challenged them. On July 1, 2025, it changed the default "to block AI crawlers unless they pay creators for their content".
A robots file states a site's preferences. RFC 9309, the robots standard, says its rules "are not a form of access authorization". That cuts both ways, since a site's terms and the law can still say more. A proxy changes where a request comes from. It grants no consent a site has withheld.
Is there an open corpus to start from?
Yes, and it is large. Common Crawl says its corpus "contains petabytes of data, regularly collected since 2008". It holds raw web pages, metadata extracts and text extracts. "Access to the corpus hosted by Amazon is free", and a URL index lets you search it.
A team that starts there collects only what the open corpus lacks, such as newer pages or narrow domains. That keeps the collection small, and the consent questions fewer.
Which proxy type fits AI data collection?
The cheapest that the source accepts, and none where an open corpus or API covers the need:
| Source | Proxy type | Why |
|---|---|---|
| Open corpora and APIs | None | the data is published |
| Sites allowing bulk crawls | Datacenter | cheapest per GB |
| Pages that change by place | Residential, by country | each market's version |
| Sites that opt out | None | respect the opt-out |
Our explainer on datacenter vs residential proxies covers the difference. Proxies for Hugging Face covers downloads of published models and datasets behind a company proxy. For the collection mechanics on sites that allow it, see our web scraping guide and proxies for LLM scraping.
What are the best residential proxies for AI data collection in 2026?
At AI scale, the price at about 1 TB matters most, and traffic that does not expire helps between runs. Each figure below comes from the seller's own pricing page, read on October 26, 2026:
| Seller | Price per GB, Oct 26 | Why it fits volume |
|---|---|---|
| Rainproxy | $0.24 at 1 TB | traffic never expires |
| HProxy Lite | $0.44/GB from 2,000 GB+ | balance never expires |
| Evomi | $0.79 from 1,000 GB | no expiry, pay as you go |
| DataImpulse | $0.80 at 1 TB | traffic never expires |
| Decodo | $2.00 on a 1 TB plan | ZIP and ASN targeting |
| MarsProxies | $2.10 at 1,000 GB | non-expiring traffic |
At about 1 TB, the five sellers we read spread almost ninefold:
Bright Data asks buyers of more than 1 TB to contact sales, and opens its residential network only to verified companies. For ten sellers compared in full, see our best residential proxies guide.
What does HProxy offer for AI data work?
Residential Lite targets by country, from $0.50/GB for one GB down to $0.44/GB at 2,000 GB+. It bills by the GB, and the balance never expires. Datacenter proxies and static ISP proxies are priced per IP with traffic unmetered, which suits heavy collection from sites that allow it. Unlimited residential proxies explains why residential traffic is metered. Our pricing page lists every product. Our scraper APIs return parsed JSON for the platforms they cover, and the free proxy checker tests an address.
What this page could not check
- No test of any network. We measured no speed or success rate at any seller, ours included, and built no dataset.
- One audit. The consent figures cover the domains behind C4, measured from 2023 to 2024.
- An older share. Cloudflare's 39% figure dates from June 2024, before its default changed.
- Prices as published. The table shows each seller's own figures of October 26, 2026, and we made no purchase to confirm them.
- No legal advice. Copyright, data protection and AI laws differ by country, and we quote no ruling here.
- Check by January 26, 2027. Robots files, terms, defaults and prices change, and every figure here needs a fresh read by then.
Sources
- Shayne Longpre and others, Data Provenance Initiative, "Consent in Crisis: The Rapid Decline of the AI Data Commons", arXiv 2407.14933, July 2024, read October 26, 2026.
- Cloudflare, "Declare your AIndependence: block AI bots, scrapers and crawlers with a single click", July 3, 2024, read October 26, 2026.
- Cloudflare, "Content Independence Day: no AI crawl without compensation!", July 1, 2025, read October 26, 2026.
- Common Crawl, Overview, read October 26, 2026.
- IETF, RFC 9309, Robots Exclusion Protocol, September 2022.
- Rainproxy, Evomi, DataImpulse, Decodo, MarsProxies and Bright Data, residential pricing pages, read October 26, 2026.
- HProxy, residential, datacenter and ISP product pages and price list, read October 26, 2026.
