Every large AI model is a compression of a very large dataset, and building that dataset is its own engineering problem, separate from training on it. The scale is easy to underestimate. Common Crawl holds petabytes of web data collected since 2008, and even a filtered slice of it can be enormous: the filtered Common Crawl behind GPT-3 was 410 billion tokens, 60% of that model's training mix. When the open corpora do not cover what you need, and increasingly they do not, you build your own, and proxies for AI training data are what make that build possible: they let you collect from defended and geo-gated sources at the scale a real dataset demands.
We run the network this collection rides on, so we see the dataset-building side directly: the teams assembling domain-specific corpora, the groups collecting under-represented languages, the labs refreshing stale data. This is the practical guide to the dataset build: why you would collect your own rather than lean on Common Crawl, how to size the bandwidth honestly, why geographic diversity is a quality requirement rather than a bonus, and which proxy type fits which source. If you want the collection mechanics first, our companion guide on proxies for LLM scraping covers the act of gathering the data this post is about shaping into a dataset.
Why build your own training data?
Because the ready-made corpora fall short in three ways that matter. They are periodic snapshots, so they are stale for anything that moves. They are general, so they under-serve a specific vertical or language. And the open commons is shrinking as sites restrict AI use, precisely for the highest-value sources. Building a current, specialized or geo-balanced dataset means collecting it yourself, and that collection needs proxies to reach sources that block datacenter ranges and declared AI crawlers.
The scale you are actually sizing for
It helps to hold the numbers in mind, because they set the whole budget. Common Crawl reached 3.1 billion web pages by April 2023 and has been cited in more than 10,000 academic studies. GPT-3's builders did not take that raw: they filtered it down and deduplicated it, using fuzzy deduplication with an MinHash LSH algorithm to strip the enormous redundancy the open web contains, and still ended with 410 billion tokens as the dominant 60% of the mix. The lesson buried in that process is the one that governs your bandwidth bill: a raw web crawl is mostly duplicate and low-value content, and the value is in what survives filtering, not what you fetched.
That is why building a dataset is not the same problem as scraping a few thousand pages. You are collecting at a scale where the redundancy, the geographic skew and the noise are structural, and where the cost of collection (bandwidth) and the cost of the noise (a worse model) are both real. The proxy layer is where you control the first, and it quietly shapes the second.
Why the open path is narrowing for the best sources
The reason self-collection keeps growing is that the open commons is closing exactly where the good data is. The Data Provenance Initiative's longitudinal audit found that in a single year, 5% or more of all tokens in the C4 corpus, and 28% or more of its most actively maintained and critical sources, became fully restricted to AI crawlers via robots.txt, with terms-of-service restrictions covering far more. Cloudflare, which found AI bots reaching roughly 39% of the top million sites it protects, moved in July 2025 to block AI crawlers by default.
Read those two findings together and the implication for a dataset builder is concrete: the highest-quality, most current sources are the ones most likely to refuse a declared crawler from a datacenter. Reaching them for a modern corpus increasingly means collecting through residential IPs that read as ordinary users, not because you are hiding, but because that is the only traffic those sources still serve.
Geographic diversity is a quality requirement
Here is the angle specific to building a dataset rather than just scraping one site. A corpus is a sample of the web, and where you collect it from decides what it samples. Pull everything through one country's IPs and you get that country's version of the web: its dominant language, its news outlets, its localized pages, its search results. A model trained on that sample inherits its lopsidedness, and no amount of downstream tuning fully removes a skew baked into the pre-training data.
If the goal is a dataset that reflects more than one market, whether for multilingual coverage, regional balance or simply a less US-centric slice of the web, you have to collect from inside those markets. That means residential exits spread across the regions you care about, which is exactly what country and city targeting on a residential pool provides. Geographic coverage stops being a networking detail and becomes a lever on what your model actually learns.
Which proxy type fits which source
The cost discipline is the same as all proxy work: cheapest tier a source will accept, escalate only on evidence.
Datacenter proxies carry the bulk of raw collection. Open repositories, public datasets, reference and government sites, and friendly APIs make up a large share of any corpus, and they do not scrutinize IP reputation, so datacenter is both cheapest and fastest for them. Spend nothing heavier where these work.
Rotating residential proxies are for the defended, high-value sources: the news, reference and community sites that block datacenter ranges and declared AI crawlers, and the geo-specific sources you need for regional balance. They read as ordinary home users and can be pinned to a country. Billed per gigabyte, which is the number that dominates a large build.
Static residential and ISP proxies cover the minority of sources that need a persistent session or login to reach the content.
Mobile proxies are the reserve tier for the few sources that refuse everything else.
| Dataset source | Proxy type | Why |
|---|---|---|
| Open repositories, public datasets, APIs | Datacenter | Bulk of the corpus, cheapest per request |
| News, reference, active communities | Rotating residential | These block datacenter and declared AI crawlers |
| Multilingual or region-balanced coverage | Residential, geo-distributed | Collecting from inside each market removes skew |
| Sources behind a login or session | Static residential / ISP | Session must persist across the collection |
| The most bot-hostile sources | Mobile | Carrier IPs shared by many users, rarely hard-blocked |
Sizing the bandwidth honestly
On a dataset build, bandwidth is the budget, because residential is metered per gigabyte and a corpus is large. Three habits keep the bill sane.
Strip early. You keep text, not the images, scripts, fonts and video that make up most of a page's weight, so fetch and retain the lean version wherever the source allows it rather than rendering full pages you will discard.
Deduplicate early. The open web is enormously redundant, which is why GPT-3's build ran fuzzy deduplication before training. Dropping duplicates near the point of collection, rather than after storing everything, saves both storage and the bandwidth of re-fetching content you already have.
Route by cost. Send everything friendly through cheap datacenter and reserve residential for the defended sources that genuinely require it. A build that routes its entire crawl through residential is paying premium rates for the open repositories that would have taken datacenter for a fraction of the price. Our pricing is pay-as-you-go per gigabyte with a balance that does not expire, which suits a build that runs in heavy bursts rather than a constant trickle.
Provenance and cleanliness
Two data-quality concerns sit on top of the collection. The first is the soft block: a defended site that serves a CAPTCHA page, an empty result or a truncated snippet instead of refusing you cleanly, which your collector then stores as if it were real content. At corpus scale this poison is invisible and it degrades whatever you train. Clean IPs that avoid triggering soft blocks are the first defense; validating that pages parsed to the structure you expected is the second. The second concern is provenance: keeping track of where each document came from, under what terms, and whether it survived your dedup and quality filters, so the dataset is auditable rather than an anonymous heap. Both are process, but both depend on collection that returns real pages in the first place, which is the part the proxy layer provides.
Keeping it legal and ethical
Building a dataset to train a model sits in a legal area still being decided, so honesty beats bravado. Collect publicly available, non-personal data. Respect the directives sources publish, including robots files, where you have agreed to them. Pace collection so you do not degrade the sources you depend on. Treat personal data and copyrighted material with real caution, because both raise questions for AI training that go well past ordinary scraping and are actively litigated. Proxies are a technical tool, not a legal shield, so keep to public, non-personal data and get proper legal advice before training a commercial model on what you collect.
Where HProxy fits
We built the network for the range a real dataset build spans. Cheap datacenter carries the open bulk, residential proxies with country and city targeting reach the defended and geo-specific sources that give a corpus its quality and balance, and ISP handles the ones behind a session. Our pricing is pay-as-you-go at $0.65/GB with a balance that does not expire, which fits the bursty economics of a large collection run. Before scaling up, point the free proxy checker at your exits to confirm they are alive and leaving from the countries your dataset needs. For the collection mechanics that feed this, see our guide on proxies for LLM scraping; for the general fundamentals, proxies for web scraping.
Sources
- Common Crawl overview (petabytes of web data collected since 2008): https://commoncrawl.org/overview
- Common Crawl (3.1 billion pages by April 2023; cited in 10,000+ academic studies; used to train Gemini): https://en.wikipedia.org/wiki/Common_Crawl
- GPT-3 training composition and deduplication (filtered Common Crawl = 410 billion tokens, 60% of the mix; fuzzy deduplication via MinHash LSH): https://en.wikipedia.org/wiki/GPT-3
- Consent in Crisis: The Rapid Decline of the AI Data Commons, Data Provenance Initiative (in one year, 5%+ of all C4 tokens and 28%+ of its most active sources became robots.txt-restricted): https://arxiv.org/abs/2407.14933
- Cloudflare, Declaring Your AIndependence and Content Independence Day (AI bots reached ~39% of the top million sites; July 2025 default block on AI crawlers): https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ and https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/