Market research teams use proxies to read prices, catalogs, reviews and ads the way each market sees them. The data changes by place: one study found prices differing by the customer's location by up to 166% on digital products. Research at scale runs on that kind of collection. By 2010, the Billion Prices Project gathered 5 million prices a day from over 300 retailers in 50 countries. Sites set their own rules for collectors, and IP databases can flag residential proxies. We read the sellers' pricing pages on October 26, 2026. Residential traffic at about 100 GB a month cost $0.40 to $3.54 per GB, and our Lite pool costs $0.50/GB from one GB.
The sections below cover the data research teams collect, why place matters, the rules, data quality, and which proxies fit. Our web scraping guide covers the collection basics this builds on.
What data does market research collect from the web?
Four kinds, mostly from other companies' public pages:
- Competitor prices. What rivals charge in each currency and region, and how it moves. Our guide to proxies for price monitoring covers this one job in depth.
- Reviews and ratings. How a product or a category is received, often across languages and regions.
- Catalogs and assortment. Listings, specs, stock and new launches, to map what competitors sell and where the gaps are.
- Ads and search results. Which ads run in which markets, and how results differ by country.
The scale can be large. Alberto Cavallo and Roberto Rigobon describe the Billion Prices Project at that scale. By 2010, it collected "5 million prices every day from over 300 retailers in 50 countries". Half a million came from the United States alone each day. By comparison, the US Bureau of Labor Statistics collected about 80,000 prices a month or two.
Why does the place of collection matter?
Because sites show different things to different places. Jakub Mikians and colleagues reported price differences "based on the geographical location of the customer", up to 166% on e-books and video games. Aniko Hannak and colleagues tested 16 popular e-commerce sites and found "some form of personalization on nine of these e-commerce sites".
So a run from one office sees one market's version of each page. Reading a market accurately means an exit inside it. Panel work has the same need, which is why survey proxies use residential addresses in the market of each questionnaire.
What do sites allow, and can they spot a proxy?
Each site sets its own rules, and many can recognize proxy traffic. Read a site's terms and its robots.txt before you collect. Trustpilot's robots.txt gives named search crawlers detailed rules. Its closing group then reads "User-agent: *" with "Disallow: /", which asks every crawler it does not name to read nothing.
Sites can also look an address up. MaxMind's Anonymous IP database flags VPNs, hosting addresses and residential proxies. It defines the last as "IP addresses on a suspected anonymizing network that are registered under residential ISPs". It sells that flag so sites can "block them or add extra checks".
A proxy changes where a request comes from. It makes no collector invisible, and it grants no permission a site has not given. Sites also slow a heavy repeat visitor, often with 429 responses. Our notes on avoiding IP bans cover pacing for sites that allow collection.
How do you keep research data trustworthy?
Check the output as hard as you check the proxies. A blocked request that returns an error is easy to spot. A page that loads with the wrong currency, half the reviews or a missing price is worse, because it looks like data.
- Check every record. Did the price parse, does the currency match the market, did the page hold the expected number of results?
- Log the context. Keep the URL, the time in UTC and the exit country with each record.
- Watch the anomaly rate. A rising share of odd records warns you before bad data reaches a report.
- Mind the client. A bare HTTP client can differ from the browser it claims to be, as our explainer on JA3 and JA4 fingerprinting shows.
Which proxy type fits which research task?
Start with the cheapest type a site accepts, and step up when blocks or wrong-market pages show you must:
| Research task | Proxy type | Why |
|---|---|---|
| Prices across countries | Residential, by country | prices change by place |
| Open catalogs and APIs | Datacenter | cheapest per request |
| Reviews where allowed | Rotating residential | spreads the load |
| Ads by market | Residential, by country | ads change by place |
| Logged-in panels | Static ISP address | the session holds |
What are the best residential proxies for market research in 2026?
The best fit covers your markets at a price that suits steady volume, and keeps unused traffic between projects. We read each seller's own pricing page on October 26, 2026:
| Seller | Price per GB, Oct 26 | Why it fits research |
|---|---|---|
| HProxy Lite | $0.50/GB from 1 GB, $0.44/GB from 2,000 GB+ | country, never expires |
| Rainproxy | $0.60 at 10 GB, $0.40 at 100 GB | traffic never expires |
| Evomi | $0.49 on a 100 GB plan | city and ISP included |
| DataImpulse | $1.00, $0.80 at 1 TB | free country targeting |
| Decodo | $2.75 at 100 GB, monthly | ZIP and ASN |
| HProxy Premium | $1.49/GB from 1 GB | state and city studies |
At about 100 GB a month, the six sellers we read spread widely:
Rainproxy falls to $0.24 per GB at 1 TB, and DataImpulse to $0.80. Bright Data's documentation opens its residential network only to verified companies. Our best residential proxies comparison sets ten sellers side by side.
What does HProxy offer for market research?
Two residential pools billed by the GB, with a balance that never expires, so a project that runs in quarterly bursts keeps its credit. Residential Lite targets by country, from $0.50/GB for one GB down to $0.44/GB at 2,000 GB+. Premium adds state, city and ASN targeting from $1.49/GB. Our pricing page lists every product. The same exits can check how your own product looks in each market, and geo-testing with proxies turns that into a test.
What this page could not check
- No test of any network. We measured no speed or success rate at any seller, ours included, and ran no research collection.
- Old studies. The location studies date from 2012 and 2014 and covered a limited set of sites. Test whether your own targets change by place.
- Sellers as printed. The seller table repeats what each pricing page printed on October 26, 2026. We bought from none of them.
- No legal advice. The robots files and policies are quoted for what they say.
- Check by January 26, 2027. Prices, robots files and terms change, and every figure here needs a fresh read by then.
Sources
- Alberto Cavallo and Roberto Rigobon, "The Billion Prices Project: Using Online Prices for Measurement and Research", Journal of Economic Perspectives, Spring 2016 (NBER Working Paper 22111).
- Aniko Hannak, Gary Soeller, David Lazer, Alan Mislove and Christo Wilson, "Measuring Price Discrimination and Steering on E-commerce Web Sites", ACM Internet Measurement Conference, November 2014.
- Jakub Mikians, Laszlo Gyarmati, Vijay Erramilli and Nikolaos Laoutaris, "Detecting price and search discrimination on the Internet", ACM HotNets, October 2012.
- Trustpilot, robots.txt of trustpilot.com, read October 26, 2026.
- MaxMind, GeoIP Anonymous IP database, read October 26, 2026.
- Rainproxy, Evomi, DataImpulse, Decodo, MarsProxies and Bright Data, residential pricing pages, read October 26, 2026.
- Bright Data documentation, Residential network access, updated September 1, 2026, read October 26, 2026.
- HProxy, residential product pages and price list, read October 26, 2026.
