Every airline sets its prices partly by watching what its competitors charge, and every competitor charges a different price depending on the market the ticket is sold in. That is the problem at the heart of airline fare intelligence: a revenue manager who reads a rival's fare from the office sees one market's version of it, and prices a route against a number no traveler in the other markets is actually shown. Proxies for airline fare intelligence exist to close that gap, letting a carrier, an online travel agency or a consolidator read competitor fares from every market they compete in, on a schedule, as a local traveler sees them.
We run a proxy network, and this is one of the more demanding jobs our residential IPs do. This page is written for the business side: what fare intelligence is, why one location is not enough, how point-of-sale and dynamic pricing shape the collection, the scale of it, and what to ask a vendor. The technical how-to is in how to scrape flight prices, the sector background is in travel fare aggregation, and the hotel-side equivalent is proxies for hotel rate monitoring.
What fare intelligence is, and why one location fails
Fare intelligence is the collection of competitor fares across routes, dates, cabins and points of sale, on a schedule, feeding revenue management and pricing. Airlines do it to price their own seats, OTAs do it to position their offers, and consolidators do it to source inventory. The whole exercise rests on seeing what competitors actually charge, and that is where a single vantage point fails.
Airlines file fares by point of sale, the market a booking appears to originate from, in that market's currency and with that market's taxes. So a competitor's fare for the same seat can differ between the German, Brazilian and Japanese markets, and a team reading it from one office in one country captures one of those and misses the rest. Pricing a route on the wrong market's competitor fare is worse than not reading it at all, because it looks like data and is not.
Read from one office location
One market's competitor fare
the office's country
One currency
not the traveler's
A partial picture
priced against a guess
Read from each point of sale
The fare each market is shown
in-market residential exit
Local currency and taxes
the real number
Price against reality
per market
Dynamic pricing makes it a continuous job
Point of sale is the geography problem; dynamic pricing is the time problem. Competitor fares are computed live from demand and change through the day, so a fare read this morning may be stale by afternoon, and a pricing decision made on yesterday's competitor data is made on a number that has moved. Fare intelligence is therefore continuous by nature: read often enough to track the movement, on the routes and markets that matter most, with the near-term departures read more frequently because they move fastest. That cadence, multiplied across markets and competitors, is what turns fare intelligence into a large, ongoing collection rather than a periodic snapshot.
The scale, and why it is a residential-proxy job
The arithmetic is what surprises people. Take one route, read against three competitors, across 60 departure dates, two cabins, and four points of sale, refreshed twice a day.
3 competitors
on one route
x 60 dates
180 reads
x 2 cabins
360
x 4 points of sale
1,440
x 2 refreshes
2,880 per route per day
Nearly three thousand reads a day for a single route, and a network has hundreds of routes, so the total runs into the millions of reads a day. That volume, from defended airline and OTA sites that price by market, is exactly what residential proxies are built to carry: an in-market address that reads as a local traveler and clears the bot management, from a pool wide enough that no single address looks like a monitor. Datacenter proxies fail the reputation check and the point-of-sale test; a single connection reads one market slowly. Only a residential pool reads many markets truthfully at scale, which is why serious fare-intelligence operations run on one. Our residential proxies cover 100+ countries for this reason.
The setup, by who is doing the collection
If you use a fare-data vendor, you do not run proxies, but the quality of what you buy is decided by how the vendor reads. The questions are: from which points of sale do you collect, and can I choose them? On datacenter or residential connections? How often, and are near-term dates read more frequently? Do you capture the dynamic movement or a daily snapshot? A vendor reading every market from one datacenter is selling you the office's view at scale.
If you build the collection, the setup is the one in how to scrape flight prices: rotating residential proxies pinned per point of sale, real browsers to render the fare pages, sticky sessions where a search spans requests, and a cadence per date band. A static ISP exit fits any logged-in or account flow. The discipline that makes the data usable is recording the market, currency, cabin and timestamp on every fare, because a competitor fare without its point of sale is not comparable to your own.
Use sanctioned data where it exists. Your own fares are in your reservation system; the GDS and NDC channels carry contracted fares to those who license them. What no feed gives you is a competitor's public fare as a traveler in each market is shown it, which is why in-market residential collection remains the core of competitive fare intelligence.
What to compare, so the intelligence is real
- Same route, same cabin, same rules. Compare like with like: the lowest available fare in the same cabin with the same fare rules, not a flexible business fare against a basic economy one.
- Total price, taxes and surcharges in. The number a traveler pays, per market, including the market's taxes and any carrier surcharges.
- Each point of sale as its own line. Never average a fare across markets; the whole point is that they differ.
- The dynamic movement, not one snapshot. A single read is a moment; the value is in the trend as departure nears.
- Ancillaries where they matter. On unbundled carriers the headline fare is a fraction of the cost, so capture the bag and seat fees, the point made in proxies for Spirit Airlines.
The limits worth knowing
Proxies let a fare-intelligence operation read each competitor fare from the market it is sold in, on a connection that market's travelers use, and spread the collection wide enough that the sites see travelers rather than a monitor. They do not model the pricing decision for you, they do not turn a scraped competitor fare into a licensed data feed, and they do not change the sites' terms of service, which competitive scraping runs against regardless of tooling. What good residential proxies buy you is an accurate, in-market, at-scale read of what competitors actually charge, which is the input revenue management cannot make good decisions without.
For a first look, our free proxy list and proxy checker cost nothing. For a fare-intelligence collection that runs on a schedule across many markets, rotating residential at $0.44/GB pay-as-you-go, pinned to each point of sale, is the setup underneath every serious competitive-fare operation, available to a small OTA on the same terms as to a network carrier.
Sources
- HProxy, how to scrape flight prices, for the technical collection setup.
- HProxy, travel fare aggregation, on point-of-sale pricing and the anti-bot defenses fare sites run.
- HProxy, proxies for hotel rate monitoring, the hotel-side equivalent of this competitive-pricing job.