Flight prices are among the most valuable data on the travel web and among the hardest to collect. There is no public fare feed for most of the market, the prices change by the hour and by the country you appear to be in, and the sites that hold them run some of the toughest bot management anywhere. This guide is the complete, practical version of how to scrape them: why it is hard, the four things a reliable pipeline gets right, the proxy setup that holds, and how to size and run it. We run a proxy network and keep this kind of traffic online, so this is written from what actually works rather than what sounds good.
The sector overview is in travel fare aggregation; the hotel-side companion is how to scrape hotel prices; and the individual sites have their own pages, from Google Flights and Skyscanner to the airlines like Delta and Lufthansa.
Why flight prices are hard to scrape
Four things make flight scraping harder than ordinary web scraping, and each one shapes the setup.
There is no public API. Google closed its QPX Express fare API in 2018 and never replaced it, and the airline and metasearch APIs that exist (Skyscanner, Kayak, the airlines' NDC connections) are partner-only, reached through a commercial agreement, not a self-serve key. So for most people the data lives on the public sites, and reading it is scraping, with the sites' terms and defenses in play.
The prices are filed by point of sale. Airlines set some fares by the market a booking appears to come from, in that market's currency and with that market's taxes, so the same seat can cost different amounts depending on where the request originates. Read a route from the wrong country and you collect a number no local traveler sees.
The fares move constantly. Cash fares and award prices are computed live from demand rather than looked up in a table, so a price you stored an hour ago may already be wrong. The value of a fare dataset lives in reading often enough to catch the movement.
The sites defend hard. Airline fare search sits behind Akamai Bot Manager and similar systems that fingerprint the TLS handshake and the browser before they weigh the IP, and the pages render fares with JavaScript into a near-empty shell, so a plain request gets a page with no prices. A standard scripting client is issued a block before its request reaches the fare engine.
No public API
partner-only feeds
Point-of-sale pricing
fare depends on market
Fares move hourly
dynamic, live-computed
Heavy bot defenses
Akamai, JS render
The four things a reliable pipeline gets right
Everything above resolves into four decisions. Get them right and flight scraping becomes routine.
1. Residential proxies, pinned per market
The point-of-sale problem and the bot-management problem have the same answer: an in-market residential exit. A residential IP reads as an ordinary traveler, which clears the reputation half of the bot score, and pinning it to the target country reads the fare that country is shown. Reading many markets means a pool of residential IPs across those countries. Datacenter ranges fail here twice, distrusted by the bot management and unable to represent a real market, which is why in published testing a datacenter proxy returns a block where the same request through residential clears. Our residential proxies cover 100+ countries for exactly this reason, and the background is in what is a residential proxy.
2. Real browsers for the JavaScript
Because the fares render client-side, a bare HTTP client gets a shell. Drive a real browser (Playwright or Puppeteer) behind the residential exit so the page runs its scripts and the prices appear, and leave the browser's own TLS and header order intact, because that handshake is what Akamai reads first. Our guides for Playwright and Puppeteer cover the proxy wiring, and the deeper anti-bot detail is in how to scrape past Akamai.
3. Sticky sessions where a search spans requests
A flight search is often not one request. Metasearch sites create a search and then poll for results as suppliers answer, over ten to thirty seconds, and a booking funnel sets a server-side session. Those multi-request flows have to come from one IP, so hold a sticky session for the length of a search, then rotate to a fresh residential IP for the next one. Rotating mid-search is the most common way a scraper gets flagged, because a session that teleports between addresses is a bot tell. The split between sticky and rotating is explained in rotating vs static residential proxies.
4. A refresh cadence matched to the fares
Because fares move, the schedule is part of the product. Read near-term departures often, because they move fastest, and far dates rarely. A cadence per date band both fits the pricing and looks more human than hammering everything at one rate, and it cuts the bandwidth bill, which on a browser-driven pipeline is where the cost lives.
Which sites to read, and how they differ
The flight data lives in three kinds of source, and a complete picture usually reads more than one.
- Metasearch (Skyscanner, Kayak, Google Flights, Momondo): the widest view, comparing many airlines and agencies at once, but the heaviest defenses and the create-then-poll flow. Start here for coverage.
- Airlines directly (Delta, United, American, Lufthansa, Emirates and the rest): the source of truth for a carrier's own fares and award space, behind Akamai, priced by point of sale. Read these when a carrier's own prices are the target.
- Online travel agencies (Expedia, Booking.com, Trip.com): another view, often with their own promotions, on their own platforms.
Each has its own page in this series with the site-specific mechanics. The important cross-site rule is that no single source is complete, so a serious fare dataset reads the metasearch for breadth and the airlines directly for depth.
Setting up the pipeline
Put the four decisions together into a run. Build the search URL directly rather than automating the search form, so you control origin, destination, dates, currency and language. Exit from the market you are reading, on a residential IP, and set the currency and language to match, because a Brazilian exit with US currency is a mixed signal. Drive a real browser, wait for the fares to render (or for a metasearch poll to complete, so you do not store the early cached subset), and parse the result. Hold the IP for the whole search, then rotate. Log the fare with its market, currency and timestamp, because a fare without those three is not comparable to another.
Sizing (rotating residential, one refresh cycle):
searches/cycle = routes x dates x markets
= 300 routes x 60 dates x 3 markets = 54,000 searches
per search = a browser render or a create-then-poll session
cadence = near-term dates read more often than far dates
per address = a few searches, then rotate
Bandwidth is the bill on a browser pipeline: block images and fonts,
keep the fare scripts, and refresh by date band. The pool spreads the load.
Staying unblocked
- Residential IP plus a real browser, always. The bot management scores both; fixing one leaves the other failing.
- Exit from the market, with matching currency and language. Accuracy and reputation in one rule.
- One search per address, then rotate. Never let a poll or a funnel step arrive from a different IP than the page.
- Guard the handshake. The TLS fingerprint is the top detection vector on Akamai sites; do not let a scripting library announce itself.
- Pace and jitter. No traveler runs a hundred searches a minute; the full checklist is in avoiding IP bans while scraping.
- Treat a 200 as unverified. A block or a challenge can arrive with a normal status; check that the fares are actually in the DOM.
Free versus paid
A free proxy is a shared datacenter address that airline bot management challenges first and that cannot represent an in-market home connection, so it is the wrong tool for a production run. It is genuinely useful for learning and for testing your parser against a saved page, and our free proxy list (100+ countries, refreshed every few minutes) plus the free proxy checker are there for that. When the data has to come back reliably, from the right market, on a schedule, paid residential is the floor.
Why residential proxies fit flight scraping specifically
This is the part worth stating directly, because it is what the whole guide adds up to. Flight scraping needs exactly what residential proxies provide: an address that reads as an ordinary traveler so the bot management lets it through, in the specific market whose fare you want to read, from a pool wide enough to spread thousands of searches so no single IP looks like a monitor. Datacenter proxies fail the reputation check and the market test; a VPN gives you one country at a time; only a residential pool does the real job of reading many markets truthfully at scale. That is why fare aggregators, price-tracking products and airline analysts run on residential proxies, and it is what ours are built for.
Our residential proxies are $0.44/GB pay-as-you-go, no KYC and no expiring balance, so a small test run costs cents and a large fare pipeline pays for the bandwidth it uses rather than a flat subscription. Start with the free proxy list to learn the shape of the job, and move to residential when the fares have to be real. The sector context is travel fare aggregation, and the hotel companion is how to scrape hotel prices.
The limits worth knowing
Proxies solve the two things flight scraping punishes hardest: one address doing too much, and a request arriving from the wrong market. They do not fix a bare client on a JavaScript page, an automation fingerprint, or a schedule that runs on the clock, and they do not restore an API the industry removed on purpose or change any site's terms of service. Scraping fares runs against those terms, a risk you own regardless of the tooling. What good residential proxies buy you is a fair, unflagged, in-market read of the fares, which is the hard part, and the rest is careful engineering.
Sources
- HProxy, proxies for Google Flights, on the absence of a public API since QPX Express closed in 2018.
- HProxy, proxies for Delta and proxies for Lufthansa, on dynamic and point-of-sale pricing and Akamai defenses.
- HProxy, how to scrape past Akamai, on the bot management airline sites run.