SERP
A search engine results page, the most location-sensitive and most defended page type on the web, and the raw material of the SEO industry.
A SERP is what a search engine answers a query with, and an entire industry exists because no two people reliably see the same one. Results vary by country, by city, by language, by device, and by the searcher's history, which means the question every SEO tool is paid to answer, where do we rank, has no single true value. It has a value PER PLACE, and collecting those is why this page type gets scraped at industrial scale.
The location sensitivity is the deepest of any page type, and it is deliberate. Engines localise ordinary queries aggressively: the same words return different businesses, different news and differently ordered organic results depending on where the searcher appears to be, and map-style local results respond to city-level position. Checking rankings for Munich from a Frankfurt address is already measuring the wrong thing, which is why rank tracking is the canonical use case for city targeting, and why the exits must be residential, since engines classify hosting-range visitors instantly.
The page itself has also stopped being ten links, which changes what collecting it means. Modern SERPs interleave ads, shopping units, map packs, news carousels, answer boxes and, since 2024, AI-generated summaries above everything else, so position three organic sits below several features that reshape what visibility at position three is worth. Serious collection parses the features alongside the ranks, because a stable rank under a growing answer box is a falling asset that a rank number alone will not show.
As a target, search engines are the hardest steady-state case in scraping. Volume is inherent, tracking a thousand keywords daily across five cities is five thousand queries before anything interesting happens, and the defences are first-rate: strict per-address pacing, instant hosting-range classification, consent and challenge interstitials, and result markup that shifts often enough to feel adversarial to parser maintenance. The practical stack is unusually settled: residential exits in the tracked geography, low per-address request rates spread across a wide pool, and constant parser upkeep, or a purpose-built endpoint that does all three behind one call.
One measurement caveat completes the picture: personalisation means even a perfectly located, perfectly clean request observes A result, not THE result. Sound tracking works from a signed-out, history-free baseline and treats small fluctuations as noise, because the page being measured is generated fresh, per query, per place, per moment.
Frequently asked questions
Why do SERP results differ from what my rank tracker says?
Because you and the tracker searched from different places, or different personalisation states. Results vary by country, city, language, device and history, so your signed-in local view and a tool's clean remote measurement are both real and both partial. Trackers measure a defined baseline, signed out, fixed location, precisely so the number is comparable over time.
What proxies do I need for SERP scraping?
Residential exits in the exact geography being measured, city-level where local results matter, spread across enough addresses that per-address query rates stay low. Engines classify hosting ranges on sight and pace individual addresses strictly, which makes datacenter supply nearly useless here and pool breadth more important than raw speed.
Is scraping search results allowed?
Engines' terms generally prohibit automated querying, and they enforce technically rather than contractually: pacing, challenges and blocks. The information itself is public, an entire industry measures it anyway, and the jurisdiction-dependent legal questions are ones for a lawyer. Our operating floor is unchanged here: public data, restrained pacing, and no escalation against a target that is actively refusing.
Why did my SERP parser break when rankings did not change?
The page changed around the rankings. Engines continuously rearrange features, ads, map packs, answer boxes, AI summaries, and adjust markup in ways that shred selectors while the organic order underneath stays put. SERP parsing is a maintenance commitment more than a build, which is much of what purpose-built SERP endpoints are actually selling.
Back to the full glossary.