XPath
A query language for addressing parts of an HTML or XML document, the more powerful of the two ways scrapers point at the data inside a page.
Once a page is fetched and parsed, extraction is the act of pointing: THAT price, THOSE titles, the link inside the third card. XPath is a language for pointing. An expression describes a path through the document tree, and evaluating it returns whichever elements, attributes or text the path reaches. The syntax reads like a filesystem with opinions: //product descends to every product element anywhere, div/span steps from parent to child, and square-bracket predicates filter what qualifies, as in //div[@class='price'] or //li[position() less than four].
Its reputation for power rests on the places the alternative cannot go. XPath matches on text content, finding the span that literally says Out of stock. It traverses upward and sideways, from the element you can identify to the parent or sibling you actually want, as in //span[text()='Total']/following-sibling::span. It counts, compares and combines conditions inside a single expression. Whole categories of extraction that are contortions in other tools are one line here.
The failure mode is brittleness you wrote yourself. Browser developer tools offer a copy-XPath convenience that emits an absolute path from the document root, every step and index hardcoded, and such a path dies on the next layout adjustment anywhere along it. Hand-written relative expressions anchored to stable attributes survive redesigns that shred absolute ones. The craft is choosing anchors that mean something, an id, a data attribute, a text label, rather than encoding today's accidental geometry.
Where it runs matters as much as what it says. Server-side, the parser libraries scrapers build on evaluate XPath natively, and crawling frameworks expose it as a first-class selector. In the browser, evaluation exists but the tooling favours its rival, and automation frameworks accept both. The practical fluency is bilingual: CSS selector syntax for the everyday shots, XPath for the ones that need text matching or upward travel, chosen per extraction rather than per project.
One honest caveat: on JavaScript-rendered pages, no expression helps until the elements exist. XPath queries the tree you parsed, and if the data arrives after load, the tree you parsed is scaffolding. Pointing precisely at an empty stage is still pointing at nothing.
Frequently asked questions
Should I use XPath or CSS selectors for scraping?
Both, chosen per job. CSS syntax is terser and universally supported for the routine cases, classes, ids, attributes, nesting. XPath earns its keep where CSS cannot reach: matching elements by their text, walking from a found element up to its parent or across to a sibling, and combining several conditions in one expression. Fluency in both costs little and removes whole categories of workaround.
Why does my copied XPath break so quickly?
Because copy-XPath from browser developer tools emits an absolute path with hardcoded steps and indexes, which encodes the page's current geometry rather than its meaning. Any structural change along the path kills it. Writing a short relative expression anchored to a stable attribute or text label produces a selector that survives the redesigns that delete copied paths wholesale.
Can XPath select elements by their text?
Yes, and it is the signature capability. Expressions can match exact text, as in //button[text()='Add to cart'], or containment via contains(), and then navigate from the matched element to whatever you actually need. CSS selectors in scraping libraries have no equivalent, which is why text-anchored extraction is the usual reason a scraper reaches for XPath.
Does XPath work on JavaScript-heavy pages?
It queries whatever tree you hand it. If the page arrives as an empty shell and fills in after scripts run, an expression evaluated against the fetched HTML finds scaffolding. Either evaluate after rendering in a headless browser, or better, check whether the data exists as a JSON feed or embedded state you can read without pointing at the DOM at all.
Back to the full glossary.