Data parsing
Turning a fetched response into structured data, the half of scraping that happens after the network and breaks for entirely different reasons.
The fetch is the loud half of scraping, with its proxies, blocks and retries; parsing is the quiet half where the value actually appears. A response arrives as bytes, markup, JSON, occasionally worse, and parsing is everything between that arrival and a clean row: decoding, tree-building, selection, normalisation, validation. Pipelines are usually judged by their fetch rate and killed by their parse quality.
For HTML, parse means building a document tree with a parser that tolerates the real web, where tags go unclosed and nesting is aspirational. Proper parsers repair such markup the way browsers do; improvised extraction with regular expressions works until the first attribute appears in an unexpected order, which is why regex-on-markup is the community's canonical cautionary tale. Once a tree exists, the pointing languages take over, and the parse's job is done when selection has something solid to stand on.
The best parsing move is often to sidestep HTML entirely. Modern pages carry their data in structured form already: JSON responses feeding the page's own scripts, state blobs embedded in the document, and the machine-readable annotations sites publish for search engines, product name, price and availability in a script tag, labelled to a public vocabulary. Reading those yields typed fields with no selector archaeology, and they change far less often than layouts do, because other machinery depends on them.
Encoding is the trap under everything. Bytes become text through a character encoding, and assuming the wrong one produces mojibake, the garbled currency signs and accented letters of a parse that technically succeeded. Declared charsets lie occasionally; compressed responses need decompressing before anything else; and numbers arrive dressed in locale, where 1.299,00 and 1,299.00 disagree about which mark is the decimal. Normalisation, prices to a number and a currency, dates to a timezone-aware form, whitespace collapsed, is parsing too, and skipping it just moves the work into every consumer of your data.
The discipline that separates durable pipelines is validating the OUTPUT, not the fetch. A parse that finds zero prices on a page that plainly sells things is a signal, either the layout moved or the site served you something other than the real page, and a pipeline that records empty fields as success will fill a warehouse with confident blanks. Parse failures are cheap when they are loud and ruinous when they are silent, so build them loud.
Frequently asked questions
Why should I not parse HTML with regular expressions?
Because markup's structure, nesting, optional attributes in any order, unclosed tags repaired by browsers, is exactly what regular expressions cannot track. Patterns that survive one page shatter on its siblings. Real HTML parsers repair the mess into a tree, selectors then address it reliably, and the regex stays useful for what it is good at: cleaning individual extracted strings.
What is the easiest data to parse from a web page?
The data the page already publishes as data. Many sites embed machine-readable annotations for search engines, name, price, availability in labelled JSON inside a script tag, and many more ship their state as JSON the page's own scripts consume. Both give typed fields without selector work, and both are more stable than layouts because other systems depend on them.
Why does my scraped text show broken characters?
An encoding mismatch: the bytes were decoded with the wrong character set, usually by trusting a wrong or missing declaration. The result is mojibake, garbled accents and currency signs on an otherwise successful parse. Detect and honour the true encoding, decompress before decoding, and test on pages with non-English text, where the failure shows immediately.
How do I know my parser is still working?
Validate outputs, not status codes. Require that expected fields actually appear, prices numeric, dates plausible, counts above zero on pages that visibly have items, and alarm when they do not. Layout changes and silently degraded responses both present as successful fetches with empty or wrong fields, so the only reliable tripwire is asserting on the data itself.
Back to the full glossary.