Parse HTML with regex: when it works and when to use a parser

Can you parse HTML with a regular expression? You can search HTML with one, but whether it finds the right things is another question, so we measured it.

On 12 October 2026 we gave 13 short pieces of HTML to four regular expressions and to two parsers, BeautifulSoup and lxml. Chrome's own HTML parser decided which links each piece really contains. The test ran on our server with Python 3.12.3, beautifulsoup4 4.15.0, lxml 6.1.3 and Chrome 143.

The short answer

• The simplest link pattern, <a href="(.*?)", was right on 2 of the 13 inputs.
• A regex we hardened rule by rule was right on 12. The 13th input still broke it.
• BeautifulSoup and lxml were right on all 13.
• The regular expressions were 9 to 39 times faster than lxml. The fastest one found only half the links.
• Regex is the right tool for text that a parser has already pulled out of the page.

The test: 13 inputs, judged by Chrome

Each input is one short piece of HTML, such as a link in single quotes or a link inside a comment. Chrome's DOMParser read each one first. MDN describes it plainly. "The DOMParser interface provides the ability to parse XML or HTML source code from a string into a DOM Document". The links Chrome found count as the right answer. Then every method read the same piece.

R1 is the simplest link pattern, and R2 drops the <a to look for href anywhere. R3 allows other attributes first, single quotes and a line break inside the tag. R4 is a hardened regex, shown further down. In the table, a method "missed" when it found nothing where Chrome found a link. It found an "extra" when it also returned an address that is not a link.

What a regex misses

Five links, each written in a way the HTML standard allows, show the problem. A regex and two parsers read the same five.

import re

import lxml.html
from bs4 import BeautifulSoup

html = """<a class="nav" href="/pricing">Pricing</a>
<a href='/docs'>Docs</a>
<A HREF="/blog">Blog</A>
<!-- <a href="/old-page">Old</a> -->
<a href="/search?q=proxy&amp;page=2">Next</a>"""

print(re.findall(r'<a href="(.*?)"', html))
print([a["href"] for a in BeautifulSoup(html, "lxml").find_all("a", href=True)])
print(lxml.html.fromstring(html).xpath("//a/@href"))

['/old-page', '/search?q=proxy&page=2']
['/pricing', '/docs', '/blog', '/search?q=proxy&page=2']
['/pricing', '/docs', '/blog', '/search?q=proxy&page=2']

The regex returned a page that was commented out. It missed three of the four real links, and the fourth still has & in it. Each miss comes from a rule in the HTML standard.

• Letter case. The standard says "tag names are case-insensitive". "In the HTML syntax, attribute names, even those for foreign elements, may be written with any mix of ASCII lower and ASCII upper alphas". So <A HREF="/blog"> is a link.
• Quotes and spaces. A value can sit in double quotes, single quotes or no quotes. The standard lists an "Unquoted attribute value syntax" and a "Single-quoted attribute value syntax". It also allows spaces on both sides of the equals sign.
• Order. Attributes can come in any order, so class can stand before href.
• Comments. Everything between <!-- and --> is a comment, not markup.
• Scripts. The standard lists script and style as raw text elements. What they hold is text, so a link written inside a script is not a link.
• Entities. "Attribute values are a mixture of text and character references". & in an address stands for &, and a regex returns it as written.

Greedy, lazy and nested

The next trap sits in the pattern itself, and Python's documentation warns about it. "The '*', '+', and '?' quantifiers are all greedy; they match as much text as possible". In its own example, <.*> run on <a> b <c> matches the whole string. Adding ? makes a quantifier lazy, and lazy has a problem of its own.

import re

html = "<h2>Plans</h2><p>Text</p><h2>Questions</h2>"
print(re.findall(r"<h2>(.*)</h2>", html))
print(re.findall(r"<h2>(.*?)</h2>", html))

nested = "<div><div>inner</div>outer</div>"
print(re.findall(r"<div>(.*?)</div>", nested))

['Plans</h2><p>Text</p><h2>Questions']
['Plans', 'Questions']
['<div>inner']

The greedy pattern swallowed both headings and the paragraph between them. The lazy one fixed that, but on the nested div it stopped at the first </div>, which belongs to the inner div. To pair each closing tag with its opening tag, a program has to count how deep it is. Python's re cannot keep that count, while a parser builds a tree and always knows.

This is the reason behind the best-known answer on the subject, posted on Stack Overflow on 13 November 2009. It opens with "You can't parse [X]HTML with regex. Because HTML can't be parsed by regex". Its short reason is "HTML is not a regular language and hence cannot be parsed by regular expressions". On 12 October 2026 it had a score of 4,394.

Fixing the regex, one rule at a time

Every miss in the test can be patched. R4 is our attempt. It removes comments and script and style blocks first. Then it ignores letter case and allows spaces around the equals sign. It accepts double quotes, single quotes or none, and it skips data-href. At the end, Python's html.unescape() turns & back into &.

import re
from html import unescape

COMMENT = re.compile(r"<!--.*?-->", re.S)
RAW_TEXT = re.compile(r"<(script|style)\b.*?</\1\s*>", re.S | re.I)
LINK = re.compile(r"""<a\s[^>]*?(?<![\w-])href\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+))""", re.I)


def links(html):
    html = RAW_TEXT.sub("", COMMENT.sub("", html))
    return [unescape(a or b or c) for a, b, c in LINK.findall(html)]


print(links("<A HREF = '/search?q=proxy&amp;page=2'>Next</A>"))
print(links('<a title="1 > 0" href="/m">M</a>'))

['/search?q=proxy&page=2']
[]

R4 was right on 12 of our 13 inputs. The 13th has a > inside a quoted title, which is valid HTML. [^>]*? stops at that > before it reaches href, so the link is lost. Fixing that means reading quoted values properly, and that is what a parser does. Every patch adds one rule the HTML standard already has, and a parser follows all of them.

In Python the usual parsers are BeautifulSoup and lxml. "Beautiful Soup is a Python library for pulling data out of HTML and XML files", says its documentation. Both were right on all 13 inputs.

Is regex faster?

Yes, by a lot. We timed every method on one page of 3,000 product cards, 487,586 bytes with 6,000 links. The times are CPU time, the middle of five runs.

R1 was the fastest and found 3,000 of the 6,000 links. Each card had a second link with class before href, and R1 skipped every one of them. R3 found all 6,000, because every link on this page used double quotes. It was still wrong on 7 of our 13 inputs.

lxml read the whole page in 35.4 ms, about 9 times faster than BeautifulSoup with lxml. If parsing speed matters, use lxml on its own or one of the faster parsers in our libraries test.

Where a regex does fit

Regular expressions are good at patterns inside text, such as prices, dates and order numbers. Let a parser find the element, then let the regex read its text.

import re

from bs4 import BeautifulSoup

html = '<div class="product"><h2>Laptop</h2><p class="price">Price: £1,299.00 (was £1,499.00)</p></div>'
soup = BeautifulSoup(html, "lxml")
text = soup.select_one("p.price").get_text()
prices = [float(p.replace(",", "")) for p in re.findall(r"£([\d,]+\.\d{2})", text)]
print(text)
print(prices)

Price: £1,299.00 (was £1,499.00)
[1299.0, 1499.0]

The parser deals with the HTML: tags, quotes, entities and nesting. The regex only sees one short string with no markup left in it. A quick regex search is also fine in HTML you wrote yourself, as long as you check what it missed.

Through a proxy

Neither a regex nor a parser downloads anything. Fetch the page first, for example with requests through a proxy, as our guide to proxies with Python requests shows. Then hand the text to BeautifulSoup or lxml. Our BeautifulSoup tutorial covers the path from download to data.

What this page could not check

• The 13 inputs are short pieces we wrote. Real pages break in more ways, and some break parsers too.
• One browser, Chrome 143, decided the right answers.
• One version each: Python 3.12.3 with its re module, beautifulsoup4 4.15.0 and lxml 6.1.3. We did not test the third-party regex module.
• The speed test used one page of our own. Other pages give other times.
• R4 is our own pattern. A better one may exist, but the 13th input shows how each patch meets the next case.
• We will rerun these tests by 12 January 2027.

Sources

• WHATWG, HTML Living Standard, the HTML syntax: html.spec.whatwg.org.
• MDN, DOMParser: developer.mozilla.org.
• Python 3.12 documentation, re: docs.python.org.
• Python 3.12 documentation, html.unescape: docs.python.org.
• Beautiful Soup documentation: crummy.com.
• Stack Overflow, "RegEx match open tags except XHTML self-contained tags", the answer of 13 November 2009: stackoverflow.com. Read through the Stack Exchange API on 12 October 2026.
• Our own tests on our server, 12 October 2026: regex_tests.py at 06:20 UTC and the code on this page at 06:23 UTC, on HTML of our own.