Web scraping with BeautifulSoup: parse HTML in Python, with examples

BeautifulSoup turns HTML into a tree of Python objects that you can search. It does not download anything. A library such as requests fetches the page, and BeautifulSoup reads what comes back.

We tested BeautifulSoup 4.15.0 on our own server on 12 October 2026, with Python 3.12.3. Every example on this page ran there, and its output is pasted below it. We also fed the same HTML to BeautifulSoup's three parsers and to Chrome 143. The parser most tutorials use built a different tree from the browser, even for valid HTML.

Install BeautifulSoup

pip install beautifulsoup4 lxml

The package is called beautifulsoup4, and you import it as bs4. Version 4.15.0 came out on 7 June 2026. lxml is a fast parser that BeautifulSoup can use. A third parser, html5lib, installs the same way.

Your first soup

BeautifulSoup reads a string or bytes. This example parses two product cards written into the code, so it runs anywhere:

from bs4 import BeautifulSoup

html = """
<div class="product featured" data-id="101">
  <h2><a href="/lamp">Desk <em>Lamp</em></a></h2>
  <p class="price">£25.00</p>
</div>
<div class="product" data-id="102">
  <h2><a href="/chair">Office Chair</a></h2>
  <p class="price">£89.00</p>
  <p class="stock">Back in stock</p>
</div>
"""

soup = BeautifulSoup(html, "lxml")
for card in soup.select("div.product"):
    name = card.h2.get_text(" ", strip=True)
    link = card.a["href"]
    price = card.select_one("p.price").get_text()
    stock = card.find("p", class_="stock")
    print(card["data-id"], name, link, price, stock.get_text() if stock else "-")

101 Desk Lamp /lamp £25.00 -
102 Office Chair /chair £89.00 Back in stock

What each part does:

• soup.select("div.product") takes a CSS selector and returns a list. select_one() returns the first match, or None.
• card.h2 is the first h2 inside the card, and card.a["href"] reads an attribute.
• card.find("p", class_="stock") is the same kind of search without CSS. It returns None when nothing matches, so the code checks before it calls get_text().
• get_text(" ", strip=True) joins the pieces of text with a space. The section on traps below shows why the space matters.

Pick the parser on purpose

BeautifulSoup does not parse HTML itself. It hands the document to a parser: Python's built-in html.parser, lxml, or html5lib. Its documentation warns that "if a document is invalid, different parsers will generate different Beautiful Soup trees for it". In our test, that happened with valid HTML too.

HTML lets you leave out some closing tags. The standard says an li element's end tag "can be omitted if the li element is immediately followed by another li element". So this list is valid HTML:

from bs4 import BeautifulSoup

items = "<ul><li>one<li>two</ul>"  # valid HTML: the closing </li> tags are optional
for parser in ("html.parser", "lxml", "html5lib"):
    soup = BeautifulSoup(items, parser)
    print(f"{parser:12}", [li.get_text() for li in soup.find_all("li")])

html.parser  ['onetwo', 'two']
lxml         ['one', 'two']
html5lib     ['one', 'two']

html.parser put the second item inside the first, so the first item's text became "onetwo". We ran four bits of HTML through the three parsers and through Chrome's own parser. Then we compared what each one put inside the body:

A closing p with no open p is a parse error. The HTML standard then tells the parser to "insert an HTML element" for a p start tag "with no attributes". That is the empty <p></p> that html5lib and Chrome built. html5lib follows the standard on purpose: it "is designed to conform to the WHATWG HTML specification, as is implemented by all major web browsers".

So which one to use:

• lxml: the fastest of the three in our test, and the same tree as Chrome in 3 of 4 cases. Use it by default.
• html5lib: the browser's tree, at about 2.3 times the time lxml took. Use it when a page only comes out right that way.
• html.parser: nothing to install, but it got both valid cases wrong. 8 of the 10 rival tutorials we read build their examples on it.

Always name the parser. Without one, BeautifulSoup takes the best parser installed, and that depends on the machine. We parsed <p>first<p>second with no parser named, in three setups. With lxml installed it used lxml, with only html5lib it used html5lib, and with neither it used html.parser and nested the paragraphs. BeautifulSoup warns about this with a GuessedAtParserWarning: "it may use a different parser and behave differently".

The tbody trap

When a table row starts straight inside a table, the HTML standard tells the browser to insert a tbody element first. So the table you inspect in the browser has a tbody, even when the HTML has none. lxml and html.parser keep the HTML as it was written:

from bs4 import BeautifulSoup

table = """<table id="prices">
  <tr><th>Plan</th><th>Price</th></tr>
  <tr><td>Basic</td><td>5</td></tr>
</table>"""

soup = BeautifulSoup(table, "lxml")
print(len(soup.select("#prices > tbody > tr")))  # written from what the browser shows
rows = [[cell.get_text(strip=True) for cell in tr.find_all(["th", "td"])] for tr in soup.select("#prices tr")]
print(rows)

0
[['Plan', 'Price'], ['Basic', '5']]

The selector with tbody in it found 0 rows with lxml and with html.parser. html5lib and Chrome found 2. The descendant selector #prices tr works with or without a tbody, and so does find_all("tr"). None of the 12 rival guides we read mentions tbody.

find() or select()?

BeautifulSoup has two ways to search, and they reach the same elements. We ran each pair below on one test card, and each pair returned the very same objects:

class_ ends with an underscore because class is a Python keyword. It matches any one of an element's classes, so class_="featured" finds <div class="card featured">.

On a test page with 3,000 product cards, find_all() collected the prices in 26.4 ms and select() in 69.2 ms. Both are small next to the 703 ms it took to parse the page, so pick the one that reads better. Our page on XPath and CSS selectors compares the two languages in more depth.

Six things that trip people up

from bs4 import BeautifulSoup

soup = BeautifulSoup('<h2 class="title big"><a href="/lamp">Desk <em>Lamp</em></a></h2>', "lxml")
print(soup.h2["class"])                   # a list: class can hold several names
print(soup.h2.string)                     # None: the text comes in more than one piece
print(soup.h2.get_text(strip=True))       # strip=True glues the pieces together
print(soup.h2.get_text(" ", strip=True))  # a separator keeps the words apart
print(soup.a.get("title"))                # None; soup.a["title"] would raise KeyError
print(soup.find("table"))                 # None: check before you call .get_text()

['title', 'big']
None
DeskLamp
Desk Lamp
None
None

1. class is a list. "The most common multi-valued attribute is class", the documentation explains, so BeautifulSoup returns its values as a list.
2. .string can be None. When a tag holds more than one piece, "so .string is defined to be None", in the documentation's words. Use get_text().
3. get_text(strip=True) glues words. It strips each piece and joins them with nothing, so "Desk" and "Lamp" became "DeskLamp". Pass a separator.
4. A missing attribute raises. tag.get("title") returns None, and tag["title"] raises a KeyError.
5. No match returns None. find() and select_one() return None, and calling .get_text() on it raises AttributeError: 'NoneType' object has no attribute 'get_text'. find_all() and select() return an empty list.
6. Whitespace is a node. In our test card, h2.next_sibling was the line break after the h2, not the next tag. find_next_sibling() skips to the next element.

A real page, start to finish

This script reads the outline of one of our own posts. It sends a User-Agent that names the scraper and sets a timeout. Then it hands BeautifulSoup the bytes in r.content, for the reason in the next section.

import requests
from bs4 import BeautifulSoup

URL = "https://hproxy.com/blog/web-scraping-with-python"
HEADERS = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/web-scraping-with-beautifulsoup)"}


def attr(soup, selector, name):
    """An attribute of the first match, or None when nothing matches."""
    tag = soup.select_one(selector)
    return tag.get(name) if tag else None


r = requests.get(URL, headers=HEADERS, timeout=30)
r.raise_for_status()
soup = BeautifulSoup(r.content, "lxml")

print(soup.h1.get_text(" ", strip=True))
print(attr(soup, 'meta[name="description"]', "content"))
for h2 in soup.select("article h2"):
    print("-", h2.get_text(" ", strip=True))
links = {a["href"] for a in soup.select('article a[href^="/blog/"]')}
print(len(soup.select("article pre")), "code blocks,", len(links), "links to other posts")

Web scraping with Python: requests, BeautifulSoup and proxies, step by step
We built and ran a Python scraper on 1,000 books, found an encoding bug the tutorials we read never explain, and tested two proxy traps in requests.
- What you need
- Step 1: download a page, then read the bytes
- Step 2: find the data with CSS selectors
- Step 3: follow the pages, slowly
- Step 4: save the rows to a CSV file
- The whole scraper
- Step 5: send the scraper through a proxy
- When requests is not enough
- What this page could not check
- Sources
- Frequently asked questions
10 code blocks, 14 links to other posts

attr() returns None instead of crashing when a page lacks the tag. That matters as soon as a script reads many pages, because some will lack it. For a scraper that follows pages and saves rows to a file, see our web scraping with Python tutorial.

Give BeautifulSoup bytes, not r.text

requests decodes r.text before BeautifulSoup ever sees it. When a server sends Content-Type: text/html with no charset, requests uses ISO-8859-1. Its documentation explains: "RFC 2616 specifies that the default charset must be ISO-8859-1. Requests follows the specification in this case."

We served three test pages with exactly that header and parsed each one both ways:

With the bytes, BeautifulSoup read the meta tag or worked out the encoding itself, and got all three right. soup.original_encoding shows its choice: utf-8, utf-8 and windows-1252. The third row is the easiest to miss. r.text looks right until the euro sign, which turned into an invisible control character. 6 of the 10 rival tutorials we read hand BeautifulSoup r.text somewhere in their code.

When BeautifulSoup is too slow

We built a product page with 3,000 cards: 824,452 bytes and 30,006 elements. Each setup parsed it five times, and we took the middle time. A fresh process for each setup measured the memory the tree added:

lxml was more than twice as fast as html.parser. html5lib was the slowest, as BeautifulSoup's documentation warns: "Very slow". lxml alone, without BeautifulSoup, parsed the same page in under a tenth of the time. Our XPath and CSS tests found the same gap on another page.

If you only need some elements, a SoupStrainer tells BeautifulSoup to build only those. With the r from the script above:

from bs4 import SoupStrainer

only_links = SoupStrainer("a", href=True)
links_soup = BeautifulSoup(r.content, "lxml", parse_only=only_links)
print(len(links_soup.find_all("a")), "links")

121 links

On the test page, keeping only the price paragraphs cut the time from 703 to 289 ms. The memory fell from 28.9 to 4.2 MB. A strainer does nothing with html5lib: "If you use html5lib, the whole document will be parsed, no matter what", the documentation says.

Old names that still work, for now

Many older examples use names that BeautifulSoup 4.15.0 still accepts, with a warning:

Soup Sieve, the library behind select(), says the :contains() alias "is deprecated moving forward and will be removed in a future version".

When find() finds nothing

If find() returns None for something you can see in the browser, check whether the server's HTML holds it at all. Search r.content for a word from it. Content that JavaScript adds after the page loads is not there, because requests does not run scripts. For those pages you need a browser tool, such as Playwright or Selenium.

If the HTML does hold it, check your parser and your selector against the two traps above.

Scraping through a proxy

BeautifulSoup never opens a connection, so a proxy belongs on the request. Pass a proxies dictionary to requests.get(), as our guide to proxies with Python requests shows, and parse r.content as above. If a site starts refusing requests, our guides to retries and to request headers cover the next steps.

What this page could not check

• Our parser tests used short HTML we wrote. Real pages break HTML in more ways.
• The times come from one server, as the middle of five runs. Your milliseconds will differ, so compare the ratios.
• One real page was tested, from our own site.
• Pages that build their content with JavaScript were not tested.
• We will rerun these tests by 12 January 2027.

Sources

• Beautiful Soup documentation: crummy.com.
• WHATWG HTML Living Standard, parsing HTML documents: html.spec.whatwg.org.
• WHATWG HTML Living Standard, grouping content (the li and p elements): html.spec.whatwg.org.
• Requests documentation, advanced usage (encodings): requests.readthedocs.io.
• Soup Sieve documentation, pseudo-classes: facelessuser.github.io.
• html5lib on PyPI: pypi.org.
• beautifulsoup4 4.15.0 on PyPI (released 7 June 2026): pypi.org.
• Our own tests on our server, 12 October 2026: bs_tests.py at 02:50 UTC and the code on this page at 02:49 UTC, on HTML of our own and on hproxy.com.