Parse XML in Python: ElementTree, lxml and sitemaps

Python can read XML with its built-in ElementTree, with lxml, or with BeautifulSoup. We tested all three on our own sitemaps on 12 October 2026. One file holds 959 URLs, and a larger one holds 44,976 URLs in 8 MB.

Three things stood out. The findall() method found nothing until we handled the namespace. One parser took 36 times longer than another. And two malicious files showed what today's parsers refuse.

Parse XML from a URL or a string

import xml.etree.ElementTree as ET

import requests

r = requests.get("https://hproxy.com/sitemap.xml", timeout=30)
root = ET.fromstring(r.content)
print(root.tag)

{http://www.sitemaps.org/schemas/sitemap/0.9}urlset

ET.fromstring() parses bytes or a string, and ET.parse() reads a file. Give it r.content, the bytes, so the parser reads the encoding from the XML itself. The root tag already shows the catch: a namespace in curly braces.

The namespace trap: why findall() returns nothing

print(len(root.findall("url")))

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
print(len(root.findall("sm:url", NS)))
print(len(root.findall("{*}url")))
print(root.find("sm:url/sm:loc", NS).text)

0
959
959
https://hproxy.com

A sitemap declares a default namespace, so the full name of every url tag is {http://www.sitemaps.org/schemas/sitemap/0.9}url. That is why findall("url") found 0 of 959 URLs. Python's documentation gives the fix: "create a dictionary with your own prefixes and use those in the search functions". Since Python 3.8, {*}url also matches a url tag in any namespace.

lxml has the same trap in XPath:

from lxml import etree

doc = etree.fromstring(r.content)
print(len(doc.xpath("//url")), len(doc.xpath("//sm:url", namespaces=NS)))

0 959

lxml is also stricter about strings. Given r.text instead of the bytes, it refused the file because of its first line, <?xml version="1.0" encoding="UTF-8"?>:

try:
    etree.fromstring(r.text)
except ValueError as e:
    print(e)

Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

BeautifulSoup ignores namespaces and found all 959 at once:

from bs4 import BeautifulSoup

soup = BeautifulSoup(r.content, "xml")
print(len(soup.find_all("url")))

959

Large files: stream with iterparse

Sitemaps can be big. The protocol allows "no more than 50,000 URLs" and 50 MB per file, and our city sitemap comes in files of about 45,000 URLs. We parsed one of them, 8 MB, five ways, each in a fresh Python process, three times:

lxml was fastest at 0.18 seconds. BeautifulSoup took 6.44 seconds, 36 times longer, and peaked at 360 MB. Its documentation calls lxml's XML parser "Very fast", and BeautifulSoup does use it, so the extra time is BeautifulSoup's own work on top.

A full parse keeps the whole tree in memory: 93 MB with ElementTree and 123 MB with lxml. Both iterparse runs never rose above the 47 MB that Python used with the libraries loaded. With ElementTree:

SM = "{http://www.sitemaps.org/schemas/sitemap/0.9}"

locs = []
for _, elem in ET.iterparse("city-sitemap-0.xml"):
    if elem.tag == SM + "url":
        locs.append(elem.findtext(SM + "loc"))
        elem.clear()
print(len(locs))

And with lxml, which also deletes the elements already read, as its documentation shows:

locs = []
for _, elem in etree.iterparse("city-sitemap-0.xml", tag=SM + "url"):
    locs.append(elem.findtext(SM + "loc"))
    elem.clear(keep_tail=True)
    while elem.getprevious() is not None:
        del elem.getparent()[0]
print(len(locs))

Both printed 44976. Here is the whole test run, from our server:

Malicious XML: what today's parsers do

Python's documentation warns that its XML modules "are not secure against erroneous or maliciously constructed data". We tried two classic attacks on each parser, in a separate process capped at 1 GB of memory:

• An entity bomb, also called billion laughs: a few lines that expand to a billion copies of "lol".
• An external entity: a reference that asks the parser to paste a local file into the document.

No parser expanded the bomb or read the file. That has changed over the years. Python's documentation notes that "Expat 2.4.1 and newer" is not vulnerable to billion laughs, and ours used expat 2.6.1. lxml's FAQ says that "Since version 5.x, lxml disables the expansion of external entities (XXE) by default".

Two results still matter. minidom and BeautifulSoup returned empty results without an error, so a bad file can look like an empty one. Check that you got the elements you expected.

For XML from sources you do not trust, Python's documentation recommends defusedxml "for any server code that parses untrusted XML data". Its ElementTree module works like the standard one, and it refused both files and named the entity:

import defusedxml.ElementTree as DET

root = DET.fromstring(r.content)
print(len(root.findall("sm:url", NS)))

959

Fetching XML through a proxy

Sitemaps and feeds are often fetched in bulk, from many sites. With requests, pass proxies= with both an http and an https key, as in our guide to proxies with Python requests. Through 3proxy on our server, the sitemap came back whole, all 959 URLs, and the proxy logged only CONNECT hproxy.com:443.

Which parser to use

• ElementTree comes with Python, was fast enough on 44,976 URLs, and streams large files with iterparse.
• lxml was the fastest and adds full XPath, but install it with pip, and give it bytes.
• BeautifulSoup is forgiving and ignores namespaces, but it was 36 times slower here and silent on bad input.
• defusedxml is for XML from sources you do not trust.

What this page could not check

• One sitemap file on one server, which was busy during the runs. The seconds are rough, but their order held over three runs.
• Peak memory is that of the whole Python process.
• Two classic attacks, not every one: we did not test decompression bombs, XPath injection or very deep nesting.
• We will rerun these tests with current versions by 12 January 2027.

Sources

• Python 3.12 documentation, XML Processing Modules (security): docs.python.org.
• Python 3.12 documentation, xml.etree.ElementTree: docs.python.org.
• lxml documentation, parsing XML: lxml.de, and the FAQ: lxml.de.
• Beautiful Soup documentation, parsers: crummy.com.
• sitemaps.org, the sitemap protocol: sitemaps.org.
• defusedxml 0.7.1 on PyPI: pypi.org.
• Our own tests on our server, 12 October 2026: xml_tests.py at 00:03 UTC and the code on this page at 00:10 UTC, on hproxy.com's own sitemaps.