pandas read_html() finds every <table> in a page and returns each one as a DataFrame. It is the shortest way from an HTML table to data you can sort, filter and save.
We tested it on our own server on 12 October 2026, with pandas 3.0.6, lxml 6.1.3 and Python 3.12.3. Two habits from older tutorials broke. Passing HTML as a plain string raised an error, and letting pandas download a page itself got a 403 from Wikipedia.
Read every table on a page
Download the page with requests, then hand pandas the text wrapped in StringIO:
from io import StringIO
import pandas as pd
import requests
URL = "https://hproxy.com/blog/python-http-clients"
HEADERS = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/pandas-read-html)"}
r = requests.get(URL, headers=HEADERS, timeout=30)
r.raise_for_status()
tables = pd.read_html(StringIO(r.text))
print(len(tables), "tables")
print(tables[0].head(3))
4 tables
Client ... On a server that never answers
0 urllib.request ... still waiting at 15 s
1 http.client ... still waiting at 15 s
2 urllib3 ... gave up at 12.1 s with request()
[3 rows x 6 columns]
read_html() always returns a list, so tables[0] is the first table. The StringIO matters in pandas 3.0. We passed the same HTML as a plain string, and pandas looked for a file by that name. It raised FileNotFoundError: [Errno 2] No such file or directory: <html><body>. Its documentation now describes the input as a "String path, path object (implementing os.PathLike[str]), or file-like object implementing a string read() function". So a string is a path or a URL, never HTML.
Pick one table
Most pages have several tables. match= keeps the tables that contain a word, and attrs= keeps the ones with a given id or class. The documentation describes match as "The set of tables containing text matching this regex or string will be returned". With the response from above:
clients = pd.read_html(StringIO(r.text), match="Sync or async")[0]
print(clients[["Client", "Install", "Sync or async"]])
Client Install Sync or async
0 urllib.request built in sync
1 http.client built in sync
2 urllib3 1 package, 0.9 MB sync
3 requests 5 packages, 3.1 MB sync
4 httpx 7 packages, 3.9 MB both
5 aiohttp 10 packages, 13.6 MB async
On our test page, attrs={"id": "plans"} returned exactly one table, and match="Preis" found the one German table among four.
Why not just read_html(url)?
read_html() takes a URL and downloads the page itself, and many tutorials use it that way. It downloads with Python's urllib. Our test server recorded the User-Agent it sent: Python-urllib/3.12. Python's own documentation warns that "some HTTP servers only allow requests coming from common browsers as opposed to scripts".
We tried it on a Wikipedia page that several tutorials use. With the default User-Agent, pandas got HTTPError: HTTP Error 403: Forbidden. With a User-Agent of our own it returned 6 tables. Wikimedia's policy says it plainly: "Scripts should use an informative User-Agent string with contact information, or they may be blocked without notice".
storage_options sets that header. In pandas' documentation, for HTTP(S) URLs "the key-value pairs are forwarded to urllib.request.Request as header options". Our test server received my-scraper/1.0 from pd.read_html(url, storage_options={"User-Agent": "my-scraper/1.0"}).
Downloading with requests is still the better habit. You get a timeout, retries, sessions and proxies, and the same text works for read_html and for BeautifulSoup.
Numbers, headers and merged cells
pandas reads cells as text unless they look like numbers. Our test table came back like this:
- Currency stays text. "$1,200.50" stayed a string, and pandas 3.0 gave the column the dtype
str. A cell with "n/a" became NaN.str.replaceandto_numericturned the column into[1200.5, 2400.0, nan]. - A table without
<th>got the column names 0 and 1.header=0made the first row the header. - Merged header cells became a two-level header:
('Region', 'Region'),('Q1', 'Jan')and('Q1', 'Feb').
European numbers need two settings. Without them, "1.234,56" stayed text:
from io import StringIO
import pandas as pd
html = "<table><tr><th>Land</th><th>Preis</th></tr><tr><td>Deutschland</td><td>1.234,56</td></tr></table>"
prices = pd.read_html(StringIO(html), thousands=".", decimal=",")[0]
print(prices, prices["Preis"].dtype, sep="\n")
Land Preis
0 Deutschland 1234.56
float64
Links in table cells
By default read_html keeps only the text of a cell. extract_links="body" changes that. In the documentation's words, "Table elements in the specified section(s) with <a> tags will have their href extracted". Each cell becomes a pair of text and address:
from io import StringIO
import pandas as pd
html = '<table><tr><th>Plan</th></tr><tr><td><a href="/plans/basic">Basic</a></td></tr></table>'
print(pd.read_html(StringIO(html), extract_links="body")[0])
Plan
0 (Basic, /plans/basic)
When read_html finds nothing
read_html raised ValueError: No tables found in two of our tests. One page had no table at all. The other had a table that a script would add after the page loads, which is not in the HTML yet. The documentation says the function "will not return an empty list", so catch the error. For tables that JavaScript builds, render the page in a browser tool such as Playwright first and pass the rendered HTML.
Which parser pandas uses
read_html needs a parser installed. In the documentation's words, "The default of None tries to use lxml to parse and if that fails it falls back on bs4 + html5lib". The user guide adds a warning. lxml "does not make any guarantees about the results of its parse unless it is given strictly valid markup". Install lxml with pip install lxml, and add beautifulsoup4 and html5lib for broken pages. Our BeautifulSoup tests show how the parsers differ on the same HTML.
Through a proxy
read_html has no proxy setting of its own. Download the page with requests through a proxy, then pass StringIO(r.text) as above. Our guide to proxies with Python requests shows the setup.
What this page could not check
- pandas 3.0.6 only. Code written for older versions may behave differently.
- The parsing tests used HTML we wrote. Real tables break in more ways.
- One public site, Wikipedia, was asked twice. Other sites decide about User-Agents in their own way.
- We will rerun these tests by 12 January 2027.
Sources
- pandas documentation, pandas.read_html: pandas.pydata.org.
- pandas user guide, IO tools, HTML gotchas: pandas.pydata.org.
- MDN, the th element: developer.mozilla.org.
- Python 3.12 documentation, urllib.request: docs.python.org.
- Wikimedia Foundation, User-Agent policy: foundation.wikimedia.org.
- Our own tests on our server, 12 October 2026: read_html_tests.py at 05:59 UTC and the code on this page at 06:00 UTC, on HTML of our own, hproxy.com and one Wikipedia page.
