Image scraping: how to download images from a website, and what is allowed

Image scraping means finding the image files a web page uses and downloading them with a program. The first scraper most people write reads every img tag's src, and on a modern page that misses much of what is there.

We tested two finders on 12 October 2026: a simple one and a fuller one. We ran them on a test page we built, with the forms real sites use, and on a page of our own site. Along the way, Python's standard robots.txt parser told us our own site forbids every image. It does not.

First, what you are allowed to take

Images are protected works. US copyright law covers "pictorial, graphic, and sculptural works", and most countries protect them too. Downloading an image to study it and publishing it again are different questions. Our page on whether web scraping is legal goes through them.

Then read the site's terms and its robots.txt. If a site keeps images behind a login or forbids copying them, ask for permission or use an official API. This page covers only images a site serves openly.

What img src misses

Our test page had eight image files and one inline image, marked up the ways real sites do. It had a plain src and a root-relative one, and a srcset with a 400 and an 800 pixel version. A lazy-load placeholder kept the real image in data-src. A picture tag held a larger source, an image sat in the HTML as data:, and another was a CSS background. Two files had the same bytes under different names.

Reading every img src gave 3 distinct images and 2 errors. It took the small version from the srcset, the placeholder instead of the lazy image, and the small picture image. It never saw the CSS background, and the two data: addresses raised InvalidSchema, because requests cannot fetch them.

The fuller finder below got all 7 distinct files. It skipped the eighth because its bytes repeated another file.

A downloader that finds them all

import base64
import hashlib
import mimetypes
import os
import re
import time
from urllib.parse import urljoin
from urllib.robotparser import RobotFileParser

import requests

CSS_URL = re.compile(r"url\(\s*['\"]?([^'\")]+)['\"]?\s*\)")


def largest(srcset):
    """The URL with the largest width (or density) in a srcset."""
    best, best_size = None, -1.0
    for part in srcset.split(","):
        bits = part.strip().split()
        if not bits:
            continue
        size = float(bits[1][:-1]) if len(bits) > 1 and bits[1][-1] in "wx" else 1.0
        if size > best_size:
            best, best_size = bits[0], size
    return best


def find_images(soup, page_url):
    found = []
    for img in soup.find_all("img"):
        src = img.get("data-src") or img.get("data-lazy-src")  # lazy loading keeps the real image here
        if not src and img.get("srcset"):
            src = largest(img["srcset"])
        src = src or img.get("src")
        if src:
            found.append(src)
    for source in soup.select("picture source[srcset]"):
        found.append(largest(source["srcset"]))
    for tag in soup.select("[style*='url(']"):
        found += CSS_URL.findall(tag["style"])
    urls = []
    for src in found:
        url = src if src.startswith("data:") else urljoin(page_url, src)
        if url not in urls:
            urls.append(url)
    return urls


def robots_for(site, headers):
    """robots.txt fetched with YOUR User-Agent: RobotFileParser.read() sends its own, which some sites refuse."""
    rp = RobotFileParser()
    r = requests.get(urljoin(site, "/robots.txt"), headers=headers, timeout=30)
    rp.parse(r.text.splitlines() if r.status_code == 200 else [])
    return rp


def download(urls, folder, headers=None, delay=1.0):
    os.makedirs(folder, exist_ok=True)
    seen, saved = set(), []
    for url in urls:
        if url.startswith("data:"):  # an inline image: its bytes are in the address
            head, _, data = url.partition(",")
            content, ctype = base64.b64decode(data), head[5:].split(";")[0]
        else:
            r = requests.get(url, headers=headers, timeout=30)
            r.raise_for_status()
            content, ctype = r.content, r.headers.get("Content-Type", "").split(";")[0]
            time.sleep(delay)
        digest = hashlib.sha256(content).hexdigest()
        if digest in seen:  # the same bytes under another name
            continue
        seen.add(digest)
        path = os.path.join(folder, digest[:16] + (mimetypes.guess_extension(ctype) or ".bin"))
        with open(path, "wb") as f:
            f.write(content)
        saved.append(path)
    return saved

Save it as images.py. Each part handles one of the cases above:

• find_images prefers data-src over a placeholder, and the widest srcset entry over the default src. It also reads picture sources and CSS url() values, and makes every relative address absolute with urljoin.
• download decodes inline data: images instead of requesting them, and waits between requests. It names each file by a hash of its bytes, so a repeated image is saved once, and picks the extension from the server's Content-Type.

Read robots.txt with your own User-Agent

Python has a robots.txt parser in its standard library. We pointed it at our own site:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://hproxy.com/robots.txt")
rp.read()
print(rp.disallow_all)

True

By that answer, our whole site was off-limits, and 0 of the 13 images on the page were allowed. The cause was the request itself. read() asks for robots.txt with Python's own User-Agent, Python-urllib/3.12, and our site answered that with a 403. With our scraper's User-Agent, the same file came back with a 200. Its rules allowed all 13 images.

When Python's parser gets a 401 or 403, its source sets disallow_all = True. The robots.txt standard says the opposite: for a file that answers in "the 400-499 range", "the crawler MAY access any resources on the server". So fetch robots.txt yourself, with the User-Agent your scraper uses, and hand it to parse(). That is what robots_for does.

Run it on a real page

import requests
from bs4 import BeautifulSoup

import images

UA = {"User-Agent": "hproxy-tutorial/1.0 (+https://hproxy.com/blog/image-scraping)"}
page = "https://hproxy.com/blog/web-scraping-with-python"

robots = images.robots_for(page, UA)
soup = BeautifulSoup(requests.get(page, headers=UA, timeout=30).content, "lxml")
urls = [u for u in images.find_images(soup, page) if u.startswith("data:") or robots.can_fetch(UA["User-Agent"], u)]
saved = images.download(urls, "images", headers=UA, delay=1.0)
print(len(urls), "allowed,", len(saved), "files saved")

13 allowed, 13 files saved

That is a page of our own site: 13 images, 274,665 bytes in all, every Content-Type matching its file's extension. Use a User-Agent that names you, and a pause of a second or more. One scraper should not flood a site with image requests. Here is the whole test run:

When this is not enough

Some galleries load their images with JavaScript after the page arrives, so the addresses are not in the HTML that requests receives. For those you need a browser tool. Our guides to Playwright and Selenium cover the setup.

Downloading thousands of images from one site can also run into rate limits. Our guide to fixing 429 errors explains when to slow down and when a proxy helps.

What this page could not check

• One test page we made and one real page of our own site. Real sites mix these forms in other ways.
• Galleries that load images with JavaScript were not tested.
• Copyright differs by country, and this page is not legal advice.
• We will rerun these tests by 12 January 2027.

Sources

• RFC 9309, Robots Exclusion Protocol (September 2022): rfc-editor.org.
• Python 3.12 documentation, urllib.robotparser: docs.python.org.
• MDN, the img element: developer.mozilla.org.
• MDN, the picture element: developer.mozilla.org.
• 17 U.S.C. 102, subject matter of copyright: law.cornell.edu.
• Our own tests on our server, 12 October 2026: image_tests.py at 02:06 UTC and the code on this page at 02:12 UTC, on a test page of our own and on hproxy.com.