Scrapy is a Python framework for crawling. It follows links, sends many requests at once, retries failures and hands every item to pipelines that clean it. This tutorial builds a small project and runs it, step by step, against a practice shop you can start on your own machine.
We ran every step on our own server on 12 October 2026 with Scrapy 2.19.0 and Python 3.12.3. Each result below comes from Scrapy's own statistics and from the shop's request log.
What you need
- Python 3.10 or newer. Scrapy 2.19.0 lists it as its minimum.
- Scrapy itself. Our install of Scrapy 2.19.0 pulled in 45 packages and came to 116 MB.
pip install scrapy
Step 1: run a practice shop
A practice shop gives everyone the same pages, and no real site carries the load. This one has 100 products on five pages, plus four traps that real sites have too. A sale link lists one product a second time, and one product has no price. One page fails twice before it works, and one always fails. Save it as shop.py.
"""A practice shop to scrape with Scrapy: five pages of 100 products, a sale link to a product listed twice,
a product without a price, a page that fails twice before it works, and one that always fails."""
import sys
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
PRODUCTS = {n: {"name": f"Lamp {n}", "price": f"£{9 + n % 40}.99", "sku": f"LMP-{n:03d}"} for n in range(1, 101)}
PRODUCTS[42]["price"] = "" # a product without a price
FLAKY = {"/product/77": 2} # answers 500 twice, then works
BROKEN = {"/product/88"} # always answers 500
class Shop(BaseHTTPRequestHandler):
def do_GET(self):
time.sleep(0.1) # a little latency, like a real server
path, _, query = self.path.partition("?")
if path == "/robots.txt":
return self.reply(200, "User-agent: *\nDisallow: /cart\n", "text/plain")
if path in BROKEN:
return self.reply(500, "server error")
if FLAKY.get(path, 0) > 0:
FLAKY[path] -= 1
return self.reply(500, "server error")
if path.startswith("/product/") and path[9:].isdigit() and int(path[9:]) in PRODUCTS:
p = PRODUCTS[int(path[9:])]
return self.reply(200, f'<h1>{p["name"]}</h1><p class="price">{p["price"]}</p><p class="sku">{p["sku"]}</p>')
if path == "/":
page = int(query.partition("=")[2] or 1)
html = "".join(f'<a class="product" href="/product/{n}">Lamp {n}</a>' for n in range(page * 20 - 19, page * 20 + 1))
if page == 3: # a sale banner links product 12 again, with a tracking parameter
html += '<a class="product" href="/product/12?ref=sale">Lamp 12, on sale</a>'
if page < 5:
html += f'<a class="next" href="/?page={page + 1}">Next</a>'
return self.reply(200, html)
self.reply(404, "not found")
def reply(self, code, body, kind="text/html; charset=utf-8"):
data = body.encode()
self.send_response(code)
self.send_header("Content-Type", kind)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
if __name__ == "__main__":
port = int(sys.argv[1]) if len(sys.argv) > 1 else 8000
ThreadingHTTPServer(("127.0.0.1", port), Shop).serve_forever()
Start it in a second terminal and leave it running. It answers on http://127.0.0.1:8000, and we ran it on another port of our server.
python shop.py
Step 2: create the project
scrapy startproject shop_scraper
cd shop_scraper
The new settings.py is polite on purpose. It sets ROBOTSTXT_OBEY = True, CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1. Scrapy's documentation explains the first one. "While the default value is False for historical reasons, this option is enabled by default in settings.py file generated by scrapy startproject command". A spider run outside a project gets none of these, as our open-source scrapers test showed.
Step 3: write the spider
Save this as shop_scraper/spiders/products.py.
import scrapy
class ProductsSpider(scrapy.Spider):
name = "products"
async def start(self):
shop = self.settings.get("SHOP", "http://127.0.0.1:8000")
yield scrapy.Request(shop + "/", callback=self.parse)
def parse(self, response):
yield from response.follow_all(css="a.product", callback=self.parse_product)
yield from response.follow_all(css="a.next", callback=self.parse)
def parse_product(self, response):
yield {
"name": response.css("h1::text").get(),
"price": response.css(".price::text").get(default=""),
"sku": response.css(".sku::text").get(),
"url": response.url,
}
start() yields the first request. parse() follows every product link and the Next link, and parse_product() turns a product page into an item. The shop address comes from a setting called SHOP, so you can point the spider elsewhere with -s SHOP=....
Many tutorials still write def start_requests(self) instead of async def start(self). That method no longer works. Scrapy 2.16.0's release notes say it "is removed and no longer called". We ran the same spider written with start_requests() on Scrapy 2.19.0. It logged "Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)" and finished without an error.
Step 4: clean the items in pipelines
A pipeline sees every item before it is written. These two drop rows that are not worth keeping. Save them in shop_scraper/pipelines.py.
from scrapy.exceptions import DropItem
class PricePipeline:
def process_item(self, item):
if not item["price"]:
raise DropItem(f"missing price: {item['url']}")
item["price"] = float(item["price"].lstrip("£"))
return item
class DuplicatesPipeline:
def __init__(self):
self.seen = set()
def process_item(self, item):
if item["sku"] in self.seen:
raise DropItem(f"duplicate {item['sku']}: {item['url']}")
self.seen.add(item["sku"])
return item
Then switch them on at the end of settings.py. Lower numbers run first.
ITEM_PIPELINES = {
"shop_scraper.pipelines.PricePipeline": 300,
"shop_scraper.pipelines.DuplicatesPipeline": 400,
}
Scrapy's documentation writes process_item(self, item) without a spider argument. Scrapy 2.14.0 deprecated that argument. With process_item(self, item, spider), our run logged a ScrapyDeprecationWarning, saying the argument "will not be passed in future Scrapy versions".
Step 5: run it
scrapy crawl products -O products.jsonl
-O writes the items to a file and overwrites an old one. These lines come from the statistics Scrapy printed at the end of our run.
'httperror/response_ignored_count': 1,
'httperror/response_ignored_status_count/500': 1,
'item_dropped_count': 2,
'item_dropped_reasons_count/DropItem': 2,
'item_scraped_count': 98,
'items_per_minute': 43.23529411764706,
'log_count/DEBUG': 209,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'log_count/WARNING': 2,
'memusage/max': 81563648,
'memusage/startup': 80121856,
'request_depth_max': 5,
'response_received_count': 107,
'responses_per_minute': 47.20588235294118,
'retry/count': 4,
'retry/max_reached': 1,
'retry/reason_count/500 Internal Server Error': 4,
The shop received 111 requests: robots.txt, five list pages, 101 product addresses and five retries. The two pipelines dropped two items, the product without a price and the sale duplicate. Scrapy's duplicate filter had let the sale link through, because ?ref=sale makes it a new URL. The pipeline caught it by its product code.
The flaky page worked on its third try. The broken page failed three times, and Scrapy logged "Gave up retrying". That is the retry middleware at work. Its RETRY_TIMES setting is the "Maximum number of times to retry, in addition to the first download", 2 by default. It retries the codes 500, 502, 503, 504, 522, 524, 408 and 429.
Step 6: speed, politeness and AutoThrottle
The whole crawl took 136.7 seconds, because the project sends one request at a time and waits about a second. We ran the same crawl with other settings.
With AutoThrottle on, the crawl took 139.0 seconds, as long as before. AutoThrottle starts at a delay of 5 seconds and then aims at the server's latency. Its documentation also says the download delay "can’t become less than DOWNLOAD_DELAY". The project keeps that at 1 second, so the crawl could not get faster than the template.
Eight requests at once without a delay finished in 4.1 seconds, 33 times faster.
scrapy crawl products -O products.jsonl -s CONCURRENT_REQUESTS_PER_DOMAIN=8 -s DOWNLOAD_DELAY=0
That pace is fine for a shop on your own machine. On someone else's site, keep the polite defaults unless the site allows more.
Step 7: cache pages while you build, without caching errors
While you work on a spider, Scrapy's HTTP cache saves every answer, so a second run does not ask the site again. Our second cached run sent 0 requests to the shop and finished in 0.3 seconds.
The cache has a trap. By default it also stores error answers. Its HTTPCACHE_IGNORE_HTTP_CODES setting, the codes it does not cache, defaults to an empty list. So in our first cached run, the flaky page's 500 went into the cache, and each retry got that same 500 back from it. Scrapy gave up on the page, and the run wrote 97 items instead of 98. Every later run from the cache lost the same product.
Leave the error codes out of the cache, and the trap is gone.
scrapy crawl products -O products.jsonl -s HTTPCACHE_ENABLED=True -s HTTPCACHE_IGNORE_HTTP_CODES=500,502,503,504,522,524,408,429
With that setting, the first run wrote all 98 items. The second run took 106 answers from the cache, and only the broken page went back to the shop.
Step 8: send the crawl through a proxy
Scrapy's proxy middleware follows the same rule as Python's urllib.request. In its documentation's words, "it obeys the following environment variables": http_proxy, https_proxy and no_proxy. So the simplest way is one variable in front of the command.
http_proxy=http://127.0.0.1:3128 scrapy crawl products -O products.jsonl
We pointed it at a test proxy of our own on another port. The proxy logged all 111 requests of the crawl, and the run wrote the same 98 items. For a proxy per request, the documentation adds "You can also set the meta key proxy per-request". In code, that is scrapy.Request(url, meta={"proxy": "http://user:password@host:port"}).
Our guide to proxies with Scrapy covers logins, rotation and HTTPS in more depth.
What this page could not check
- A practice shop of our own, with 100 products and 0.1 seconds of latency. Real sites are slower and larger.
- One run per setting, on one server. The times will differ elsewhere.
- Scrapy 2.19.0 only. Older versions still call
start_requests(). - The proxy was a test proxy on our own server, with plain HTTP to the shop.
- We will check this page again by 12 January 2027.
Sources
- Scrapy documentation, spiders and
start(): docs.scrapy.org. - Scrapy documentation, item pipelines: docs.scrapy.org.
- Scrapy documentation, downloader middleware (retries, proxies, the HTTP cache): docs.scrapy.org.
- Scrapy documentation, AutoThrottle: docs.scrapy.org.
- Scrapy documentation, settings and the command line tool: docs.scrapy.org.
- Scrapy release notes (2.13.0, 2.14.0, 2.16.0 and 2.19.0): docs.scrapy.org.
- Scrapy 2.19.0 on PyPI: pypi.org.
- Our own tests on our server, 12 October 2026: run_scrapy.py at 09:08 UTC and run_old_api.py at 09:19 UTC, on a practice shop of our own.
