Pagination
How a site splits a long list across many pages or loads, and the part of a scrape most likely to silently miss data or trap a crawler.
Pagination is how a site serves a long list a piece at a time rather than all at once, and it is where scrapes quietly go wrong. Getting the first page is easy; getting all the pages, exactly once each and knowing when you have them all, is the actual job, and it is deceptively full of traps. Most incomplete datasets are not blocked collections, they are pagination handled a little bit wrong.
The mechanism varies and the right approach varies with it. Classic numbered pages or a next link are the friendly case: follow the link until it disappears. Offset-based paging, page 2, page 3, is simple but breaks when the list changes underneath you, silently skipping or duplicating items as things shift between requests. Cursor-based paging, where each response hands you a token for the next, is the robust modern form and the one that most rewards calling the data endpoint directly. And infinite scroll has no pages at all: content loads as you scroll, which usually means a background request you can find and call rather than a page you must render.
That last point is the highest-leverage move, and it connects to the rendering and API entries. Infinite scroll and dynamically loaded lists look like they demand a headless browser scrolling forever, but the scroll is almost always firing a paged data request underneath, and calling that request directly with the page or cursor parameter gets you the whole list in clean fragments, faster and cheaper than driving a browser. Finding the pagination in the network tab is frequently the moment a hard scrape becomes an easy one.
Two failure modes deserve naming because they are so common. The first is stopping too early, treating an empty page or a missing next link inconsistently and quietly ending with three quarters of the data, which is why validating the total count against what the site claims matters. The second is the crawler trap the crawler entry describes: faceted filters and calendars generate endless valid pages from finite content, so paginating without scope limits walks a crawler into an infinite space. Pagination is where coverage is won or lost, on both the too-little and the too-much side.
Frequently asked questions
How do I scrape all pages of a list, not just the first?
Match the method. Follow next links until they vanish for classic paging; increment offsets carefully for numbered pages, watching for items shifting between requests; follow the token each response gives you for cursor-based paging, which is the most reliable. For infinite scroll, find the background request firing as you scroll and call it directly with its page or cursor parameter.
How do I scrape infinite scroll pages?
Usually without scrolling at all. Infinite scroll fires a paged data request in the background as you reach the bottom, so open the network tab, find that request, and call it directly with its page or cursor parameter to pull the whole list in clean fragments. That is faster and cheaper than driving a headless browser to scroll endlessly, and less fragile.
Why is my scraper missing data across pages?
Most often pagination handled slightly wrong: stopping early on an inconsistent empty page or missing next link, or using offsets against a list that changes between requests so items are skipped or duplicated. Validate the total collected against the count the site reports, and prefer cursor-based paging or a direct data endpoint, which are far less prone to silent gaps.
Back to the full glossary.