Pagination sounds like a solved problem: split a long list across pages and tell Google which one to rank. Then you look at a log file for a large publisher and find that 431,000 of the 485,000 URLs Googlebot requested in a month were paginated list pages with no content of their own. Eighty-nine percent of the crawl, spent reading tables of contents instead of articles. On betting and sports news sites, which publish dozens of pieces a day, this is not an edge case. It is the default state unless someone deliberately prevents it. This article covers how we diagnosed it, why the textbook fixes were not enough, and the three approaches that actually worked.
In this article
How pagination became a crawl budget problem
For years the accepted answer was rel="next" and rel="prev": attributes that told Google to treat a paginated series as one unit and rank the first page. Google retired support in March 2019 with very little fanfare. The practical consequence was that page 2, page 3 and page 47 became ordinary pages in Google’s eyes: crawled, potentially indexed, potentially ranked, each competing with the others and with the articles they list.
For a news or tips site producing hundreds of articles a week, the series grow daily and never shrink. Author archives, tag archives, league archives, each with its own pagination, multiplied by every filter or sort parameter that creates a new series. The number of paginated URLs grows faster than the number of articles.
The standard recommendations that followed the 2019 change were canonicalising every page in a series to page 1, or setting paginated pages to noindex, follow. Both are defensible. Both have real drawbacks at scale:
- Canonical to page 1 tells Google that page 14 is a duplicate of page 1, which is not true, and Google is free to ignore a canonical it does not believe. When it does believe it, the articles linked only from page 14 lose their crawl path.
- Noindex, follow keeps the pages out of the index but Googlebot still has to visit them to read the directive, so it does nothing for crawl budget. There is also long-running debate about whether links on long-term noindexed pages keep passing value.
Diagnosis: read the logs, not the crawler
A site crawler tells you what URLs exist. A log file tells you where Googlebot actually spends its time, which is the question that matters. We used Screaming Frog’s Log File Analyser on roughly thirty days of server logs from one publisher site. Paginated URLs were identifiable by a parameter, which made the split clean.
| Googlebot requests, ~30 days | URLs | Share |
|---|---|---|
| Total crawled | 485,000 | 100% |
| Paginated list pages | 431,000 | 89% |
| Everything else, including every article | 54,000 | 11% |
When we surfaced that number, it reframed the conversation. This was not a pagination problem. It was a crawl budget crisis hiding inside a pagination problem.
That reframing matters for how you get the fix prioritised. “Our pagination is suboptimal” is a ticket that waits a year. “Googlebot spends nine-tenths of its time on this site not reading our articles” is a ticket that gets a product owner.
Three approaches that worked
1. Intentional de-optimisation
Keep every paginated page indexable with a self-referencing canonical, but make everything except page 1 deliberately unappealing to rank. Remove the H1 from pages 2 onward. Differentiate the meta title per page so they are not duplicates. Strip any intro copy or content module that is repeated from page 1. Google can crawl them and link equity flows through them, but the signal about which page to rank is unambiguous.
This worked best on sites with evergreen archives, where the articles linked from deep pages keep earning traffic for years and you do not want to cut their crawl path. It does not, on its own, reduce crawl volume much. It reduces index bloat and cannibalisation, which for that site was the more damaging half of the problem.
2. Archive with a hard cut-off
Set a maximum pagination depth, twenty pages say, and return a 410 or redirect for anything beyond it. Old content does not disappear. It lives in a dedicated archive section organised by month or competition, which is a better way to browse it anyway. Pair that with the unavailable_after robots directive on articles past a certain age and you have a layered crawl-management system that maintains itself.
We tested unavailable_after separately on a high-volume news site: across 1,122 URLs, every page was de-indexed after its expiry date and crawl frequency dropped measurably afterwards. The one methodological wrinkle was that the expiry date coincided with the article rotating off the main pagination, so we could not fully separate the two effects. The tag works; the honest statement is that we do not know how much of the crawl reduction it accounted for on its own.
3. Infinite scroll, implemented for crawlers
One site had an author-page problem: a few prolific writers had thousands of paginated archive pages each. We replaced classic pagination with infinite scroll, but implemented properly. Component pages exist in the background at real URLs Googlebot can request. The scroll loads them progressively. rel="next" and rel="prev" were re-added as hints, because even though Google no longer uses them, other crawlers do and they cost nothing. The user experience improved and each author’s archive collapsed from thousands of crawlable URLs to a single entry point with component pages behind it.
“Infinite scroll is bad for SEO” is a rule of thumb that describes the naive implementation, where content is only reachable through client-side events. With addressable component pages it is a pagination scheme like any other. The distinction is whether every piece of content has a URL a crawler can request without executing a scroll handler.
The iGaming-specific angle: users hate it too
Here is what purely technical discussions of pagination underweight. On a betting tips site, the tip a user wants is the one for this evening’s match. Within hours of publishing, on a site with several leagues and multiple angles per fixture, that tip can be three or four pages deep. Nobody, on mobile, taps through to page 14. The crawler problem and the user problem are the same problem seen from two sides.
The most durable fix we shipped on one sportsbook affiliate was a hybrid built around that observation:
- Two tabs on the listing page: active tips and past tips, driven by a timestamp against kick-off time.
- Unified pagination that collapses to nothing when there are no current tips, instead of paginating an empty state.
- Past tips organised by competition and month rather than by an endless reverse-chronological list.
- Everything still crawlable through addressable URLs; the total number of paginated pages fell by an order of magnitude.
Crawl volume on list pages dropped, the tips users actually wanted were one tap away, and the editorial team stopped fielding complaints that “the site is slow to show new tips” when the real problem was that new tips were on page 3. That is the version of technical SEO I find most satisfying: fixing the crawler problem made the product better for humans.
A checklist for high-volume publishers
- Pull thirty days of server logs before you touch anything. Measure the share of Googlebot requests going to list pages. If it is above about a third, you have a budget problem, not a tidiness problem.
- Count your pagination series, not your paginated pages. Every archive type and every parameter that creates a new series is a multiplier.
- Decide per series whether deep pages have value. Evergreen archives: de-optimise and keep. Time-sensitive feeds: cap depth and archive.
- Use
unavailable_afterfor content with a natural expiry, and decouple its timing from other structural changes so you can measure it. - Redesign the listing around what users are looking for today. If the answer is “the newest thing”, pagination is the wrong primary navigation.
- Re-read the logs a month later. The only proof that crawl budget moved is the crawl budget moving.