Technical SEO

Crawl budget

Crawl budget is the number of pages a search engine is willing and able to crawl on a site in a given period, shaped by server capacity and how much demand there is for its pages.

Google describes crawl budget as a mix of crawl capacity (how much crawling your server can handle) and crawl demand (how much Google wants to crawl your pages, based on popularity and freshness).

In more detail

For most sites with a few thousand pages, crawl budget is not a concern. It matters for large sites, such as stores with many filter combinations, marketplaces and publishers, where wasted crawling on duplicate, thin or broken URLs can delay discovery of important pages. Fixes include blocking useless parameter URLs, removing redirect chains, fixing errors, improving server speed and keeping sitemaps clean.

Why it matters

On large sites, managing crawl budget decides how quickly new and updated pages appear in search, which directly affects revenue for stores and news publishers.

Example

A marketplace with 2 million listings finds Googlebot spends 60 percent of its visits on sorted and filtered URLs. Blocking those parameters in robots.txt shifts crawling to new listings, which now get indexed in days instead of weeks.

FAQ

Crawl budget: quick answers

Does my small website need to worry about crawl budget?

Usually not. Google can crawl a few thousand well-linked pages easily.

How do I see crawl activity?

Search Console's Crawl Stats report shows requests per day, response codes and file types.

Free audit

Find out where you stand in search and AI answers.

Get a free audit of your SEO, AI visibility and email setup. We send a short, specific report with the three changes that would move the needle most.