What is Crawl Budget?
TL;DR
Crawl budget is the number of pages a crawler will visit on your site within a given time period. Wasting it on low-value pages means important pages get crawled less often.
Last updated: 2026-03-09
Definition#
Crawl budget is the number of pages a web crawler (from Google, Bing, or an AI platform) will visit on your site within a given time period. Every crawler has limited resources, so it allocates a budget to each site based on the site's size, authority, and server capacity.
If your crawl budget is 1,000 pages per day but your site has 10,000 pages, it takes ten days for the crawler to see everything. Pages that return errors, redirect chains, or duplicate content waste crawl budget that could be spent on your important pages.
Crawl budget matters most for large sites (1,000+ pages). Small sites are usually crawled completely without issues. For larger sites, efficient crawl budget usage means your best content gets discovered and updated faster.
Why It Matters for AI Readiness#
AI crawlers have crawl budgets too. Wasted crawl budget means your most important pages may not be crawled, indexed, or cited. The Crawl Health factor in your AgentReady™ score evaluates how efficiently your site uses its crawl budget.
See How to Fix Crawl Health Issues for practical improvements.
Related Concepts#
Crawl budget is affected by redirect chains, canonical tags, and sitemaps. It is measured in the Crawl Health factor.
Related Pages
Was this page helpful?