- Category
- Technical SEO
- Level
- Advanced
- Depends on
- Server capacity and how much the engine wants to crawl
- Matters most for
- Very large or fast-changing sites
- Check it free
- Sitemap Generator
What is crawl budget?
Search engines limit how much they crawl each site. Google describes two parts: the crawl capacity limit (how much crawling your server can handle without slowing down) and crawl demand (how much Google wants to crawl, based on popularity and freshness).
Google's documentation says crawl budget is mainly a concern for very large sites, around a million pages or more, or sites with tens of thousands of pages that change daily. Most small and medium sites don't need to worry about it.
What wastes crawl budget
- Endless URL variations from filters, sorting and tracking parameters.
- Redirect chains and long lists of redirected URLs.
- Duplicate pages and soft 404s.
- Slow server responses.
Best practices
- Keep one URL per piece of content, using canonicals where needed.
- Link internally to final URLs, not redirected ones.
- Keep sitemaps to indexable, canonical pages.
- Improve server response times.