What is a Crawl Budget, and How to Best Utilize it

While it may come as a surprise to some, Google doesn’t always crawl every page of a website instantly. Learn how to optimize this to your advantage.

How Google’s Crawl Budget Impacts SEO on Your Website

If you include new content or an optimized landing page, Google may not be index it right away. That can be frustrating on many levels as you have spent time and energy improving your website. Luckily, there are several ways you can plan for your crawl budget to get the biggest impact on your site.

What is a Crawl Budget?

Simply put, crawl budget is a certain number of pages Google crawls a website in a given time frame. If you have more pages than your website’s crawl budget, there will pages on your website that aren’t indexed. If Google doesn’t index a page, it won’t rank in search results.

Do I Need to Worry about Crawl Budget?

There are a few instances where you need to pay attention to your crawl budget:

• The size of the website: Large website with several thousand pages can sometimes wait several days for Google to notices any changes or updates.
• Errors on the pages: If your website slows down or responds to Google with several errors, it lowers the number of pages that are crawled to avoid overloading your server.
• Be direct: When several redirects go from one URL to another, it takes Google longer to crawl your site and can limit your crawl budget.

Industry Best Practices for Crawl Budget

There are several best practices to follow to improve the number of pages Google crawls.

Address Errors

Reducing the number of errors you have on your website is a great first step. Use Google Search Console or your website’s server logs to view errors that have not returned with a 200 (meaning ‘OK’) and 301 (redirects) and fix them. The less number of errors Google sees, the more pages it crawls.

Review Redirects

When you have several 301 redirects, Google will see each individual URL and include them all in the list to crawl. In addition, if there are redirect chains on your website (for example if you direct a non-www to a www, then an http to an https website) it takes Google even longer to crawl your pages. Reducing the number of redirects will help Google put a greater emphasis on crawling pages with fresh content.

Not Crawling Sections

You can also control sections of your website that don’t need to be crawled. By utilizing the robots.txt rule on your site, you are able to block pages that will be crawled by Google. Simply marking a page as ‘nofollow’ on your pages is not a guarantee that the page will not be crawled by Google.

Link Strategy

Another best practice is to get more links on your website. Google has a tendency to prioritize pages that have lots of internal and external links pointing to them. Internal linking is also key as it sends Google to different pages that you want indexed on your site. This option is more difficult, as it is a slower and more manual method of increasing your crawl budget.

Keep Everything Up To Date

If you are keeping up with your website’s maintenance well and the best practices listed above, then you are well on your way to maximizing the results of Google’s crawl budget. If you have any questions about the information here or need help on where to being, reach out to our team of SEO experts today. We can help you focus your efforts on the aspects of your website that matter most to you and your visitors.

NICK ERHART
Digital Marketing Strategist

The original version of this page was published at:  https://www.geonetric.com/seo/crawl-budget/