While it may come as a surprise to some, Google doesn’t always crawl every page of a website instantly. Learn how to optimize this to your advantage.
If you include new content or an optimized landing page, Google may not be index it right away. That can be frustrating on many levels as you have spent time and energy improving your website. Luckily, there are several ways you can plan for your crawl budget to get the biggest impact on your site.
Simply put, crawl budget is a certain number of pages Google crawls a website in a given time frame. If you have more pages than your website’s crawl budget, there will pages on your website that aren’t indexed. If Google doesn’t index a page, it won’t rank in search results.
There are a few instances where you need to pay attention to your crawl budget:
There are several best practices to follow to improve the number of pages Google crawls.
Reducing the number of errors you have on your website is a great first step. Use Google Search Console or your website’s server logs to view errors that have not returned with a 200 (meaning ‘OK’) and 301 (redirects) and fix them. The less number of errors Google sees, the more pages it crawls.
When you have several 301 redirects, Google will see each individual URL and include them all in the list to crawl. In addition, if there are redirect chains on your website (for example if you direct a non-www to a www, then an http to an https website) it takes Google even longer to crawl your pages. Reducing the number of redirects will help Google put a greater emphasis on crawling pages with fresh content.
You can also control sections of your website that don’t need to be crawled. By utilizing the robots.txt rule on your site, you are able to block pages that will be crawled by Google. Simply marking a page as ‘nofollow’ on your pages is not a guarantee that the page will not be crawled by Google.
Another best practice is to get more links on your website. Google has a tendency to prioritize pages that have lots of internal and external links pointing to them. Internal linking is also key as it sends Google to different pages that you want indexed on your site. This option is more difficult, as it is a slower and more manual method of increasing your crawl budget.
If you are keeping up with your website’s maintenance well and the best practices listed above, then you are well on your way to maximizing the results of Google’s crawl budget. If you have any questions about the information here or need help on where to being, reach out to our team of SEO experts today. We can help you focus your efforts on the aspects of your website that matter most to you and your visitors.
The original version of this page was published at: https://www.geonetric.com/seo/crawl-budget/
At Geonetric, we create compelling digital experiences for healthcare organizations that make lasting patient connections and drive organizational growth. We solve your challenges with expert UX, ... Read more
