This overview shows how your crawl budget (also called crawl volume) has changed on a monthly base. We chose the duration of an month on purpose to make it easier comparable and to identify trends at a glance. Extreme increase and decrease indicates crawling problems or Google updates. Thanks to log file storing up to 5 years you are able to identify trends, long-term developments and optimization successes.

We purposely define crawl budget different than Google: for us the term crawl budget stands for the total number of Googlebot requests during one month.

Statistics show you further current crawl trends compared to the previous day, week and month. This is the place where all hard facts are collected and compared:

• the total amount of Googlebot request
• amount of crawled unique URLs
• amount of crawled “Business Relevant URLs”
• amount of crawled errors (status codes 4xx and 5xx)
• average download speed of an URL
• average download size of an URL in kilobytes

Finally you are able to check wether your robots.txt and XML-sitemap(s) has been crawled.

What are Business Relevant Pages?

We define Business Relevant Pages as all pages of a website, that have importance from a business angle and are meant to fulfill an economic purpose (for instance news articles, category pages, brand pages, detailed product pages and many more). To fulfill its economic purpose, those pages need to produce organic search engine traffic within google search results and last but not least satisfy search intentions of searchers to the maximal extent. Those are the reasons why only Business Relevant Pages should be crawled by bots, every other crawl of non Business Relevant Pages would be a total waste of already limited and expensive Googlebot requests.

This is how you define your Business Relevant Pages with crawlOPTIMIZER

Note: defining Business Relevant Pages is a one time event and should be done at the start.

If your website has one or more XML-sitemap(s), you can add it or them via SETTINGS to crawlOPTIMIZER and all URL entries out of your sitemaps are automatically marked as business relevant in the system.

Go to crawlOPTIMIZER’s SETTINGS to add URLs and or parameters manually in a comments section as Business Relevant Pages (for instance URLs to resources, that Googlebot needs for rendering or old URLs that contain backlinks. Another example are detailed product pages with special tracking parameters that are designed for Google Shopping – these URLs are business relevant too, because they are indexed by the general Googlebot). You can mark URLs as business relevant through REGEX commands too if you go to SETTINGS.

Once defined, analyzing starts

1. Business Relevant Pages (BRP) (see point 1) shows you yesterday’s crawl. Also it shows you the percentage of Business Relevant Pages that Googlebot was able to crawl. With a single click on the pie chart you are able to see your raw data in the log file explorer.
2. Trend (see point 2) shows you the development of the last 30 days (default) on a daily basis. At a glance you are able to check if your optimization efforts have already produced results or if there are new crawling issues. If you click on a bar in the graph you will automatically see pre-filtered raw data in the log file explorer. So you are able to see which for instance NON-BRP-URLs were crawled the day before with one click.
3. Current Crawl (see point 3) shows you the percentage of Business Relevant Pages that have been crawled since the first of the month. We are always looking at the whole month to guarantee comparability. If you want to see all URLs just click on the button “Show Pages”.
4. If one month has passed you are able to see the values of the previous month under Crawl Last Month (see point 4). View the list in Logfile Explorer with help of the buttons “Show Pages”.
5. Trend Analysis (see point 5) gives you an image of previous collected information on a monthly basis so you are able to see long-term development.
6. Full Crawls (see point 6) gives you the option to self-define at which percentage a “full crawl” is achieved. 100% are difficult to reach if you have a big website or online-shop because of various page fluctuations. Because of that you should choose a value that represents a “full craw” realistically (for instance 90%). As soon as the value has bin added under SETTINGS, crawlOPTIMIZER starts counting the days until the value (for instance 90%) is reached. As soon as your value has been reached, the days it took are saved and shown in the graph. So you will have insight how long it takes for Google to do a full crawl. Your goal should be to reduce the amount of days with help of various optimization measures.

If your website contains XML sitemap(s), you should have deposit them under SETTINGS by now (see point 3 above).

If necessary you may also have deposited Business Relevant Pages manually.

If yes, perfect! Because all of these URLs will be crawled by us once a week. Yes, you have heard right. They will be checked by us regularly.

For this case we especially developed our crawler, that crawls your XML sitemap(s) one a week nightly. Our crawler checks every URL entry inside your sitemap and also your manually deposited URLs. We always check the following values:

• Are URLs included that are blocked because of your robots.txt?
• Are URLs included that shouldn’t be, for instance 404 URLs or redirecting URLs?
• Are URLs included that are marked as NOINDEX in the meta robots?
• Are URLs included that are canonizing to other pages via canonical tags?

If yes, please check these pages, correct or delete them from your sitemaps, because such incorrect URLs waste your precious crawl budget.

You can see incorrect URLs by clicking one of the buttons labeled “Show URLs” whenever you want.

In contrast to the previous overview here crawled URLs are not listed, but the most crawled resources (js/css/etc.). Also with the last current Status Code.

By clicking on any URL you will be also transferred to the Logfile Explorer (RAW DATA), where all single requests of this URL (resource) are listed.

Crawled parameters

This graph shows all crawled parameters, crawled in a defined period of time.

Which parameters do significantly influence the content of your website and which can be excluded from crawling? Analyze found parameters with the greatest concern in detail and monitor the bot with a clean website architecture and/or cautiously by using Robots.txt.

There are many types of Googlebots. Here you can see all Googlebots that have crawled your website and the frequency. We reasonably clustered different Googlebots for you, Googlebot Desktop, Googlebot Smartphine and Googlebot others to be specific.