The Crawl stats report: how much Google fetches and where it struggles

Open Google Search Console. Pick a root-level property. In the left menu, click Settings, then Crawl stats.

The report covers the last 90 days. It shows total crawl requests, total download size and average response time. It also shows host status and request breakdowns.

• Total crawl requests: volume over time. A steady line is normal on a stable site.
• Total download size: bytes fetched. A jump can mean heavier pages or many large assets.
• Average response time: how fast your server answered Google. A climb means your server, code or network slowed.
• Host status: recent checks for robots.txt fetch, DNS resolution and server connectivity.

Scan the 90‑day trends. Mark spikes and drops. Then open each breakdown to find what changed: response codes, file types, purpose and Googlebot type.

Host status checks three basics: Google could fetch robots.txt, resolve DNS and connect to your server. If one shows red, Google had recent failures.

Fix the red check first. Google slows crawling when it cannot fetch robots.txt or when your host fails to answer. When green again, crawling ramps back by itself.

Open each breakdown. You get requests by response code, by file type, by purpose and by Googlebot type. Compare the timelines to your spikes or drops.

Click a response code to see example URLs. Copy a few and test them in your browser and server logs. Fix patterns, not single URLs.

If your site has fewer than tens of thousands of URLs, crawl budget is rarely the problem. Google can fetch and refresh that scale without special help.

• Steady crawl requests with small weekly swings.
• Average response time stable within a narrow band.
• Most requests on HTML and core assets. Few 404s.
• Discovery bumps when you add a section, for example /guides/.

A healthy report for yourproduct.com shows most crawl time on real pages like /, /pricing and /guides/getting-started. Old redirects and querystring pages are low or gone.

• 5xx spikes: server errors waste crawl and stall indexing.
• Most requests on parameters: Google exploring endless URLs like /list?sort= and /list?page=.
• Crawl time on old redirects: long chains or loops after a migration.
• Average response time climbing: slower app, busy database or network issues.
• Total download size surging: oversized images or assets from a release.

1. 1

   Confirm the fault window

   Match the spike dates to deploys, traffic surges or hosting incidents. Check your uptime monitor and server logs.

2. 2

   Stabilise the host

   Add capacity or roll back. Remove rate limits that block Google. Ensure your load balancer and WAF allow Googlebot.

3. 3

   Serve lighter pages while hot

   Delay heavy jobs in requests. Cache HTML for anonymous users. Use a CDN for static assets.

4. 4

   Trim expensive endpoints

   Audit routes with long times to first byte. Paginate costly listings. Reduce third‑party calls in critical paths.

5. 5

   Retest in Crawl stats

   When average response time returns to normal and 5xx fall, Google will crawl more again.

If a page is gone, return 410 or 404, not a 500. Keep your error budget for real faults, not content removals. A clean 404 does not harm crawl health.

Parameter sprawl is common. You see many requests for /products?ref=, /blog?utm_source= and /search?page=. These add near‑duplicate pages and infinite spaces.

• Canonical: point parameter variants to the clean URL when the content is the same.
• Noindex: add noindex on thin results, for example filtered lists that no one should land on.
• Internal links: stop linking to tracked URLs. Strip UTM parameters on site links.
• Pagination: cap the highest page you link to. Use rel=next/prev patterns in your UI, not as signals to Google.
• Faceted navigation: only link combinations that have search demand and unique value.
• Robots.txt: disallow only when you are sure you do not need Google to crawl those patterns.

After a week, Crawl stats should show Discovery falling on parameter URLs and Refresh rising on your core pages like /collections/shoes and /blog/how-we-built-search.

Redirect chains and loops burn crawl. You see many 301 or 302 in the response breakdown. Typical after a rebrand or during a CMS change.

1. 1

   Map the chains

   Pick examples from the report. Use a redirect checker to trace hops from /old to /new.

2. 2

   Flatten to one hop

   Redirect old URLs directly to the final destination. Remove intermediate hops.

3. 3

   Fix internal links

   Update menus, footers, sitemaps and body links to the final URLs. Do not link to redirected URLs.

4. 4

   Expire stale routes

   If content is gone, return 410 or 404. Do not redirect everything to /.

5. 5

   Verify in Crawl stats

   Requests should shift from redirects to 200. Average response time often improves too.

Large images and heavy JS inflate download size. If those lines jump without a change you planned, a theme or plugin shipped more assets than before.

• Images: compress and resize. Use modern formats. Lazy‑load non‑critical images.
• JS and CSS: remove unused code. Split bundles. Serve with caching and compression.
• Page resources: avoid querystrings on static assets. Set long cache headers and consistent URLs.

Check Googlebot type. Smartphone should lead on most sites. If Desktop is higher, your mobile site may be thin or blocked. Fix parity between mobile and desktop content and internal links.

On purpose, Discovery rising after you ship new pages or submit a sitemap is fine. Discovery rising on junk patterns is not. Refresh should dominate on mature sections like /pricing.

Give Google clean paths to what matters. Sitemaps and internal links are the levers you control. The Crawl stats report tells you if Google is following them.

• XML sitemaps: submit a sitemap or index in the Sitemaps report. Google reads lastmod when it is consistently accurate.
• Scope: a sitemap may hold up to 50,000 URLs or 50 MB uncompressed and can only list URLs on its own host or an approved cross‑host setup.
• Coverage: include canonical, indexable URLs only. Exclude redirects, noindex and parameters.
• Internal links: link to pages you want crawled. Avoid orphan pages. Update links after a move.

A fixed site shows Crawl stats requests clustering on the URLs you linked in your navigation and sitemap, not on /?page= or /tag/uncategorised/?sort=oldest.

Do not chase a flat line. Crawl demand fluctuates. Treat Crawl stats as an operational signal. Only act when you see clear waste or failure patterns.

Cross‑check with the Performance report and the Core Web Vitals report. If clicks and vitals are steady, a short‑term crawl spike may be nothing. Keep notes near deploys and incidents.