Crawl error
A crawl error is a failure when a crawler tries to fetch a URL: a server error, a timeout, a DNS failure, a refused connection, a robots.txt that could not be fetched, or a redirect that loops. It is a fetching problem, not a content problem, and the site-level ones matter most.
By Théophile Louvart, founder of Porteur · Updated 15 September 2026 · Markdown
The kinds, and which ones are urgent
| Error | What it means | Urgency |
|---|---|---|
| robots.txt unreachable | Google cannot fetch the file, so it slows or stops crawling the whole site | Highest |
| DNS failure | The host does not resolve; nothing can be fetched | Highest |
| Server error 5xx | The server answered with a failure | High if it repeats or affects many URLs |
| Timeout or refused connection | The server took too long or rejected the request | High if it repeats |
| Redirect error | A loop, a chain that is too long, or an empty target | Medium |
| 404 on a deleted page | The page is gone, which is expected | None |
Where they show
- The Page indexing report in Search Console lists per-URL failures under Not indexed, with reasons such as Server error (5xx) and Redirect error.
- The Crawl stats report, under Settings, shows host status: robots.txt fetch, DNS resolution and server connectivity, plus responses by status code over 90 days.
- The URL Inspection tool shows what happened on the last crawl of one URL, and Test live URL tries again now.
- Your own server or CDN logs show every fetch, including the ones Search Console samples.
How to work through them
Check host status first
Open Crawl stats. If robots.txt, DNS or connectivity is red, fix that before looking at any URL.
Look for a shape
Errors on one template, or on one day, point at a deploy or an incident. Scattered single errors are usually transient.
Reproduce the fetch
Request the URL from outside your network and read the status. A page that works in your browser may fail for a crawler behind a firewall rule or a bot filter.
Fix the cause, not the symptom
A 5xx under load is a capacity problem, a 429 is a rate limit hitting the crawler, and a redirect error is a chain to flatten.
Validate and watch
Use Validate fix in the report and check the Crawl stats trend over the following weeks.
Questions
A failure to fetch a URL: a server error, a timeout, a DNS or connection failure, an unreachable robots.txt, or a redirect that loops. Content problems such as thin pages are reported separately.
Not in the sense that needs fixing. A 404 for a page you deleted is correct. It matters only when internal links or the sitemap still point at it, or when the page had traffic and links.
When Google cannot fetch robots.txt it does not know what it may crawl, so it slows or stops crawling the site. It is the one error that affects every URL at once.
Isolated ones do not. Repeated 5xx responses, timeouts or an unreachable robots.txt reduce how much Google crawls, so new and updated pages take longer to be seen.
Fetch the URL from outside your network with a crawler user agent. If it answers for a browser and fails for the crawler, a firewall, a CDN rule or a bot filter is the cause.
Check my site, free
The free check fetches your pages the way a crawler does and names what fails, in about thirty seconds.
- Free check, no card
- Read-only, your own accounts
- Readable by your agent