# Crawl error

A crawl error is a failure when a crawler tries to fetch a URL: a server error, a timeout, a DNS failure, a refused connection, a robots.txt that could not be fetched, or a redirect that loops. It is a fetching problem, not a content problem, and the site-level ones matter most.

Updated 2026-09-15 · Source: https://porteur.ai/glossary/crawl-error

## The kinds, and which ones are urgent

| Error | What it means | Urgency |
| --- | --- | --- |
| robots.txt unreachable | Google cannot fetch the file, so it slows or stops crawling the whole site | Highest |
| DNS failure | The host does not resolve; nothing can be fetched | Highest |
| Server error 5xx | The server answered with a failure | High if it repeats or affects many URLs |
| Timeout or refused connection | The server took too long or rejected the request | High if it repeats |
| Redirect error | A loop, a chain that is too long, or an empty target | Medium |
| 404 on a deleted page | The page is gone, which is expected | None |

> A 404 is not a crawl error in the sense that matters. Every site that has ever deleted a page has 404s, and Google expects them.

## Where they show

- The Page indexing report in Search Console lists per-URL failures under Not indexed, with reasons such as Server error (5xx) and Redirect error.
- The Crawl stats report, under Settings, shows host status: robots.txt fetch, DNS resolution and server connectivity, plus responses by status code over 90 days.
- The URL Inspection tool shows what happened on the last crawl of one URL, and Test live URL tries again now.
- Your own server or CDN logs show every fetch, including the ones Search Console samples.

## How to work through them

1. **Check host status first** Open Crawl stats. If robots.txt, DNS or connectivity is red, fix that before looking at any URL.
2. **Look for a shape** Errors on one template, or on one day, point at a deploy or an incident. Scattered single errors are usually transient.
3. **Reproduce the fetch** Request the URL from outside your network and read the status. A page that works in your browser may fail for a crawler behind a firewall rule or a bot filter.
4. **Fix the cause, not the symptom** A 5xx under load is a capacity problem, a 429 is a rate limit hitting the crawler, and a redirect error is a chain to flatten.
5. **Validate and watch** Use Validate fix in the report and check the Crawl stats trend over the following weeks.

## Questions

### What is a crawl error in Search Console?

A failure to fetch a URL: a server error, a timeout, a DNS or connection failure, an unreachable robots.txt, or a redirect that loops. Content problems such as thin pages are reported separately.

### Are 404 errors crawl errors?

Not in the sense that needs fixing. A 404 for a page you deleted is correct. It matters only when internal links or the sitemap still point at it, or when the page had traffic and links.

### Why does an unreachable robots.txt matter so much?

When Google cannot fetch robots.txt it does not know what it may crawl, so it slows or stops crawling the site. It is the one error that affects every URL at once.

### Do crawl errors hurt rankings?

Isolated ones do not. Repeated 5xx responses, timeouts or an unreachable robots.txt reduce how much Google crawls, so new and updated pages take longer to be seen.

### How do I tell a crawl error from a bot block?

Fetch the URL from outside your network with a crawler user agent. If it answers for a browser and fails for the crawler, a firewall, a CDN rule or a bot filter is the cause.

## Read next

- [Index coverage](https://porteur.ai/glossary/index-coverage): Index coverage is the old name of the Search Console report now called Page indexing. What it lists, which reasons to act on, and how to read the trend.
- [The Crawl stats report: how much Google fetches and where it struggles](https://porteur.ai/guides/search-console-crawl-stats-report): Find Crawl stats in Search Console Settings. Read the four charts, host status and breakdowns. Spot 5xx spikes and wasted crawls. Know what is normal.
- [Crawling](https://porteur.ai/glossary/crawling): Crawling is how Googlebot discovers and fetches your URLs. See what it fetched, fix slow or blocked areas, and know when crawl budget matters.
- [Redirect chain](https://porteur.ai/glossary/redirect-chain): A redirect chain is a URL that redirects to a URL that redirects again. What each hop costs, how many Google follows, and how to flatten one.
- [HTTP status codes for SEO: what each one tells Google](https://porteur.ai/guides/http-status-codes-for-seo): See how 2xx, 3xx, 4xx and 5xx codes change crawling and indexing, when to use 301 vs 302, handle soft 404s, 503 for maintenance, and how to check URLs.
- [robots.txt](https://porteur.ai/glossary/robots-txt): robots.txt tells crawlers which URLs they may fetch. See what it does not do, how to test it, what to put in it, and how to handle AI bots.
- [How to use the URL Inspection tool in Search Console](https://porteur.ai/guides/url-inspection-tool): Read each panel, run Test live URL, and know when to request indexing. Fix new pages, dropped pages, and canonicals Google ignores.
- [Soft 404](https://porteur.ai/glossary/soft-404): A soft 404 answers with a success status while being an error or an empty page. How Google detects it, where it shows, and the fix for each cause.

The free check fetches your pages the way a crawler does and names what fails, in about thirty seconds. Free check: https://porteur.ai/