Googlebot

Googlebot fetches pages from the web so Google can index and rank them. It follows links and revisits to keep the index fresh.

There are two main versions: Googlebot Smartphone and Googlebot Desktop. Since mobile-first indexing, the smartphone version reads most sites first.

Google also runs specialised crawlers such as Googlebot-Image, Googlebot-Video, AdsBot, and Google-Extended which you can use to set AI training controls.

Googlebot obeys robots.txt at the root of your domain. It identifies itself in the user agent when it requests a URL.

It renders JavaScript with an evergreen Chromium. It needs access to your JS and CSS to see your layout, links, and content that client code inserts.

• If robots.txt blocks a path, Googlebot will not crawl those URLs.
• If critical resources are blocked, the rendered page can be incomplete.
• If links only appear after a user action Googlebot cannot perform, it may never find them.

User agents can be faked. Verify Googlebot by checking its IP, not by reading the user agent string alone.

1. 1

   Collect the requester IP

   From your server logs or firewall, copy the IP address that hit /pricing or another page.

2. 2

   Do a reverse DNS lookup

   Confirm the IP resolves to a googlebot host. Then do a forward lookup on that host and check it points back to the same IP.

3. 3

   Alternatively, check published ranges

   Compare the IP against Google’s published IP ranges for crawlers if you manage allow lists.

4. 4

   Record the user agent

   Note if it is Googlebot Smartphone or Desktop. Use it to spot patterns in Crawl stats later.

Use Search Console’s Crawl stats report to see what Googlebot fetched, response codes, and whether requests skew to smartphone or desktop. This shows crawl health over 28 days.

• Look for spikes in 5xx responses after a deploy.
• Check if most crawl is on the canonical host, for example yourproduct.com not www.yourproduct.com if you prefer non-www.
• Scan fetches by file type to confirm Googlebot can fetch JS and CSS.

• Keep robots.txt simple: allow crawling of pages, JS and CSS. Only disallow areas you do not want in Search.
• Serve the same primary content to users and Googlebot. Do not gate content behind interactions Googlebot cannot do.
• Make internal links HTML and present on load. For example, link /guides/getting-started from /pricing in the markup.
• Avoid endless URL spaces. Limit faceted combinations and add nofollow to links that create infinite variants if you cannot prune them.
• Ensure pages render quickly and fully. Avoid client code that fetches the main content late or only after a click.
• If you do not want content used for AI training, review Google-Extended controls alongside your robots rules.

A fixed page loads with all key text in the DOM on first render, links are crawlable, and robots.txt does not block assets.

• When testing, do not assume “view as Googlebot” with a spoofed UA reflects reality. Verify with IP and check the rendered HTML.
• After major changes, recheck Crawl stats and fetch a sample URL you changed to confirm a 200 and full render.