# Googlebot Googlebot is Google’s web crawler, the software that fetches your pages for the index. You need to know which version visits, what it can see, and how to confirm real requests. This keeps your pages crawlable, renderable and eligible to rank. Updated 2026-09-14 · Source: https://porteur.ai/glossary/googlebot ## What Googlebot is and its versions Googlebot fetches pages from the web so Google can index and rank them. It follows links and revisits to keep the index fresh. There are two main versions: Googlebot Smartphone and Googlebot Desktop. Since mobile-first indexing, the smartphone version reads most sites first. Google also runs specialised crawlers such as Googlebot-Image, Googlebot-Video, AdsBot, and Google-Extended which you can use to set AI training controls. ## What Googlebot obeys and how it renders Googlebot obeys robots.txt at the root of your domain. It identifies itself in the user agent when it requests a URL. It renders JavaScript with an evergreen Chromium. It needs access to your JS and CSS to see your layout, links, and content that client code inserts. - If robots.txt blocks a path, Googlebot will not crawl those URLs. - If critical resources are blocked, the rendered page can be incomplete. - If links only appear after a user action Googlebot cannot perform, it may never find them. ## How to verify real Googlebot traffic User agents can be faked. Verify Googlebot by checking its IP, not by reading the user agent string alone. 1. **Collect the requester IP** From your server logs or firewall, copy the IP address that hit /pricing or another page. 2. **Do a reverse DNS lookup** Confirm the IP resolves to a googlebot host. Then do a forward lookup on that host and check it points back to the same IP. 3. **Alternatively, check published ranges** Compare the IP against Google’s published IP ranges for crawlers if you manage allow lists. 4. **Record the user agent** Note if it is Googlebot Smartphone or Desktop. Use it to spot patterns in Crawl stats later. > Rule: verify Googlebot via reverse DNS of the requesting IP. Do not trust the user agent string. ## How to read Googlebot activity Use Search Console’s Crawl stats report to see what Googlebot fetched, response codes, and whether requests skew to smartphone or desktop. This shows crawl health over 28 days. - Look for spikes in 5xx responses after a deploy. - Check if most crawl is on the canonical host, for example yourproduct.com not www.yourproduct.com if you prefer non-www. - Scan fetches by file type to confirm Googlebot can fetch JS and CSS. ## What to do on a small site - Keep robots.txt simple: allow crawling of pages, JS and CSS. Only disallow areas you do not want in Search. - Serve the same primary content to users and Googlebot. Do not gate content behind interactions Googlebot cannot do. - Make internal links HTML and present on load. For example, link /guides/getting-started from /pricing in the markup. - Avoid endless URL spaces. Limit faceted combinations and add nofollow to links that create infinite variants if you cannot prune them. - Ensure pages render quickly and fully. Avoid client code that fetches the main content late or only after a click. - If you do not want content used for AI training, review Google-Extended controls alongside your robots rules. A fixed page loads with all key text in the DOM on first render, links are crawlable, and robots.txt does not block assets. - When testing, do not assume “view as Googlebot” with a spoofed UA reflects reality. Verify with IP and check the rendered HTML. - After major changes, recheck Crawl stats and fetch a sample URL you changed to confirm a 200 and full render. ## Questions ### What is Googlebot in simple terms? It is Google’s crawler, the software that visits your pages to add and refresh them in Google’s index. It follows links and renders JavaScript to see content. ### Can I block Googlebot? Yes, with robots.txt rules. If you block crawling, the blocked pages are unlikely to be indexed. Only disallow areas you truly want out of Search. ### Why is Googlebot checking my site? It is discovering new pages and refreshing known ones. Spikes usually follow a deploy, a sitemap change, or new internal links that expose more URLs. ### How often does Googlebot crawl a site? It varies by site and URL. Important pages may be fetched often, others less. The Crawl stats report shows your recent pattern and any errors. ### How do I view a page as Googlebot? Do not rely on a spoofed user agent. Verify real Googlebot visits by reverse DNS of the IP, and check your server’s response and the final rendered HTML. ### What is the difference between smartphone and desktop Googlebot? Both fetch pages, but the smartphone version is primary for most sites under mobile-first indexing. Use it as your reference for rendering and layout. ## Read next - [Crawling](https://porteur.ai/glossary/crawling): Crawling is how Googlebot discovers and fetches your URLs. See what it fetched, fix slow or blocked areas, and know when crawl budget matters. - [Rendering](https://porteur.ai/glossary/rendering): Rendering is how Google executes your JavaScript and CSS after crawling HTML, which can delay indexing. Here is what to check and how to fix it. - [robots.txt](https://porteur.ai/glossary/robots-txt): robots.txt tells crawlers which URLs they may fetch. See what it does not do, how to test it, what to put in it, and how to handle AI bots. - [The Crawl stats report: how much Google fetches and where it struggles](https://porteur.ai/guides/search-console-crawl-stats-report): Find Crawl stats in Search Console Settings. Read the four charts, host status and breakdowns. Spot 5xx spikes and wasted crawls. Know what is normal. - [How to use the URL Inspection tool in Search Console](https://porteur.ai/guides/url-inspection-tool): Read each panel, run Test live URL, and know when to request indexing. Fix new pages, dropped pages, and canonicals Google ignores. - [Mobile-first indexing](https://porteur.ai/glossary/mobile-first-indexing): Mobile-first indexing means Google uses your mobile page to crawl, index and rank. Here is what it hides on desktop-only pages and how to fix it. Get a free check that reads your site and the searches around it from a URL in about thirty seconds and shows three findings you can use today. Free check: https://porteur.ai/