Gemini and your site: grounding, Google-Extended, and what to allow

AI Mode sits inside Google Search. It fans out your query, pulls from the same index that powers organic results, and writes an answer with links. It cites pages that Google’s ranking systems already surface. There is no separate crawler for it and no special opt-out beyond snippet controls.

Gemini is the assistant at gemini.google.com and in the app. It answers from the model. When it needs to look things up, it grounds the answer with Google Search and shows links. Whether it can use your pages for training and for grounding is controlled by Google-Extended in robots.txt. That switch does not affect Google Search or AI Overviews.

Google-Extended is a robots token from Google. It lets you allow or disallow use of your pages for Gemini model training and for grounding in Gemini apps and the Vertex AI API. As of 2026, it does not change Google Search ranking and it does not change which pages appear in AI Overviews or AI Mode.

If you block Google-Extended, Gemini should not use new or updated content for training or for grounding its answers. It can still mention your brand if it sees it elsewhere. Blocking does not remove content already collected. The control is voluntary, like all robots rules for these vendors.

If you allow Google-Extended, you make it easier to be cited in Gemini answers when it searches. That may send referral visits from gemini.google.com. It will not change your Search Console metrics by itself. AI Mode and AI Overviews clicks arrive as google.com like any search click.

You might block training for policy or legal reasons. You might also want Gemini to ground on your latest docs and cite you. Those aims conflict. Decide which matters more for your site.

• If you block Google-Extended: fewer chances to be cited by Gemini, but your content is not used for Gemini training or grounding going forward.
• If you allow Google-Extended: more visibility in Gemini answers when it searches, with potential referral clicks, but your pages can be used for Gemini training and grounding.

For a small SaaS or store, visits and mentions often matter more than training control. Many founders allow Google-Extended and focus on being quotable. If you must block, you can still win AI Mode and AI Overviews because they rely on Googlebot and ranking, not Google-Extended.

Gemini answers from the model first. When it needs facts, it runs a search, selects sources, and grounds the answer. You see links to those sources. This is closer to ChatGPT with search and to Claude with web search than to a pure chat reply with no citations.

AI Mode expands your query into sub-queries across Google’s index and writes a single answer. It links to a handful of pages, often those near the top of the organic set. There is no separate crawler for AI Mode, so the same on-page signals that help SEO also decide which links appear there.

Practical read: you get Gemini referral clicks from gemini.google.com. You get AI Mode and AI Overviews clicks as google.com in analytics. In Search Console, those clicks appear inside the Performance report with the rest of web search, not broken out, so read queries and landing pages to infer impact.

Assistants and answer engines rely on crawlers and search indexes. Allow the answering crawlers if you want citations. Control the training crawlers if you need to. Blocking the answering and search crawlers makes your site invisible to assistants, even if you allow training.

• Googlebot powers Google Search and the pages AI Overviews cite. Do not block it.
• Google-Extended controls Gemini training and grounding. Allow it if you want Gemini citations.
• OAI-SearchBot indexes for ChatGPT search. Allow it for ChatGPT search results. You can still disallow GPTBot for training.
• Claude-SearchBot indexes for Claude web search. Allow it to appear in Claude answers.
• PerplexityBot builds Perplexity’s index. Allow it to be cited there.
• Bingbot powers Bing Search and Microsoft Copilot. Allow it to appear in Copilot answers.
• Per-request fetchers like ChatGPT-User, Claude-User and Perplexity-User load a page when a user asks. They respect your robots rules, so do not block them unless you mean it.

# Keep Google Search and AI Mode/Overviews eligible
User-agent: Googlebot
Disallow:

# Allow Gemini training and grounding (change to Disallow to block)
User-agent: Google-Extended
Allow: /

# Allow ChatGPT search results, but keep out of training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /

# Allow Claude search
User-agent: Claude-SearchBot
Allow: /

# Allow Perplexity index
User-agent: PerplexityBot
Allow: /

# Allow Bing and Copilot
User-agent: Bingbot
Allow: /

# Optional: allow per-request user fetchers
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /

Older or other tokens you may still see: anthropic-ai, Claude-Web, Applebot-Extended, CCBot, Bytespider, meta-externalagent, meta-externalfetcher, Amazonbot, cohere-ai, DuckAssistBot. Check your needs before blocking them. Blocking a training crawler does not remove pages already collected.

Generative engine optimisation means being named and cited in AI answers. The term was popularised in a 2023 paper by Aggarwal and colleagues. As of 2026, assistants retrieve from a web index and cite pages that state the answer early and plainly. There is no submission form.

• Match the question in your h1 and first paragraph. Example: a page at /guides/gemini-grounding that opens with a one sentence answer.
• Use consistent names for your product and people across pages. Assistants map names to entities.
• Lay out lists, steps and simple tables. Answers quote lists and figures. Avoid burying facts in long prose.
• Add a visible date and an author. Assistants prefer fresh, attributable sources.
• Answer the buyer question in the first 100 words. Then add depth. Perplexity, ChatGPT search and Claude reward early clarity.
• Write comparison and alternatives pages that name rivals fairly. Assistants pull from roundups and comparison lists.
• Get on the roundups that already rank and get cited. Being on those lists is the most reliable route to be named.

A fixed page looks like this: /pricing opens with a one line summary of plans, then a comparison table with rows for limits, support and billing. The names match your product UI. A clear date and contact are on the page. If someone asks Gemini for “yourproduct pricing”, it can quote and link that table.

When Gemini searches, it wants pages it can ground to. Publish canonical, crawlable pages for your main facts: pricing, features, integrations, support policies, security, and API limits. Put the answer near the top, then detail. Keep URLs stable and human-readable, like /security or /api/rate-limits.

• Avoid hiding key facts behind tabs or accordions that need JavaScript to render. Many crawlers do not execute scripts.
• Avoid PDF-only information if you want citations. Give the HTML page first.
• Avoid vague feature names. Name the feature and its status in plain text, like “SSO: SAML 2.0 and Okta supported”.
• Avoid boilerplate pages with no dates or author. Assistants prefer dated, attributable sources.
• Avoid mismatched names. If your product is Acme Forms, call it that on every page, not Forms by Acme on some pages.

If you run a docs site, expose the most cited answers in one page per topic. Example: /guides/google-forms-to-sheets, with a two step list and an image alt text that states the action. Assistants quote steps and short lists. Your brand name sits near the answer, so it is named when cited.

No assistant publishes a report of where you were cited. You measure mentions, clicks and presence with a mix of checks. Keep it simple and regular. Build a list of buyer questions and track them every 28 days.

1. 1

   Make a question list

   Write the 20 searches buyers use. Example: “best time tracking for agencies”, “Acme Forms pricing”, “Acme vs Typeform”. Cover how-to, comparisons and pricing.

2. 2

   Ask the assistants on a schedule

   Use ChatGPT search, Perplexity, Gemini and Claude. Log which pages they cite. Screenshots help. Note the date and the exact wording used.

3. 3

   Check analytics referrers

   Segment chatgpt.com, perplexity.ai, copilot.microsoft.com and gemini.google.com. Watch landing pages and events. Google AI surfaces arrive as google.com.

4. 4

   Read Search Console

   Use the Performance report. Track queries and clicks for your key pages. AI Mode and AI Overviews are included here with other web clicks.

5. 5

   Automate if you can

   Tools like Profound, Peec AI, Otterly, Scrunch, and the AI suites from Semrush and Ahrefs, and DataForSEO’s LLM mentions data can run the checks for you.

A useful report is one table per assistant: query, cited domains, your rank among citations, your URL if cited, and a note. Over time you will see which page types win. Fix gaps with targeted pages, not broad content pushes.

• Robots: allow Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot. Decide on Google-Extended and document the choice.
• Pages: add one comparisons page, one alternatives page, and one pricing clarity pass. Start with /compare/acme-vs-rival and /alternatives/acme.
• Structure: move the core answer to the first 100 words on your top five pages. Use matching headings.
• Findability: link from your homepage to pricing, integrations, docs and comparisons. Avoid orphan pages.
• Citations: get listed on three roundups that already rank and get cited. Ask for an update if they used old names or facts.
• Rendering: ensure your core content loads in HTML without JavaScript. Test with curl or a text-only browser.
• Governance: add a visible date and an author byline to your key guides and docs.

After this pass, ask Gemini and AI Mode a buyer query you care about, like “Acme time tracking pricing”. Check who is cited. Refine the page that should win that citation and update the roundup that names you. Repeat each month.

Short answers to the questions founders ask on this topic.

They serve different jobs. AI Mode is part of Search and shows links chosen by Google’s ranking systems. Gemini is the assistant that can search and ground answers. For citations from Google surfaces, optimise like you would for organic Search. For Gemini citations, allow Google-Extended and make pages that are easy to ground to, with clear early answers and consistent names.

Accuracy varies by query. AI Mode rides on the same index and ranking signals as organic results. Gemini can answer from the model and then ground with sources. You influence both by publishing up to date, specific pages that state the answer early and plainly and that match the query wording in headings.

If you want to be cited by Gemini when it searches, allow it. If you cannot accept use for Gemini training or grounding, disallow it. It does not affect Google Search ranking, AI Overviews or AI Mode. You can still be cited by AI Mode and AI Overviews through Googlebot and good on-page work either way.

Let it use your pages for grounding with Google-Extended. Publish pages that answer the question in the first paragraph, with clear headings, lists and dates. Get named on roundups and comparisons Gemini already cites. There is no submission form, so your route in is being easy to quote and already present on the lists it pulls from.

In analytics, segment referrers: gemini.google.com shows Gemini clicks, google.com covers AI Mode, AI Overviews and regular search. In Search Console, read the Performance report and track queries to your key pages. No assistant publishes a citations report, so keep a manual or automated log of who is cited for your buyer questions each month.