Claude’s web search: how it fetches, cites and what to allow

Claude web search arrived in March 2025. It answers a question, and when it needs the web it searches and reads sources. You see inline citations with links inside the answer.

Two fetch modes matter. Claude-SearchBot crawls to build a search index. Claude-User fetches pages on demand when a user asks about a topic or a URL. ClaudeBot is for training and improving models, not for answering a live question.

Reports suggest Claude’s index uses Brave Search as a base. Anthropic also crawls with Claude-SearchBot. There is no submission form. You get in by being crawlable and by being linked in pages that already rank for the query.

Assistants cite pages that answer the question directly. They prefer clear structure, consistent names for products and people, lists and tables, and figures they can quote. A visible date and an author help disambiguate and build trust.

Claude shows inline citations. If your site is named, users click through from the link. Those visits appear in analytics with referrers like claude.ai or the chat host. You will not get a separate report from Anthropic as of 2026.

For a small site, one named citation on a high intent query can bring meaningful visits. Your job is to publish a page Claude can quote, and to be on the lists and comparisons it already trusts.

Three current tokens to know as of 2026. Claude-SearchBot builds Anthropic’s search index. Claude-User fetches a page a user asks about. ClaudeBot crawls for training and model improvement. Older tokens, anthropic-ai and Claude-Web, are still seen in robots files and can be kept for clarity.

# Allow Claude’s answering and search crawlers
User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Optional: allow or block training crawlers
User-agent: ClaudeBot
Disallow: /

# Legacy tokens some sites still mention
User-agent: anthropic-ai
Allow: /
User-agent: Claude-Web
Allow: /

Allow Claude-SearchBot and Claude-User if you want visibility in Claude’s answers. Decide on ClaudeBot based on your training preference. Vendors honour robots.txt voluntarily. Blocking a training crawler does not remove pages already collected.

Keep the file simple. Put these rules at the top of /robots.txt. Test with a plain fetch. Update your allowlist when vendors rename a token. Do not guess a name you have not seen documented.

Training crawls feed models. Answering crawls fetch or index pages to ground an answer for a user. They are different purposes and often different bots.

For Claude, ClaudeBot is training. Claude-SearchBot is index. Claude-User is on demand fetch. Allow the last two if you want citations and clicks. You can disallow training without losing search visibility in Claude.

Across the field as of 2026, OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User. Perplexity separates PerplexityBot and Perplexity-User. Google separates Googlebot from Google-Extended. The pattern is similar: answering access is separate from training access. Your robots file should reflect that intent on a vendor by vendor basis.

Claude retrieves from its index and search partners. As reported, Brave Search underpins the index. Claude-SearchBot also crawls. There is no inclusion request. You earn inclusion by being linked and crawlable.

• Publish pages that answer a single question early and plainly. Use headings that match the question.
• Use consistent product and company names. Avoid nicknames on key pages.
• Show a recent date and an author. Keep the page updated.
• Add a short list or table that a model can quote without paraphrase.
• Get listed on independent lists, comparisons and reviews your buyers already read.

Being on the pages that answers already cite is the reliable route. For example, if answers to “best database for small SaaS” often cite a roundup on smalldev.tools, you need a mention there. Then your own “/compare/our-db-vs-x” can be quoted for details and specs.

Start with the buyer’s phrasing. If buyers ask “pricing for YourProduct”, your “/pricing” page should say “YourProduct pricing” in the h1 and in the first sentence. Do not bury the price chart under tabs.

1. 1

   State the answer in the first 2 paragraphs

   Put the key figure or choice first. On “/guides/getting-started”, write “Set up YourProduct in 5 minutes. You need an email, a card, and these three steps.”

2. 2

   Name entities consistently

   Use the exact product name, not a shorthand. Use the same spelling for models, plans and integrations across pages.

3. 3

   Structure with quotable blocks

   Add a list of steps, a bulleted feature list, or a 2-column table of limits. Keep each item concrete and short.

4. 4

   Add a visible date and author

   Show “Updated September 2026” and who wrote it. Update the page when details change.

5. 5

   Link the comparisons you want to rank for

   Publish “/alternatives/competitor” and “/compare/yourproduct-vs-competitor”. Use honest, sourced tables and clear pro and con lists.

A fixed page looks like “/alternatives/competitor”: an h1 with the exact query, a two paragraph summary, a 6 item pros and cons list, and a table that states plan limits and key specs. Claude can lift lines with confidence and cite you inline.

Treat answering access as the default allow. Then make explicit choices for training crawlers. Keep rules vendor specific. Here is a compact pattern that covers the major assistants as of 2026 without blocking search visibility.

# Claude answering and index
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Claude training
User-agent: ClaudeBot
Disallow: /

# OpenAI answering and index
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
# OpenAI training
User-agent: GPTBot
Disallow: /

# Perplexity answering and index
User-agent: Perplexity-User
Allow: /
User-agent: PerplexityBot
Allow: /

# Google Search and Gemini control
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /

# Microsoft Bing and Copilot
User-agent: Bingbot
Allow: /

# Other common crawlers
User-agent: CCBot
Disallow: /
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Disallow: /
User-agent: Amazonbot
Allow: /
User-agent: Bytespider
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: DuckAssistBot
Allow: /

Test with a robots tester and a curl. Watch server logs for the exact user agent string and IP range before you make strict allowlists. Vendors can change user agent formats over time, so review monthly.

You cannot get a citation report from Anthropic as of 2026. So you measure it yourself. Use a fixed list of buyer questions and track who Claude cites over time.

1. 1

   Write 20 buyer questions

   Use the wording buyers use. Examples: “best payroll for UK startups”, “how to migrate from X to YourProduct”, “YourProduct pricing”.

2. 2

   Ask Claude on the same cadence

   Run the set weekly. Paste the questions as separate prompts. Save the answers with dates.

3. 3

   Log names and links

   Record every domain Claude cites for each question. Mark when your domain appears, and on which queries.

4. 4

   Segment analytics by referrer

   Create segments for claude.ai and any Anthropic chat host. Compare against chatgpt.com, perplexity.ai, copilot.microsoft.com and gemini.google.com.

5. 5

   Use a tracker if you need scale

   Tools like Profound, Peec AI, Otterly, Scrunch, and the AI toolkits in Semrush and Ahrefs, or DataForSEO’s LLM mentions, automate the asking.

A fixed process looks like a sheet with 20 rows for queries and weekly columns for “cited domains”. Your domain is green when named. You add a note with the page cited and any change you made that week.

If Claude skips your site, start with access and structure. Then fix content. Then earn mentions on third party pages Claude already trusts for the query.

• Check robots.txt for Claude-User and Claude-SearchBot allows.
• Fetch the target URL as Claude-User with curl to confirm access.
• Read the top cited pages. Note the headings, lists, date and author blocks they use.
• Match the question in your h1 and first paragraph. Answer in plain terms in the first 2 paragraphs.
• Add a short list or a table with the exact figures the answer needs.
• Add internal links from related pages so the page is easy to find and crawl.
• Pitch the page to roundups and comparison posts that already rank for the query.

For example, if Claude cites “/compare/tool-a-vs-tool-b” pages and you only have a blog post, publish “/compare/yourproduct-vs-tool-b” with a table of plan limits and a dated summary. Link it from “/alternatives/tool-b”.

• Single page apps: assistants cannot read content that only appears after scripts run. Render critical content server side.
• Stale dates: if your page shows 2023, assistants may prefer a 2026 page with the same facts. Update and date visibly.
• Inconsistent naming: if your “Starter” plan is called “Basic” on one page, assistants may misquote. Standardise names first.
• Thin comparison pages: if your compare page only says “we are better”, assistants will skip it. Add sourced specs and limits.
• Country variants: if pricing varies by country, state the region on the page and show the currency in the table.