Meta robots tag
The meta robots tag tells crawlers what to do with a page. You will use it to keep thin or private pages out of search, or to control snippets. This page shows the directives, how to set them, how to test them, and the common traps.
By Théophile Louvart, founder of Porteur · Updated 14 September 2026 · Markdown
What the meta robots tag does
The meta robots tag is <meta name="robots" content="…"> in the <head>. It sets crawler rules for that page only. A crawler specific name like <meta name="googlebot" …> narrows it to one bot.
- noindex: leave the page out of results
- nofollow: do not follow its links
- none: both noindex and nofollow
- noarchive: do not show a cached copy
- nosnippet: do not show a text snippet
- max-snippet:[n]: limit snippet characters
- max-image-preview:[none|standard|large]
- max-video-preview:[n]
- notranslate: do not offer translation
- noimageindex: do not index images on the page
- unavailable_after:[date]: stop showing after a date
- indexifembedded: allow indexing when embedded
Use it when a page should exist for users but not be searchable, like /account, /cart, or a filter page with near duplicate content. Keep high intent pages indexable, like /pricing and core docs.
How to set it and what a correct page looks like
<!-- Sitewide default allow (omit tag) -->
<!-- Page level control -->
<meta name="robots" content="noindex, follow">
<meta name="googlebot" content="max-snippet:160, max-image-preview:large">
<!-- Example: stop after a date (UTC) -->
<meta name="robots" content="unavailable_after: 31 Dec 2026 23:00:00 UTC">You can send the same directives in the X-Robots-Tag HTTP header. Use this for PDFs, images, video or when you control headers but not HTML.
HTTP/1.1 200 OK
X-Robots-Tag: noindex
X-Robots-Tag: googlebot: max-snippet: 160
X-Robots-Tag: noimageindexA fixed page matches your intent. For example, /guides/internal-tools has <meta name="robots" content="noindex, follow">, still linked from /guides for users, and is not in search results.
How to test and monitor it
- View source: confirm one robots tag with the values you expect. Avoid duplicates.
- Check HTTP headers for X-Robots-Tag using curl -I or your browser’s network panel.
- Use the URL Inspection tool to see if Google saw a noindex and whether the page is indexed.
- In Search Console, watch Excluded by ‘noindex’ and trends after changes. Reinspect changed URLs.
Common traps and conflicts
- Do not use noindex in robots.txt. It is ignored. Use meta robots or X-Robots-Tag.
- Do not mix index and noindex on the same URL. One tag, one intent.
- Avoid sitewide noindex during a launch. Remove it before going live.
- indexifembedded helps only when the page is embedded. It does not override a normal noindex for standalone pages.
- nofollow does not hide a page. It only affects link following.
- unavailable_after removes the page from results after the date. It does not unpublish the page for users.
Which directive to use when
| Goal | Use |
|---|---|
| Keep a utility page out of results | noindex, follow |
| Stop images from being indexed | noimageindex |
| Limit long snippets on /changelog | max-snippet: 160 |
| Prevent cached copy on /pricing | noarchive |
| Let an embed be indexed while page stays noindex | indexifembedded |
| Expire a promo page after a date | unavailable_after: 31 Dec 2026 23:00:00 UTC |
Questions
It is an HTML meta tag that tells crawlers what to do with a page. You set directives like noindex or nosnippet in the content attribute.
No. Omit it for normal pages you want indexed. Add it only where you need control, like noindex, snippet limits or image rules.
No. Robots.txt controls crawling only. A noindex there will not work. Use a meta robots tag or an X-Robots-Tag header instead.
X-Robots-Tag is the HTTP header version. It supports the same directives and works for any file type, like PDFs and images.
Send X-Robots-Tag: noindex in the HTTP response for that PDF. You cannot add HTML meta tags to a PDF file.
The bot may not be able to crawl the page. If robots.txt blocks it, the noindex cannot be seen. Remove the block, let it crawl, then recheck.
Sources
Check my site, free
Check your site’s index controls in seconds: drop in a URL to see three findings on crawling and meta robots from your pages and rivals, free.
- Free check, no card
- Read-only, your own accounts
- Readable by your agent