# X-Robots-Tag X-Robots-Tag is an HTTP response header that tells crawlers how to index a URL. Use it when you need robots directives on files without HTML heads, or when control is easier at the server or edge. Updated 2026-09-14 · Source: https://porteur.ai/glossary/x-robots-tag ## What X-Robots-Tag is X-Robots-Tag is an HTTP response header carrying the same directives as the meta robots tag. It works for any file type, including PDFs and images. Typical directives: noindex, nofollow, nosnippet, max-snippet, and unavailable_after. You can scope it to a crawler name too, for example googlebot. ```http HTTP/1.1 200 OK Content-Type: application/pdf X-Robots-Tag: noindex, nofollow HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Robots-Tag: googlebot: noindex ``` ## When a small site should use it Use X-Robots-Tag when you must control indexing for non-HTML files. Think /legal/terms.pdf, /docs/guide.pdf, or image variants you do not want in search. Use it when templating meta tags is hard. For example, you serve files from a CDN bucket or a static export where adding an HTML head is not possible. Keep meta robots for normal pages. It is simpler to read in templates like /pricing or /guides/getting-started. Use the header where HTML is not practical. ## Where and how to set it Set the header per path on your server or at the edge. Most stacks let you add response headers by route, file extension or folder prefix. ```text # Example rules you might apply # All PDFs under /docs are noindex X-Robots-Tag: noindex for path starts_with /docs/ and content-type = application/pdf # Block only Googlebot for a test image X-Robots-Tag: googlebot: noindex for path = /images/test.jpg ``` If you use a CDN, set it in the CDN behaviour for that path or content type. If you render from an app, add the header in the controller or middleware for those routes. For static hosts, many allow custom headers via a config file at build. Keep rules narrow so you do not hide your whole site by mistake. ## How to read and measure it First verify the header on the wire. Use curl or your browser’s network panel. You are checking the response of the final URL, after redirects. ```bash curl -I https://yourproduct.com/docs/guide.pdf # Look for a line like: # X-Robots-Tag: noindex, nofollow ``` Then confirm how Google sees it. Use Search Console’s URL Inspection for the exact URL. It shows whether indexing is allowed by a robots meta or header. Watch coverage for unintended changes. If many pages move to Excluded by ‘noindex’, review recent header deploys by path. Roll back the rule and recrawl a sample. ## Common traps to avoid - Blocking crawl with robots.txt and expecting noindex to work. The header is only read when the URL is crawlable. - Setting the header on a 301 or 302 and assuming it applies after the hop. Check the target URL’s own response. - Adding noindex globally on */*.pdf. Audit which PDFs you do want indexed, like product sheets or whitepapers. - Forgetting other directives. If you only need to hide a snippet, use nosnippet or a max-snippet value, not noindex. - Sending the header with the wrong casing or duplicated values. Check the single, final header Googlebot receives. > Rule: keep robots controls in one place per URL. Avoid mixing a meta tag and a conflicting header on the same page. ## Questions ### What is the X-Robots-Tag header? It is an HTTP response header that tells crawlers how to index a URL. It supports the same directives as a meta robots tag, but works for any file type. ### What does the error “no noindex detected in x robots tag http header” mean? The response did not include an X-Robots-Tag with a noindex directive. Check the final URL with curl or your network panel and add the header on that response. ### What is a robots tag? It is a set of directives that control indexing and crawling behaviour for a URL. You can send it as a meta tag in HTML or as an X-Robots-Tag header. ### Should I use X-Robots-Tag or meta robots? Use meta robots for normal HTML pages. Use X-Robots-Tag when the file has no head, like a PDF or image, or you manage rules at the server or edge. ### Does robots.txt handle noindex? No. Robots.txt can allow or disallow crawling, but it does not set noindex. Use a meta robots tag or the X-Robots-Tag header for noindex. ### How do I target only Googlebot? Prefix the directive with the crawler name. For example: X-Robots-Tag: googlebot: noindex. Keep scoped rules rare and documented. ## Read next - [Meta robots tag](https://porteur.ai/glossary/meta-robots-tag): The meta robots tag tells crawlers what to do with a page. See each directive, how to set it, how to test it, and the robots.txt trap. - [Noindex](https://porteur.ai/glossary/noindex): Noindex tells search engines not to index a page. Use it for pages that should never rank, and avoid adding it to pages that earn. - [How to use the URL Inspection tool in Search Console](https://porteur.ai/guides/url-inspection-tool): Read each panel, run Test live URL, and know when to request indexing. Fix new pages, dropped pages, and canonicals Google ignores. - [Blocked by robots.txt: what Google can still do with the URL](https://porteur.ai/guides/blocked-by-robots-txt): What “Blocked by robots.txt” and “Indexed, though blocked by robots.txt” mean, how to test a URL, the common accidents, and the clean fixes. - [robots.txt](https://porteur.ai/glossary/robots-txt): robots.txt tells crawlers which URLs they may fetch. See what it does not do, how to test it, what to put in it, and how to handle AI bots. - [robots.txt tester](https://porteur.ai/tools/robots-txt-tester): Paste a site, paths and a crawler: what its robots.txt allows by the rules Google reads with, the rule that decided each, why it won, and the file mended. - [Viewport](https://porteur.ai/glossary/viewport): The viewport is the visible area of a page in the browser. What the meta tag does, what a missing one costs on a phone, and what above the fold means. Paste one URL and get a free check that reads your site and rivals on your searches, in about thirty seconds, and shows three findings whole. Free check: https://porteur.ai/