The X-Robots-Tag is an HTTP response header that tells search engines and AI crawlers how to treat a file — index it, exclude it, or follow its links. Because it travels with the server response instead of inside the page, it also controls PDFs, images, and other non-HTML files a robots meta tag cannot reach.
What is the X-Robots-Tag?
The header carries the same directives as a robots meta tag — noindex, nofollow, noarchive — plus extensions like nosnippet and unavailable_after, and it can be scoped to specific user agents, so a rule reaches named bots without affecting everyone else. Applied once at the server or CDN, it covers every PDF, image, and spreadsheet in a directory, which is what makes it the standard control for assets that live outside an HTML template.
X-Robots-Tag: googlebot: noindex
When should you use the X-Robots-Tag over a meta robots tag?
Reach for the header whenever the file cannot hold a meta tag: PDFs, images, spreadsheets, and anything served outside an HTML template. It is also the efficient way to apply one rule across an entire directory at the server or CDN level. For a single HTML page, either method works — the meta tag simply sits closer to the content.
At our Miami studio, both halves of this meet daily: the photography craft produces the visual assets, and the AI-visibility work decides which of those assets search engines and answer engines are allowed to keep — the digital equivalent of marking a contact sheet. One line in a response header is often the difference between a published file and a private one.
The X-Robots-Tag is an HTTP response header that controls how search engines crawl and index a resource, supporting directives like noindex, nofollow, and noarchive. Unlike the robots meta tag, it works for non-HTML files such as PDFs, images, and other documents.
Related terms
Questions people ask
No. robots.txt controls whether a crawler can fetch a URL at all, while the X-Robots-Tag controls what happens after the file is fetched — whether it is indexed, archived, or shown as a snippet. The two work together: one gates access, the other gates presentation.
Yes. Directives can target specific user agents, so you can serve rules to named bots while leaving general access intact. It complements robots.txt for fine-grained control over what crawlers may access. Verify supported directives in each crawler's official documentation.
Open your browser's DevTools, load the URL, and inspect the response headers in the Network panel — the X-Robots-Tag line appears alongside headers like Content-Type and Cache-Control. From a terminal, curl -I https://example.com prints the response headers without downloading the page body.
See if AI actually names your brand
Acromatico runs live AI-visibility audits across ChatGPT, Perplexity, Gemini, and Claude.
Run your free AI Visibility Audit →From our Miami studio: Bright and citrusy? Reach for the Ecolosophy Citrus Burst kit. Scent-sensitive? Ecolosophy Unscented Oasis was made for you. Small-batch, made with care.