The X-Robots-Tag is an HTTP header used to control crawler access and indexing for non-HTML files like PDFs and images. It applies server-level rules such as noindex or nofollow, allowing precise management of how search engines and AI bots consume your content without editing individual files.
What is the x robots tag?
Applied at the server or CDN level, it lets teams set noindex on entire directories, block AI-specific bots, or prevent caching of sensitive resources with precise, scalable rules. In AI-visibility engagements, Acromatico uses X-Robots-Tag to draw clean boundaries around what generative crawlers may consume. Directives can target named user agents, so a site can welcome answer engines onto public content while keeping gated or duplicate resources out of both traditional indexes. For indexing control, rely on the specific documentation provided by each AI crawler operator.
The X-Robots-Tag is an HTTP response header that controls how search engines crawl and index a resource, supporting directives like noindex, nofollow, and noarchive. Unlike the robots meta tag, it works for non-HTML files such as PDFs, images, and other documents.
Related terms
Questions people ask
Use the header when you need to control non-HTML resources like PDFs, images, or spreadsheets that cannot hold a meta tag, or when applying rules across many files at once at the server level. For single HTML pages, either method works.
Yes. Directives can target specific user agents, so you can serve rules to named bots while leaving general access intact. It complements robots.txt for fine-grained control over what crawlers may access. Verify supported directives in each crawler's official documentation.
See if AI actually names your brand
Acromatico runs live AI-visibility audits across ChatGPT, Perplexity, Gemini, and Claude.
Run your free AI Visibility Audit →