Because it lives in the server response rather than the page body, this header can govern indexing for file types that have no place to put a meta tag. Applied at the server or CDN level, it lets teams set noindex on entire directories, block AI-specific bots, or prevent caching of sensitive resources with precise, scalable rules.
In AI-visibility engagements, Acromatico uses X-Robots-Tag to draw clean boundaries around what generative crawlers may consume. Directives can target named user agents, so a site can welcome answer engines onto public content while keeping gated or duplicate resources out of both traditional indexes and model training pipelines.
Related terms
Questions people ask
Use the header when you need to control non-HTML resources like PDFs, images, or spreadsheets that cannot hold a meta tag, or when applying rules across many files at once at the server level. For single HTML pages, either method works.
Yes. Directives can target specific user agents, so you can serve noindex or noarchive rules to named bots like GPTBot while leaving general access intact. It complements robots.txt for fine-grained control over what generative engines may index or cache.
See if AI actually names your brand
Acromatico runs live AI-visibility audits across ChatGPT, Perplexity, Gemini, and Claude.
Get a free AI Visibility Audit →