AI crawlers behave much like traditional search spiders but serve generative systems. Some gather text to train future models; others retrieve pages in real time when a user asks a question, then cite what they find. Site owners control access through robots.txt directives that allow or block specific bot user-agents.
Deciding which AI crawlers to permit is now a strategic choice. Blocking them protects content from training use but can also remove your brand from the answers those engines generate. Acromatico helps clients audit crawler access so their pages stay eligible for citation in ChatGPT, Perplexity, Claude, and Google's AI surfaces.
Related terms
Questions people ask
Use your robots.txt file to allow or disallow each bot by its user-agent name, such as GPTBot, ClaudeBot, PerplexityBot, or Google-Extended. You can grant full access, block specific bots, or restrict certain directories. Server-level rules add another enforcement layer.
It depends on your goals. Blocking prevents your content from training models, but it also removes your pages from being cited in AI answers, cutting visibility. Many brands allow answer-fetching crawlers for reach while weighing training-focused bots separately.
See if AI actually names your brand
Acromatico runs live AI-visibility audits across ChatGPT, Perplexity, Gemini, and Claude.
Get a free AI Visibility Audit →