ClaudeBot follows robots.txt conventions, and Anthropic publishes documentation so publishers can identify and manage the bot. Like other training-oriented crawlers, it gathers text at scale; the decision to allow it shapes whether your content contributes to Claude's knowledge base.
Anthropic also operates separate agents for user-initiated fetching, so blocking ClaudeBot does not necessarily block every Claude-related request. For brands aiming to be represented accurately when people ask Claude about their category, Acromatico weighs crawler permissions against the value of appearing in those AI-generated answers.
Related terms
Questions people ask
ClaudeBot is Anthropic's crawler that gathers publicly available web content which may help develop and improve the Claude family of models. It respects robots.txt directives, and Anthropic documents the user-agent so site owners can identify, verify, and control its access to their pages.
Blocking ClaudeBot limits your content's use in training, but Anthropic uses separate agents for live user-initiated fetching. The effect on answers is nuanced, so review Anthropic's current crawler documentation before deciding, and weigh training exclusion against the visibility of accurate brand representation.
See if AI actually names your brand
Acromatico runs live AI-visibility audits across ChatGPT, Perplexity, Gemini, and Claude.
Get a free AI Visibility Audit →