ClaudeBot
Anthropic's web crawler for Claude AI. Allows or blocks Claude from accessing your website content.
What is ClaudeBot?
ClaudeBot is the user agent Anthropic uses to crawl the web for Claude. It honours robots.txt, and Anthropic publishes its user-agent strings and guidance so that site owners can allow or restrict it. Anthropic has since documented additional agents alongside it, including Claude-User, which fetches a page when a Claude user's request needs live web content, and Claude-SearchBot, which supports Claude's search functionality. A robots.txt review for Claude should therefore check all of Anthropic's published agents, not just the one most people know.
For a site that wants to be citable in Claude's web-enabled answers, the requirement is the same as for other engines: the fetching agents must be allowed, the page must return its content in the initial HTML, and the answer to the page's question must be stated plainly near the top. Claude with web search attaches source links to its answers, so a page it can fetch and quote can earn a citation in the same way it would on Perplexity or ChatGPT search.
ClaudeBot traffic is only visible in server or CDN logs. Like other AI agents it does not execute JavaScript, so tag-based analytics never records it. Log analysis shows which pages it fetches and how often, which is a direct indicator of whether the site is being considered as a source.
Why it matters for AI search
Claude is one of the major assistants buyers use, and enterprise adoption makes it especially relevant for B2B categories. Blocking its agents removes the brand from Claude's live answers and, over time, from what Claude knows. Because Anthropic's agent list has grown, sites that set a single ClaudeBot rule years ago should re-check that their intent, whether allow or deny, still covers the agents that matter for retrieval.
Related terms
Claude
Anthropic's AI assistant known for nuanced, thoughtful responses. One of the major AI platforms for brand visibility.
Robots.txt
A file that tells web crawlers (including AI bots) which pages to access or avoid. Critical for managing AI visibility.
Synthetic Traffic
Web traffic where the primary visitor is an AI agent or crawler acting on behalf of a user.
GPTBot
OpenAI's web crawler that collects data for ChatGPT. Managing GPTBot access via robots.txt affects how ChatGPT understands your content.
Frequently asked questions
Should I allow ClaudeBot?+
If you want Claude to be able to retrieve and cite your content, yes, along with Anthropic's user-initiated and search agents. If your concern is model training rather than retrieval, check Anthropic's current documentation for which agent controls which use, since the split is documented per agent.
How do I see whether ClaudeBot is visiting my site?+
Search server or CDN access logs for the ClaudeBot user-agent string (and Claude-User and Claude-SearchBot). Client-side analytics will not show these visits because the agents do not run JavaScript.