Robots.txt
A file that tells web crawlers (including AI bots) which pages to access or avoid. Critical for managing AI visibility.
What is Robots.txt?
robots.txt is a plain-text file served at the root of a domain (for example https://example.com/robots.txt) that lists rules for automated crawlers. Each rule block names a user agent and then allows or disallows URL paths for it. The format is defined by the Robots Exclusion Protocol, formalised as RFC 9309, and is honoured voluntarily by well-behaved crawlers; it is not an access control mechanism, and a disallowed page can still be fetched by anything that ignores the file.
AI crawlers have made the file newly important. OpenAI, Anthropic, Perplexity, Google, Common Crawl and others each identify their agents by name and document which robots.txt rules they respect. The critical distinction is between training crawlers (GPTBot, CCBot, Google-Extended) that collect content for model training, and retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and their newer siblings) that fetch pages so an assistant can cite them in a live answer. Blocking the first group is a policy choice about content reuse; blocking the second removes the site from AI answers.
Many sites adopted blanket AI-blocking lists in 2023 and 2024 that disallow every agent. Auditing the file against each vendor's current agent documentation, and checking server logs for which agents actually visit, is the first technical step in any GEO programme.
Why it matters for AI search
robots.txt is the single most common reason a brand is absent from AI answers. One line copied from a "block AI" template can stop every retrieval agent from reaching the site, and nothing in normal analytics will reveal it. Getting the file right, allowing retrieval agents while making a deliberate choice about training crawlers, is the cheapest, fastest GEO fix available, and it must be re-checked as vendors add new agents.
Related terms
GPTBot
OpenAI's web crawler that collects data for ChatGPT. Managing GPTBot access via robots.txt affects how ChatGPT understands your content.
ClaudeBot
Anthropic's web crawler for Claude AI. Allows or blocks Claude from accessing your website content.
PerplexityBot
Perplexity AI's web crawler that indexes content for use in Perplexity's AI-powered search responses.
Synthetic Traffic
Web traffic where the primary visitor is an AI agent or crawler acting on behalf of a user.
Frequently asked questions
Which AI crawlers should I allow in robots.txt?+
If you want AI visibility, allow the retrieval agents: OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and the user-initiated agents each vendor documents. Decide separately about training crawlers such as GPTBot, CCBot and Google-Extended based on your policy on model training.
Does blocking a crawler in robots.txt remove existing content from AI models?+
No. It stops future fetches by crawlers that honour the file. Content already used in training stays in the model, and pages already indexed may persist until the index refreshes.
How do I test my robots.txt for AI agents?+
Read the file agent by agent against each vendor's documented user-agent names, use a robots.txt tester to confirm key pages are allowed for those agents, and check server logs to see which agents are fetching which pages.