GPTBot
OpenAI's web crawler that collects data for ChatGPT. Managing GPTBot access via robots.txt affects how ChatGPT understands your content.
What is GPTBot?
GPTBot is the user agent OpenAI uses to crawl the web for content that may be used to train and improve its models. It identifies itself with a GPTBot user-agent string, respects robots.txt, and OpenAI publishes the IP ranges it crawls from so that sites can verify requests. It is one of three OpenAI agents site owners need to know about, and the distinction between them is where most robots.txt mistakes happen.
The other two are OAI-SearchBot, which crawls to build the index behind ChatGPT search, and ChatGPT-User, which fetches a page at the moment a user's request needs it. Those two are retrieval agents: they determine whether a site can appear as a cited source in ChatGPT answers. GPTBot is a training crawler: it determines whether content may feed future model training. OpenAI documents them as separate agents with separate robots.txt controls precisely so that a site can allow search while declining training, or the reverse.
A robots.txt rule that disallows GPTBot has no effect on OAI-SearchBot or ChatGPT-User unless they are listed too. Conversely, sites that copied a blanket "block all AI" list in 2023 often blocked all three and are now invisible in ChatGPT search without realising it. Reviewing the file agent by agent, and checking server logs for which agents actually visit, resolves this.
Why it matters for AI search
GPTBot is the crawler most site owners have heard of, and decisions about it are often made without distinguishing training from retrieval. For GEO the practical rule is: decide deliberately about GPTBot based on how you feel about model training, but keep OAI-SearchBot and ChatGPT-User allowed if you want ChatGPT to cite you. Content that reaches training also shapes what the model says about a brand in plain chat, so many brands allow all three.
Related terms
Robots.txt
A file that tells web crawlers (including AI bots) which pages to access or avoid. Critical for managing AI visibility.
Synthetic Traffic
Web traffic where the primary visitor is an AI agent or crawler acting on behalf of a user.
Training Data
The text data used to train AI models. Being in training data affects how AI understands your brand.
ChatGPT Search
OpenAI's feature allowing ChatGPT to search the web in real-time for current information, combining conversational AI with live search results.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?+
Not from ChatGPT search. Search retrieval uses OAI-SearchBot and ChatGPT-User, which have their own robots.txt rules. Blocking GPTBot only limits use of your content for model training, which affects what future models know about you from memory rather than what they can retrieve live.
How do I verify a request is really from GPTBot?+
OpenAI publishes the IP address ranges its crawlers use. Compare the requesting IP against that list rather than trusting the user-agent string alone, since any client can claim to be GPTBot.