Synthetic Traffic
Web traffic where the primary visitor is an AI agent or crawler acting on behalf of a user.
What is Synthetic Traffic?
Synthetic traffic is the growing share of requests to a website that come from automated agents rather than people in a browser. In the AI era that includes several distinct kinds of visitor. Training crawlers such as GPTBot, CCBot and Google-Extended fetch pages to build or refresh model training data. Retrieval agents such as OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot fetch pages at answer time so an assistant can ground and cite a response. Task agents go further, browsing, filling forms, comparing products or completing purchases on a user's instruction.
Each type needs a different policy. Blocking training crawlers is a legitimate choice about how content may be reused, but blocking retrieval agents removes the site from AI answers entirely, and many robots.txt files do the second by accident while intending the first. Task agents raise their own questions about rate limits, bot detection and whether a site's checkout or forms work without a human in the loop.
Synthetic traffic also needs to be measured separately. Server logs, CDN analytics and bot-management tools can identify these agents by user agent and, for the major ones, by published IP ranges. Client-side analytics usually cannot see them at all, which is why a site can be cited heavily by AI engines while its web analytics show nothing.
Why it matters for AI search
A page an AI agent cannot fetch cannot be cited, recommended or purchased from. Getting synthetic traffic policy right, allowing retrieval agents, deciding deliberately about training crawlers, and making sure pages render without JavaScript, is the technical foundation of AI visibility. Log-level measurement of which agents visit which pages is also one of the few direct signals of whether GEO work is causing engines to retrieve the site more often.
Related terms
LLM.txt
A proposed standard file (similar to robots.txt) that provides a Markdown-formatted map of a website for AI crawlers to consume efficiently.
RAG (Retrieval-Augmented Generation)
A technique that allows LLMs to pull in real-time information from external sources (like a website) before generating an answer.
LLM (Large Language Model)
A type of AI trained on vast amounts of text data to understand and generate human-like language (e.g., GPT-4, Claude 3).
AEO (Answer Engine Optimization)
The practice of optimizing content to be cited and recommended by AI-driven search engines and conversational assistants like ChatGPT and Perplexity.
Robots.txt
A file that tells web crawlers (including AI bots) which pages to access or avoid. Critical for managing AI visibility.
Frequently asked questions
Should I block AI crawlers?+
Decide per crawler type. Blocking training crawlers (GPTBot, CCBot, Google-Extended) limits reuse of your content for model training. Blocking retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot) removes you from AI answers. Most brands that want AI visibility allow retrieval agents.
Why does my analytics tool not show AI crawler visits?+
Most AI agents do not execute JavaScript, so tag-based analytics never fires. Use server logs, CDN analytics or bot-management reporting to see them.