Robots.txt Validator + AI Crawler Check
Validate your robots.txt syntax and instantly see whether ChatGPT, Claude, Perplexity, and Google AI are allowed to crawl your site — the difference between being cited by AI or invisible to it.
Paste your robots.txt or fetch it by URL to validate + check AI crawler access.
Frequently asked questions
What does this robots.txt validator check?+
It parses your robots.txt, verifies the syntax of User-agent groups and Allow/Disallow rules, checks for a Sitemap directive, and — uniquely — shows whether each major AI crawler (ChatGPT, Claude, Perplexity, Google AI) is allowed or blocked.
Why does AI crawler access matter?+
To be cited in ChatGPT, Perplexity, or Google AI answers, those engines must be allowed to crawl your site. Many sites accidentally block GPTBot, ClaudeBot, or PerplexityBot, which makes them invisible to AI answer engines. This tool flags exactly which are blocked.
Which AI crawlers do you check?+
GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI); ClaudeBot and Claude-User (Anthropic); PerplexityBot; Google-Extended (Gemini and AI Overviews); Applebot-Extended; Amazonbot (Rufus); Meta-ExternalAgent; CCBot (Common Crawl); Bingbot (Copilot); and DuckAssistBot.
How do I allow AI crawlers?+
Add a User-agent group that permits them, e.g. "User-agent: GPTBot" followed by "Allow: /". Avoid a blanket "User-agent: *" with "Disallow: /", which blocks everything including AI engines.
Can I check a live robots.txt by URL?+
Yes. Use "Fetch by URL" and enter your domain or the robots.txt URL. You can also paste the contents directly. It is free, needs no signup, and runs in your browser.