Robots.txt Checker for AI Crawlers

Syntax and access. On syntax it flags an empty file, a UTF-8 BOM, lines without a colon, a missing User-agent group, a Sitemap that is not an absolute URL, a missing Sitemap directive, a Noindex line and any unrecognised directive. On access it resolves 14 AI agents, including GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Bingbot, to the one group each obeys and reports ALLOWED, PARTIAL or BLOCKED with the deciding rule.

A validator tests whether the file is well-formed; a checker tests what the file does to specific crawlers. This tool is both: the syntax checks are the validator and the AI crawler panel is the checker. Google’s robots.txt report in Search Console is a third thing, a log of what Googlebot fetched, and it says nothing about AI agents.

Because that crawler has its own named group, and a crawler obeys only the most specific group that matches it. If "User-agent: GPTBot" is followed by "Allow: /" and nothing else, GPTBot reads that one line and ignores every Disallow under "User-agent: *". Repeat the Disallow block inside the named group, or stack the User-agent lines so the agents share one group.

BLOCKED means the group that agent obeys contains Disallow: / and no Allow: /, so the whole site is closed. PARTIAL means the group disallows a narrower path such as /admin/ while the root stays open, the healthy state for most sites. ALLOWED means no Disallow at all: fine for a wildcard group with nothing to hide, suspicious for a named AI group that was supposed to inherit your private paths.

No. GPTBot is OpenAI’s training crawler. ChatGPT search results come from OAI-SearchBot and on-demand page reads from ChatGPT-User, both documented as separate from training. Block GPTBot and you are excluded from future training; block OAI-SearchBot and you disappear from ChatGPT search citations.

No. Google-Extended is an opt-out token read from Googlebot’s crawl, not a separate crawler, and it governs whether your pages are used for Gemini training and grounding. Google documents that it has no effect on inclusion or ranking in Search, and AI Overviews are part of Search. Applebot-Extended works the same way for Apple’s models.

Almost always a CDN or a plugin. Cloudflare’s managed robots.txt prepends a block marked with a "BEGIN Cloudflare Managed content" comment that disallows AI crawlers, and some SEO plugins inject rules of their own. Your repository file is unchanged; the served file is not. Use Fetch by URL to see the served version, then decide whether the CDN or your file owns the policy.

No. It judges each agent against the site root: Disallow: / is blocked, a narrower Disallow is partial, no Disallow is allowed. It does not evaluate a path against wildcard patterns, and it reads Disallow: / literally, so Disallow: /* reports as partial. For per-URL answers, Google’s open-source robots.txt parser runs the exact matching logic offline.

In paste mode, no; the parser runs in your browser and the text stays on your machine, so an unreleased file is safe to test. In Fetch by URL mode our server retrieves the public robots.txt at the domain you enter, because a browser cannot read another site’s file directly, and returns it to your browser for checking.

Yes. Content-Signal is a proposal for stating search, AI-input and AI-training preferences inside robots.txt, and this site’s own file carries one. The checker parses it as a known directive and does not warn on it. It does not enforce it, because the signal is a request to crawlers rather than an access rule, and support varies by vendor. Treat it as a statement of policy, and keep using Allow and Disallow for actual access control.