Is Your Site Blocking AI Crawlers? How to Check and Fix It

Your website might be invisible to ChatGPT, Perplexity, and Gemini right now -- and you would have no idea.

Here is the uncomfortable reality: 79% of major news publishers block at least one AI training bot. That part is intentional. What is not intentional is that 71% of those same sites simultaneously block AI search and retrieval bots -- the ones that actually drive referral traffic and citations.

The result? They thought they were protecting their content from being scraped for model training. Instead, they erased themselves from the fastest-growing search channel on the web.

This guide walks you through how to check whether your site is blocking AI crawlers, understand which bots you actually need to allow, and fix the problem in under 30 minutes.

Why This Matters More Than You Think

AI search is not a future trend. It is a current traffic source -- and it is accelerating fast.

Adobe reported a 10x increase in AI referral traffic between July 2024 and February 2025. ChatGPT referrals grew 52% year-over-year. Gemini referrals grew 388% YoY.

Across hundreds of thousands of AI responses we analyzed at Asva AI, company websites directly drive 21.3% of all AI citations -- the single biggest lever brands control. But that lever only works if AI bots can actually reach your pages.

Every day your site blocks AI search crawlers is a day you are conceding citations, referrals, and brand visibility to competitors who have their access configured correctly.

The 3-Tier Bot Architecture You Need to Understand

Most people treat "AI crawlers" as a single category. They are not. Every major AI company operates three distinct bot types, and each one serves a different purpose:

Provider	Training Bot	Search Bot	User-Browse Bot
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Google	Google-Extended	Googlebot	Googlebot

Google-Extended controls Gemini training access. Blocking it does NOT affect Googlebot, Google Search, or AI Overviews.

This distinction is critical. Blocking GPTBot in your robots.txt stops OpenAI from using your content for model training. It does not stop your site from appearing in ChatGPT search results -- that is handled by OAI-SearchBot and ChatGPT-User.

But most robots.txt configurations do not make this distinction. They block everything from a given provider, throwing away search visibility along with training access.

The Cloudflare Problem

If your site runs behind Cloudflare, pay close attention.

Cloudflare serves approximately 20% of all websites globally. In July 2024, Cloudflare rolled out a one-click toggle to block AI bots -- and defaulted it to ON for many configurations.

The result: sites on Cloudflare block AI bots at 2.3x the baseline rate compared to non-Cloudflare sites.

Many site owners never consciously chose this. The toggle was enabled during a platform update, and unless someone in your team actively checked the Bots settings in the Cloudflare dashboard, your site may have been blocking AI crawlers for months without anyone noticing.

This is the single most common "hidden" cause of AI crawler blocking we see across the sites we analyze.

6-Step Audit: Is Your Site Blocking AI Crawlers?

Here is a step-by-step checklist to diagnose whether your site is accidentally blocking AI crawlers.

Step 1: Check Your robots.txt

Navigate to yourdomain.com/robots.txt and look for any Disallow directives targeting these user agents:

• GPTBot
• OAI-SearchBot
• ChatGPT-User
• ClaudeBot
• Claude-SearchBot
• Claude-User
• Google-Extended
• PerplexityBot
• Bytespider

If you see a blanket block like this, you have a problem:

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /

That third rule is blocking your site from ChatGPT search results -- likely not what you intended.

Step 2: Test With a Live Fetch

Use curl to simulate an AI crawler request and check the HTTP response:

curl -A "OAI-SearchBot" -I https://yourdomain.com/

A 200 status means access is granted. A 403 or 401 means your server or CDN is blocking the request at the infrastructure level, independent of robots.txt.

Step 3: Check Cloudflare AI Bot Settings

If you use Cloudflare:

• Log in to your Cloudflare dashboard.
• Navigate to Security > Bots.
• Look for the AI Bots toggle.
• If it is enabled (blocking), you need to decide which bots to allow.

Cloudflare's bot management operates independently of your robots.txt, so both must be configured correctly.

Step 4: Check Server-Level Blocks

Review your server configuration for AI bot blocks:

• Apache: Check .htaccess for RewriteCond %{HTTP_USER_AGENT} rules targeting AI bots.
• Nginx: Check nginx.conf for if ($http_user_agent ~* "GPTBot|ClaudeBot") blocks.
• WAF rules: Check any Web Application Firewall rules that may be filtering by user agent.

Step 5: Verify Meta Robots Tags

Check your key pages for meta robots tags that could prevent indexing:

While these primarily affect traditional search engines, some AI crawlers respect them as well. Ensure your high-value pages are not inadvertently tagged.

Step 6: Run an AI Visibility Check

The ultimate test: query your brand name and core topics in ChatGPT, Perplexity, and Gemini. If your site never appears in citations while competitors do, you likely have an access problem.

For a systematic approach, audit your AI visibility across all major platforms.

The Selective Blocking Template

You do not have to choose between "block everything" and "allow everything." The smart approach: block training bots, allow search and browse bots.

Here is a robots.txt template you can adapt:

# ============================================
# AI TRAINING BOTS - BLOCKED
# Prevents content from being used in model training
# ============================================
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
# ============================================
# AI SEARCH & BROWSE BOTS - ALLOWED
# Enables your site to appear in AI search results
# ============================================
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /

This configuration protects your content from being ingested into training datasets while keeping your site visible in AI-powered search results. It is the approach we recommend for most brands.

Important: If you use Cloudflare, you must also configure bot access there. Robots.txt alone is not enough -- Cloudflare's infrastructure-level blocking takes precedence.

What to Do After You Fix It

Unblocking AI crawlers is step one. Step two is monitoring whether it works.

After updating your configuration:

Re-test with curl to confirm search bots now receive 200 responses.

Wait 1-2 weeks for AI platforms to re-crawl your site.

Monitor your AI citations to see if your site starts appearing in results.

Understanding how LLMs discover brands will help you go beyond fixing access issues and start actively optimizing for AI visibility.

If you want to track which AI platforms cite your brand, which queries trigger your content, and where you are losing citations to competitors -- see what Asva AI shows you.