How to check whether AI crawlers can read your site

If ChatGPT, Claude, Perplexity or Google's AI features can't fetch your pages, they can't quote them. Access can be blocked at three different layers, and a site can pass one test while failing another.

You can run all three checks at once with our free AI crawler checker. Here is what it does and how to do it by hand.

Layer 1: robots.txt

Each AI company publishes the user-agent tokens its crawlers obey. The main ones:

• OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user-triggered fetches)
• Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-triggered)
• Perplexity: PerplexityBot, Perplexity-User
• Google: Googlebot for Search and AI Overviews; Google-Extended controls use for Gemini training
• Others: bingbot (Bing and Copilot), Applebot-Extended, CCBot (Common Crawl), meta-externalagent

A crawler follows the group that names it; if none does, it follows User-agent: *. Within the group, the most specific (longest) matching rule wins. Blocking training crawlers while allowing search crawlers is a legitimate choice; just make it on purpose.

Layer 2: your CDN's bot settings

CDNs such as Cloudflare offer one-click AI bot blocking and bot-management rules. These act before robots.txt is ever consulted. If someone enabled them, crawlers get a 403 or a challenge page no matter what robots.txt says.

Layer 3: the origin server

This is the one most audits miss. Hosting firewalls and security plugins often block user-agents they don't recognize. If your CDN caches the homepage, crawlers hitting it get a clean 200, while uncached pages (blog posts, new pages, URLs with query strings) go to the origin and get blocked.

On one site we audited, the cached homepage returned 200 to every crawler, but every blog post, the sitemaps and llms.txt returned 403 to ClaudeBot from the origin.

Testing by hand

Send a request with the crawler's user-agent, once to a cached page and once with a cache-busting query string:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "https://example.com/?nocache=1"

A 403, 429 or 503, or an HTML page saying "Just a moment" or "One moment, please", means that crawler is blocked or challenged.

One caveat: these tests use the crawler's name, not its real IP addresses. Firewalls that verify crawler IPs may let the real bot through. Treat a block as a strong signal and confirm it in your server or CDN logs.

Then help them understand you

Access is step one. An llms.txt file at your site root, a short Markdown summary linking to your key services, pricing and location pages, helps language models find what matters. Keep it curated: an auto-generated list of recent posts is not much help.