AI crawler access guide: who to allow, who to block, and how to test it

Which AI-related crawlers exist, and what do vendors say about robots.txt?

Only crawlers with an official vendor page we opened on 2026-10-28 are listed. The robots.txt column records what the vendor says, not what we observed.

What should I allow if I want to be cited, and block to stay out of training?

Hypothesis: allowing the search bots is a prerequisite for being fetched by those products. We have found no vendor statement that it increases how often you are cited.

What is a safe robots.txt to start from?

# Search / answer engines: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# Training opt-outs
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

How can my CDN or WAF undo my robots.txt?

• robots.txt only states a preference. A WAF rule, bot-fight setting or challenge page can return 403 or a CAPTCHA to a bot you allowed, so the bot never sees your Allow.
• Cloudflare describes verified bots as bots it has confirmed are transparent about their identity and well-behaved, kept on a global allowlist. Its page does not say this exempts them from every WAF rule, and says customers can still set block or allow rules for AI bots.
• Cloudflare's AI Crawl Control lets you allow or block individual crawlers from a Crawlers table and enforces blocks through a WAF custom rule it creates or edits on your zone. It is available on all plans; the free plan identifies crawlers by user agent only.
• Cloudflare's managed robots.txt, enabled via Bot Preference Sync, prepends its directives to your existing file (or creates one) when at least one AI policy is set to block. Check your live file after enabling it.
• Anthropic warns that IP-blocking its crawlers may stop them reading robots.txt at all.
• Other CDNs and WAFs: we have not opened their documents, so check yours separately.

What is a 10-step test routine with curl?

1. Fetch your live file: curl -s https://yourdomain/robots.txt and confirm it is the file you intended, over HTTP 200.
2. Confirm no CDN-injected managed block has changed it (diff against your repo copy).
3. Open the audit spreadsheet below and list each bot you allow or block.
4. For each, copy the user agent string from the vendor page linked in the sheet (not all vendors publish a full string; skip those).
5. Request your homepage with that string using the command below and note the status code.
6. Repeat for a deep URL and for /robots.txt itself; a 403 on robots.txt is the worst case.
7. Also request with a plain curl user agent as a control. A different result tells you the WAF is acting on user agent.
8. Open your CDN or WAF event log for the time of your test and see which rule fired; record the rule name.
9. Fetch each vendor's published IP list and compare it with real bot hits in your logs when you have them. A curl test from your own IP cannot prove the real crawler gets through if your CDN verifies bots by IP.
10. Record dates, fix rules, retest, and rerun monthly: vendors rename bots and move IP files.

UA="paste vendor user agent string here"
for path in / /robots.txt /your-deep-page/; do
  printf "%s " "$path"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" "https://example.com$path"
done