--- title: "AI crawler access guide: robots.txt and WAF | CiterLabs" url: "https://citerlabs.com/resources/ai-crawler-access-guide" description: "Table of AI crawlers by purpose and robots.txt stance, a decision table, a copy-paste robots.txt, CDN/WAF pitfalls and a 10-step curl test, from vendor docs." last_updated: "2026-10-28" --- # AI crawler access guide: who to allow, who to block, and how to test it Vendors now split AI crawling into training bots, search-index bots and user-initiated fetchers, and most document that only the first two follow robots.txt. To be citable in AI search while staying out of training, allow the search and user-fetch bots, disallow the training tokens, then confirm your CDN or WAF is not silently blocking the bots you allowed. The table, starter file and 10-step curl routine below do that in about 20 minutes. Last verified 28 October 2026 · Reviewed by Adarsh Kant - [Crawler access audit spreadsheet (XLSX) ↓](https://citerlabs.com/downloads/crawler-access-audit.xlsx)One row per crawler with user agent, purpose, doc URL and blank columns for your robots rule, WAF status, test date and notes. Contains no test results. ## Which AI-related crawlers exist, and what do vendors say about robots.txt? Only crawlers with an official vendor page we opened on 2026-10-28 are listed. The robots.txt column records what the vendor says, not what we observed. | Crawler / token | Vendor | Purpose (vendor) | Honours robots.txt? (vendor) | | --- | --- | --- | --- | | GPTBot | OpenAI | Training | Disallow signals no training use | | OAI-SearchBot | OpenAI | Search index (ChatGPT search) | Yes: allow to appear | | ChatGPT-User | OpenAI | User-initiated fetch | Rules may not apply | | ClaudeBot | Anthropic | Training | Yes; supports Crawl-delay | | Claude-SearchBot | Anthropic | Search index | Yes (bots follow robots.txt) | | Claude-User | Anthropic | User-initiated fetch | Yes (bots follow robots.txt); behaviour for user fetches not detailed | | Googlebot | Google | Search index; basis for AI Overviews and AI Mode eligibility | Yes, for automatic crawls | | Google-Extended (token) | Google | Controls Gemini training and grounding use; no own user agent | Token read in robots.txt | | PerplexityBot | Perplexity | Search index; stated not used for foundation-model training | Yes | | Perplexity-User | Perplexity | User-initiated fetch | Generally ignores robots.txt | | Applebot | Apple | Search features; may help train Apple models and AI answers | Yes; falls back to Googlebot rules; ignores crawl-delay | | Applebot-Extended (token) | Apple | Controls training use of Applebot data; does not crawl | Token read in robots.txt | | CCBot | Common Crawl | Open web corpus | Block with a CCBot Disallow | Bing and Microsoft: not tabled Microsoft's Bing crawler help pages did not render for us, so Bingbot is not in this table. Microsoft's Oct 2023 webmaster blog describes NOCACHE and NOARCHIVE meta tags for controlling Bing Chat usage: NOARCHIVE excludes content from chat answers and training, NOCACHE limits it to URL, title and snippet. ## What should I allow if I want to be cited, and block to stay out of training? Hypothesis: allowing the search bots is a prerequisite for being fetched by those products. We have found no vendor statement that it increases how often you are cited. | Your goal | Allow | Disallow | Caveat | | --- | --- | --- | --- | | Be eligible for AI search citations, no training use | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot | Google-Extended also covers Gemini grounding. Applebot may still use data for Apple features unless you add the token | | Maximum exposure | All of the above | Nothing | Training use follows | | Keep out of everything the vendors let you control | Nothing | All listed bots | User-initiated fetchers may ignore robots.txt; you need firewall rules for those | | Hide only some pages or text from Google AI features | Googlebot | Use nosnippet, max-snippet or data-nosnippet | See the Google-Extended page for scope | ## What is a safe robots.txt to start from? Starter: citable, not trainable. Edit to match the decision table ``` # Search / answer engines: allowed User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: / # Training opt-outs User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / # Everyone else User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` ## How can my CDN or WAF undo my robots.txt? - robots.txt only states a preference. A WAF rule, bot-fight setting or challenge page can return 403 or a CAPTCHA to a bot you allowed, so the bot never sees your Allow. - Cloudflare describes verified bots as bots it has confirmed are transparent about their identity and well-behaved, kept on a global allowlist. Its page does not say this exempts them from every WAF rule, and says customers can still set block or allow rules for AI bots. - Cloudflare's AI Crawl Control lets you allow or block individual crawlers from a Crawlers table and enforces blocks through a WAF custom rule it creates or edits on your zone. It is available on all plans; the free plan identifies crawlers by user agent only. - Cloudflare's managed robots.txt, enabled via Bot Preference Sync, prepends its directives to your existing file (or creates one) when at least one AI policy is set to block. Check your live file after enabling it. - Anthropic warns that IP-blocking its crawlers may stop them reading robots.txt at all. - Other CDNs and WAFs: we have not opened their documents, so check yours separately. ## What is a 10-step test routine with curl? 1. Fetch your live file: curl -s https://yourdomain/robots.txt and confirm it is the file you intended, over HTTP 200. 2. Confirm no CDN-injected managed block has changed it (diff against your repo copy). 3. Open the audit spreadsheet below and list each bot you allow or block. 4. For each, copy the user agent string from the vendor page linked in the sheet (not all vendors publish a full string; skip those). 5. Request your homepage with that string using the command below and note the status code. 6. Repeat for a deep URL and for /robots.txt itself; a 403 on robots.txt is the worst case. 7. Also request with a plain curl user agent as a control. A different result tells you the WAF is acting on user agent. 8. Open your CDN or WAF event log for the time of your test and see which rule fired; record the rule name. 9. Fetch each vendor's published IP list and compare it with real bot hits in your logs when you have them. A curl test from your own IP cannot prove the real crawler gets through if your CDN verifies bots by IP. 10. Record dates, fix rules, retest, and rerun monthly: vendors rename bots and move IP files. Status-code check per user agent (replace the string and URL) ``` UA="paste vendor user agent string here" for path in / /robots.txt /your-deep-page/; do printf "%s " "$path" curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" "https://example.com$path" done ``` Honest limits ## What we do not know. - We have not yet published an analysis of CiterLabs' own server logs, so we make no claim about real-world crawl frequency or compliance. - We could not open Microsoft's Bingbot and Copilot crawler help pages (they did not render), so Bingbot's documented stance and verification method are not covered here. - Whether vendors actually behave as documented is untested by us; the table records stated policy only. - Anthropic's page lacks full user agent strings; Apple's page gives only a short example string, so step 4 cannot be done for every row. - Cloudflare's pages do not state the default action for AI crawlers or whether verified bots bypass all WAF rules. - We have no source showing that any robots.txt or WAF tactic changes how often AI products cite a site. - Other CDNs and WAFs (Akamai, Fastly, AWS and similar) are not covered. - Common Crawl's corpus can be used by third parties we cannot enumerate. ## Sources 1. [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (opened 28 October 2026) 2. [Anthropic Help Center: crawling and blocking](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (opened 28 October 2026) 3. [Google Search Central: Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) (opened 28 October 2026) 4. [Google Search Central: Verifying Googlebot and other crawlers](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) (opened 28 October 2026) 5. [Perplexity: Bots guide](https://docs.perplexity.ai/guides/bots) (opened 28 October 2026) 6. [Apple: About Applebot](https://support.apple.com/en-us/119829) (opened 28 October 2026) 7. [Common Crawl: CCBot](https://commoncrawl.org/ccbot) (opened 28 October 2026) 8. [Bing Webmaster Blog: New options to control content usage in Bing Chat (Oct 2023)](https://blogs.bing.com/webmaster/october-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat) (opened 28 October 2026) 9. [Cloudflare: Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) (opened 28 October 2026) 10. [Cloudflare: AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) (opened 28 October 2026) 11. [Cloudflare: Manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) (opened 28 October 2026) 12. [Cloudflare: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) (opened 28 October 2026) How this page was made: An AI assistant drafted this page from the cited vendor documents, and it is awaiting human review. ## Questions Does a user-initiated fetcher respect robots.txt? OpenAI and Perplexity say rules may not apply or are generally ignored for user-requested fetches; Google says the same for its user-triggered fetchers. Anthropic says its bots follow robots.txt without separating Claude-User. Is blocking training bots going to hurt my citations? We found no vendor source saying so for the bots listed, but we have not tested it. Google-Extended is the one to read carefully because it also covers grounding. Related - [OpenAI crawlers and robots.txt](https://citerlabs.com/resources/gptbot-robots-txt) - [ClaudeBot and robots.txt](https://citerlabs.com/resources/claudebot-robots-txt) - [Google-Extended and robots.txt](https://citerlabs.com/resources/google-extended-robots-txt) - [AEO checklist](https://citerlabs.com/resources/aeo-checklist) - [GEO method](https://citerlabs.com/geo-method) Free, no call needed ## See who AI namesin your category today. Tell us your domain and category. We run 10 buyer questions and send a short report with the raw answers attached. [Get a free AI Visibility Check](https://citerlabs.com/ai-visibility-check) Or write to [hi@citerlabs.com](mailto:hi@citerlabs.com)