ClaudeBot, Claude-SearchBot and Claude-User: what each does and how to set robots.txt
Anthropic documents three bots: ClaudeBot collects public content that may be used for model training, Claude-SearchBot crawls to improve search results, and Claude-User fetches pages when a Claude user's question needs it. Anthropic says its bots follow robots.txt, so you can opt out of training while leaving the other two allowed.
Last verified 28 October 2026 · Reviewed by Adarsh Kant
What does each Anthropic crawler do?
| Bot | Purpose per Anthropic | What Anthropic says disabling it does |
|---|---|---|
| ClaudeBot | Gathers public web content that may feed training of Anthropic's generative models | Signals that your future material should be left out of training datasets |
| Claude-SearchBot | Crawls the web to improve the quality and relevance of search results | Stops indexing for search optimisation; Anthropic says this may reduce your visibility and accuracy in user search results |
| Claude-User | Accesses sites when a Claude user asks something that needs a page | Stops retrieval for user queries; may reduce visibility in user-directed web search |
What robots.txt rules should I copy?
Anthropic's example blocks ClaudeBot with a Disallow of the whole site and says to add the rule for each subdomain you want covered. The snippets below use that same syntax.
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /How do I block every Anthropic bot, or slow ClaudeBot down?
# Scenario B
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
# Scenario C (instead of blocking ClaudeBot): slow it down
User-agent: ClaudeBot
Crawl-delay: 1What does blocking do, and not do?
- Anthropic warns against blocking its IP addresses instead of using robots.txt: that may stop the bot from reading robots.txt at all, so the opt-out may not work reliably.
- Anthropic says its bots do not try to get past measures such as CAPTCHAs.
- Opting out of ClaudeBot is described as applying to future material; the article does not say anything about data already collected.
- Hypothesis, not a sourced finding: allowing Claude-SearchBot and Claude-User may help your pages be retrievable in Claude answers. Anthropic states only that blocking them may reduce visibility.
How do I verify a request is from Anthropic?
- Take the client IP from your access log.
- Compare it to Anthropic's published list at claude.com/crawling/bots.json; Anthropic says a match means the traffic is theirs.
- Treat a matching user agent name with an unlisted IP as unverified.
- Check that your firewall allows the listed IPs to fetch /robots.txt, since Anthropic notes blocking the IPs can stop it reading your rules.
Honest limits
What we do not know.
- We have not published any analysis of CiterLabs' own server logs, so we make no claim about how often these bots visit real sites.
- Anthropic's help article does not list full user agent strings for the three bots, so we give none and do not provide a curl test string.
- The article does not say whether Claude-User honours Disallow in practice for a user-triggered fetch, beyond saying its bots follow robots.txt.
- It does not say how Claude-User or Claude-SearchBot treat Crawl-delay.
- We found no Anthropic statement that allowing Claude-SearchBot improves citation frequency.
- We did not test the snippets against live Anthropic crawlers, and we did not open the bots.json file itself.
Sources
- Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (opened 28 October 2026)
How this page was made: An AI assistant drafted this page from the cited vendor document, and it is awaiting human review.
Questions
- Can I opt out of training but stay in Claude search?
- Per Anthropic's description of the three bots, yes: they are separate user agents, so disallow ClaudeBot and leave the others allowed.
- Should I block Anthropic's IP addresses?
- Anthropic advises against it, because it may stop the bot reading robots.txt. Use robots.txt as the opt-out.
Free, no call needed
See who AI namesin your category today.
Tell us your domain and category. We run 10 buyer questions and send a short report with the raw answers attached.
Get a free AI Visibility CheckOr write to [email protected]