---
title: "Anthropic crawlers and robots.txt: ClaudeBot | CiterLabs"
url: "https://citerlabs.com/resources/claudebot-robots-txt"
description: "What ClaudeBot, Claude-SearchBot and Claude-User each do per Anthropic's help docs, with robots.txt allow/block snippets, Crawl-delay and IP verification."
last_updated: "2026-10-28"
---

# ClaudeBot, Claude-SearchBot and Claude-User: what each does and how to set robots.txt

Anthropic documents three bots: ClaudeBot collects public content that may be used for model training, Claude-SearchBot crawls to improve search results, and Claude-User fetches pages when a Claude user's question needs it. Anthropic says its bots follow robots.txt, so you can opt out of training while leaving the other two allowed.

Last verified 28 October 2026 · Reviewed by Adarsh Kant

## What does each Anthropic crawler do?

| Bot | Purpose per Anthropic | What Anthropic says disabling it does |
| --- | --- | --- |
| ClaudeBot | Gathers public web content that may feed training of Anthropic's generative models | Signals that your future material should be left out of training datasets |
| Claude-SearchBot | Crawls the web to improve the quality and relevance of search results | Stops indexing for search optimisation; Anthropic says this may reduce your visibility and accuracy in user search results |
| Claude-User | Accesses sites when a Claude user asks something that needs a page | Stops retrieval for user queries; may reduce visibility in user-directed web search |

## What robots.txt rules should I copy?

Anthropic's example blocks ClaudeBot with a Disallow of the whole site and says to add the rule for each subdomain you want covered. The snippets below use that same syntax.

Scenario A: stay findable in Claude search and user fetches, opt out of training

```
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /
```

## How do I block every Anthropic bot, or slow ClaudeBot down?

Scenario B: block all three; Scenario C line: Crawl-delay

```
# Scenario B
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

# Scenario C (instead of blocking ClaudeBot): slow it down
User-agent: ClaudeBot
Crawl-delay: 1
```

Crawl-delay is non-standard

Anthropic says it supports Crawl-delay but only shows an example for ClaudeBot; it does not state behaviour for Claude-User or Claude-SearchBot. Do not rely on it for those two.

## What does blocking do, and not do?

-   Anthropic warns against blocking its IP addresses instead of using robots.txt: that may stop the bot from reading robots.txt at all, so the opt-out may not work reliably.
-   Anthropic says its bots do not try to get past measures such as CAPTCHAs.
-   Opting out of ClaudeBot is described as applying to future material; the article does not say anything about data already collected.
-   Hypothesis, not a sourced finding: allowing Claude-SearchBot and Claude-User may help your pages be retrievable in Claude answers. Anthropic states only that blocking them may reduce visibility.

## How do I verify a request is from Anthropic?

1.  Take the client IP from your access log.
2.  Compare it to Anthropic's published list at claude.com/crawling/bots.json; Anthropic says a match means the traffic is theirs.
3.  Treat a matching user agent name with an unlisted IP as unverified.
4.  Check that your firewall allows the listed IPs to fetch /robots.txt, since Anthropic notes blocking the IPs can stop it reading your rules.

Honest limits

## What we do not know.

-   We have not published any analysis of CiterLabs' own server logs, so we make no claim about how often these bots visit real sites.
-   Anthropic's help article does not list full user agent strings for the three bots, so we give none and do not provide a curl test string.
-   The article does not say whether Claude-User honours Disallow in practice for a user-triggered fetch, beyond saying its bots follow robots.txt.
-   It does not say how Claude-User or Claude-SearchBot treat Crawl-delay.
-   We found no Anthropic statement that allowing Claude-SearchBot improves citation frequency.
-   We did not test the snippets against live Anthropic crawlers, and we did not open the bots.json file itself.

## Sources

1.  [Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (opened 28 October 2026)

How this page was made: An AI assistant drafted this page from the cited vendor document, and it is awaiting human review.

## Questions

Can I opt out of training but stay in Claude search?

Per Anthropic's description of the three bots, yes: they are separate user agents, so disallow ClaudeBot and leave the others allowed.

Should I block Anthropic's IP addresses?

Anthropic advises against it, because it may stop the bot reading robots.txt. Use robots.txt as the opt-out.

Related

-   [AI crawler access guide](https://citerlabs.com/resources/ai-crawler-access-guide)
-   [OpenAI crawlers and robots.txt](https://citerlabs.com/resources/gptbot-robots-txt)
-   [Google-Extended and robots.txt](https://citerlabs.com/resources/google-extended-robots-txt)
-   [Glossary](https://citerlabs.com/glossary)

Free, no call needed

## See who AI namesin your category today.

Tell us your domain and category. We run 10 buyer questions and send a short report with the raw answers attached.

[Get a free AI Visibility Check](https://citerlabs.com/ai-visibility-check)

Or write to [hi@citerlabs.com](mailto:hi@citerlabs.com)