GPTBot, OAI-SearchBot and ChatGPT-User: what each does and how to set robots.txt
OpenAI documents separate bots for separate jobs: GPTBot for crawling content that may train its models, OAI-SearchBot for surfacing sites in ChatGPT search, and ChatGPT-User for fetches a person triggers inside ChatGPT. Per OpenAI, the first two are controlled independently in robots.txt, while robots.txt rules may not apply to ChatGPT-User. So a site can opt out of training and still allow the search bot.
Last verified 28 October 2026 · Reviewed by Adarsh Kant
What does each OpenAI crawler do?
The table paraphrases OpenAI's bot documentation as opened on 2026-10-28. OpenAI says user agent version numbers may change, so match on the product token (for example OAI-SearchBot), not the full string.
| Bot | Purpose per OpenAI | robots.txt per OpenAI | IP list OpenAI publishes |
|---|---|---|---|
| OAI-SearchBot | Lets sites show up in ChatGPT search features | Allow it (and its IP ranges) if you want to appear; OpenAI says search systems can take about 24 hours to reflect a change | https://openai.com/searchbot.json |
| GPTBot | Crawls content that may be used to train generative AI foundation models | Disallowing it signals your content should not be used for training; independent of OAI-SearchBot | https://openai.com/gptbot.json |
| ChatGPT-User | Visits pages when a user asks ChatGPT or a custom GPT something, and for GPT Actions; not automatic crawling | OpenAI says robots.txt rules may not apply because a user initiated the action; not used to decide search inclusion | https://openai.com/chatgpt-user.json |
| OAI-AdsBot | Checks landing pages submitted as ChatGPT ads; not used to train foundation models | Only visits submitted ad pages | https://openai.com/adsbot.json |
What robots.txt rules should I copy?
OpenAI's page describes the policy but gives no sample directives, so these are standard robots.txt syntax written by us. Pick one scenario. Each bot gets its own group.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /How do I block all OpenAI crawlers?
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /What does blocking do, and not do?
- Disallowing GPTBot signals that future crawls should not feed training. OpenAI does not say it removes content already collected.
- Disallowing OAI-SearchBot means, per OpenAI, the site will not appear in ChatGPT search answers, though it may still show as a navigational link.
- GPTBot and OAI-SearchBot are independent settings. OpenAI notes that if both are allowed, a single crawl may serve both purposes.
- robots.txt is a request. OpenAI states its own bots' behaviour; it does not control other companies' crawlers that reuse public data.
- Hypothesis, not a sourced finding: allowing OAI-SearchBot is necessary to be eligible for ChatGPT search, but we have found no OpenAI statement that it raises how often you are cited.
How do I verify a request really came from OpenAI?
- Take the client IP from your access log (behind a CDN, use the original-client header your CDN sets).
- Download the relevant JSON from the table above and check whether the IP falls inside a listed range.
- Treat a matching user agent with a non-listed IP as unverified; user agent strings are trivially copied.
- Fetch /robots.txt as OAI-SearchBot would and confirm your firewall does not return 403 or a challenge page (see the access guide for a curl routine).
- Re-download the IP JSON periodically; OpenAI does not state a schedule, so we do not assume the ranges are static.
How can I test that my site serves the search bot?
This only tests how your site reacts to a user agent header from your own IP. A CDN that verifies bots by IP may treat the real crawler differently, so a pass here is not proof.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
https://example.com/robots.txtHonest limits
What we do not know.
- We have not published any analysis of CiterLabs' own server logs, so we make no claim about how often these bots visit real sites.
- OpenAI's page does not say whether disallowing GPTBot removes previously collected content from training sets.
- We found no OpenAI statement that allowing OAI-SearchBot improves citation frequency or ranking inside ChatGPT answers.
- OpenAI's page does not say how ChatGPT-User behaves for a robots.txt Disallow in practice, only that rules may not apply.
- We did not test any of the snippets against live OpenAI crawlers.
- Bot versions, user agent strings and IP list locations change; the page was checked on 2026-10-28 only.
Sources
- OpenAI: Overview of OpenAI crawlers (bots) (opened 28 October 2026)
How this page was made: An AI assistant drafted this page from the cited vendor documents, and it is awaiting human review.
Questions
- Does blocking GPTBot remove me from ChatGPT search?
- Per OpenAI, no: GPTBot and OAI-SearchBot are separate settings, and search inclusion is governed by OAI-SearchBot.
- Will ChatGPT-User obey robots.txt?
- OpenAI says robots.txt rules may not apply to it because a user initiates the visit. Plan on that basis.
- How long until a robots.txt change takes effect?
- OpenAI says its search systems can take about 24 hours to reflect a change for OAI-SearchBot. It gives no figure for the others.
Free, no call needed
See who AI namesin your category today.
Tell us your domain and category. We run 10 buyer questions and send a short report with the raw answers attached.
Get a free AI Visibility CheckOr write to [email protected]