--- title: "OpenAI crawlers and robots.txt: GPTBot guide | CiterLabs" url: "https://citerlabs.com/resources/gptbot-robots-txt" description: "What GPTBot, OAI-SearchBot and ChatGPT-User each do per OpenAI's docs, how robots.txt affects them, copy-paste allow/block rules and how to verify them." last_updated: "2026-10-28" --- # GPTBot, OAI-SearchBot and ChatGPT-User: what each does and how to set robots.txt OpenAI documents separate bots for separate jobs: GPTBot for crawling content that may train its models, OAI-SearchBot for surfacing sites in ChatGPT search, and ChatGPT-User for fetches a person triggers inside ChatGPT. Per OpenAI, the first two are controlled independently in robots.txt, while robots.txt rules may not apply to ChatGPT-User. So a site can opt out of training and still allow the search bot. Last verified 28 October 2026 ยท Reviewed by Adarsh Kant ## What does each OpenAI crawler do? The table paraphrases OpenAI's bot documentation as opened on 2026-10-28. OpenAI says user agent version numbers may change, so match on the product token (for example OAI-SearchBot), not the full string. | Bot | Purpose per OpenAI | robots.txt per OpenAI | IP list OpenAI publishes | | --- | --- | --- | --- | | OAI-SearchBot | Lets sites show up in ChatGPT search features | Allow it (and its IP ranges) if you want to appear; OpenAI says search systems can take about 24 hours to reflect a change | https://openai.com/searchbot.json | | GPTBot | Crawls content that may be used to train generative AI foundation models | Disallowing it signals your content should not be used for training; independent of OAI-SearchBot | https://openai.com/gptbot.json | | ChatGPT-User | Visits pages when a user asks ChatGPT or a custom GPT something, and for GPT Actions; not automatic crawling | OpenAI says robots.txt rules may not apply because a user initiated the action; not used to decide search inclusion | https://openai.com/chatgpt-user.json | | OAI-AdsBot | Checks landing pages submitted as ChatGPT ads; not used to train foundation models | Only visits submitted ad pages | https://openai.com/adsbot.json | ## What robots.txt rules should I copy? OpenAI's page describes the policy but gives no sample directives, so these are standard robots.txt syntax written by us. Pick one scenario. Each bot gets its own group. Scenario A: appear in ChatGPT search, opt out of training ``` User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: GPTBot Disallow: / ``` ## How do I block all OpenAI crawlers? Scenario B: block all documented OpenAI bots ``` User-agent: OAI-SearchBot Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / ``` Blocking ChatGPT-User may not be honoured OpenAI says robots.txt may not apply to user-initiated fetches. If you must stop them, a firewall rule on the user agent and the published IP list is the stronger lever, with the trade-offs described in the next section. ## What does blocking do, and not do? - Disallowing GPTBot signals that future crawls should not feed training. OpenAI does not say it removes content already collected. - Disallowing OAI-SearchBot means, per OpenAI, the site will not appear in ChatGPT search answers, though it may still show as a navigational link. - GPTBot and OAI-SearchBot are independent settings. OpenAI notes that if both are allowed, a single crawl may serve both purposes. - robots.txt is a request. OpenAI states its own bots' behaviour; it does not control other companies' crawlers that reuse public data. - Hypothesis, not a sourced finding: allowing OAI-SearchBot is necessary to be eligible for ChatGPT search, but we have found no OpenAI statement that it raises how often you are cited. ## How do I verify a request really came from OpenAI? 1. Take the client IP from your access log (behind a CDN, use the original-client header your CDN sets). 2. Download the relevant JSON from the table above and check whether the IP falls inside a listed range. 3. Treat a matching user agent with a non-listed IP as unverified; user agent strings are trivially copied. 4. Fetch /robots.txt as OAI-SearchBot would and confirm your firewall does not return 403 or a challenge page (see the access guide for a curl routine). 5. Re-download the IP JSON periodically; OpenAI does not state a schedule, so we do not assume the ranges are static. ## How can I test that my site serves the search bot? This only tests how your site reacts to a user agent header from your own IP. A CDN that verifies bots by IP may treat the real crawler differently, so a pass here is not proof. Curl check using OpenAI's documented OAI-SearchBot string (version may be outdated) ``` curl -s -o /dev/null -w "%{http_code}\n" \ -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \ https://example.com/robots.txt ``` Honest limits ## What we do not know. - We have not published any analysis of CiterLabs' own server logs, so we make no claim about how often these bots visit real sites. - OpenAI's page does not say whether disallowing GPTBot removes previously collected content from training sets. - We found no OpenAI statement that allowing OAI-SearchBot improves citation frequency or ranking inside ChatGPT answers. - OpenAI's page does not say how ChatGPT-User behaves for a robots.txt Disallow in practice, only that rules may not apply. - We did not test any of the snippets against live OpenAI crawlers. - Bot versions, user agent strings and IP list locations change; the page was checked on 2026-10-28 only. ## Sources 1. [OpenAI: Overview of OpenAI crawlers (bots)](https://developers.openai.com/api/docs/bots) (opened 28 October 2026) How this page was made: An AI assistant drafted this page from the cited vendor documents, and it is awaiting human review. ## Questions Does blocking GPTBot remove me from ChatGPT search? Per OpenAI, no: GPTBot and OAI-SearchBot are separate settings, and search inclusion is governed by OAI-SearchBot. Will ChatGPT-User obey robots.txt? OpenAI says robots.txt rules may not apply to it because a user initiates the visit. Plan on that basis. How long until a robots.txt change takes effect? OpenAI says its search systems can take about 24 hours to reflect a change for OAI-SearchBot. It gives no figure for the others. Related - [AI crawler access guide](https://citerlabs.com/resources/ai-crawler-access-guide) - [ClaudeBot and robots.txt](https://citerlabs.com/resources/claudebot-robots-txt) - [Google-Extended and robots.txt](https://citerlabs.com/resources/google-extended-robots-txt) - [AEO checklist](https://citerlabs.com/resources/aeo-checklist) Free, no call needed ## See who AI namesin your category today. Tell us your domain and category. We run 10 buyer questions and send a short report with the raw answers attached. [Get a free AI Visibility Check](https://citerlabs.com/ai-visibility-check) Or write to [hi@citerlabs.com](mailto:hi@citerlabs.com)