ChatGPT-User, GPTBot, and OAI-SearchBot are not three names for the same crawler. They represent different OpenAI web-access purposes: a user-triggered visit, automatic search discovery, and potential model-training collection. The right configuration depends on the outcome you want—not on whether you broadly “allow AI bots.”
Decision in one sentence: allow OAI-SearchBot if you want content eligible for ChatGPT search summaries and citations; set a separate GPTBot policy for potential model training; and treat ChatGPT-User as user-triggered access that is not the control for search inclusion.
Short answer: ChatGPT-User vs GPTBot vs OAI-SearchBot
| Agent | Primary purpose | Automatic crawler? | Main site-owner decision | Does it control ChatGPT search inclusion? |
|---|---|---|---|---|
| OAI-SearchBot | Discover and retrieve public pages for ChatGPT search features | Yes | Allow or disallow search crawling by path | Yes—the relevant OpenAI crawler control |
| GPTBot | Crawl content that may be used to improve and train OpenAI generative models | Yes | Allow or opt out of potential training use | No |
| ChatGPT-User | Visit a page for certain actions initiated by a ChatGPT or Custom GPT user | No, not an automatic web crawler | Make user-requested pages technically reachable when appropriate | No |
These settings are independent. A publisher can allow OAI-SearchBot while disallowing GPTBot. That combination supports search discoverability while expressing that the site’s content should not be collected for model training. OpenAI also notes that when both are allowed, it may avoid duplicate fetching by using one crawl for both purposes; the policy signals remain separate.
Definitions and boundaries
OAI-SearchBot: the search-discovery crawler
OAI-SearchBot is the agent website owners should evaluate when the goal is visibility in ChatGPT search. OpenAI says sites opted out of OAI-SearchBot will not be shown inside ChatGPT search answers, although a navigational link may still appear in limited circumstances. Allowing it removes an access barrier; it does not guarantee ranking, selection, a quotation, or a citation.
A valid robots.txt allowance is only the first layer. The host, CDN, WAF, bot manager, rate limiter, authentication rules, redirects, and origin server must also let requests from OpenAI’s published search-bot IP ranges reach useful HTML. OpenAI advises that robots.txt changes may take roughly 24 hours to be reflected by its systems.
GPTBot: the model-training crawler
GPTBot is used to crawl content that may be used to make OpenAI’s generative AI foundation models more useful and safe. Disallowing GPTBot communicates that the affected site content should not be used for training those models. It is not the search opt-out switch, and blocking it does not by itself block OAI-SearchBot.
ChatGPT-User: a user-triggered visit
ChatGPT-User is used for certain actions initiated by people using ChatGPT or Custom GPTs, such as asking the product to visit a particular page. OpenAI states that it is not used for automatic web crawling and is not used to determine whether content can appear in Search. Because the request is user initiated, robots.txt rules may not apply in the same way they do to automatic crawlers.
Important: do not write a robots.txt policy for ChatGPT-User and assume you have configured ChatGPT search. Search inclusion is managed with OAI-SearchBot; potential training collection is managed separately with GPTBot.
Side-by-side comparison: purpose, control, evidence, and limits
| Comparison point | OAI-SearchBot | GPTBot | ChatGPT-User |
|---|---|---|---|
| Trigger | OpenAI automatic crawl for search | OpenAI automatic crawl for potential training use | A user action in ChatGPT or a Custom GPT |
| robots.txt relevance | Primary site-owner control | Primary site-owner control | May not apply because the request is user initiated |
| Published IP list | openai.com/searchbot.json | openai.com/gptbot.json | openai.com/chatgpt-user.json |
| Useful evidence | robots decision, verified IP, 2xx response, rendered content, server/CDN logs | robots decision, verified IP, server/CDN logs | reproducible user action, request log, response and page usability |
| What success proves | The crawler can access the tested resource | The training crawler can access the tested resource | A user-triggered visit can retrieve the tested resource |
| What success does not prove | That ChatGPT will rank or cite the page | That the page appears in ChatGPT search | That the site is crawled automatically or eligible for search |
| Direct monetary cost | No crawl fee from OpenAI; operational work may have a cost | No crawl fee from OpenAI; policy review may have a cost | No crawl fee from OpenAI; site delivery costs may apply |
Discovery and access implications
Website owners often collapse five different states into a single idea of “AI visibility.” Keep them separate:
- Policy: does robots.txt allow the relevant automatic crawler on this path?
- Network access: does the request survive CDN, firewall, WAF, rate-limit, geo, and bot-management rules?
- Delivery: does the final URL return a successful response and useful HTML rather than a challenge, redirect loop, login wall, or empty shell?
- Eligibility: do index controls and page-level directives permit the intended use?
- Selection: is the content relevant, clear, trustworthy, and useful enough to be surfaced for a particular query?
OAI-SearchBot access affects the first four stages, but the fifth remains query- and system-dependent. GPTBot concerns a different downstream use. ChatGPT-User tests whether a user-triggered retrieval works; it cannot substitute for automatic search-crawler evidence.
Robots.txt configurations for common policies
Use explicit groups when you want the intent to remain readable during reviews. The following examples are policy illustrations; confirm them against your own legal, privacy, content, and infrastructure requirements.
Allow search discovery but opt out of model training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Allow both automatic OpenAI crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /Allow public content but block a private section
User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/
User-agent: GPTBot
Disallow: /Do not put secrets behind robots.txt. It is a crawler instruction, not access control. Protect private material with authentication and authorization, and avoid exposing sensitive URLs in public files.
Measurement, evidence quality, and repeatability
A reliable crawler test records evidence at every layer. A screenshot of robots.txt alone is weak because a correct directive cannot show whether a request was blocked later by a CDN or origin rule. A successful curl request with a copied user-agent is also incomplete because anyone can spoof that string.
| Evidence | What it can establish | Strength | Main limitation |
|---|---|---|---|
| robots.txt snapshot with timestamp | Published policy for a named agent and path | Useful | Does not prove network or origin access |
| Controlled request with relevant user-agent | How your stack treats that identity at test time | Useful diagnostic | Does not prove the requester is genuinely OpenAI |
| CDN/WAF event | Security action, rule, status, challenge, or rate limit | Strong | May not show origin rendering quality |
| Origin/server access log | Final request path, response status, time, and bytes | Strong | Identity still needs network verification |
| Published-IP match | Source IP belongs to the current OpenAI range list | Strong identity evidence | Ranges can change; use current data |
| Rendered response comparison | Crawler receives materially useful content | Strong delivery evidence | Still does not prove ranking or citation |
Best policy by scenario
| Scenario | Recommended focus | Why |
|---|---|---|
| You want pages surfaced and cited in ChatGPT search | Allow and verify OAI-SearchBot | It is the relevant automatic search crawler |
| You want search visibility but not potential model-training use | Allow OAI-SearchBot; disallow GPTBot | The controls are independent |
| You want to permit potential training collection too | Allow OAI-SearchBot and GPTBot | Both intended automatic uses are permitted |
| A user says ChatGPT cannot open a URL | Test the user-triggered path and page accessibility | This is not automatically an OAI-SearchBot problem |
| Your robots.txt is correct but logs show 403 or 429 | Audit CDN/WAF/rate limits and published IP allowlisting | Downstream infrastructure is overriding the apparent policy |
| There are no verified crawler requests in logs | Check discovery paths, sitemaps, internal links, log retention, and the correct IP ranges | Absence of a log event is not proof of a block |
Combined workflow: use the three signals without mixing them up
- Define the business policy. Decide separately whether you want ChatGPT search eligibility and whether you permit potential model-training use.
- Publish explicit automatic-crawler rules. Configure OAI-SearchBot and GPTBot independently in robots.txt.
- Check delivery beyond robots.txt. Review redirects, final status, response body, canonical/noindex controls, CDN rules, WAF challenges, rate limits, and authentication.
- Verify genuine traffic. Match observed source IPs against the current official JSON lists rather than trusting the user-agent alone.
- Test user-triggered access separately. Reproduce the specific ChatGPT-User action and record what the page returned.
- Retest the same URLs. Keep the URL, timestamp, request identity, source IP, status, response size, security event, and rendered-content observation together.
Test it yourself with a controlled request
A controlled user-agent request helps reveal whether your infrastructure treats the identities differently. It is a diagnostic simulation—not proof of a real OpenAI crawl.
curl -I -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://example.com/page/
curl -I -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://example.com/page/
curl -I -A "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" https://example.com/page/Repeat the checks on a homepage, a commercial page, and a knowledge article. Compare the final status, redirect chain, important headers, response size, and whether the meaningful page content exists in the returned HTML. If a normal browser gets 200 but one simulated identity receives 403, 429, a JavaScript challenge, or a much smaller body, inspect the matching CDN and WAF events.
Pass condition: the intended agent is explicitly permitted where robots.txt applies, the request reaches the correct final URL, returns a stable 2xx response without a challenge, and delivers the page’s meaningful content. For genuine traffic, the source IP must also match OpenAI’s current published range.
Evidence to capture in a real audit
- The exact robots.txt group and path decision, saved with date and time.
- The requested URL, redirect chain, final HTTP status, response headers, and response size.
- The CDN or WAF event ID, matched rule, action, and any bot score or challenge.
- The source IP checked against the correct current OpenAI JSON range.
- The relevant origin log line, with sensitive information redacted.
- A rendered-content comparison showing whether the title, main copy, links, structured data, and canonical/index controls are present.
- The fix applied, deployment time, cache state, and result from the same retest.
Common mistake: trusting the user-agent string
A user-agent is a self-declared label. A malicious scanner, SEO tool, or test command can claim to be GPTBot, OAI-SearchBot, or ChatGPT-User. Do not create a broad security bypass based only on that string. Verify the source IP against the published range for the claimed agent, keep the list current, and scope any infrastructure exception as narrowly as possible.
The reverse mistake is also common: teams allow an IP range at the firewall but leave a conflicting robots rule, origin deny rule, redirect, noindex directive, login requirement, or empty client-rendered response. Authenticity and access are separate checks; both must pass.
Frequently asked questions
Can I allow OAI-SearchBot and block GPTBot?
Yes. OpenAI documents the controls as independent. This is the clearest configuration when you want eligibility for ChatGPT search but do not want the affected content collected for potential model training.
Does blocking GPTBot remove my site from ChatGPT search?
Not by itself. OAI-SearchBot is the relevant crawler for ChatGPT search. Confirm that it is allowed and that your infrastructure accepts requests from its published IP ranges.
Does allowing OAI-SearchBot guarantee a citation?
No. It removes an important access barrier, but selection depends on relevance, reliability, content quality, query fit, system behavior, and other signals.
Should ChatGPT-User have a robots.txt group?
Do not use it as a substitute for OAI-SearchBot configuration. OpenAI describes ChatGPT-User as user initiated and says robots.txt rules may not apply. Test the particular user-triggered retrieval separately.
How quickly will a robots.txt change take effect?
OpenAI says its systems may take about 24 hours to adjust after a robots.txt update. Cache and infrastructure changes can have their own propagation times, so record deployment and retest timestamps.
How do I know a request really came from OpenAI?
Match the observed source IP to the current official JSON range for the claimed agent. Combine that with the request path, timestamp, user-agent, CDN/WAF event, and origin log rather than relying on any single field.
Next step: check your website’s AI discoverability
Start with the policy you actually want, then test the complete delivery path. Use the GPTBot crawl test for a deeper diagnostic procedure and the AI search readiness checklist to evaluate access, rendering, index controls, content clarity, and observable platform evidence together.
Official references
Reviewed against OpenAI’s official documentation on 28 July 2026. Crawler names, example version numbers, IP ranges, and product behavior can change; verify current documentation before applying production rules.
Leave a Reply