An AI crawler user agent list helps website owners identify automated requests from AI search, user-triggered retrieval and model-development systems. But a bot name in a log is only a claim: user-agent strings are easy to copy. A reliable policy combines the official token, the crawler’s purpose, robots.txt behavior, published IP information and evidence from your server or CDN logs.
Updated: 28 July 2026. Crawler documentation changes frequently. Treat this page as a practical reference, then confirm sensitive firewall or compliance decisions against the linked first-party documentation.
Before you use this AI crawler user agent list
Choose representative URLs before testing: your homepage, one commercial page, one knowledge article and any area you intentionally restrict. Gather the current robots.txt file, CDN or WAF events, origin access logs, HTTP response headers and the rendered HTML returned to each request. This baseline stops you from mistaking one successful homepage request for site-wide access.
Also separate three decisions. Search and answer discovery can create citation or referral opportunities. User-triggered retrieval fetches a page because a person requested it. Model-development or training controls govern a different use of content. Allowing one does not automatically require allowing the others.
AI crawler user agent list: verified tokens and purposes
| Operator | Token | Primary purpose | Verification/control |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT search discovery | robots.txt + published IP JSON |
| OpenAI | GPTBot | Potential model-development data | robots.txt + published IP JSON |
| OpenAI | ChatGPT-User | User-requested retrieval | Published IP JSON; robots rules may not apply |
| Anthropic | Claude-SearchBot | Claude search quality | robots.txt + official IP list |
| Anthropic | ClaudeBot | Potential model-development data | robots.txt + official IP list |
| Anthropic | Claude-User | User-requested retrieval | robots.txt policy + official IP list |
| Perplexity | PerplexityBot | Search indexing and links | robots.txt + published IP JSON |
| Perplexity | Perplexity-User | User-requested retrieval | Published IP JSON; generally ignores robots.txt |
| Google-Extended | Control token for Gemini/Vertex AI use | robots.txt token; not a standalone crawler | |
| Apple | Applebot-Extended | Control for foundation-model training | robots.txt token; does not crawl itself |
| Meta | Meta-ExternalAgent | AI model training or product improvement | robots.txt + official crawler guidance |
| Common Crawl | CCBot | Open web crawl dataset | robots.txt + operator documentation |
The table is intentionally purpose-led. For example, OpenAI documents OAI-SearchBot for ChatGPT search, GPTBot for content that may be used to improve foundation models, and ChatGPT-User for certain user actions. Anthropic likewise documents Claude-SearchBot, ClaudeBot and Claude-User as separate robots. Those distinctions matter when you want search visibility without granting every type of reuse.
Check 1 — match the exact crawler token
Search raw logs for the short product token rather than relying on a fixed full browser string. Versions and surrounding browser identifiers can change. OpenAI explicitly notes that version numbers may change, while Perplexity publishes full strings for PerplexityBot and Perplexity-User. Your detection rule should therefore identify the documented token and separately validate the request source.
OAI-SearchBot
GPTBot
ChatGPT-User
Claude-SearchBot
ClaudeBot
Claude-User
PerplexityBot
Perplexity-User
Meta-ExternalAgent
CCBotSecurity rule: Never allow a request through a WAF solely because its User-Agent header contains a trusted bot name. Any client can send “GPTBot,” “ClaudeBot” or “PerplexityBot.” Pair the token with the operator’s current IP ranges or prescribed DNS verification method.
Check 2 — test robots.txt and technical delivery
Read the robots.txt file from the exact hostname being tested. A rule on example.com does not automatically govern shop.example.com. Check whether a specific group matches the token, whether a later wildcard group changes the practical outcome, and whether the requested path is allowed. Then send a controlled request and record the final status after redirects.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /This example expresses a policy, not a guarantee. A CDN challenge, firewall rule, rate limit, geo restriction, authentication wall or origin error can still return 403, 429 or 5xx. A 200 response can also be misleading if it contains an empty shell, consent wall or JavaScript-only content that the crawler cannot use.
Check 3 — understand special control tokens
Some names commonly included in an AI crawler user agent list are not ordinary fetchers. Apple says Applebot-Extended does not crawl webpages; it controls whether content gathered by Applebot may be used to train Apple’s general-purpose foundation models. Google-Extended is also a robots.txt product token that controls whether content may be used to improve Gemini apps and Vertex AI generative APIs. Do not expect either token to appear as a standalone request in access logs.
Important: Blocking a training-control token is not the same as blocking search discovery. Keep your legal, content-licensing and visibility decisions separate, and document who approved each policy.
Check 4 — verify IP, HTTP response and server logs
For operators that publish IP ranges, retrieve the current JSON from the official endpoint and compare it with the source address seen by your edge. If the vendor prescribes reverse and forward DNS verification, complete both directions. Avoid copying a static range into documentation and forgetting it; IP allocations can change.
- Capture the source IP and complete User-Agent header at the CDN or origin.
- Match the source to the operator’s current published ranges or official DNS method.
- Confirm robots.txt allows the exact path and crawler token.
- Record redirects and the final HTTP status, not only the first response.
- Inspect returned HTML, canonical, robots meta directives and essential content.
- Check WAF events and origin logs for blocks, challenges, throttling or timeouts.
Prioritization: critical, important and improvement items
| Priority | Finding | Recommended action |
|---|---|---|
| Critical | Search crawler blocked across public content | Fix robots/WAF access and retest representative URLs |
| Critical | Spoofed bot is allowlisted by name only | Require official IP or DNS verification |
| Important | Crawler gets 200 but unusable or empty HTML | Fix rendering and content delivery |
| Important | Unexpected training crawler policy | Review with content/legal owner and set explicit rules |
| Improvement | No repeatable evidence or monitoring | Save logs, dates and scheduled verification |
Evidence to save for every crawler test
A defensible test record includes the robots.txt snapshot, request timestamp, tested URL, source IP, full User-Agent, verification result, redirect chain, final status, response headers, rendered or raw HTML, WAF event and origin log entry. Store the operator documentation URL and the date checked. This evidence lets another person reproduce the result after a platform or security configuration changes.
Common mistake: confusing a user-agent string with identity
The most damaging mistake is creating an allow rule such as “User-Agent contains GPTBot → bypass security.” That exposes the rule to spoofing. The opposite mistake is blocking an unfamiliar address without checking the operator’s current range, which can quietly remove search access. Identity is a chain of evidence: documented token, verified network source, expected behavior and matching logs.
Crawler access is only the first stage of AI visibility. A verified 200 response does not prove indexing, retrieval, mention or citation. Those later outcomes depend on platform behavior, content relevance, authority, freshness and the quality of the answer your page can support.
Frequently asked questions
Which AI crawler should I allow for ChatGPT search?
OpenAI identifies OAI-SearchBot as the crawler used to surface sites in ChatGPT search. GPTBot is a separate control for content that may be used to improve foundation models, while ChatGPT-User supports certain user-initiated actions.
Does allowing a crawler guarantee an AI citation?
No. Access makes retrieval possible, but it does not guarantee indexing, selection, citation or referral traffic. Test those outcomes separately with fixed prompts, dates, URLs and logs.
Can I verify an AI crawler by its User-Agent header?
Not by the header alone. User-agent strings are spoofable. Use the vendor’s official IP ranges or prescribed DNS process, and confirm the request in edge and origin logs.
Should I block all AI training crawlers?
That is a policy decision, not a universal SEO rule. Consider licensing, privacy, server load and strategic visibility. Where operators provide separate search, user-fetch and training controls, set each deliberately.
Official sources
- OpenAI crawler documentation
- Anthropic crawler guidance
- Perplexity crawler documentation
- Google crawlers and fetchers
- Applebot documentation
- Meta web crawler documentation
- Common Crawl CCBot documentation
Next step: check your website’s AI discoverability
Use this AI crawler user agent list to audit access on representative pages, then continue with the AI Crawler Access and Blocking Guide. Next, learn how to verify an AI crawler IP address and record a repeatable pass/fail result. Visible Pilot is being built to make these technical checks easier to run, explain and prioritize.
Leave a Reply