How to Verify an AI Crawler IP Address

How to verify an AI crawler IP address is a practical security question, not a user-agent lookup. A request can call itself GPTBot, OAI-SearchBot, PerplexityBot, or another crawler because the HTTP user-agent header is easy to copy. Reliable verification begins with the source IP recorded by infrastructure you trust and checks that address against the operator’s current official network ranges.

What “how to verify an AI crawler IP address” means in practical terms

Verification answers one narrow question: did this request originate from network space that the named crawler operator currently publishes? It does not prove that the crawler indexed the page, understood it, or will cite it. Those are later stages.

Keep identity, access, and content tests separate. Identity compares a trusted source IP with official ranges. Access checks whether the verified bot receives a usable response. Content testing checks whether the returned HTML carries the page’s essential meaning. Mixing these stages creates false conclusions, such as treating a successful spoofed curl request as proof of a real crawler visit.

Question	Required evidence	What it proves
Is the visitor genuine?	Trusted source IP plus official CIDR match	The request came from a published provider network.
Can it access the URL?	HTTP status, redirects, WAF event and final response	Infrastructure allowed or denied the request.
Can it use the page?	HTML, index controls and meaningful main text	The response is usable—not guaranteed to be selected.

Step 1 — Establish a clean baseline and choose representative URLs

Choose three URLs: the homepage, one important commercial page, and one recent article. This small sample helps reveal whether a failure affects the domain, a template, or one resource. Record each canonical URL, expected status, robots rule, and whether cookies, authentication, or JavaScript are required.

Capture a normal browser-style request. Preserve the timestamp, response code, redirect locations, content type, cache status, server or WAF request ID, and final URL. This is your control. Without it, a crawler failure can be mistaken for an ordinary outage, redirect loop, or origin error.

curl -sS -D headers.txt -o page.html -L https://example.com/important-page/

Step 2 — Compare a normal request with a controlled crawler-identity request

Repeat the request with the claimed crawler’s published user-agent format. This is a diagnostic simulation: it can expose a user-agent-based block, challenge, redirect, or altered HTML response. It cannot authenticate a real crawler because the connection still comes from your IP.

curl -sS -D bot-headers.txt -o bot-page.html -L \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://example.com/important-page/

Compare status, final URL, response size, title, canonical tag, robots meta directives, and primary text. A browser response of 200 and simulated bot response of 403 suggests a WAF or bot-management policy. Two 200 responses with missing main copy indicate a rendering or delivery problem—not an identity result.

Step 3 — Inspect the evidence and verify the source IP

Start with a real log event containing the claimed crawler user-agent. Extract the client address from the authoritative edge or origin field. Next, open the operator’s current official documentation and JSON range file. OpenAI publishes separate lists for OAI-SearchBot, GPTBot, and ChatGPT-User. Perplexity publishes separate lists for PerplexityBot and Perplexity-User. Keep these identities separate because they represent different product actions.

Crawler	Official verification source	Key distinction
OAI-SearchBot	OpenAI SearchBot JSON ranges	Search discovery; independent of GPTBot controls.
GPTBot	OpenAI GPTBot JSON ranges	Training crawler; not the ChatGPT Search control.
PerplexityBot	PerplexityBot JSON ranges	Automatic search crawler; separate from Perplexity-User.
Googlebot	Google crawler JSON or confirmed reverse DNS	Google documents both CIDR and DNS verification.

CIDR represents a network block. Verification passes when the logged address is contained in at least one current range. Do not compare text prefixes: IPv4 and IPv6 membership needs proper address arithmetic. Python’s standard library provides a safe one-off check:

python3 - <<'PY'
import ipaddress
source = ipaddress.ip_address("203.0.113.27")
published = ipaddress.ip_network("203.0.113.0/24")
print(source in published)
PY

For production automation, fetch the official endpoint on a schedule, validate its structure, build a new allow set, and swap it atomically only after validation. Retain the last known-good set so a timeout or malformed response does not erase working rules.

Step 4 — Apply the smallest safe fix and document the change

If a genuine crawler is blocked, identify the exact control responsible: robots.txt, firewall, managed bot rule, rate limit, geographic restriction, CDN challenge, origin ACL, or authentication layer. Change only the narrowest rule needed.

A safe WAF exception normally requires both the expected crawler identity and membership in the provider’s official IP set. Allowing every request containing a bot name creates an easy bypass. Allowing all traffic from a large cloud provider is also unnecessarily broad.

Finding	Smallest practical action	Risk to avoid
Official IP gets a 403 at the edge	Allow verified identity plus provider CIDR	Do not disable the WAF site-wide.
Edge allows but origin denies	Correct origin ACL or proxy IP handling	Do not trust arbitrary forwarded headers.
200 response has no useful body	Deliver essential content in accessible HTML	Do not treat status 200 alone as success.
IP is outside official ranges	Keep normal security policy; label unverified	Do not whitelist by name alone.

Step 5 — Retest with the same inputs and define a pass condition

Retest the same three URLs after caches and configuration changes propagate. Use identical methods and compare before-and-after evidence. A strong pass condition is explicit: the logged IP matches a current official range; the intended robots policy allows the crawler; edge and origin return the expected 200 without challenge; redirects terminate correctly; and final HTML contains the title, canonical URL, main copy, and index controls.

Worked example — inputs, observations, fix, and verified result

A SaaS company sees two requests claiming to be OAI-SearchBot. Its CDN log shows one source address inside the current OpenAI SearchBot list and another outside it. Both present nearly identical user-agent strings.

The team labels the first request verified and the second unverified. The genuine request receives a managed challenge and never reaches WordPress, while the browser control returns 200. The team creates a narrow edge rule that allows the OAI-SearchBot identity only when the address belongs to the refreshed official set. Unrelated traffic remains protected.

On retest, the verified request reaches the origin, returns 200, follows one canonical redirect, and receives the same substantive HTML as the control. The team saves the CDN event, source IP, matched CIDR, rule revision, final headers, and HTML comparison. That proves resolved access—not a future citation.

Evidence and screenshots to include

• An edge or origin log row with timestamp, request ID, path, user-agent, source IP, and status.
• The provider documentation URL and a dated snapshot or hash of the JSON ranges used.
• CIDR membership output showing the exact address and matching network.
• The relevant robots.txt group, including any wildcard rule that may apply.
• The CDN or WAF event showing the rule that allowed, blocked, or challenged the request.
• The complete redirect chain and final response headers before and after the fix.
• A content comparison confirming that main text and metadata reached the crawler response.

Common interpretation mistake — trusting the user-agent string

The most common error is to see a familiar bot name in analytics and assume the visitor is genuine. User-agent strings are claims, not credentials. A spoofed request may be harmless noise, a scraper, or an attacker testing whether named bots bypass security.

A second error is using a copied, stale address list. Providers update ranges and may separate automatic crawlers from user-triggered fetchers. Use the operator’s official current endpoint. Google also documents forward-confirmed reverse DNS: reverse-resolve the IP to an approved hostname, then resolve that hostname forward and confirm it returns the original address.

FAQs

Can I verify an AI crawler using only its user-agent?

No. Pair the claimed user-agent with the source IP from trusted infrastructure and the operator’s current published range.

Should I use reverse DNS for every AI crawler?

Only when the operator documents expected hostnames and forward confirmation. Google does. For providers publishing JSON CIDR ranges, direct membership is the clearer documented check.

Does a matching IP guarantee a citation?

No. It authenticates the network origin. The page must still be accessible and understandable, and the platform independently decides whether to crawl, index, select, or cite it.

How often should official lists be refreshed?

Treat them as changing operational data. Refresh automatically on a sensible schedule, monitor the provider documentation, validate updates, and keep a last known-good set.

What if CDN and origin logs show different client IPs?

Trace the proxy chain and rely only on documented client-IP fields. Misconfigured forwarding can cause the origin to test the CDN address rather than the visitor.

Next step — Get the AI Search Readiness checklist

After verifying identity and delivery, use the AI Search Readiness Checklist to assess rendering, index controls, entity clarity, citation signals, and platform evidence.

For related crawler controls, read ChatGPT-User vs GPTBot vs OAI-SearchBot and PerplexityBot Returns 403 on My Website.

Official references

• OpenAI crawler documentation and official IP endpoints
• Perplexity crawler, address-range, and WAF guidance
• Google CIDR and forward-confirmed reverse-DNS verification