If PerplexityBot returns 403 on my website describes your problem, the request is reaching a system that refuses to serve it. The denial may come from Cloudflare, another CDN or WAF, a WordPress security plugin, the web server, or application code. This guide helps you identify the exact layer, make the smallest safe change, and prove that genuine Perplexity traffic can reach the intended public pages.
A 403 is not the same as a robots.txt disallow. Robots rules express crawl preferences; an HTTP 403 is an enforced response. Perplexity currently documents PerplexityBot as its search-indexing crawler and publishes its IP ranges. Treat user-agent and IP evidence together before changing security controls.
Fast answer: reproduce the 403, locate the matching security event or origin log, verify the request against Perplexity’s current published IP ranges, then allow only the necessary public paths. Retest the same URLs and keep dated evidence.
Quick diagnosis: why PerplexityBot returns 403 on my website
| Observation | Likely layer | Next check |
|---|---|---|
| Browser gets 200; crawler identity gets 403 | Bot rule, WAF, CDN or security plugin | Find the matching security event and rule ID |
| Both browser and crawler get 403 | Path permissions, authentication or origin rule | Test another public URL and inspect origin logs |
| robots.txt disallows the path but HTTP is 200 | Crawl policy, not transport blocking | Review the matching robots group |
| Edge says 200 but origin logs show no request | Cached response or edge handling | Inspect cache status and request trace |
| Only repeated requests fail | Rate limiting | Check threshold, period and mitigation duration |
The three-layer symptom map
Separate the problem into policy, delivery and visibility. Mixing them leads to fixes that look successful but change nothing.
- Policy: the applicable robots.txt group allows or disallows the requested path.
- Delivery: the CDN, WAF and origin return a usable 2xx response with the real page content.
- Visibility: Perplexity later discovers, selects and possibly cites the page. Access is necessary, but it does not guarantee inclusion.
Important: a curl request with a PerplexityBot user-agent is a controlled diagnostic, not proof that the request came from Perplexity. Anyone can copy a user-agent string.
Test 1: reproduce the 403 on representative URLs
Choose a homepage, a public commercial page and a useful article. Record the URL, timestamp, status, redirect chain, response headers and response size. This shows whether the failure is site-wide, template-specific or path-specific.
curl -I -L https://example.com/
curl -I -L https://example.com/service/
curl -I -L https://example.com/guides/example/If the browser baseline is already 403, solve ordinary public access first. If only one directory fails, inspect rules that match that path rather than adding a domain-wide allow.
Test 2: compare browser and crawler-identity requests
Repeat the same request with Perplexity’s documented user-agent. Keep method, URL and timing consistent so the identity header is the main changed variable.
curl -I -L \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
https://example.com/guides/example/A browser 200 paired with a crawler-identity 403 strongly suggests a bot-sensitive control, but it still does not reveal which one. Save useful headers such as server, cache status, request identifiers and any Cloudflare Ray ID, then find the same event in the security dashboard or server logs.
Root-cause checks from robots.txt to the origin
1. Check the applicable robots.txt group
Open /robots.txt and find a group for PerplexityBot. If none exists, inspect the wildcard group. Confirm the exact path, because a public article may be allowed while a search, account or parameterized path is intentionally restricted.
User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /internal-search/This example expresses a policy; it does not override a WAF. Do not expose private or low-value areas merely to eliminate an audit warning.
2. Inspect Cloudflare and WAF events
Search the event window using the request time, host, path, status and Ray ID. Identify the product and rule that acted: AI Crawl Control, a custom WAF rule, managed rules, Bot Fight Mode, Super Bot Fight Mode or rate limiting. Cloudflare documents that AI Crawl Control can deliberately return 403 and that its controls are enforced through WAF.
Do not stack a broad allow rule above every security control. First confirm that the intended crawler is genuine and that the requested path is safe for public retrieval.
3. Check origin and application controls
- Web-server deny rules, basic authentication or IP restrictions.
- WordPress security plugins that block unusual agents or countries.
- Application middleware requiring cookies, JavaScript or a session.
- Rate limits shared across crawler traffic, monitoring and API requests.
- Redirect destinations with stricter rules than the original URL.
Verify that the request is genuinely from Perplexity
Perplexity publishes separate JSON endpoints for PerplexityBot and Perplexity-User IP ranges and says those ranges can change. Use the current endpoint as the source of truth. Match both the expected user-agent and source IP range before creating an exception.
Security warning: never allow an arbitrary request merely because its user-agent contains “PerplexityBot.” Spoofing that string is trivial. A safe exception should combine authenticated network evidence with narrow path and method conditions.
Perplexity-User is different: it supports user-triggered page visits rather than ordinary crawling and, according to Perplexity, generally ignores robots.txt because a user requested the fetch. Diagnose the exact agent shown in the verified log instead of treating the two identities as interchangeable.
Fixes ordered by impact, effort and risk
| Fix | Impact | Effort | Risk |
|---|---|---|---|
| Correct an accidental PerplexityBot block in AI Crawl Control | High | Low | Low when intentional policy is confirmed |
| Add a narrow WAF exception using current IP ranges plus user-agent | High | Medium | Low to medium |
| Exclude public GET/HEAD pages from an overbroad rate limit | Medium | Medium | Medium |
| Remove a security-plugin bot block for verified traffic | Medium | Low | Medium |
| Disable bot protection across the entire site | High | Low | High—avoid unless temporary and controlled |
Prefer a path-limited, method-limited exception that can be logged and reversed. Preserve protection for login, checkout, account, admin, search-result and API endpoints unless Perplexity genuinely needs them.
Verification: the evidence that proves the 403 is resolved
Retest the same three URLs using the same commands. A defensible pass condition includes all of the following:
- The intended public URLs return 200–299 after redirects.
- The response contains the expected article or commercial content, not a challenge page.
- The matching edge and origin logs show which rule allowed the request.
- Verified source IP and user-agent evidence are stored with a timestamp.
- Sensitive paths remain blocked and ordinary security monitoring remains active.
After deployment, monitor fresh genuine requests rather than relying only on your spoofed test. Recheck Perplexity’s published ranges on a schedule so a static allowlist does not become stale.
When access is healthy but Perplexity still does not cite the site
A clean 200 response establishes delivery, not selection. Perplexity may not have discovered the URL yet, may prefer another source, or may judge the page less relevant to a particular query. Check indexable content, canonical and noindex controls, internal links, entity clarity, freshness and whether the page directly answers the target question.
Use the broader AI search readiness checklist and the guide to test whether a website is ready for AI search. These separate technical access from content and citation readiness.
Frequently asked questions
Does a 403 mean robots.txt blocked PerplexityBot?
No. A robots disallow is a crawl directive; a 403 is an HTTP refusal. Both can exist, but they must be diagnosed separately.
Should I allowlist PerplexityBot by user-agent?
Not by user-agent alone. Perplexity recommends combining its documented identity with the current IP ranges it publishes. Keep the exception narrow.
Can Cloudflare intentionally return 403 to AI crawlers?
Yes. Cloudflare AI Crawl Control supports allow and block actions, and a configured block can return 403. Custom WAF, bot and rate-limiting rules can also produce a denial.
Will a successful retest guarantee a Perplexity citation?
No. It proves the tested delivery barrier is removed. Discovery, relevance and source selection remain platform-dependent.
Next step: If PerplexityBot returns 403 on your website, capture one affected URL, its request time, response headers and matching security event. Then use the Visible Pilot checklist to verify the rest of the readiness path.
Official references
- Perplexity crawler documentation
- Current PerplexityBot IP ranges
- Cloudflare: manage AI crawlers
- Cloudflare: analyze AI crawler traffic
- Cloudflare WAF rate limiting rules
Reviewed 28 July 2026. Recheck current crawler documentation and IP ranges before changing production security rules.
Leave a Reply