AI Crawler HTTP Status Code Benchmark

The AI crawler HTTP status code benchmark shows how often automated AI requests reach content, redirect, hit an access rule, or fail at the server. In Cloudflare’s network-wide snapshot published on 17 April 2026, about 74% of observed AI-crawler requests received a successful 2xx response. The remaining requests were split across redirects, client errors, and server errors—a reminder that crawl visibility is partly an HTTP delivery problem, not only a content problem.

Key AI crawler HTTP status code benchmark findings

• 74% returned 2xx: most observed AI-crawler requests were served successfully.
• 26% were not successful responses: redirects and errors together represented roughly one request in four, using the rounded published groups.
• 13.7% returned 4xx: client-error responses formed the largest non-2xx class.
• 11.3% returned 3xx: redirects were common enough to affect crawl efficiency and canonical delivery.
• 1.2% returned 5xx: server failures were the smallest group, but they usually deserve the fastest operational response.
• 403 alone represented 8.3%: blocking or denied access explained a large portion of the aggregate response mix.
• GPTBot reached 83% 2xx: its published subset had more successful responses and fewer redirects than the aggregate AI-crawler sample.

Methodology and limitations

This Visible Pilot report is a dated synthesis of Cloudflare Radar data, not a claim that Visible Pilot crawled the entire web. Cloudflare identifies verified AI bots and crawlers across traffic handled by its network, then groups the HTTP responses they receive. The published example reported individual codes—including 200, 301/302, 403, and 404—and broader 2xx, 3xx, 4xx, and 5xx classes. We use those published shares and calculate only simple percentage-point comparisons.

The source did not publish an absolute request count, number of websites, CMS distribution, company-size breakdown, or a fixed domain list for the illustrated snapshot. Rounded class shares sum to 100.2%, which is normal rounding rather than extra traffic. Coverage is limited to requests visible to Cloudflare, and the mix reflects crawler activity as well as website configuration. These constraints make the dataset useful for operational benchmarking, but not for estimating the share of all websites accessible to AI.

Overall response results

The dominant outcome was 2xx success at nearly 74%. At the individual-code level, just over 70% of requests returned 200 OK. A successful status is necessary for retrieving a page, but it does not prove that the returned body contains the intended content, renders correctly, is indexable, or will be cited by an AI system.

Client errors were the largest non-success class at 13.7%. Cloudflare reported 403 Forbidden for 8.3% of requests and 404 Not Found for 3.7%. Redirects accounted for 11.3% as a class, with 301 and 302 responses together representing 10.1%. Finally, 5xx server errors made up 1.2%. The pattern suggests that access policy, stale URLs, and redirect architecture are more common friction points than outright origin failure.

Breakdown by crawler and technical context

GPTBot’s subset differed materially from the all-crawler view. Its requests received 83% 2xx, nearly 10% 4xx, 5.1% 3xx, and 2.2% 5xx. At the named-code level, just over 80% returned 200, 4.7% redirected, 2.7% returned 404, and nearly 6% returned 403. Compared with the aggregate sample, GPTBot gained about nine percentage points in 2xx and encountered 6.2 points fewer redirects, while its 5xx share was one point higher.

This does not prove one crawler is inherently better. Request paths, crawl purpose, site mix, verification, rate, and timing can all change the distribution. Cloudflare Radar can filter by industry set and crawl purpose, and zone owners can examine crawler, operator, hostname, and path. A responsible internal benchmark should segment along those dimensions before attributing a difference to the bot itself.

Failure-pattern analysis

• 403 plus low 2xx: inspect AI Crawl Control, WAF, bot-management, IP or signature verification, and origin allow rules. A 403 may be intentional policy or an accidental false positive.
• 404 plus redirects: audit deleted URLs, malformed links, sitemap freshness, canonical targets, and redirect chains. Crawlers may be discovering outdated paths faster than your redirects are maintained.
• 429 bursts: compare request rate with edge and origin limits. Rate limiting should be measured separately from permanent blocking and paired with a clear retry policy.
• 5xx across several bots: prioritize origin capacity, timeouts, application errors, and CDN-to-origin failures. Widespread 5xx is more likely an availability problem than a crawler-specific rule.
• 200 with weak visibility: verify the response body, content type, rendered HTML, canonicals, robots directives, and meaningful page content. Status success alone is not search success.

What website owners should do

Start with your own logs. Calculate status shares for verified AI crawlers over a stable period, then compare them with ordinary search crawlers and human traffic. Review top paths inside every non-2xx class. A redirect from an old URL to its canonical replacement may be healthy; a chain through three URLs is avoidable waste. A planned 403 can express policy; an unexpected 403 on public documentation is a discoverability defect.

Fix incidents in this order: widespread 5xx, accidental 403/401, repeated 429, broken 404 targets, and unnecessary redirect chains. Re-test with a verified crawler identity and record both edge and origin responses. For the broader architecture, use Visible Pilot’s technical architecture for AI search visibility guide. Cloudflare users can also consult the official AI Crawl Control analytics documentation.

Reproducibility: fields and calculations

Store one row per request with: UTC timestamp, verified crawler, operator, crawl purpose, hostname, normalized path, final URL, exact HTTP status, response class, content type, edge status, origin status, cache result, WAF action or rule ID, response bytes, latency, retry result, CMS, CDN, site category, and consent policy. Remove health checks and duplicate retries only under a documented rule.

For each segment, divide requests in a response class by all eligible requests in that segment. Publish the numerator and denominator beside the percentage. Keep request-weighted and site-weighted results separate: request-weighted data describes crawler experience, while site-weighted data describes how many domains exhibit a condition. Report rounded values at consistent precision and retain the unrounded calculation for trend comparisons.

Update policy

Visible Pilot will treat this page as a dated baseline. Future editions should preserve definitions, publish counts, use the same observation window, identify crawler-list changes, and separate trend data from new cohorts. If CDN classification or bot verification changes, the methodology note must change before percentages are compared.

Frequently asked questions

What is a good AI crawler success rate?

There is no universal target because intentional blocking changes the correct result. For content you want AI systems to access, monitor verified-bot 2xx separately and investigate material drops. Compare like-for-like paths and crawler purposes rather than chasing the network-wide 74% as a score.

Does a 403 mean an AI crawler is blocked?

Usually it means the request was understood but access was denied. The cause may be an explicit AI-crawler policy, a WAF rule, authentication, geofencing, or a bot-management false positive. Trace the edge action and origin response before changing the rule.

Is every 3xx response a failure?

No. A single permanent redirect to the correct canonical URL can be healthy. Long chains, loops, temporary redirects that never settle, and redirects to irrelevant destinations create crawl friction and should be fixed.

Sources

• Cloudflare: Redirects for AI Training and response-status analysis — 17 April 2026.
• Cloudflare Radar AI Insights — live response-status explorer.
• Cloudflare AI Crawl Control: Analyze AI traffic — filters, status distribution, and exports.
• Cloudflare AI Crawl Control: Manage AI crawlers — allow, 403, and 402 controls.
• OpenAI crawler documentation — crawler identities and controls.