AI Search Readiness Benchmark Report

AI search readiness benchmark report findings should help website owners make better decisions—not decorate a page with an unexplained score. This July 2026 pilot establishes a reproducible baseline by reviewing the public crawler policies of ten prominent SaaS and business websites.

The narrow conclusion is useful: every sampled site published a reachable robots.txt file, but explicit AI-agent rules were uncommon and inconsistent. That does not mean most sites blocked AI search. When a named crawler is absent, the wildcard group may govern it. It also does not prove that an allowed crawler can render, understand, index or cite a page.

Key findings from the pilot benchmark

Finding	Result	Denominator	What it means
Public robots.txt file reachable	10	10 sites	The basic policy file could be inspected for every sampled domain.
GPTBot explicitly named	3	10 sites	Most files did not create a named GPTBot group; wildcard rules may still apply.
OAI-SearchBot explicitly named	1	10 sites	Only Zapier explicitly named OpenAI’s search crawler in this snapshot.
ClaudeBot explicitly named	3	10 sites	Named Claude access rules appeared on Squarespace, Canva and Zapier.
PerplexityBot explicitly named	2	10 sites	Canva and Zapier explicitly addressed PerplexityBot.
Different explicit AI-policy patterns	3	3 sites with named AI rules	The observed patterns were general-path rules, selective-path access and broad explicit access.

The strongest signal is not that one policy is universally “best.” It is that crawler intent, inherited rules and path scope must be read together. OpenAI documents OAI-SearchBot for search visibility and GPTBot for potential training use as independent controls. Counting either name without interpreting the associated rules would produce a misleading readiness score.

Methodology

We selected ten recognisable SaaS and business websites with public, crawlable marketing properties: Shopify, Wix, Squarespace, HubSpot, Salesforce, Stripe, Notion, Canva, Zapier and Webflow. The sample is purposive rather than random, so results describe these domains only.

• Collection period: one observation window on 28 July 2026.
• Source: each domain’s public /robots.txt file.
• Availability test: whether the policy file could be retrieved and read.
• Agent checks: literal presence of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot.
• Policy interpretation: named group, wildcard inheritance, and whether the rule broadly allowed, broadly disallowed or selectively scoped paths.
• Exclusions: no WAF/IP verification, rendered-page test, index-state measurement, prompt testing or citation-rate claim.

The observations were manually checked against current official crawler guidance. A named-agent count measures configuration explicitness, not effective access by itself. If an agent is absent, robots exclusion rules normally fall back to the applicable wildcard group; server, CDN and security controls can still produce a different real-world outcome.

Overall results for the AI search readiness benchmark report

Ten out of ten sampled sites exposed a readable robots.txt file. That is a useful minimum control, but it is only the first layer of readiness.

Three out of ten explicitly named GPTBot: Squarespace, Canva and Zapier. Three out of ten explicitly named ClaudeBot: the same three domains. Only one out of ten explicitly named OAI-SearchBot, and only two out of ten explicitly named PerplexityBot.

The explicit policies were not equivalent. Zapier grouped several AI agents and allowed the root path. Canva broadly disallowed several training-oriented agents while allowing selected public directories and separately allowing Perplexity agents. Squarespace grouped many named agents with the wildcard rules that restricted sensitive, search, account, API and parameter paths while leaving other public paths available.

Site-level robots.txt snapshot

Domain	GPTBot named	OAI-SearchBot named	ClaudeBot named	PerplexityBot named	Observed policy pattern
Shopify	No	No	No	No	Wildcard rules; public areas and selected paths governed generally
Wix	No	No	No	No	Wildcard rules with many product and utility exclusions
Squarespace	Yes	No	Yes	No	Named AI agents grouped with general path restrictions
HubSpot	No	No	No	No	Large wildcard policy; no selected named agents found
Salesforce	No	No	No	No	No selected named agents found in this snapshot
Stripe	No	No	No	No	No selected named agents found in this snapshot
Notion	No	No	No	No	No selected named agents found in this snapshot
Canva	Yes	No	Yes	Yes	Training-oriented agents broadly restricted; selected paths allowed; Perplexity allowed
Zapier	Yes	Yes	Yes	Yes	Selected AI search, user and training agents explicitly allowed
Webflow	No	No	No	No	Compact wildcard policy; no selected named agents found

This table is a configuration snapshot, not a ranking. A domain with no named rule may be accessible through User-agent: *. A domain with a named allow can still fail at the transport layer. The next full edition should therefore pair policy parsing with verified requests and rendered-content evidence.

Breakdown by policy approach

1. Wildcard-led configuration: 7 of 10 sites

Seven domains did not explicitly name the selected AI agents. Their public crawler treatment therefore depends on the applicable general rules plus any controls outside robots.txt. This approach is simpler, but it makes the site owner’s intent less explicit to reviewers and can hide accidental inheritance.

2. Named agents with shared path rules: 1 of 10 sites

Squarespace listed multiple AI and conventional agents before a shared rule set. The pattern restricts sensitive and low-value paths rather than applying a blanket block to the public site.

3. Named agents with deliberate allow/disallow choices: 2 of 10 sites

Canva and Zapier showed more explicit agent-level choices. Canva separated certain agents and path exceptions; Zapier grouped search, user and training agents under a broad allow. These examples show why a binary “has AI rules” field is too crude for a useful benchmark.

Failure-pattern analysis: what clusters together

Even in a robots-only pilot, three recurring interpretation failures emerged.

• Name-counting without rule parsing: finding GPTBot in a file says nothing until the matching allow/disallow paths are evaluated.
• Training and search controls collapsed into one score: OAI-SearchBot and GPTBot serve different documented purposes; one cannot substitute for the other.
• Policy availability treated as end-to-end readiness: robots.txt cannot prove a 200 response, rendered content, correct canonical, clear entities or platform citation.

These failures cluster because simple audit tools prefer a single boolean. A defensible readiness model should preserve separate fields for policy, transport, rendering, index controls, entity clarity and observed platform behaviour.

What the findings mean for website owners

Action	Why it matters	Evidence to keep
Map product goal to crawler	Search visibility, user-triggered retrieval and training are different controls	Official agent documentation and dated policy decision
Test representative paths	Rules can vary by directory and template	Homepage, commercial page and article results
Verify beyond robots.txt	CDN/WAF responses may contradict policy intent	Status, headers and security event
Inspect delivered content	A 200 response may still contain an empty shell	Initial HTML and rendered DOM
Separate readiness from citation	Eligibility does not guarantee source selection	Fixed prompt, date, locale and cited URL

For a practical end-to-end process, use the AI search readiness checklist and the guide on how to test if a website is ready for AI search.

How to reproduce this benchmark

A future researcher should be able to reach the same classification from the stored inputs. Record one row per domain with the following raw fields.

• Domain, requested robots.txt URL, final URL, retrieval timestamp and HTTP status.
• Raw policy text or cryptographic hash, plus any redirect details.
• Literal presence of each tested user agent.
• Matched group for the root path and for at least one representative knowledge-page path.
• Effective allow/disallow conclusion with the exact rule that produced it.
• Reviewer, parser version and a manual-review note for ambiguous syntax.

Percentages should always retain their numerator and denominator. For example, report “3 of 10 sites explicitly named GPTBot,” not “30% are GPTBot-ready.” The first statement is observable; the second adds an unsupported causal label.

Limitations of this pilot

Robots files can change at any time, and platform documentation can change independently. Some policy syntax is also difficult to interpret without the crawler’s exact precedence behaviour. The public files and timestamp are therefore part of the result, not footnotes.

Update policy

Visible Pilot will keep future editions comparable by preserving the original ten-domain panel, publishing the observation date, retaining the same core fields and labeling every methodology change. A larger edition should add a stratified sample by CMS, company size and site type while reporting the original panel separately.

• Monthly: recheck crawler documentation and agent names.
• Quarterly: rerun the ten-domain robots snapshot and publish changes.
• Major edition: add verified HTTP, WAF, rendering and representative-page tests.
• Versioning: never overwrite prior denominators; publish a dated comparison table.

Frequently asked questions

What is an AI search readiness benchmark?

It is a defined comparison of observable conditions that affect whether AI search systems can discover and use website content. A useful benchmark publishes its sample, fields, denominators, date and limitations rather than relying only on a proprietary score.

Does missing OAI-SearchBot from robots.txt mean ChatGPT search is blocked?

Not automatically. If no specific group exists, the applicable wildcard rules may govern the crawler. Effective access also depends on the tested path and controls such as CDN or WAF rules.

Is GPTBot the crawler for ChatGPT search?

OpenAI documents OAI-SearchBot for ChatGPT search visibility and GPTBot for potential training use. Their controls are independent, so a benchmark should not collapse them.

Can robots.txt prove that a page will be cited?

No. It can express crawl permissions for compliant agents. Citation also depends on successful delivery, rendering, discovery, relevance, source selection and platform behaviour.

Why use a ten-site pilot?

A small transparent sample is more useful than invented large-sample statistics. It tests the measurement fields, exposes classification problems and creates a baseline that can be expanded without rewriting history.

Downloadable charts and dataset

The pilot’s site-level table, field definitions and calculation notes are included above so the result can be checked immediately. A future downloadable dataset should preserve the raw policy snapshot, dated classifications and change log rather than publishing only a chart.

Official sources and observed policy files

• OpenAI crawler documentation
• Anthropic crawler guidance
• Google AI features and website controls
• Perplexity crawler documentation
• Shopify robots.txt, Wix robots.txt, Squarespace robots.txt, HubSpot robots.txt, and Salesforce robots.txt
• Stripe robots.txt, Notion robots.txt, Canva robots.txt, Zapier robots.txt, and Webflow robots.txt

Reviewed 28 July 2026. This report documents a point-in-time public-policy snapshot. Recheck current files before changing production controls.