An llms.txt before and after experiment sounds simple: publish the file, repeat a few AI searches, and compare the answers. The hard part is separating a useful signal from ordinary variation. AI results can change because of the prompt, platform, retrieval index, content updates, crawler access, or time. This guide shows a controlled, evidence-first test that a SaaS team or website owner can reproduce without pretending that llms.txt is a confirmed ranking factor.
Experiment rule: change one main variable, preserve the evidence, and report absolute counts. A better result after publication is a correlation worth investigating—not automatic proof that llms.txt caused it.
Baseline: capture the site before publishing llms.txt
Start with a website whose important pages are public, canonical, indexable, and accessible without JavaScript interaction or authentication. Record the test date, platform, product mode, location if relevant, and exact URLs. Then save the current robots.txt rules, XML sitemap, response headers, page HTML, canonical tags, and CDN or firewall status. These details matter because a blocked crawler or empty server-rendered response can explain poor discovery more directly than a missing llms.txt file.
Create a fixed prompt set before making changes. Include brand prompts, problem-led prompts, and research questions that the selected pages genuinely answer. Run each prompt several times, record every mention, linked URL, citation, and incorrect statement, and keep screenshots or exports. Also capture AI referral sessions and crawler requests from server or CDN logs. OpenAI advises publishers who want content eligible for ChatGPT search summaries and links to allow OAI-SearchBot; that crawler-access control is separate from llms.txt.
Diagnosis: test the hypothesis before accepting it
The initial hypothesis may be, “AI systems cannot understand the site because it lacks llms.txt.” Test simpler explanations first. Can the relevant crawler fetch the page? Does the server return the same useful content to a non-browser client? Are directives contradictory? Is the answer stated clearly in the main content? Do authoritative sources support the claim? If one of these checks fails, repair it before the experiment or document it as a confounding factor.
What llms.txt is: a proposed Markdown convention at /llms.txt that summarizes a site and points to selected resources. The proposal does not require every AI platform to fetch or use it, so the file should be evaluated as an optional machine-readable guide.
Intervention plan: what changed and what stayed fixed
Publish one concise file at the root of the domain. Follow the proposed format: an H1 naming the organization, a blockquote summary, brief context, and curated links grouped under clear H2 sections. Select canonical pages that explain the business, product, documentation, evidence, and policies. Avoid dumping the sitemap into the file; a focused resource map makes the experiment easier to interpret and maintain.
- Day 0: freeze the prompt list and archive the technical baseline.
- Day 1: publish llms.txt, confirm HTTP 200, and validate every linked URL.
- Days 2–28: monitor requests, citations, referrals, and unexpected errors without rewriting the test pages.
- Day 29: repeat the original prompt set under the same documented conditions.
Deliberately avoid simultaneous redesigns, mass content updates, major digital PR campaigns, robots.txt changes, or CDN migrations. If another change is unavoidable, timestamp it and explain how it could affect the outcome.
Measurement method for an llms.txt before and after experiment
Use the same prompts, platforms, URLs, repetition count, and observation window. Score a mention only when the correct organization appears. Score a citation only when the answer links to the tested domain. Keep crawler requests, referral visits, and conversions as separate measures; one does not prove another. A crawler request confirms access to a resource, not indexing or future selection.
Results: how to report the before-and-after table
The worked example below demonstrates the reporting format; the figures are illustrative and are not claimed as Visible Pilot customer results. Reporting the denominator prevents a small change from looking larger than it is.
| Measurement | Baseline | After | Interpretation |
|---|---|---|---|
| Correct brand mentions | 3 of 25 | 6 of 25 | Positive correlation; repeat before concluding |
| Links to tested domain | 1 of 25 | 2 of 25 | Too few events for a strong claim |
| Requests for /llms.txt | 0 | 4 | Confirms retrieval attempts only |
| AI referral sessions | 0 | 2 | Useful signal, not proof of causation |
What likely caused improvement—and what cannot be proven
If logs show a compatible system fetched llms.txt and then fetched linked pages, the file likely helped that retrieval path navigate the selected resources. If mentions or citations also increased, the timing supports a hypothesis worth retesting. It still cannot prove that llms.txt created a platform-wide ranking improvement. The answer may have changed because the retrieval index refreshed, a third-party source mentioned the brand, the model changed, or ordinary output variation favored the site.
For a stronger design, compare similar pages or sites, stagger publication dates, and run several post-change rounds. Preserve failed and unfavorable observations. The goal is not to “win” the test; it is to learn whether the file adds measurable value beyond healthy crawling, strong content, clear entities, and independent authority.
Transferable lesson: llms.txt is most useful as a clean, testable interface to pages that are already accurate and accessible. It cannot rescue blocked delivery, thin evidence, stale facts, or unclear ownership of an entity.
Evidence and screenshots to preserve
- The live llms.txt file, syntax, headers, HTTP status, and publication timestamp
- The canonical status and readable main content of every linked page
- Exact prompts, platform modes, answers, citations, and repetition counts
- Crawler, CDN, and firewall logs with timestamps, user agents, URLs, and response codes
- A dated change log for content, links, directives, and infrastructure
Common interpretation mistake
The most common mistake is presenting a proposed convention as a guaranteed ranking or citation mechanism. The llms.txt specification explains a way to offer LLM-friendly context; it does not dictate how every AI search product must behave. Do not treat the file as a replacement for robots.txt, XML sitemaps, structured data, technical SEO, or high-quality source pages.
Frequently asked questions
How long should an llms.txt experiment run?
There is no universal window. Four weeks is a practical minimum for a small observational test, but low-volume sites may need longer. Technical access can be checked immediately; discovery, citations, referrals, and conversions require repeated observations.
What is the best control for the experiment?
Use comparable pages or sites that do not receive llms.txt during the same period, or stagger publication across groups. Keep prompts and measurements fixed, and document every unrelated change that could influence retrieval.
Does a crawler request prove llms.txt improved AI visibility?
No. It proves that a client requested the file. It does not prove indexing, answer selection, citation, ranking, or business impact. Track each stage separately.
Should you publish llms.txt before fixing crawler access?
No. A navigation file cannot override authentication, firewall challenges, disallow rules, noindex directives, broken canonicals, or missing server-rendered content. Fix access and delivery first.
Next step: request a Visible Pilot audit
Use the llms.txt and machine-readable website guide to build the file, then read whether llms.txt improves AI visibility for the evidence boundaries. If you want the full discovery path checked—from crawler access and rendering to content clarity and citations—request a Visible Pilot audit.
Sources: the llms.txt proposal and OpenAI’s publisher guidance. Guidance reviewed 6 August 2026.
Leave a Reply