llms.txt: Can AI Use What It Fetched? (The Visibility Chain, Part 2 — Readable)

What this piece covers

A successful fetch of /llms.txt proves the file was obtainable. It does not prove that any AI system parsed, retained, or used it. Fetch and use are four separate technical events, and only the first is visible from the publishing side.

This is Part 2 of the Visibility Chain series. It sits at Readable.

The five-stage chain, in this exact order, is Reach → Readable → Retrieved → Recalled → Impact.

1. Reach: Can a system access the site?
2. Readable: Can it parse pages, files, structure, and other technical signals once it has fetched them?
3. Retrieved: Does it retrieve the site for a relevant prompt or task?
4. Recalled: Does it surface or name the site when a user asks about the topic?
5. Impact: Does earlier visibility appear as demand a site owner can observe?

This entry reports no measurement, no crawl, and no dataset. It addresses one Readable question and stops there: does an AI system fetching a file mean that system uses the content inside it? Fetch and use are separate events. llms.txt on its own only speaks to whether a condensed file exists to be fetched. It does not answer Retrieved, Recalled, or Impact.

Reach — whether a named crawler can access the site at all — is a separate question, with a crawler-access verification method published separately. How to author and publish an llms.txt file is also separate; see what llms.txt is and how to publish one. This piece does not repeat those how-tos.

Why this is hard to pin down

Vendor and community language about llms.txt often collapses four distinct technical events into one word: "used." A crawler making an HTTP request to /llms.txt is one event. That file's content being tokenized and inserted into a model's context window is a second. That content persisting across a session or being written into a retrieval index is a third. That content showing up, paraphrased or cited, in a model's output is a fourth. Tooling copy routinely implies that solving the first event solves all four, and that doing so advances every stage of the Visibility Chain.

Those four events are how Readable can fail after Reach has already succeeded. They are not a substitute Visibility Chain. They do not rename Reach, Readable, Retrieved, Recalled, or Impact.

The claim this entry examines is the framing of llms.txt as a de facto standard that AI systems consume — common in tooling marketing and some SEO commentary. It is a community-proposed convention: a plain-text or markdown file intended to give crawlers a condensed summary of a site. Whether any vendor fetches it, let alone parses, retains, or cites it, is a separate open question that the file's existence cannot answer.

Method (for this entry)

We ran no crawl, no dataset, and no scoring exercise for this piece. What follows is the set of reproducible checks we would run, and that a reader can run against their own site or against any site they audit:

• Locate the site's llms.txt file (typically at the domain root) and confirm what it contains.
• Check robots.txt for AI-specific user-agent directives (for example, entries naming known AI crawler user agents) and note whether /llms.txt is disallowed or allowed.
• Review server access logs for requests to /llms.txt filtered by named AI crawler user-agent strings, with request timestamps.
• Check the developer documentation published by relevant LLM vendors for any explicit statement that they fetch, parse, or otherwise consume llms.txt.

These are verification steps a reader can perform. They are not a measurement we performed, and we report no results from running them here.

Four technical events inside Readable

Name each event in exact terms. The ambiguity in this space comes from treating them as one thing, or from treating them as the Visibility Chain.

Fetch. An HTTP request is made to a URL, in this case /llms.txt, and a response is returned. This is the same category of event as any other crawl request logged on a web server. A successful fetch is closer to Reach than to Readable: it says a file was obtainable. It says nothing about what happens to the response body afterward.

Parse. The fetched content is processed by some system — tokenized, structured, or otherwise interpreted — so that it can be used in a downstream process such as building a prompt context or populating an index. A file can be fetched and never parsed, for example if a crawler stores raw bytes for a purpose unrelated to a given model's context construction. Parse is the core Readable question for this file.

Retain. Parsed content persists somewhere beyond a single request-response cycle: in a session's context window across turns, in a retrieval index, or in some other stored representation that a later query can draw on. Fetch and parse can both occur without retention if the content is discarded after immediate use. Retention is a precondition for later Retrieved or Recalled checks; it is not itself Retrieved or Recalled.

Use. The retained or parsed content appears, in some recognizable form, in a system's output — as a citation, a paraphrase, or a factual claim the output draws on. That output-side event is what most vendor claims about AI visibility are gesturing at. Confirming it requires a Retrieved or Recalled test, not the existence of llms.txt.

llms.txt, as a file format and convention, addresses only the first of these four events: it is one candidate location a crawler could fetch to obtain a condensed summary of a site. Publishing the file does not guarantee that any system parses it, retains it, or uses it in output. Those remaining events depend on decisions inside AI vendors' own systems, which are not visible from the publishing side.

What is and is not documented

We scoped our documentation review to the public crawler and developer documentation that each vendor publishes for site owners — the pages describing crawler user agents, crawl controls, and publisher-facing behaviour. Within that scope we did not locate documentation that names llms.txt specifically and confirms that the vendor fetches, parses, retains, or cites its contents.

This is a null finding: an absence of evidence, not evidence of absence. Such documentation may exist outside the scope we checked, a vendor may support the convention without documenting it publicly, or no vendor may currently do either. None of those possibilities can be distinguished from the outside, and we do not guess which is true. State your scope when you run this check; a reader who checks different pages may reach a different answer.

For the community proposal itself, independent of any one product, see the specification at llmstxt.org, which documents the proposed file format and its intended purpose.

Limitations

Every statement above about fetch, parse, retain, and use is conceptual — drawn from how HTTP requests, tokenization, and model context construction generally work — and not an empirical finding from a study run for this piece. Nothing here is a ranking, citation, mention-rate, traffic, or trust outcome.

Vendor documentation practices for AI crawlers change without notice. Any statement here about what is or is not documented is time-bound to when we prepared this entry. A reader verifying this later should re-run the documentation check rather than rely on our null finding indefinitely.

The TrustGrowth proof page separately reports a published score for trustgrowth.ai, dated on that page. That score is an unrelated, separately measured artefact. It is not evidence about llms.txt, it says nothing about fetch, parse, retain, or use, and we do not use it here.

How to reproduce (verification, not measurement)

Check for the presence of /llms.txt with a simple HTTP request:

curl -I https://example.com/llms.txt
# Look for a 200 response and a text/plain or text/markdown content-type.
# A 404 means no file is published at the conventional path.

Grep server access logs for requests to /llms.txt from named AI crawler user agents:

grep '/llms.txt' access.log | grep -Ei 'gptbot|claudebot|perplexitybot|ccbot'
# Adjust user-agent strings to match whichever crawlers you are checking for.
# A match confirms a fetch request occurred; it does not confirm parse, retain, or use.

A match in the access log confirms a fetch and nothing beyond it. Run the documentation check against each vendor's current developer docs rather than against this entry. No score or outcome is promised, and none is claimed here.

What publishing llms.txt does not change

Publishing /llms.txt does not, by itself, move a site through the rest of the Visibility Chain.

• Reach is still a crawler-access question: robots.txt, HTTP response, CDN/WAF rules, and access logs. A file the crawler is disallowed from fetching is not Readable.
• Retrieved still requires a later check that the system actually selects the site for a relevant prompt or task. File presence is not retrieval.
• Recalled still requires a later check that the system names or surfaces the brand when the prompt does not supply it. File presence is not unprompted naming.
• Impact still requires observable demand, such as tagged referrals compared with branded search. File presence is not demand.

FAQ

Does a 200 response on /llms.txt mean an AI system can use the file?

No. A 200 OK confirms only that the HTTP fetch succeeded. Parse, retain, and use are separate events. Use in output is a Retrieved or Recalled question, not a Readable one.

Is llms.txt a replacement for robots.txt?

No. robots.txt states crawl intent for named user agents. llms.txt is a community-proposed condensed summary of a site. Allowing a crawler in robots.txt does not mean that crawler consumes llms.txt. Disallowing /llms.txt in robots.txt means Reach fails for that file before Readable can be tested.

Does this article claim that major LLM providers ignore llms.txt?

No. We report that, within the documentation scope described above, we did not locate vendor documentation confirming consumption. That is an absence of evidence, not evidence of absence.

Is a logged request the same as a successful fetch?

No. An access-log line records that a request arrived. A successful fetch also requires a success response carrying the file body. A request that a CDN, WAF, or missing-file handler rejected appears in the log as a request and delivered nothing. Read the status code beside the user agent, not the user agent alone.

Do this next

Before you treat a published llms.txt as AI visibility, run the four checks in this entry against your own domain and write down what each one returned:

• Confirm the file exists at the conventional path, and record the status code and content type.
• Confirm robots.txt does not disallow the file for the crawlers you care about.
• Grep your access logs for requests to the file, and record the user agent, status code, and timestamp for each match.
• Check the current public crawler documentation for each vendor you care about, and record which pages you checked.

Those four answers bound Readable for this file, and no result from them should be published as an AI visibility outcome. The later stages need their own tests: Part 3 takes up Retrieved, Part 4 takes up Recalled, and Part 5 takes up Impact. If you want the stages diagnosed together rather than one file at a time, that is what AI visibility diagnostics are for.

Last reviewed

This article is a definitional method for the Readable stage. It contains no measured fetch, parse, retain, or use result, and no dated crawl. The documentation-review scope is stated in "What is and is not documented" above; re-run that check against current vendor documentation rather than relying on our null finding.