What llms.txt is and how to publish one

This tutorial is about one file: llms.txt. I fetched a live copy, checked that it parses against the current proposal, and followed every listed URL. That is file presence, parseability, and listed-URL status on one domain on one date. It does not measure rankings, citations, or revenue.

Jeremy Howard's proposal (llmstxt.org, accessed 2026-08-19) is a public convention, not a guarantee that any named model will fetch, quote, or mention your brand.

What llms.txt is (and is not)

The current proposal (v2) is a Markdown file named llms.txt at the site root, or at a subpath such as /docs/llms.txt. It is meant to give an agent a small, curated map of the pages you actually want it to read.

Required and recommended pieces, from that proposal:

Piece Required? What it is H1 Yes The project or site name Blockquote Optional A short summary an agent can lift H2 sections Recommended Grouped Markdown lists of links Linked pages Recommended Clean, LLM-readable URLs, often .md

That is not a crawl-permission file. The same proposal gives robots.txt and llms.txt different jobs: robots.txt states what automated access is considered acceptable; llms.txt is used on demand when an agent wants a guided path into a site. Publishing llms.txt does not, by itself, make a model train on your pages, rank you, or mention you.

A 404 on /llms.txt is a missing file. It is not a block.

How to author one against the current proposal

Write the file in Markdown. Keep it short. Point at the pages that explain what you do, not at every URL in the sitemap.

A minimal shape that matches the proposal:

# Example site name
> One-sentence summary of what this site is for.

## Start here
- [Overview](https://example.com/): what the product does
- [Docs](https://example.com/docs): how to use it

Then check three things before you call it done:

1. The H1 exists. A file that starts with a paragraph and no heading is not the proposal.
2. Every listed URL is a URL you control and want an agent to read. Do not list /dashboard, /auth, or other application paths you disallow elsewhere.
3. The listed URLs actually respond. A curated map that 404s is worse than no map.

I treat rel="alternate" / .md copies as optional follow-on work. The proposal recommends them; they are not required to validate that a root llms.txt exists and parses.

How I validated trustgrowth.ai on 2026-08-19

I ran this check against https://trustgrowth.ai/llms.txt on 2026-08-19 UTC. I am reporting what I saw that day, not a ranking outcome.

curl -sS -D - -o llms.txt https://trustgrowth.ai/llms.txt

File presence. HTTP 200. Final URL stayed https://trustgrowth.ai/llms.txt (no redirect). Content-Type: text/plain. Body length 725 bytes. Last-Modified: Wed, 19 Aug 2026 01:42:00 GMT.

Parseability. The file had:

• an H1 (# TrustGrowth)
• a blockquote summary about AI visibility measurement
• H2 sections (Free tools, Product, Evidence, Writing) whose list items were Markdown links

That matches the proposal's required H1 plus the recommended section-and-link pattern. I did not run a third-party linter; this is a human read of the live bytes.

Listed-URL status. I requested each URL listed in that file:

Listed URL HTTP status on 2026-08-19 https://trustgrowth.ai/tools/free-scan 200 https://trustgrowth.ai/tools 200 https://trustgrowth.ai/ 200 https://trustgrowth.ai/pricing 200 https://trustgrowth.ai/proofs 200 https://trustgrowth.ai/blog 200

None of those paths appear in the Disallow list of https://trustgrowth.ai/robots.txt as fetched the same day (/dashboard, /onboarding, /monitoring, /auth, /rails/, /scan). So the curated map and the crawl-permission file were not pointing at opposite policies for those six URLs on that date.

A 200 on /llms.txt is file presence. It is not proof that ChatGPT, Claude, or Gemini fetched the file at inference time.

How to repeat the check on your domain

Record HTTP status, final URL after redirects, fetch time, and a short parse note:

curl -sS -D - -o llms.txt https://example.com/llms.txt

Then extract the listed URLs and request each one. Note status. If a listed path is Disallow'd for the relevant crawler in robots.txt, that is a conflict between your two files, not a ranking signal.

Expected outputs:

• 200 plus Markdown that contains an H1 → file present and at least minimally formed
• 404 → missing file
• 200 with HTML, JSON, or an empty body → the path exists, but it is not the Markdown map the proposal describes

Keep the date on the record. The proposal can change; OpenAI, Anthropic, and Google do not publish a conformance test that says a named production model will honor your file.

Crawl eligibility is a separate control

llms.txt does not replace robots.txt. Allowing or blocking GPTBot is a third, independent control. OpenAI documents GPTBot, OAI-SearchBot, and ChatGPT-User as separate product tokens at developers.openai.com/api/docs/bots (accessed 2026-08-19). A GPTBot directive alone says nothing about ChatGPT search. An llms.txt file alone says nothing about GPTBot.

Control Job What a pass means What it does not mean llms.txt Curated on-demand map The file exists, parses, and listed URLs respond A model will fetch, quote, or mention you robots.txt Crawl eligibility for named product tokens A conforming crawler can parse allow/disallow groups Access control, rankings, or ChatGPT search inclusion GPTBot OpenAI training-crawler eligibility That crawler is allowed or disallowed for the matching paths ChatGPT search (OAI-SearchBot) or user-initiated visits (ChatGPT-User)

If you need to verify whether AI crawlers can fetch your site — user-agent, robots.txt groups, and log-level IP checks — use the existing method: Check if AI crawlers can access your site. I am not restating that tutorial here. This page stops at the llms.txt file. A tool that reports "blocked" can also be failing to reach you; how to tell whether an SEO tool measured your site or just failed to reach it covers that failure mode.

Limitations

• One domain, one date. The worked example is trustgrowth.ai on 2026-08-19 UTC. It does not generalize to other domains or later proposal versions.
• Proposal, not a standard. llmstxt.org is a public convention. It is not RFC 9309, and it is not an OpenAI, Anthropic, or Google product contract.
• No outcome measurement. This method measures file presence, parseability, and listed-URL HTTP status. It does not measure rankings, citations, or revenue.
• No inference-time proof. A 200 on /llms.txt does not prove that a named model read the file while answering a prompt.

FAQ

Does publishing llms.txt replace a robots.txt rule?

No. llmstxt.org (accessed 2026-08-19) describes llms.txt as an on-demand map for agents, not a crawl-permission file. GPTBot eligibility is still a robots.txt group. Validate the file with the checks above; verify crawler access with the AI crawler access method.

Will ChatGPT or another model definitely read my llms.txt?

Not measured here. The proposal says agents are expected to look for the file; it does not publish a conformance test against any named production model. A 200 on /llms.txt is file presence, not proof of inference-time use.

Where should I put the file?

The proposal's default is /llms.txt at the site root. A subpath such as /docs/llms.txt is allowed. I checked the root URL on trustgrowth.ai because that is the location an agent will try first.

What if /llms.txt returns 404?

The file is missing. That is not a block, and it is not a ranking event. Publish a Markdown file that matches the proposal if you want the map to exist; skip it if you do not.

Can I list URLs that robots.txt disallows?

You can type them. I would not. On 2026-08-19 the six URLs listed on trustgrowth.ai were all HTTP 200 and none matched a Disallow path. A listed URL that a crawler is told not to fetch is a conflict in your own files.

Summary

llms.txt is a curated Markdown map. Author it against the current llmstxt.org proposal, then verify three things on a dated record: the file returns 200, it parses as Markdown with an H1, and every listed URL responds. Crawl eligibility stays in robots.txt. If you need the crawler-side method, follow the AI crawler access tutorial instead of duplicating it here.