Compose a robots.txt that states allow or block for AI crawlers.

We fetch the live file, keep Sitemap lines and non-AI groups, and emit explicit User-agent rules for the same crawlers the access check already evaluates.

No account. One bounded check, never the full crawl or a site-wide score.

Free tools / AI robots.txt Policy deterministic · compose northwind.example Example data

A copyable robots.txt with a named group for each AI crawler.

8composed allow

1composed block

1sitemap line

1other group kept

GPTBot live: allowedAllow
PerplexityBot live: blockedBlock
User-agent: GPTBotAllow: /User-agent: PerplexityBotDisallow: /Sitemap: https://northwind.example/sitemap.xml

Reading live robots.txt…

Fetch robots.txt from the public origin after resolving the homepage URL.
Preserve Sitemap lines and unrelated User-agent groups.
Emit allow or block groups for the maintained AI crawler list.
Leave deployment to you — nothing is written back to the origin.

One bounded check. It returns or it fails, and this page moves to the result when it returns.

What does this check do?

Fetch robots.txt from the public origin after resolving the homepage URL.
Preserve Sitemap lines and unrelated User-agent groups.
Emit allow or block groups for the maintained AI crawler list.
Leave deployment to you — nothing is written back to the origin.

Deterministic

It fetches the page and reads what is in it. Same input, same answer.

User-agent: PerplexityBotDisallow: /
Run 1Blocked
Run 2Blocked

The full version lives in AI Visibility

Browse all free tools

What does this composer actually do?

The crawler-access check reports which AI bots may fetch you today. This page composes a robots.txt you can deploy: explicit allow or block groups for those same crawlers, with Sitemap lines and unrelated User-agent groups left intact. It describes the current file, then a copy you can edit in the browser. It does not publish anything to your origin.

What is kept, and what is rewritten?

Sitemap lines stay
Every Sitemap: directive from the live file is copied through. The composer does not invent a sitemap URL.
Non-AI groups stay
Googlebot, *, and any other group that is not on the tracked AI list keep their rules. Only the named AI crawler groups are composed.
Toggles stay in the browser
Allow/block switches rewrite the composed text on this page. They are not a second server request, and they are not live on your site until you deploy the file.

Why does it matter for search and for AI answers?

Crawler access is a policy choice, not a ranking trick. A wildcard written for one scraper can silently cover every AI engine. This composer makes that choice visible as named groups you can copy, without claiming that deploying the file will get you cited.

What does the full product add?

This is one slice of Reach, the first stage of TrustGrowth's AI visibility funnel. The product re-checks live crawler access on a schedule, then follows whether engines can use what they fetched, cite you, or name you unprompted.

Which other check answers my question?

Each one is its own free tool: one URL, one check, no account.

KindToolQuestionRun it
Reach and be understood
AI Crawler Access Checkdeterministic · one fetch Which AI crawlers can reach my site? Check crawler access
llms.txt Creatordeterministic · bounded fetch What should my llms.txt say? Create my llms.txt
Sitemap llms.txtdeterministic · sitemap locs Can my sitemap become an llms.txt? Create llms.txt from sitemap
How AI Sees Your Sitemodel-assisted · one read What does my homepage tell AI about me? See the AI read
llms.txt Graderdeterministic · bounded fetch Is my llms.txt a briefing or a stub? Grade my llms.txt
Build trust
E-E-A-T Quick Checkmodel-assisted · one read Does my site show visible trust signals? Check trust signals
Are You Named by AI?sampled · three engines Does AI name my site when nobody prompts it? Ask three engines
KindToolQuestionRun it
Publish correctly
Schema Checkdeterministic · one fetch Is my structured data valid? Check my schema
FAQ JSON-LDdeterministic · one fetch Can my FAQ become FAQPage markup? Build JSON-LD
SERP Previewdeterministic · one fetch What will my listing look like in search and social? Preview my listing
Canonical Inspectordeterministic · bounded fetch Do my redirects agree with my canonical tag? Inspect my canonical
Page Speeddeterministic · PageSpeed How fast is my page for real mobile visitors? Check page speed
Link Graphdeterministic · bounded crawl Which pages are orphaned or buried deep? Map my links
Create and convert
sameAs Packdeterministic · homepage + about Can I mark up the profiles my page already links to? Build sameAs JSON-LD
HTML to Markdowndeterministic · one fetch How does my page read to an LLM? Convert this page

Want one report on your whole site?

The Free Site Audit puts top issues, ranked keywords, closest competitors, authority and AI readiness in one dated report. No account.