FREE TOOL · DETERMINISTIC · COMPOSE

Compose a robots.txt that states allow or block for AI crawlers.

We fetch the live file, keep Sitemap lines and non-AI groups, and emit explicit User-agent rules for the same crawlers the access check already evaluates.

No account. One bounded check — never the full crawl or a site-wide score.

WHAT YOU GET

ILLUSTRATIVE

A copyable robots.txt with named AI crawler groups

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://example.com/sitemap.xml

What this composer actually does

The crawler-access check reports which AI bots may fetch you today. This page composes a robots.txt you can deploy: explicit allow or block groups for those same crawlers, with Sitemap lines and unrelated User-agent groups left intact. It describes the current file, then a copy you can edit in the browser. It does not publish anything to your origin.

What is preserved, and what is rewritten

Sitemap lines stay: Every Sitemap: directive from the live file is copied through. The composer does not invent a sitemap URL.
Non-AI groups stay: Googlebot, *, and any other group that is not on the tracked AI list keep their rules. Only the named AI crawler groups are composed.
Toggles stay in the browser: Allow/block switches rewrite the composed text on this page. They are not a second server request, and they are not live on your site until you deploy the file.

Why it matters for search — and for AI answers

Crawler access is a policy choice, not a ranking trick. A wildcard written for one scraper can silently cover every AI engine. This composer makes that choice visible as named groups you can copy, without claiming that deploying the file will get you cited.

What the full product adds

This is one slice of Reach, the first stage of TrustGrowth's AI visibility funnel. The product re-checks live crawler access on a schedule, then follows whether engines can use what they fetched, cite you, or name you unprompted.

Convinced? It takes one URL.

TRY ANOTHER CHECK

Check my schema → Build JSON-LD → Build sameAs JSON-LD → Preview my listing → Inspect my canonical → Check crawler access → Create my llms.txt → See the AI read → Check trust signals → Ask three engines → Convert this page → Grade my llms.txt →

Browse all free tools The full version lives in AI Visibility →