We fetch the live file, keep Sitemap lines and non-AI groups, and emit explicit User-agent rules for the same crawlers the access check already evaluates.
No account. One bounded check, never the full crawl or a site-wide score.
A copyable robots.txt with a named group for each AI crawler.
8composed allow
1composed block
1sitemap line
1other group kept
User-agent: GPTBotAllow: /User-agent: PerplexityBotDisallow: /Sitemap: https://northwind.example/sitemap.xml
Reading live robots.txt…
One bounded check. It returns or it fails, and this page moves to the result when it returns.
It fetches the page and reads what is in it. Same input, same answer.
User-agent: PerplexityBotDisallow: /
The crawler-access check reports which AI bots may fetch you today. This page composes a robots.txt you can deploy: explicit allow or block groups for those same crawlers, with Sitemap lines and unrelated User-agent groups left intact. It describes the current file, then a copy you can edit in the browser. It does not publish anything to your origin.
Sitemap: directive from the live file is copied through. The composer does not invent a sitemap URL.Googlebot, *, and any other group that is not on the tracked AI list keep their rules. Only the named AI crawler groups are composed.Crawler access is a policy choice, not a ranking trick. A wildcard written for one scraper can silently cover every AI engine. This composer makes that choice visible as named groups you can copy, without claiming that deploying the file will get you cited.
This is one slice of Reach, the first stage of TrustGrowth's AI visibility funnel. The product re-checks live crawler access on a schedule, then follows whether engines can use what they fetched, cite you, or name you unprompted.
Each one is its own free tool: one URL, one check, no account.
| Kind | Tool | Question | Run it |
|---|---|---|---|
| Reach and be understood | |||
| AI Crawler Access Checkdeterministic · one fetch | Which AI crawlers can reach my site? | Check crawler access | |
| llms.txt Creatordeterministic · bounded fetch | What should my llms.txt say? | Create my llms.txt | |
| Sitemap llms.txtdeterministic · sitemap locs | Can my sitemap become an llms.txt? | Create llms.txt from sitemap | |
| How AI Sees Your Sitemodel-assisted · one read | What does my homepage tell AI about me? | See the AI read | |
| llms.txt Graderdeterministic · bounded fetch | Is my llms.txt a briefing or a stub? | Grade my llms.txt | |
| Build trust | |||
| E-E-A-T Quick Checkmodel-assisted · one read | Does my site show visible trust signals? | Check trust signals | |
| Are You Named by AI?sampled · three engines | Does AI name my site when nobody prompts it? | Ask three engines | |
| Kind | Tool | Question | Run it |
|---|---|---|---|
| Publish correctly | |||
| Schema Checkdeterministic · one fetch | Is my structured data valid? | Check my schema | |
| FAQ JSON-LDdeterministic · one fetch | Can my FAQ become FAQPage markup? | Build JSON-LD | |
| SERP Previewdeterministic · one fetch | What will my listing look like in search and social? | Preview my listing | |
| Canonical Inspectordeterministic · bounded fetch | Do my redirects agree with my canonical tag? | Inspect my canonical | |
| Page Speeddeterministic · PageSpeed | How fast is my page for real mobile visitors? | Check page speed | |
| Link Graphdeterministic · bounded crawl | Which pages are orphaned or buried deep? | Map my links | |
| Create and convert | |||
| sameAs Packdeterministic · homepage + about | Can I mark up the profiles my page already links to? | Build sameAs JSON-LD | |
| HTML to Markdowndeterministic · one fetch | How does my page read to an LLM? | Convert this page | |
The Free Site Audit puts top issues, ranked keywords, closest competitors, authority and AI readiness in one dated report. No account.