How to Allow AI Crawlers in robots.txt—and Prove the Rule Works

How to Allow AI Crawlers in robots.txt—and Prove the Rule Works

A correctly served robots.txt can prove that a named crawler is permitted to request specified paths; it cannot prove that the crawler requested, indexed, retrieved, cited, or used those pages. This tutorial tests one production hostname, one captured file, several crawler tokens, and representative URLs at a recorded timestamp.

Prerequisites

Before starting, you need:

• Control of the production web server, CDN, hosting platform, or framework-generated robots.txt configuration.
• A terminal with curl, a text editor, and permission to inspect access logs.
• A browser and a verified Google Search Console property if Search Console observations are included.
• Familiarity with HTTP status codes, response headers, DNS, TLS, and user-agent strings.
• One public URL and one deliberately restricted test path.
• Knowledge that robots.txt is a voluntary crawl preference, not authentication or authorization.
• A staging or preview hostname for drafting. You must verify the exact production hostname.
• Official crawler documentation checked immediately before publication. Do not add an invented access date.

1. Define what an allow rule proves

An Allow: / rule for OAI-SearchBot, served from https://example.com/robots.txt, states that the named crawler may request paths under that host. The result depends on the exact file, redirects, cache state, parser behavior, and whether the operator honors robots.txt.

It does not prove crawler identity. A user-agent header can be spoofed. It does not prove a request occurred, content was indexed, an AI service retrieved it, or a response influenced an answer. Treat permission, successful fetch, indexing, retrieval, and citation as five separate evidence layers.

Your evidence scope is therefore: hostname, protocol, response status and headers, complete response body, crawler token, representative URLs, parser and version, and test timestamp. The expected result is a reproducible interpretation of permission, not proof of AI visibility.

2. Prepare a reproducible test log

Record one row for each crawler and URL. Use these fields: crawler, URL, expected result, observed result, HTTP status, response timestamp, parser, parser version, and evidence limitation. Include the production hostname and protocol; example.com and www.example.com are different robots policies.

Google Search Console can provide owned-site crawl, indexing, and URL observations. It does not expose private OpenAI, Anthropic, Perplexity, or other AI crawler logs. Record the Search Console property and observation date separately from your robots test.

Expected result: a dated evidence log exists before deployment, and staging content is not mistaken for production evidence.

3. Identify the exact crawler token

Separate AI search or retrieval crawlers from model-training crawlers and general search crawlers. One policy does not automatically apply to all three. Use the exact token published by the operator; do not invent a generic AI token.

Check these primary documentation URLs immediately before publication:

• OpenAI: GPTBot
• OpenAI: OAI-SearchBot
• Anthropic: ClaudeBot
• Perplexity: PerplexityBot
• Google: Google-Extended

Google-Extended is a product control token. It is not necessarily a separate network crawler identity. Your internal inventory should contain token, operator, stated purpose, intended policy, official URL, and verification date.

Expected result: you have a dated inventory based on current OpenAI, Anthropic, Perplexity, and Google documentation, with no assumed token names.

4. Write the minimum rule

The syntax is a group containing a crawler token and directives:

User-agent: CrawlerToken
Disallow: /private/
Allow: /

To allow only OpenAI’s search crawler across the host:

User-agent: OAI-SearchBot
Allow: /

For different policies, use separate groups:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

User-agent grouping and specificity can produce surprising results. A named group may not inherit the wildcard group as you expect, and implementations differ around Allow, wildcards, encoding, and redirects. Keep rules simple and test every target explicitly.

A sitemap identifies URLs; it does not grant crawl permission. robots.txt also does not protect private content. Use authentication, authorization, a WAF, firewall rules, or server configuration for access control. Allow: / cannot override a login wall, application error, or network block.

Expected result: the production policy contains documented tokens, narrow intent, and no unsupported directives or accidental broad block.

5. Publish at the correct location

Publish the file at the origin root: https://example.com/robots.txt. A file at /blog/robots.txt does not govern the site. Test HTTP and HTTPS, and test example.com and www.example.com independently.

After deployment, require a successful response, valid TLS, readable plain text, no login wall, and no unnecessary redirect chain. Check CDN caching, generated framework routes, middleware, rewrites, and the possibility that an application route returns HTML instead of the file.

Expected result: the exact production URL returns the intended body with a successful HTTP status.

6. Capture and verify the response

Run these commands against the production URL:

curl -iL --max-redirs 5 https://example.com/robots.txt
curl -sS https://example.com/robots.txt | sed -n '1,120p'

Inspect status, Location, Content-Type, cache headers, ETag, Last-Modified, and the complete body. Save evidence and calculate a checksum:

curl -sS -D robots.headers \
  https://example.com/robots.txt \
  -o robots.txt

sha256sum robots.txt
date -u

The SHA-256 value identifies the captured file at that time. It does not establish crawler activity. To inspect user-agent-dependent behavior, make a request with:

curl -i -A 'OAI-SearchBot' https://example.com/robots.txt

curl -A changes only the request header. It does not simulate OpenAI infrastructure or prove identity. Fetch a public URL and restricted URL separately; Allow: / does not prove every application path works.

Expected result: you have robots.txt, robots.headers, a timestamp, checksum, redirect chain, and a documented interpretation.

7. Test parsing and inspect access logs

Use a small matrix. For example, test OAI-SearchBot and GPTBot against / and /private/, recording expected and observed results. The following uses Python 3.12.2’s standard-library urllib.robotparser; confirm this exact Python version and behavior before publication, especially for Allow, wildcards, encoding, and redirects.

from urllib.robotparser import RobotFileParser

ROBOTS_URL = "https://example.com/robots.txt"
TEST_URLS = [
    "https://example.com/",
    "https://example.com/private/",
]
CRAWLER_TOKENS = ["OAI-SearchBot", "GPTBot"]

robots = RobotFileParser(ROBOTS_URL)
robots.read()

for crawler_token in CRAWLER_TOKENS:
    for test_url in TEST_URLS:
        result = robots.can_fetch(crawler_token, test_url)
        print(f"{crawler_token}\t{test_url}\t{'ALLOW' if result else 'DENY'}")

This parser result is not universal operator behavior. Compare it with the captured file and, where available, an operator-provided tester. Pin the runtime in your log: python --version should report Python 3.12.2 for this example.

Inspect server logs for timestamp, path, status, source IP, validated reverse DNS where appropriate, and user-agent. A user-agent alone is spoofable. No matching record means no observed request in the inspected logs, not proof that no request occurred elsewhere.

Expected result: each matrix row has a parser result and, only when real traffic exists, a corresponding log record.

8. Add Search Console and TrustGrowth evidence

In Google Search Console, confirm the verified property and inspect URL Inspection, indexing, and crawl signals for tested pages. These observations concern Google’s systems and your owned property; they do not prove that ClaudeBot, PerplexityBot, GPTBot, or OAI-SearchBot accessed or cited a page.

Run TrustGrowth’s free keyless check at trustgrowth.ai/tools/ai-crawler-access. It reads the site’s robots.txt and performs a live fetch per AI crawler user-agent without requiring an account. Compare its output with your curl capture, recording hostname, timestamp, token, status, and path. A live fetch is a test request, not evidence that the operator made the same request.

TrustGrowth’s read-only Google Search Console connection adds an owned-site audit layer for technical, performance, content, and answer-readiness signals. It does not provide private logs from OpenAI, Anthropic, Google, Perplexity, or another operator.

Expected result: your evidence bundle contains the response capture, parser output, TrustGrowth output, available server-log records, and dated Search Console observations with limitations.

9. Troubleshoot by layer

1. 404, 403, 5xx, HTML, or login: fix origin routing, middleware, permissions, or deployment configuration.
2. Different hostnames: test each protocol and hostname independently; update the correct production file.
3. Stale CDN content: purge the cache and compare Age, ETag, Last-Modified, and capture timestamps.
4. No crawler log entry: permission is not a request. Check operator policy, DNS, TLS, rate limits, scheduling, and log retention.
5. Rule ignored: robots.txt is voluntary. Use authentication, a WAF, or firewall controls when protection is required.
6. Broad block: test public pages, private paths, assets, and canonical URLs after changing Disallow: / or wildcard rules.
7. Conflicting syntax: reduce the file to documented directives and retest with the pinned parser.
8. Different tool results: compare fetched body, redirects, token, parser version, and timestamp before concluding that a rule differs.

Allow: / does not override noindex, canonicalization, robots meta tags, WAF policy, authentication, or application errors. Identify the failing layer instead of attributing every failure to AI crawler access.

10. Verification checklist and next steps

Re-run after deployment and after relevant cache expiry. Record date, hostname, protocol, exact body, headers, token, URLs, parser and version, TrustGrowth output, logs, and limitations. You have demonstrated stated crawl permission and its interpretation—not AI visibility, indexing, retrieval, or citation. Next, repeat the matrix when crawler documentation or production routing changes.