Check if AI crawlers can access my site: a step-by-step verification method

Short answer: to check whether AI crawlers can access your site, test three configuration layers — robots.txt directives, per-crawler HTTP response, and CDN/WAF bot rules — then confirm against server access logs, and date every result, because each layer can change on its own schedule and disagree with the others.

Prerequisites

Before starting, confirm you have:

• Terminal access with curl installed
• Access to the site's robots.txt and, ideally, the hosting control panel
• Basic familiarity with HTTP status codes (200, 403, 429, 503)
• Server access logs (nginx, Apache, or CDN export) — optional, needed for Step 6
• A CDN/WAF dashboard (e.g., Cloudflare) if the site uses one — optional, needed for Step 5
• A Google Search Console–verified property, if using the automated check in Step 7

Introduction

AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Amazonbot, Applebot-Extended, and CCBot — can be blocked by robots.txt, CDN bot-management rules, or server configuration without the owner noticing. Traffic logs stay quiet, nobody files a ticket, and the block persists.

This method checks access at four layers: robots.txt directives, HTTP-level response to crawler user-agents, CDN/WAF bot rules, and live access-log evidence. Steps 2–5 read what the configuration says should happen; only Step 6 shows what a real crawler fetch actually returned. The status codes in the tables below are illustrative placeholders showing the format to fill in — not measured results — because a crawler's verdict depends on source IP and TLS fingerprint that a terminal request cannot reproduce. Run each command against your own domain and timestamp every result.

Step 1: Identify which AI crawlers to test

Start with a defined list rather than every string you've seen mentioned online. The table lists seven AI-related crawlers with published, named operators. Check each source directly before you rely on it and record the date you checked it — operators add or rename crawlers without a changelog, so this is a starting reference, not a live registry.

Crawler User-agent string Operator Source Date checked (name) (HTTP User-Agent value) (company) (documentation URL) (YYYY-MM-DD) GPTBot GPTBot OpenAI OpenAI – GPTBot (fill in) ClaudeBot ClaudeBot Anthropic Anthropic – crawler docs (fill in) Google-Extended Google-Extended Google Google Search Central – crawlers (fill in) PerplexityBot PerplexityBot Perplexity Perplexity – bots (fill in) Amazonbot Amazonbot Amazon Amazon Developer – Amazonbot (fill in) Applebot-Extended Applebot-Extended Apple Apple Support – Applebot-Extended (fill in) CCBot CCBot Common Crawl Common Crawl – CCBot (fill in)

Step 2: Locate and read robots.txt

curl -s https://yourdomain.com/robots.txt

A 200 returns the raw file. A 404 means no robots.txt exists, which under the Robots Exclusion Protocol (RFC 9309, September 2022) implies no crawl restrictions at this layer. Read the file for User-agent blocks matching each name from Step 1 and note the Disallow/Allow paths. A block reading User-agent: GPTBot / Disallow: / asks GPTBot not to fetch any path. A generic User-agent: * block applies to every crawler not named elsewhere in the file.

Step 3: Test HTTP response per crawler using curl

robots.txt states intent; it does not prove the server serves or blocks a given user-agent. Test each string against the live endpoint:

curl -A "GPTBot" -I https://yourdomain.com/
curl -A "ClaudeBot" -I https://yourdomain.com/
# ...repeat for each user-agent from Step 1

Record the status line from each response. The table below is illustrative — it shows the format to fill in, not measured results.

Crawler User-agent sent HTTP status Notes Test timestamp (name) (string sent via -A) (status code) (free text) (ISO 8601)

Interpretation: 200 means the server responded as it would for any client. 403/401 means the server or an intermediary rejected the request by identity. 429 means rate-limited, not permanently blocked. 503 can indicate a bot-challenge page rather than an outage on some CDNs.

Step 4: Check the X-Robots-Tag header and meta robots tag

curl -sI https://yourdomain.com/page | grep -i x-robots-tag

Then view page source and search the <head> for a <meta name="robots" content="noai, noimageai"> tag. The noai and noimageai values are not IETF-standardized robots directives; they originate from a 2023 publisher proposal and are honored only by operators who opt in. Page-level directives can supplement or override robots.txt scope, so check both layers before concluding a crawler is blocked or allowed.

Step 5: Check CDN/WAF-level bot management

In Cloudflare: Security → Bots and Security → WAF → Custom rules, looking for rules matching AI crawler user-agents or a "Block AI Bots" toggle. Fastly, Akamai, and AWS WAF offer equivalent products; consult each vendor's bot-management documentation for the exact path.

This is a configuration check, not a behavioral test. A curl request with a spoofed user-agent does not replicate a crawler's real source IP or TLS fingerprint, and these products can key decisions on both — so read the dashboard rule directly rather than inferring it from curl output.

Step 6: Cross-reference with server access logs

grep -iE "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Amazonbot|Applebot-Extended|CCBot" /var/log/nginx/access.log | tail -50

This shows crawler visits that actually occurred, with the status the server actually returned — the one layer that reflects real crawler behavior rather than inferring it. Log retention limits how far back this evidence reaches; a server rotating logs weekly cannot answer about traffic from two months ago. State the retention window next to any finding from this step.

Step 7: Verify with an automated check

TrustGrowth's crawler access check automates the robots.txt parse, per-crawler HTTP fetch, and meta-tag inspection from Steps 2–4, run from TrustGrowth's own infrastructure against a Google Search Console–verified property, and outputs a timestamped proof page listing crawler-by-crawler status with the method and date attached. It does not read your CDN/WAF dashboard (Step 5) or your server access logs (Step 6) — those require credentials TrustGrowth does not have — so treat the proof page as covering three of the four layers, not all four.

Step 8: Document and interpret findings

Combine Steps 2–6 into one dated record. Template — reader-completed, not measured data.

Crawler robots.txt HTTP (curl) CDN status Log evidence Date checked (name) (Allow/Disallow/none) (status code) (Allowed/Blocked/Unknown) (Yes/No) (YYYY-MM-DD)

Date every entry independently, since robots.txt, CDN rules, and log windows change on different schedules. A 403 from curl correlates with likely blocking but does not prove the crawler itself received a 403; Step 6 log evidence narrows that gap because it reflects a request the crawler actually made.

Limitations

This method establishes whether a crawler can fetch a URL under current configuration. It does not measure inclusion in AI answers, citations, or rankings (a 200 confirms only that the request succeeded); the behavior of a crawler's real traffic, which arrives from published IP ranges a terminal request cannot reproduce (only Step 6 reflects that); whether an operator honors robots.txt, which RFC 9309 defines but does not enforce; or any history beyond your log retention window.

Common pitfalls and troubleshooting

• JavaScript-rendered content is invisible to curl. curl returns raw HTML and does not execute JavaScript, so client-injected content may be absent even when curl returns 200.
• Caching can serve a stale robots.txt. Test with a cache-busting query (?cachebust=$(date +%s)) or purge the CDN cache before concluding a recent edit took effect.
• 429 is not 403. Rate-limiting and blocking are different decisions; record which you observed and when.
• Log grep can miss rotated or renamed user-agent strings. Cross-check any log finding against the operator's current documentation.

Frequently asked questions

Does a 200 status confirm my content will be indexed or cited?

No. A 200 confirms only that the HTTP request succeeded and the server returned content. Whether that content is later used in a model's training set, retrieval index, or a generated answer is a separate, unmeasured outcome this method does not test.

Is robots.txt compliance enforced?

No. RFC 9309 (September 2022) defines the file format and how compliant crawlers should interpret it, but nothing in the protocol forces an operator's crawler to honor it. Treat any claim that a specific crawler ignores robots.txt as unconfirmed until you have Step 6 log evidence showing it on your own domain.

Are Google-Extended and Googlebot the same?

No. Googlebot controls eligibility for Google Search indexing; Google-Extended controls use of content for Gemini and other AI training, per Google's crawler documentation. A site can Allow one and Disallow the other in the same file, so check them independently.

Why can a curl test and a CDN dashboard rule disagree?

A WAF or bot-management product may decide on IP reputation or TLS fingerprint, not user-agent string alone. A curl -A "GPTBot" request from your terminal is not sent from OpenAI's published IP ranges, so it can receive a different response than the crawler's real traffic. Read the CDN rule directly (Step 5).

Conclusion

The four layers — robots.txt (Step 2), per-crawler HTTP response (Step 3), CDN/WAF rules (Step 5), and access-log evidence (Step 6) — answer different questions and can disagree. Record a timestamp and a source next to every finding, since crawler lists, CDN rules, and log retention windows all change on independent schedules. An automated proof page (Step 7) covers Steps 2–4 on a repeatable schedule but does not substitute for checking your own CDN dashboard and logs directly.