AI crawler checker
See which AI crawlers can reach your site. We read your robots.txt for each bot, then request your homepage as that bot to catch the blocks robots.txt does not show.
The checker reads your /robots.txt and tells you, for each of the 24 AI crawlers a squirrelscan audit knows, whether the file blocks it. Then it requests your homepage as a browser and as each crawler, and compares the answers. That second step finds the block robots.txt cannot show: a CDN or firewall rule that refuses a bot your robots.txt lets in.
Results are grouped by what each bot is for, because "block AI" covers four different decisions. Blocking a training crawler is a policy choice. Blocking an AI search indexer or a user-action fetcher costs you citations and live visits, and it is usually an accident.
What it checks
robots.txt, bot by bot
Reads the file the way RFC 9309 groups it and marks each crawler allowed or blocked at the site root. It also says whether the bot has its own group or falls back to User-agent: *.
The edge, user agent by user agent
Requests the homepage as a browser and as 22 crawlers, and flags a 401, 403 or 5xx, a bot challenge, or a near-empty page the browser did not get. An audit's ax/agent-blocking rule sends two of these user agents.
How to use it
- 1
Enter your domain
Type a domain such as example.com. The checker reads /robots.txt at the root and requests the homepage. It fetches nothing else and stores nothing.
- 2
Read the warnings first
A warning means an AI search or user-action bot is blocked, an archive crawler is blocked, or the edge refuses a bot that robots.txt allows. Blocking only training crawlers is shown as information.
- 3
Compare the two columns
robots.txt is what you asked crawlers to do. Edge is what the bot got when we sent its user agent. When they disagree, the edge wins: a bot that gets HTTP 403 never reads your pages.
- 4
Fix it and check again
Change the firewall rule, or edit robots.txt in the generator, which opens pre-filled from your live file. Then run the checker again. Each network gets three checks a minute.
How to fix what it finds
An AI search or user-action bot is blocked in robots.txt
Give the bot its own group with Allow: /, or take it out of the group that ends in Disallow: /. Training bots are separate tokens, so they can stay blocked.
robots.txt
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Reported by ax/ai-crawlers
robots.txt allows a bot, but the edge answers 403 or a challenge
Something in front of the site matches the bot's user agent. On Cloudflare, look at the AI bot policies under Security settings and at any WAF custom rule on the User-Agent header. Scope the block to the bots you mean. If assistant users should reach you, allow ChatGPT-User, Claude-User and Perplexity-User.
Reported by ax/agent-blocking
An archive crawler is blocked
Blocking CCBot keeps the site out of Common Crawl, which many training sets are built from. Blocking the Internet Archive bots removes it from the Wayback Machine. If you only meant to opt out of one vendor's training, block that vendor's training bot instead.
Reported by ax/ai-crawlers
robots.txt could not be read
A 401, 403 or 429 on /robots.txt usually means a firewall is challenging the file itself. Under RFC 9309 a crawler that gets a 4xx may treat the site as unrestricted, and one that gets a 5xx must stay out, so neither answer says what you meant. Serve /robots.txt as plain text to every client.
Reported by crawl/robots-txt
Why a robots.txt reader is not enough
On 23 October 2026 squirrelscan.com answered HTTP 403 to GPTBot, ClaudeBot and CCBot while its robots.txt allowed every crawler. PerplexityBot, OAI-SearchBot and a browser got 200. Four of the six free checkers we tried that day reported those bots as allowed, because they only read the file. We have since lined the edge up with robots.txt, and on 24 October all three got 200.
The 403 was served by Cloudflare, in front of the app. That is the case this checker is built for: the policy you wrote and the answer a bot gets are two separate things, and only the answer reaches the bot.
What this does not do
It sends each bot's user agent from squirrelscan's servers, not from the vendor's network. Plenty of edge rules match on the user agent alone, and Cloudflare's free-plan AI crawler detection works from user agent strings, so a block on the bot's name shows up here. Where a vendor publishes no full user agent string, we send a short one that carries the bot's token. A rule that also checks the vendor's published IP ranges can treat our request as a fake bot and the real crawler differently, in either direction.
It checks the homepage and the site root. Like the ax/ai-crawlers rule, it counts a bot as blocked when robots.txt disallows / for it, so a rule that blocks only /blog/ shows as allowed. A full audit checks every crawled page against robots.txt.
Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind them. Google crawls with its usual user agents and Apple says Applebot-Extended does not crawl pages; the token only decides how crawled content is used for training. There is nothing to request, so the edge column says not probed.
robots.txt is a request, not a lock. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them, because a person asked for the page. Keeping those out takes an edge rule.
Keep going
What each AI crawler does, its token, and robots.txt recipes.
robots.txt generatorBuild a file that blocks training bots and keeps AI search open.
GEO checker guideWhat free GEO checkers test, and what they miss.
llms.txt validatorCheck the file that points AI agents at your best pages.
ax/ai-crawlers ruleHow an audit classifies the AI crawlers robots.txt allows or blocks.
All free toolsThe other checks, from llms.txt to security headers.
Frequently asked questions
- Does blocking GPTBot stop ChatGPT from citing my site?
- No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and pages a person asks ChatGPT about are fetched as ChatGPT-User. OpenAI documents them as separate tokens, so you can block GPTBot and keep the other two allowed.
- Why does a bot show as blocked when my robots.txt allows it?
- Something in front of the site refused the request: a CDN bot setting, a WAF rule or a bot management product. robots.txt asks crawlers to stay away; the edge decides who gets a response. Find the rule that matches the bot's user agent and narrow it.
- Is the edge result what the real crawler gets?
- Not always. We send the bot's user agent from our own servers, so the result is what the site answers to that name from our network. A rule that matches only the user agent treats us like the real bot. A rule that also checks the vendor's IP ranges, which Perplexity recommends for its allow rules, can answer our request and the real crawler differently.
- Which AI crawlers does it check?
- The same 24 the ax/ai-crawlers audit rule knows, from OpenAI, Anthropic, Google, Common Crawl, Internet Archive, Perplexity, Apple, ByteDance, Amazon, Meta, Cohere, DuckDuckGo, Mistral, Allen AI, Diffbot and You.com. 22 of them are also requested at the edge; Google-Extended and Applebot-Extended have no crawler of their own to request.
- How is this different from the robots.txt generator?
- The generator writes a file. The checker reads the one you deployed and tests the edge. After a check, the generator button carries your live file's decisions over, so you can flip a few bots and copy a new file.
- Is it free?
- The llms.txt validator, the AI crawler checker, the robots.txt generator and the instant security headers check are free and need no account. The other checks run inside a squirrelscan audit. The squirrel CLI runs audits on your machine for free when you are signed out or pass --offline, with no cap on how many. Signed in, CLI and cloud audits spend credits: every account gets 500 free credits a month. An audit costs 50 credits plus 2 per audited page, and 1 per external link the cloud verifies when link checks are on.
Check every page, not just the root
A squirrel audit runs ax/ai-crawlers and ax/agent-blocking with the rest of its agent experience rules, and checks every crawled page against your robots.txt.
In the cloud
squirrelscan crawls the site for you and keeps each report in your dashboard. Every account gets 500 free credits a month. An audit costs 50 credits plus 2 per audited page, and 1 per external link the cloud verifies.
On your machine
The squirrel CLI runs the same rules locally, free while you are signed out or with --offline. Install it, then run:
squirrel audit https://example.com