AI crawlers explained: GPTBot, ClaudeBot, PerplexityBot, and how to control them

Someone on our team once blocked "all the AI bots" in robots.txt with a single User-agent: * rule pointed at a list they'd copy-pasted from a blog post, and then couldn't figure out why the site had stopped showing up in ChatGPT search results at all, not just stopped being used for training. That's the trap: AI crawlers aren't one thing. They're three categories doing three different jobs, and a robots.txt rule that treats them as interchangeable usually breaks the thing you wanted to keep.

Three categories, not one blob

Before the list of bots, the framework that actually matters:

• Training crawlers scrape your content to include in a future model's training data. Blocking these opts you out of training, permanently and retroactively for future crawls, with no effect on how any AI product answers questions about your site today.
• Search and answer crawlers index your content so an AI product can cite or link to it in real-time answers, the AI equivalent of Googlebot. Blocking these removes you from that product's answers entirely.
• User-triggered fetchers fire only when a human explicitly asks an assistant to open a specific URL. Blocking these means the assistant can't open a link a user pastes in, even though the user is asking on purpose.

Confusing category two for category one is the single most common AI-crawler mistake, because a lot of vendors reuse a similar-sounding name (GPTBot vs OAI-SearchBot vs ChatGPT-User) across all three.

The field guide

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

Per OpenAI's crawler documentation:

• GPTBot (training): identifies itself as GPTBot/1.3, crawls to improve future foundation models, respects robots.txt. IP list at openai.com/gptbot.json.
• OAI-SearchBot (search): powers ChatGPT's search features, not used for training, respects robots.txt. IP list at openai.com/searchbot.json.
• ChatGPT-User (user-triggered): fires when someone asks ChatGPT or a Custom GPT to visit a URL, does not respect robots.txt since it's a direct user action, not automated crawling. IP list at openai.com/chatgpt-user.json.

Allowing one doesn't allow the others. That's by design, per OpenAI: each token is its own switch.

Anthropic: ClaudeBot, Claude-User, Claude-SearchBot

Per Anthropic's crawler documentation, Anthropic runs three separate bots with a shared UA pattern:

• ClaudeBot (training): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). Anthropic's primary web-scale crawler for model development.
• Claude-User (user-triggered): fires when someone asks Claude to open a specific page, same pattern with Claude-User swapped in.
• Claude-SearchBot (search): indexes content quality to power Claude's retrieval and search-style answers, not a training crawler.

All three respect robots.txt and are documented against Anthropic's published IP ranges.

Perplexity: PerplexityBot, Perplexity-User

Per Perplexity's crawler docs:

• PerplexityBot (search): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot), surfaces and links sites in Perplexity's answers, respects robots.txt. IP list at perplexity.com/perplexitybot.json.
• Perplexity-User (user-triggered): fires when a user's question makes Perplexity fetch a specific page live, generally ignores robots.txt since it's a direct user action. IP list at perplexity.com/perplexity-user.json.

Worth flagging: Cloudflare published research in August 2025 showing Perplexity also running undeclared crawlers that rotate user-agents and IPs to get around disallow rules on some sites, separate from the two documented tokens above. Documented and honest isn't the same as exhaustive.

Google-Extended

This one's different in kind. Per Google's crawler documentation, Google-Extended isn't a crawler with its own HTTP requests. It's a robots.txt-only control token: Google still crawls with the regular Googlebot user agent, but a site can add a Google-Extended group to opt content out of training and grounding for Gemini apps and Vertex AI, independent of whether that same content stays indexed in Google Search. Blocking Google-Extended has zero effect on your Search rankings; it's a training opt-out lever, not a visibility lever.

CCBot

Common Crawl's bot, CCBot/2.0 (https://commoncrawl.org/faq/), feeds the open dataset that underlies a large share of the LLM training corpora in use industry-wide, including models that never crawl your site directly. It respects robots.txt. Common Crawl warns that impersonators exist, so verify by reverse-DNS: legitimate requests resolve to [ip].crawl.commoncrawl.org.

Bytespider

ByteDance's crawler, Bytespider (+http://www.bytedance.com), gathers data for the company's LLMs. Unlike the others on this list, ByteDance doesn't publish a dedicated crawler documentation page or an IP verification list, and site owners have widely reported it continuing to fetch paths disallowed in robots.txt. Cloudflare's own numbers have repeatedly put Bytespider at or near the top of both crawl volume and how often a bot gets blocked outright. If you're only going to disallow one crawler by name and mean it, this is the one where robots.txt alone probably isn't enough; pair it with a server-side or WAF-level block if you want it to actually stop.

robots.txt recipes

Opt out of training, stay visible in AI answers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

This blocks the training crawlers by name while leaving OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, and Perplexity-User uncovered by any specific rule, so they fall through to the * group and stay allowed. That's usually the right default if you want to show up when someone asks ChatGPT or Perplexity a question, but don't want your content baked into the next model release.

Block everything, training and answers both:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

A legitimate choice if you'd rather opt out of the AI ecosystem entirely, just go in knowing it also removes you from ChatGPT search, Perplexity answers, and Claude citations, not only training sets.

Verifying a crawler is who it says it is

User-agent strings are just a header a client sends; nothing stops a scraper from claiming to be GPTBot. The documented mitigation is checking the source IP against each operator's published list (linked above for OpenAI, Anthropic, and Perplexity) or doing a reverse-DNS lookup for crawlers like CCBot that use that pattern instead. Both approaches work but require you to maintain an allowlist and refresh it as IP ranges change.

The newer, more durable approach is Web Bot Auth, a standard that has crawlers sign their requests with HTTP message signatures tied to a published key, so you can verify identity cryptographically instead of trusting a header or an IP range that might be stale by the time you check it.

squirrelscan checks this automatically

Working out which of 20-plus known AI-agent tokens your robots.txt allows or blocks by hand is exactly the kind of thing that should be automated. squirrelscan's ax/ai-crawlers rule runs on every audit, parses your robots.txt, and reports which crawlers are explicitly allowed, explicitly blocked, or just falling through to your * rule, broken out by vendor and purpose. It's informational, not scoring, because blocking these bots is a legitimate choice; the point is making sure it's a choice you made on purpose rather than one a copy-pasted robots.txt snippet made for you.

Pair it with the llms.txt guide if you're also curating what AI agents see when they do have access, and the generative engine optimization pillar for the rest of the AI-visibility picture, including showing up in ChatGPT's answers once you've confirmed the right crawlers can actually reach your content.

Check which AI crawlers your own robots.txt currently allows and blocks:

curl -fsSL https://install.squirrelscan.com | sh
squirrel audit https://example.com --format llm

Or run it from the browser: check your site free or kick off a full audit and squirrelscan will break down your AI-crawler access alongside 249 other rules.