robots.txt generator
Toggle AI crawlers, search bots, and scrapers from a curated, up-to-date list, then copy a ready-to-ship robots.txt. Blocks by default — model training and unknown bots are disallowed while search, AI search, and live user fetches stay allowed.
Everything else (User-agent: *)
Default rule for any bot not named below. Blocked by default — the named search, AI search, and agent bots stay allowed.
# robots.txt generated by squirrelscan
# https://squirrelscan.com/tools/robots-txt-generator
#
# robots.txt is voluntary: well-behaved crawlers obey it, but nothing
# enforces it. To actually stop non-compliant bots, pair it with a WAF.
# AI training crawlers
# Note: Meta-ExternalAgent, Bytespider often ignore robots.txt — enforce with a WAF/edge rule.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: cohere-ai
User-agent: AI2Bot
User-agent: Diffbot
Disallow: /
# AI search & answer engines
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
User-agent: YouBot
Allow: /
# AI agents & on-demand fetchers
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
Allow: /
# Archive crawlers
User-agent: CCBot
User-agent: ia_archiver
User-agent: archive.org_bot
Allow: /
# Traditional search engines
User-agent: Googlebot
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: Applebot
User-agent: YandexBot
Allow: /
# SEO & marketing tools
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DataForSeoBot
User-agent: MJ12bot
User-agent: rogerbot
Disallow: /
# Scrapers & data aggregators
User-agent: Scrapy
User-agent: SeekportBot
User-agent: DotBot
Disallow: /
# Everyone else
User-agent: *
Disallow: /
How to block AI bots without losing AI search
"Block AI" isn't one setting — it's several policies wearing the same name, and each one has its own user-agent. A training crawler like GPTBot or ClaudeBot collects pages to train a future model. An AI-search indexer like OAI-SearchBot is what puts you in an assistant's cited answers. A user-action fetcher like ChatGPT-User or Claude-User loads a page live because a real person just asked about it.
They're separate tokens, so blocking one never blocks another: disallowing ClaudeBot (training) does nothing to Claude-User (a live customer). The common mistake is a blanket block that quietly removes a site from AI search results and breaks live lookups people made inside an assistant. The defaults here reflect the useful middle ground — opt out of training, stay answerable — but every bot is yours to override.
Archive crawlers are their own case. CCBot feeds the Common Crawl corpus that most AI labs train from, and ia_archiver feeds the Wayback Machine. Blocking CCBot opts you out of every downstream model at once, not just one vendor — a bigger decision than blocking a single training bot, which is why they default to allowed.
One caveat worth repeating: robots.txt is voluntary. Reputable crawlers obey it, but some — Bytespider and Meta's crawlers among them — routinely don't. For those, a Disallow line is a request, not a wall; you need a WAF or edge rule to enforce it. This tool flags the bots that fall into that category.
Frequently asked questions
- How do I block AI bots in robots.txt?
- Add a group naming the crawler's user-agent token followed by Disallow: /. For example, to block OpenAI's training crawler you write 'User-agent: GPTBot' then 'Disallow: /'. This generator builds those groups for you from a curated, up-to-date list of AI crawlers, so you copy a finished file instead of hand-writing tokens.
- Does blocking GPTBot stop ChatGPT from citing my site?
- No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot and live user fetches use ChatGPT-User — both separate user-agents. You can block training while staying citable by blocking GPTBot but allowing OAI-SearchBot and ChatGPT-User, which is exactly what the default settings here do.
- Is robots.txt actually enforced?
- No. robots.txt is a voluntary standard. Well-behaved crawlers (Google, OpenAI, Anthropic) honor it, but some — notably Bytespider and Meta's crawlers — routinely ignore it. To truly block those you need a WAF or edge rule; the generator flags the bots that need it.
- Where does robots.txt go?
- At the root of your domain, at /robots.txt (for example example.com/robots.txt). It must be served as plain text. Save the generated file, deploy it there, and re-run a squirrelscan audit to confirm crawlers see what you intended.