robots.txt generator
Toggle AI crawlers, search bots, and scrapers from a curated, up-to-date list, then copy a ready-to-ship robots.txt. Blocks by default: model training and unknown bots are disallowed, while search, AI search, and live user fetches stay allowed.
Everything else (User-agent: *)
Default rule for any bot not named below. Blocked by default. The named search, AI search, and agent bots stay allowed.
# robots.txt generated by squirrelscan
# https://squirrelscan.com/tools/robots-txt-generator
#
# robots.txt is voluntary: well-behaved crawlers obey it, but nothing
# enforces it. To actually stop non-compliant bots, pair it with a WAF.
# AI training crawlers
# Note: Meta-ExternalAgent, Bytespider often ignore robots.txt; enforce with a WAF/edge rule.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: cohere-ai
User-agent: AI2Bot
User-agent: Diffbot
Disallow: /
# AI search & answer engines
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
User-agent: YouBot
Allow: /
# AI agents & on-demand fetchers
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
Allow: /
# Archive crawlers
User-agent: CCBot
User-agent: ia_archiver
User-agent: archive.org_bot
Allow: /
# Traditional search engines
User-agent: Googlebot
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: Applebot
User-agent: YandexBot
Allow: /
# SEO & marketing tools
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DataForSeoBot
User-agent: MJ12bot
User-agent: rogerbot
Disallow: /
# Scrapers & data aggregators
User-agent: Scrapy
User-agent: SeekportBot
User-agent: DotBot
Disallow: /
# Everyone else
User-agent: *
Disallow: /
How to block AI bots without losing AI search
"Block AI" isn't one setting. It's several policies wearing the same name, and each one has its own user-agent. A training crawler like GPTBot or ClaudeBot collects pages to train a future model. An AI-search indexer like OAI-SearchBot is what puts you in an assistant's cited answers. A user-action fetcher like ChatGPT-User or Claude-User loads a page live because a real person just asked about it.
They're separate tokens, so blocking one never blocks another: disallowing ClaudeBot (training) does nothing to Claude-User (a live customer). The common mistake is a blanket block that quietly removes a site from AI search results and breaks live lookups people made inside an assistant. The defaults here take the middle ground: opt out of training, stay answerable. Every bot is yours to override.
Archive crawlers are their own case. CCBot feeds the Common Crawl corpus that most AI labs train from, and ia_archiver feeds the Wayback Machine. Blocking CCBot opts you out of every downstream model at once, not just one vendor. That is a bigger decision than blocking a single training bot, which is why they default to allowed.
One caveat worth repeating: robots.txt is voluntary. Reputable crawlers obey it, but some, Bytespider and Meta's crawlers among them, routinely don't. For those, a Disallow line is a request, not a wall; you need a WAF or edge rule to enforce it. This tool flags the bots that fall into that category.
Keep going
- AI crawlers guide: what each major AI crawler does, its user-agent token, and robots.txt recipes.
- AI crawler checker: check the robots.txt you deployed and what your CDN or firewall answers each AI crawler. It can open this generator pre-filled from your live file.
- llms.txt validator: check the file that points AI agents at your most useful pages against the llmstxt.org spec.
- AI crawlers rule: how a squirrelscan audit classifies the AI crawlers your robots.txt allows or blocks.
- robots.txt rule: the checks an audit runs on the file itself, from syntax errors to a site-wide block.
Frequently asked questions
- How do I block AI bots in robots.txt?
- Add a group naming the crawler's user-agent token followed by Disallow: /. For example, to block OpenAI's training crawler you write 'User-agent: GPTBot' then 'Disallow: /'. This generator builds those groups for you from a curated, up-to-date list of AI crawlers, so you copy a finished file instead of hand-writing tokens.
- Does blocking GPTBot stop ChatGPT from citing my site?
- No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot and live user fetches use ChatGPT-User, both separate user-agents. You can block training while staying citable by blocking GPTBot but allowing OAI-SearchBot and ChatGPT-User, which is exactly what the default settings here do.
- Is robots.txt actually enforced?
- No. robots.txt is a voluntary standard. Well-behaved crawlers (Google, OpenAI, Anthropic) honor it, but some, notably Bytespider and Meta's crawlers, routinely ignore it. To truly block those you need a WAF or edge rule; the generator flags the bots that need it.
- Where does robots.txt go?
- At the root of your domain, at /robots.txt (for example example.com/robots.txt). It must be served as plain text. Save the generated file, deploy it there, and re-run a squirrelscan audit to confirm crawlers see what you intended.
- Is it free?
- The llms.txt validator, the AI crawler checker, the robots.txt generator and the instant security headers check are free and need no account. The other checks run inside a squirrelscan audit. The squirrel CLI runs audits on your machine for free when you are signed out or pass --offline, with no cap on how many. Signed in, CLI and cloud audits spend credits: every account gets 500 free credits a month. An audit costs 50 credits plus 2 per audited page, and 1 per external link the cloud verifies when link checks are on.
Check the robots.txt you deployed
A squirrel audit reads your live robots.txt. It lists the AI crawlers you allow and block, warns when an AI search or user-action fetcher is blocked, and checks the file for syntax errors.
In the cloud
squirrelscan crawls the site for you and keeps each report in your dashboard. Every account gets 500 free credits a month. An audit costs 50 credits plus 2 per audited page, and 1 per external link the cloud verifies.
On your machine
The squirrel CLI runs the same rules locally, free while you are signed out or with --offline. Install it, then run:
squirrel audit https://example.com