The term generative engine optimization comes from a research paper, GEO: Generative Engine Optimization, submitted to arXiv on 16 November 2023 by Pranjal Aggarwal and co-authors and presented at KDD 2024. In marketing it has since come to mean "make ChatGPT mention my company". We build an audit tool, not a content agency, so our interest in GEO is narrower and more literal: which checkable things on a site make an AI system more or less likely to read and cite a page. This is that list, with the reasoning behind each item and the tool we'd use for it.
What generative engine optimization actually is
Traditional search optimization gets a page ranked in a list of ten blue links. GEO gets a page cited, quoted, or paraphrased inside a generated answer, whether that answer comes from ChatGPT, Perplexity, Google's AI Overviews, or an agent doing research on your behalf. The output isn't a ranking position anymore, it's a sentence in someone else's answer with your domain attached (or not attached, which is its own problem).
That changes what "winning" looks like. You can't check a GEO rank the way you check a SERP position, because the same query can produce a different answer each time, sometimes citing you and sometimes not. What you can control is everything upstream of that: whether the answer engine's crawler was allowed to fetch your page, whether it could parse the content once it did, and whether the passage that answers the query still makes sense once it's lifted out and dropped into a chat window.
How big this channel is for your site is a question for your own analytics, not for a GEO vendor's chart. Look at your referral split before you rearrange a content strategy around it, and treat everything below as additive to your existing SEO work.
How answer engines actually pick sources
None of these companies publish a ranking algorithm. What they do publish is crawler and product documentation, which tells you the mechanics even when it won't tell you the weighting: which bots exist, what each one is for, and what a page needs to be eligible. That's enough to act on.
OpenAI runs three crawlers with three jobs, documented in its overview of OpenAI crawlers: GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a user's request makes ChatGPT visit it. Each one is its own robots.txt token, so you can block training and still show up in ChatGPT search.
Anthropic splits Claude the same way. Per Anthropic's crawler documentation, ClaudeBot collects content that may be used for training, Claude-SearchBot indexes pages to improve Claude's search results, and Claude-User fetches pages when a person asks Claude a question. Anthropic says blocking either of the last two may reduce your visibility in Claude's answers.
Google doesn't run a separate AI crawl for Search. Its guidance on AI features and your website says a page must be indexed and eligible to show in Google Search with a snippet to appear as a supporting link in AI Overviews or AI Mode, and that there are no additional requirements, nor any special schema.org markup you need to add. If Googlebot can already index a page with a snippet, it's eligible. The separate Google-Extended token controls whether your content is used to train Gemini models and for grounding. Google says it has no effect on inclusion in Search and isn't a ranking signal.
Perplexity documents two bots in its crawler docs. PerplexityBot surfaces and links websites in Perplexity's search results, and Perplexity says it isn't used to crawl content for foundation models. Perplexity-User visits a page when a user's question needs it, so the answer can link to that page.
So every one of these systems has to fetch your page (or already have it indexed), parse it, and pull out a chunk that answers a question. Block the fetch and none of the rest matters. Bury the answer under five paragraphs of preamble and the model has to work harder to find it. We can't prove that lowers your odds, since nobody publishes the weighting, but it's hard to see how it helps.
GEO vs SEO: the overlap is bigger than the hype admits
Most GEO advice reduces to "write clear, authoritative, well-structured content", which has also been good SEO advice for a decade. If your site already ranks because it's fast, crawlable, and useful, you're most of the way there.
The parts that are actually new:
- AI crawlers are a separate allow list from search crawlers. You can rank first in Google and still be missing from ChatGPT search if
OAI-SearchBotis blocked in robots.txt. A blanket "block all AI bots" rule copied from a blog post is the usual way it happens, because it catches the search and user-fetch bots along with the training ones. - Passages get lifted out of context, so each one has to stand alone. A search snippet links back to the full page. A generated answer often doesn't, or the link sits below the summarized text. Put the answer to "what is X" in the first sentence of the section about X, not at the end of a five-paragraph buildup.
- Machine-readable summaries are a new surface.
llms.txtand Markdown versions of your pages give agents a shortcut past your nav and cookie banner. Nothing forces an answer engine to use them, but they're cheap to publish.
What's still unproven
We'd rather undersell this than oversell it. Nobody outside OpenAI, Anthropic, Google, and Perplexity knows the retrieval weighting, and a lot of the GEO advice going around, including plenty from vendors selling GEO tools, is inference written up as fact. Two things to keep in mind:
- Citation counts are noisy. Ask the same question twice and you can get different sources cited, because these systems generate answers rather than look them up. Watch trends over weeks, not single queries.
llms.txtis a community proposal published at llmstxt.org, not a standard any answer engine has committed to reading. We recommend it because it's cheap and plausible, not because we have evidence a particular engine consumes it.
The checklist
Run through this in order. Each item can be checked, and most of them map to a squirrelscan audit rule.
1. Confirm the right AI crawlers can reach you. Open robots.txt and look at each named bot by job, the same split our AI crawler guide and the ax/ai-crawlers rule use:
- Training:
GPTBot,ClaudeBot,Google-Extended, plusApplebot-Extended,Meta-ExternalAgent, andBytespider. - Search:
OAI-SearchBot,Claude-SearchBot,PerplexityBot. - User-triggered fetch:
ChatGPT-User,Claude-User,Perplexity-User.
To opt out of training and stay answerable, block the training bots by name and leave the search and user-fetch bots allowed. Blocking a training bot is a policy choice, and the rule only reports it. Blocking a search or user-fetch bot is usually a mistake, so the rule warns. It also warns on CCBot, which feeds the Common Crawl corpus rather than one vendor's model, so blocking it reaches further than blocking one company's crawler.
2. Publish an llms.txt. A plain Markdown file at your domain root: an H1 title, a one-line summary, then linked sections pointing at your most useful pages, in the format defined at llmstxt.org. It takes a few minutes to write and gives an agent a map of your site without crawling all of it. The llms.txt guide has real examples.
3. Make each section answer one question. Lead each H2 with the direct answer, then explain. Compare "Our platform was built by a team who spent years thinking deeply about developer workflows before we ever wrote a line of code, and eventually that led us to..." with "squirrelscan is a CLI that audits websites against 300+ rules." The second sentence is the one that survives being pasted into a chat window. It's also just better writing, whatever it does for GEO.
4. Add schema markup for the content you actually have. Google says AI Overviews and AI Mode need no special structured data, so treat schema.org markup (Article, FAQPage, Product) as a way to say plainly what a page is, not as a ticket into the answer. We ship a faq block on guides like this one because FAQPage markup is already shaped like the question-and-answer pairs these systems produce. It won't guarantee a citation.
5. Keep dates honest and visible. No answer engine documents how it weighs a page's date, so we won't pretend to know. Show an "updated" date, and only change it when the content has actually changed. A date that changes while the page doesn't is a small lie to every reader.
6. Make sure content renders without JavaScript. Vercel's analysis of crawler traffic, The rise of the AI crawler, found that none of the major AI crawlers it measured executed JavaScript, including OpenAI's three bots, ClaudeBot, Meta-ExternalAgent, Bytespider, and PerplexityBot. Gemini was the exception, because it uses Googlebot's rendering infrastructure. If your main content only appears after a script runs, those crawlers don't see it. The ax/content-without-js rule checks for this.
7. If you're behind bot protection, verify crawlers instead of blocking them all. Web Bot Auth lets a crawler prove its identity with HTTP Message Signatures (RFC 9421) instead of a user-agent string anyone can fake. Cloudflare accepts it as a verification method for verified bots and agents, and Shopify uses it to authorize crawlers on storefronts. That way you can let specific verified crawlers through without opening the door to every bot that claims to be one of them.
Tools for each step
GEO tools do different jobs, and no single one covers the whole checklist. Here's where to go for each step:
| Step | Guide | What it covers |
|---|---|---|
| Check which AI crawlers you allow | AI crawlers explained | Every major bot's user-agent and job, robots.txt recipes, and the robots.txt generator |
Publish and validate llms.txt | What is llms.txt? | The format, real examples, and the free llms.txt validator |
| Check the whole site | GEO checker | What a full GEO check tests, the audit rule behind each item, and how to run one from the command line |
| Track whether answer engines mention you | AI visibility trackers | Third-party trackers that run prompts against ChatGPT, Perplexity, and Google, with prices from each vendor's page |
| Compare the whole category | GEO tools | Trackers, content tools, and site auditors side by side |
squirrelscan sits in the "check the whole site" row. It doesn't track mentions or citations in any answer engine, so if you need those numbers, pick a tracker.
How to audit this instead of guessing
Most of the checklist is checked automatically. squirrelscan's agent experience (AX) rules cover AI crawler access (ax/ai-crawlers), llms.txt presence and basic format (ax/llms-txt), whether your main content is there without JavaScript (ax/content-without-js), and how many LLM tokens a page's HTML costs compared with its actual text (ax/token-weight). They run alongside the 300+ other audit rules for SEO, performance, and content, which still matter because GEO didn't replace them. For the rest of the AI-search picture, there's a separate guide on getting cited in ChatGPT.
curl -fsSL https://install.squirrelscan.com | bash
squirrel audit https://example.com --format llmThe --format llm flag renders the report as plain, agent-readable text instead of a dashboard. Which is the same advice this guide gives you, applied to our own output: write for the reader that's actually going to consume it, human or model.