What is llms.txt? The practical guide, with real examples

I went looking for who actually ships an llms.txt file expecting to find a handful of docs sites and Anthropic. What I found instead was a file format that's simultaneously more widely adopted than I expected (tens of thousands of domains by mid-2026) and less useful than its advocates claim, according to the crawl-log data that's now public. Both things are true at once, so let's get into what the file is, what it looks like in practice, and what happens when you publish one.

What llms.txt is

llms.txt is a proposed standard, published at llmstxt.org, created by Jeremy Howard of Answer.AI in September 2024. The pitch is simple: language models have small context windows relative to the size of a real website, and the HTML on most pages is full of navigation chrome, ads, and script tags that waste tokens without adding meaning. llms.txt gives a site a single, curated, plain-Markdown entry point that says "here's what this is, and here's where the useful pages live," so an agent doesn't have to guess from a sitemap or scrape a nav bar.

It's explicitly modeled on robots.txt: same idea of a well-known file at a fixed root path, same spirit of "a small text file that machines can trust more than they can trust guessing."

The exact format

The spec is deliberately minimal. A valid llms.txt is:

# Project or brand name
 
> A one-line summary of what this is.
 
Optional free-text context paragraph, still Markdown.
 
## Section name
 
- [Page title](https://example.com/page): short description
- [Another page](https://example.com/other): short description
 
## Optional
 
- [Less essential page](https://example.com/extra): short description

Four rules matter: the file starts with an H1, the H1 can be followed by a blockquote summary, content is grouped under H2 headings, and each entry under a heading is a Markdown link with an optional colon-separated description. An ## Optional section is a convention for pages worth including only if the consumer has room to spare, since not every agent will fetch everything listed.

The llms-full.txt variant

The same spec defines a second, optional file: /llms-full.txt. Where llms.txt is an index of links, llms-full.txt inlines the actual content those links point to, concatenated into one Markdown document. It's meant for tools with large enough context windows to just ingest the whole thing in one request instead of following twelve separate links. Not every site needs both; a llms.txt index is the baseline, llms-full.txt is the upgrade for content-heavy docs sites where round-tripping dozens of pages is the bottleneck.

Real examples worth reading

The most useful way to learn the format is to read a few. Anthropic, Stripe, Cursor, Zapier, and Vercel, all publish one, mostly because their documentation platforms generate it automatically. That's a pattern worth noticing: llms.txt adoption clusters heavily around developer docs, where the file doubles as a table of contents for exactly the audience (coding agents, MCP clients) most likely to fetch it.

We publish one too, because we'd have felt ridiculous auditing other people's sites for a rule we didn't pass ourselves. Here's the real thing, unedited, at squirrelscan.com/llms.txt:

# squirrelscan
 
> The website QA tool for your coding agent. 249+ SEO, performance,
> security and agent experience rules, run from the CLI, your
> coding agent, the cloud, or over MCP.
 
squirrelscan finds broken links, missing meta tags, slow pages,
and security gaps, then hands you the fix, not just a score.
 
## Docs
 
- [Getting started](https://docs.squirrelscan.com/quickstart): install the CLI and run your first audit
- [CLI reference](https://docs.squirrelscan.com/cli): commands, flags, and config
- [MCP and developers](https://docs.squirrelscan.com/developers/mcp): connect an agent over MCP or the HTTP API
- [Rules](https://docs.squirrelscan.com/rules): the full audit rule catalog

Nothing clever: an H1, a two-sentence summary, and links grouped by what an agent would actually want to do first. That's the whole trick.

How many sites have one

Adoption grew fast off a tiny base. Casey Burridge's crawl of the web found something like 951 domains publishing llms.txt in mid-2025; by May 2026, that had climbed to roughly 36,000, an 8.8x jump in under a year. An SE Ranking scan of 300,000 domains put overall adoption at just over 10%, and closer to 6% among the top 10,000 sites, which tells you the file skews toward developer-facing tools and docs platforms rather than the median website. If you're on Mintlify, Docusaurus with the right plugin, or a handful of other docs generators, you probably already have one and never noticed.

The honest controversy: does anyone read it?

Here's where I have to walk back some of the enthusiasm. In July 2025, Google's Gary Illyes said at Search Central Live that Google does not support llms.txt and isn't planning to. John Mueller went further, comparing it to the keywords meta tag: a signal so easy to game that no serious search engine has ever used it. Google's own AI-features guidance now lists llms.txt by name in a mythbusting section, as a tactic that doesn't help.

The bigger surprise came from the crawl-log side. An Ahrefs analysis of over 137,000 sites publishing llms.txt found that 97% of those files received zero requests from AI crawlers in a given month, and where requests did happen, SEO audit tools accounted for more of the traffic than actual AI assistants did. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip straight to crawling HTML and never touch the file at all.

So who does read it? Mostly IDE agents and MCP-connected tools: Cursor, Continue, Cline, and anything that treats "fetch this URL and summarize it" as a first-class action rather than a background crawl. That's a narrower audience than "every AI system on the internet," but it's a real one, and it happens to overlap heavily with the exact people evaluating a developer tool like squirrelscan.

Worth being honest about the numbers before you spend a sprint on this: llms.txt is not an SEO lever, it will not move you in AI Overviews, and most of the crawlers you're picturing when you build one will never request it. It's a small, cheap convenience for a narrower set of consumers than the hype suggests, and it's worth doing for exactly that reason and no more.

How to create one

1. Write the H1 and one-line summary first. If you can't summarize your site in one sentence, that's useful information on its own.
2. List your five to ten most important pages under one or two H2 sections, ordered by what a new visitor (human or agent) would want first: quickstart, reference, pricing, contact.
3. Keep descriptions to a phrase, not a paragraph. The point is a map, not a mirror of the page.
4. Serve it as static plain text at /llms.txt, no auth, no redirect.
5. If your docs are large and you want a single-fetch option, generate /llms-full.txt by concatenating the same pages' content. Docs platforms like Mintlify do this automatically; for a hand-rolled site, a small build script that walks your content directory works fine.
6. Skip the keyword-stuffing temptation. It reads as spam to the handful of tools that do parse it, and it does nothing for the tools that don't.

The most common mistake I see is treating llms.txt like a second sitemap.xml and dumping every URL on the site into it. That defeats the point. A sitemap tells a crawler everything exists; llms.txt is supposed to tell an agent what matters. If your llms.txt has 200 links in undifferentiated bullet lists, you've built a worse sitemap, not a better one.

Validate yours

Once you've written one, don't trust it blind. Run it through the llms.txt validator to check the H1, summary, and link structure parse correctly, or point a full squirrelscan audit at your site: the ax/llms-txt rule checks for the file's presence and format automatically as part of every crawl, alongside the rest of our agent experience rules. It's an info-level check, not a scoring one, which is honestly the right call given everything above.

If you're building out a broader AI-visibility strategy rather than just this one file, the generative engine optimization guide covers where llms.txt fits (a small piece) next to the things that matter more, like structuring content for AI crawlers and showing up in ChatGPT's answers.

Want a second opinion on your own llms.txt, or want to know whether AI crawlers can even reach your site in the first place? Run a free audit and squirrelscan will check both, along with 249 other things, in one pass. Or grab the CLI and run it locally:

curl -fsSL https://install.squirrelscan.com | sh
squirrel audit https://example.com