AX: Defining and Auditing Agent Experience

Until recently websites were primarily built for humans - the user experience. They are optimized for search discovery with SEO. Today AI agents make up an ever-increasing proportion of web traffic - which requires a new optimization paradigm - the agent experience, or AX.

Why it's important to build for the agent experience:

1. Bots and agents are now a huge share of web traffic: Bot traffic has already overtaken human traffic on the web. In 2025 AI action bot traffic was already equal to Google bot traffic. AI coding agents make up 45% of traffic to tech documentation. All of these numbers are increasing.
2. AI makes up a tiny part of search, but is rapidly growing: Google still makes up 87% of website referrals, but AI traffic grew 16x in two years.
3. Google traffic is now AI traffic: Google AI Overviews now have 2 billion monthly active users. AI Overviews are halving organic traffic as fewer search users click through organic results.
4. Developers are increasingly interfacing with agents, not dashboards: Coding agents now have millions of users, and users are increasingly choosing to interact with their tools via their agents. The best tools don't need a dashboard, they need a good agent experience. Developer tools like Sentry, PostHog and Grafana are leaning heavily into providing agent-first experiences.

So what makes a good agent experience?

There have been attempts to define this. From Cloudflare, Vercel and others. The latest version of Google Lighthouse incorporates an agentic browsing score.

There are many emerging standards - llms.txt, WebMCP, AGENTS.md, Web Bot Auth, CIMD, MCP Server Cards, Link Headers, auth.md, and more. We attempt to define what is currently worth implementing and auditing.

squirrelscan defines a category of audit rules for websites as Agent Experience / AX - and in developing and defining these audit rules we have attempted to narrow down what works for website owners and developers to enhance agent compatibility.

So we are naming the category and, more usefully, shipping the audit for it. squirrelscan now has an ax category that scores how ready a site is for agents to read it, reach it, and act on it.

In summary they can be categorized as:

1. Reachability: Allowing search and agent crawlers to access your site
2. Readability: Server side rendering and content negotiation with markdown to suit agents
3. Discoverable: Structured data and formats - OpenAPI, sitemaps, schemas, .well-known
4. Actionable: Allow the agent to easily take actions - auth negotiation, MCP, APIs, payments and more

Here is what we consider important for the agent experience of a website.

1. Reachability

Bot blocking

Most AI crawlers - especially those that make up a large majority of traffic - respect robots.txt. In our audits we have found that many websites have inadvertently blocked user action crawls from their websites with the intent of blocking only training or search, not knowing that their websites now no longer appear in AI search results.

There are three types of AI bot, based on use case:

• Search - Finds and indexes your content for later retrieval. Equivalent to GoogleBot
• Agent - Search and retrieval acting on behalf of a user
• Training - Crawls your website in order to permanently train a model

Content signals are a new Cloudflare standard where robots.txt is extended to signal to all AI bots what type of AI use case is allowed.

The big providers distinguish their bots and as a site owner you can allow or block based on use case. It is recommended to allow for ai-input and search purposes, both via explicit crawler allows and via content signals.

An example robots.txt that allows user intent and search from the major AI providers, but blocks training:

# Allow search and user-triggered AI fetches. Block AI training.
 
Content-Signal: search=yes, ai-input=yes, ai-train=no
 
# Allowed: Google Search, OpenAI Search, OpenAI user fetches,
# Anthropic Search, Anthropic user fetches.
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
 
# Block Google AI-use token, OpenAI training, Anthropic training.
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
 
# Everyone else: default allow. Flip this to Disallow for an
# allowlist-only policy, knowing it blocks every unlisted crawler
# (Bing, DuckDuckGo, archive.org, ...).
User-agent: *
Allow: /

This allows (good) bots to index your site and act on behalf of users - most importantly, keeping you discoverable in ChatGPT, Claude and other AI apps. You can build a policy like this for your own site with our robots.txt generator.

2. Readable to agents

Most agent fetchers are not full browsers. ChatGPT, Claude, et al. make requests with plain HTTP clients that can't render JavaScript. It's essential that important information and pages for agents are server-side rendered.

When content is returned, the weight of the content is important as agents have limited context windows and large responses will harm agent retrieval.

HTML is not well suited for agents for this reason - and we recommend returning content to agents in markdown via content negotiation.

An agent that prefers markdown sends an Accept header. A site with content negotiation answers in kind:

GET /blog/defining-ax HTTP/1.1
Host: squirrelscan.com
Accept: text/markdown, text/html;q=0.9
User-Agent: Claude-User/1.0

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Cache-Control: public, max-age=3600
Content-Length: 1782
 
# AX: Defining Agent Experience
 
UX is how people experience your site. AX is how AI agents experience it.
 
## Why it matters
 
Bot traffic has already overtaken human traffic on the web...
[..]

The same URL, requested by a browser, still gets the full HTML page:

GET /blog/defining-ax HTTP/1.1
Host: squirrelscan.com
Accept: text/html
User-Agent: Mozilla/5.0

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Vary: Accept
Content-Length: 48213
 
<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>AX: Defining Agent Experience</title>
...
[..]

Same content, two representations. The markdown response is 1,782 bytes; the HTML is 48,213. The Vary: Accept header tells caches the two are distinct, so a CDN won't hand a browser the markdown or vice versa.

In addition you can direct agents to a markdown alternative served at a .md URL and discoverable via a Link header. Content negotiation on the Accept header is the cleanest version of this, but not every client sends Accept: text/markdown, and an agent that doesn't ask can't be answered. The fallback is to publish the markdown at a stable, guessable address, the page URL with a .md suffix, and point at it from the HTML response so any agent can find it without negotiating:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Link: </blog/defining-ax.md>; rel="alternate"; type="text/markdown"

The same signal works in the document head for agents that parse HTML before deciding what to fetch:

<link rel="alternate" type="text/markdown" href="/blog/defining-ax.md">

Agents now have two options: they can request markdown up front and get it, or take the HTML, read the Link header, and follow the pointer to a clean copy. The .md file is worth serving on its own terms too: it is the URL a developer will paste into an agent by hand, so it should be the real content, not a redirect back to the app shell.

3. Discoverable content

llms.txt

There is a growing family of standards around assisting agents with discovering content on websites and webapps.

The first and most common is the llms.txt standard - a plaintext file in markdown to guide agents. Google said llms.txt has no bearing on search ranking. Naive implementations that just copy the entire website content (most often seen in llms-full.txt style files) or just duplicate content are more likely to harm agents.

llms.txt is best used as an index for agent resources where you introduce the website or projects and point to the most frequently accessed resources.

See the llms.txt for this website as an example. There is enough information there to guide the agent to next steps without blowing up context.

well-known resource paths

The .well-known/ directory (RFC 8615) is the web's convention for machine-discoverable metadata: a predictable path prefix where a client can look for a file without being told it exists. For agents that matters because guessable beats crawlable.

A handful of these files make up the actionable and discoverable end of AX.

The pattern to notice is that the mature entries are the ones that predate the agent web: OAuth and OIDC discovery were built for machine clients years ago, and agents inherit them for free. For websites that have an API or authentication methods these standards are important to agent discovery.

4. Actionable

This is where the bleeding edge is today with agent experience: guiding agents on how they can take action with APIs, payments, authentication, skills integrations and more. The field is early and our audit coverage here is still light - a few rules check API discovery and payment signals - but it's important to know what these are as the standards and implementations are rapidly evolving.

OpenAPI

OpenAPI describes an API in a machine-readable format, and generating it is built into most API frameworks. Specs are often at /openapi.json (example) or at the API host, but you can point to them with a well-known entry. For example:

https://squirrelscan.com/.well-known/api-catalog

contains the pointer to the squirrelscan API:

{
  "linkset": [
    {
      "anchor": "https://squirrelscan.com/",
      "service-desc": [
        {
          "href": "https://squirrelscan.com/openapi.json",
          "type": "application/vnd.oai.openapi+json;version=3.1",
          "title": "squirrelscan Public API"
        }
      ]
    }
  ]
}

This can also be served with a Link header in responses:

Link: <https://squirrelscan.com/openapi.json>; rel="service-desc"; type="application/vnd.oai.openapi+json;version=3.1"; title="squirrelscan Public API"

MCP discovery

The Model Context Protocol has become the default way agents call tools, and a growing number of developer tools now ship an MCP server alongside their API. Today most of those servers are discovered through configuration, a user pastes a URL into their client, but the web-native story is arriving. Auth is the part that is already standardized: an MCP server points at its authorization server through /.well-known/oauth-protected-resource (RFC 9728), so an agent can negotiate access without a human wiring up tokens.

Server cards (manifests describing what a server exposes) and public registries are the next piece. If you run a developer tool, an MCP endpoint is fast becoming an essential offering as more developers migrate to utilizing agents.

What we shipped

All of that is theory until something checks it against your actual site, so that is what we built. The ax category now runs 14 rules across all four pillars. You can audit with squirrelscan by:

squirrel audit --rule-include ax squirrelscan.com

This will run an audit against our agent experience rules.

What we're deliberately not checking yet

Part of defining a category is saying what doesn't belong in it yet, so here is what we looked at and left out.

• DNS for AI Discovery (DNS-AID), which publishes agent entrypoints as SVCB records under _agents.<domain>, is an IETF draft with adoption close to zero
• Web Bot Auth - we support it in audits but we're not checking for it yet.
• The agentic commerce stack (x402, MPP, UCP, ACP) is four competing protocols with no clear winner and almost no live deployments; until real agents are paying real money through one of them, telling a site to pick a horse is not an audit finding, it's a bet.
• We do already detect a 402 Payment Required response and pay-per-crawl wiring where it exists, which covers the part of that story that is actually observable today.

The bar for adding a rule is that failing it costs a real site something now, or demonstrably will on a known date. All of these are worth watching, and none of them clear that bar yet. When one does, it goes in.

The goal

This is how I measure the goal of agent experience - am I able to point a fresh agent (Codex, Cursor, Claude et al) to a website and have it understand the service and be able to interact with it efficiently? i.e. this prompt:

>>> go to squirrelscan.com and setup squirrelscan here and run an audit against my website 

With our AX score improved to ~100 I tested this across OpenCode, Codex CLI and Claude Code and it was able to successfully install the CLI, set up the agent skills, set up the MCP and prompt for auth and then run an audit. That's the agent experience.