v0.0.134

Robots.txt is read the way crawlers read it, a redirect can no longer steer a crawl at a private address, and nested pages stop being slow.

Added

• [crawler] robots_user_agent sets the token robots.txt groups are matched against. It defaults to SquirrelScan. See robots_user_agent and the behavior change under Changed.
• social/og-image-size checks image formats. It reads the format of an Open Graph or Twitter card image from og:image:type or the file extension, without fetching it. SVG warns because preview clients may not render it, AVIF is recorded as an info check because support is inconsistent, and JPEG, PNG, GIF and WebP pass. twitter:image is recognized as name or property. See og:image size and format.
• crawl/robots-txt reports search crawler delays. A Crawl-delay of 10 seconds or more for * or Bingbot warns and estimates how many URLs a day Bing can then fetch, and says that Google ignores the directive while Bing and Yandex obey it. Every crawler named in a stacked User-agent group is checked. A shorter delay, or one for Googlebot, MSNBot or Yandex, is recorded as an info check. Zero delays and delays for other bots are ignored. See robots.txt.
• i18n/hreflang checks language and region codes. A value that is not x-default or a valid language and region code warns, such as en_US (use a hyphen) or en-uk (the region is GB). It checks the code format only, not return links.

Changed

• A crawl that respects robots.txt now follows a site's User-agent: SquirrelScan rules. The docs already said so. Before, the default browser user agent was matched by its first token, Mozilla, so in practice only the * rules applied. Requests still send user_agent. If you set user_agent to your own bot and respect robots.txt, the crawl is now matched as SquirrelScan too: set robots_user_agent to that bot's token, or to "" to match on user_agent as before.
• Config validation runs on zod 4. The same settings are accepted.
• install.sh and install.ps1 check release metadata first. The version, asset filename and SHA-256 are validated before any download URL is built, so malformed metadata cannot retarget the download. A checksum matches in either letter case. See Installation.

Fixed

• A viewer or billing member who runs a signed-in audit is told why it was not tracked. The organization's role refused it (forbidden_role). The run used to be missing from the dashboard with no word, and a publish failed with an error that had no message. The local audit and its results are unchanged. See Publishing to the dashboard.
• robots.txt groups are parsed as RFC 9309 describes. Consecutive User-agent lines share the rules that follow them, a named group with an empty Disallow: explicitly allows every path and no longer falls through to *, and inline comments are ignored. AI crawler and indexability checks now match how crawlers read those files.
• A Crawl-delay that is negative or too large to be a number is ignored, such as -1 or 1e999, like one that is not a number. A published report no longer carries it as a negative value or as null.
• A crawl that stops at its page cap no longer makes link rules look clean. links/orphan-pages, links/weak-internal-links and links/no-contextual-inbound skip, because an uncrawled page may hold the inbound links that would clear them. They show as Not checked with a hint to raise --max-pages, and a finding one of them reported on an earlier audit is not closed by a capped crawl. See output.
• A cloud audit uploaded in parts is scored like one uploaded whole. The server used to count every page a rule skipped, and every rule that checked no page, as a pass, so a large audit could score a few points above the same crawl run locally, with categories such as video or local at 100. It now scores from the per-check counts the audit publishes, as Rules that do not apply describes.
• A cloud smart audit no longer carries an older run's rule timeout or error onto a page it did not re-audit. A fresh error from the current run still reports.
• Deeply nested pages no longer slow an audit down. Each link's place in the page (header, navigation, footer, content) is worked out once per page, not by walking up the tree for every link: a page with 20,000 nested links went from 12.5 s to parse to 250 ms (benchmark).
• Nested tables are judged one table at a time, and fast. a11y/th-has-data-cells and a11y/td-headers-attr treat a table inside another table's cell as its own table, so a bad headers reference there is reported once, not once per enclosing table. These two and a11y/table-headers no longer slow down on deep nesting: 200 nested tables took th-has-data-cells 1.4 s and now take 1 ms. a11y/link-in-text-block counts a link in nested lists or quotes once, and 5,000 nested quotes went from 827 ms to 3 ms (benchmark).
• A colspan or rowspan of Infinity no longer hangs a11y/th-has-data-cells. It used to run until the audit ran out of memory. It now counts as 1, and a span past the HTML limits counts as the limit.
• a11y/aria-text and a11y/aria-hidden-focus are fast on pages that nest many role="text" or aria-hidden elements. Each rescanned its whole subtree for every nested element: 1,000 nested aria-hidden containers took 56 s and now take 22 ms. The findings are unchanged (benchmark).
• A custom header that differs only in case from one squirrel sets now replaces it. -H "user-agent: ..." or accept in [crawler] headers applies to page fetches, robots.txt, sitemaps, llms.txt, the well-known and RSL probes and asset checks. Before, both went out as one combined value such as squirrel/..., my-agent. A -H flag also wins over a [crawler] headers entry with the same name in another casing. The cloaking probe's Googlebot fetch keeps its Googlebot agent when a custom User-Agent is set, so agent-based cloaking can show again, and the markdown probe keeps its text/markdown Accept. See headers.
• A second audit --resume or crawl --resume no longer writes into a crawl a live process owns. It starts a new crawl and names the owner's PID and host. A stale lease, or an owner that exited on the same host, can still be resumed.
• squirrel report keeps the original audit's scan scope when it rebuilds from the project database: the page limit, a clamped requested limit, the runner, and the URLs discovered but not fetched. Audits stored before this render without an inferred scope.
• perf/lcp-hints shows its evidence. It records the affected page, the likely-LCP candidate URLs and the image preload URLs it compared. The JSON report's grouped check shows one affected page's evidence, and stored rule results keep every page's.
• The opt-in gaps/keywords and gaps/content rules run when you ask for them with [rules] enable, --rule-include gaps or [rule_options] enabled = true, and stay off under the default rule selection.
• security/x-frame-options flags an unrecognised value. X-Frame-Options: ALLOWALL or a typo warns, because browsers ignore it, unless CSP frame-ancestors covers the page. A repeat joined in one header such as DENY, DENY counts as one value, and frame-ancestors is matched as a directive name, not as text anywhere in the policy.
• content/keyword-stuffing skips function words in more languages. German, Spanish, French, Portuguese, Italian and Dutch pages are matched by <html lang>, so natural copy no longer reports those words as overused.
• The crawler names managed challenge pages served with HTTP 403. Cloudflare, Akamai and other detected providers report Blocked by <provider> challenge (403). A plain or stalled 403 stays Request blocked by server. Only the first 64 KB of the body is sampled.
• Glossary JSON-LD no longer leaks into llms-full.txt or per-page agent Markdown as raw script text, and it stays in rendered pages.
• HTML report issue cards show each detected technology once, in the order each first appears.

Security

• A redirect found while crawling the audited site can no longer send the CLI to a cloud metadata or link-local address. The page crawl, the root files (robots.txt, sitemaps, llms.txt, the well-known and markdown probes) and the internal link check refuse a redirect hop that leaves the site's hosts for an address such as 169.254.169.254, fe80::/10 or metadata.google.internal, before sending it, as the asset checks already did. A sitemap that robots.txt or a sitemap index names at such an address is not requested either, and security/http-to-https checks each redirect it follows the same way. Private networks stay allowed, so localhost, LAN and staging audits still crawl their own pages and root files. A redirect of the start URL itself, resolved before the crawl begins, is not covered yet. See follow_redirects.
• With [project] domains set, the crawl and the internal link check stay on the audited URL's port. A link to another port on an allowed host, such as http://example.com:6379/, is no longer crawled or checked as an internal link, where it could reach another service on the audited machine. When the audited URL uses the default port, 80 and 443 both still count as the site. See Project settings.

Docs

• A new FAQ answers the common first questions.
• Team roles are documented. The organization role matrix says what an owner, admin, editor, viewer and billing member can do, that the CLI and MCP sign-ins follow it, what the 403 with forbidden_role looks like, that API keys are the organization's and outlive a demoted creator, and how invite limits work.
• White-label reports, credits and reports pages are updated. White-label reports are in the Team docs, a second free organization from the same account starts with 0 credits, and published reports redact secrets in crawled URLs.
• AGENTS.md for contributors and coding agents explains how to work on the repo.
• Glossary pages carry structured data, and a few rule pages are corrected.