v0.0.135

Robots.txt groups are read the way crawlers read them when a Sitemap line sits between User-agent lines, a redirect with a Location that cannot be parsed no longer fails a crawl, the egress guard refuses two more cloud metadata addresses, and a custom user-agent header now reaches the link check and the soft-404 confirmation. Two new info rules: images/hotlinked and content/h1-title-alignment.

Added

• images: new rule images/hotlinked reports, as info, a content image (<img>, <picture> source, inline background) that loads from another site's server, grouped per host. First-party hosts, known image CDNs and site-builder storage, launch and badge directories, tracking pixels and avatar services are not reported, and allowed_hosts adds your own. An og:image or twitter:image on another site is reported too (#932).
• content: new rule content/h1-title-alignment reports, as info, a page whose H1 and <title> share no meaningful word (stopwords, generic words, stemming and the brand affix are read out; both strings are shown, and details.confident marks the stricter cases). It is info only because a minority of such pages (roughly 10 to 20 percent by hand judgement) are real mismatches; the rest are marketing headlines. Homepages and generic marketing pages, identical title and H1, and pages with no or several H1s, noindex pages, titles that are only the brand, glued or non-Latin text and languages without a stopword list are read as info or skipped (English, Spanish, French, German, Portuguese, Italian and Dutch are read) (#929).

Changed

• Cloud API: an API key missing a scope gets the typed error envelope. The 403 is now { "error": { "code": "missing_scope", "message": "API key missing required scope: audits:write", "missing_scope": "audits:write" } } instead of a bare string error, so a client can switch on the code. See Refusals for a role or a scope.

Fixed

• -H "user-agent: ..." (or a User-Agent in [crawler] headers) is now also the agent of the internal link check and the soft-404 confirmation, which send no other custom header. They used to send the crawl's random browser agent beside a crawl that sent yours. The reachability check, the cloaking probe and the AI agent access probe still use their own agents on purpose. See crawler headers (#1152).
• A role or scope refusal is explained in either shape the cloud sends. squirrel audit (the run is not tracked) and squirrel report --publish name a forbidden_role refusal in the typed error: { code, message } envelope as well as the bare error: "forbidden_role", and an API key missing a scope (missing_scope) says which scope a key needs. Before, only the bare shape was recognised.
• A redirect whose Location cannot be parsed no longer fails the whole crawl. A Location: http://[ threw inside the page fetch, the error escaped as a crash, and the audit stopped. That page now ends its redirect chain like a redirect with no Location, and the crawl goes on. The plain fetcher, which rejected the whole fetch with "Invalid URL", does the same.
• squirrel audit with -f json, xml, llm, markdown, text or html sends its progress and log lines ("New crawl: ...", "Found N URLs in sitemap") to stderr, so stdout starts with the report and squirrel audit <url> -f json | jq works. The console format is unchanged.
• robots.txt groups are read the way RFC 9309 and Google read them when a Sitemap, Crawl-delay or unknown line sits between User-agent lines: the group continues, so User-agent: GPTBot, Sitemap: ..., User-agent: CCBot, Disallow: / blocks both bots. Before, the second User-agent started a new group and the first bot was read as unlisted. A file with CR-only line endings is read too, where before it parsed as empty. AI crawler, indexability and crawl-delay checks read these files the way crawlers do, in the CLI and in cloud audits.
• squirrel audit -f <unknown> exits 1 with "Unknown format", like squirrel report -f, before crawling. Before, it ran the whole audit, printed no report and exited 0.

Security

• A robots.txt built to multiply no longer exhausts memory. Each group's rules are given to every user agent that heads it, so a file of a few thousand stacked User-agent lines over a few thousand rules, or over one very long rule, inside the 512 KiB fetch limit, grew to hundreds of millions of entries or gigabytes of rule text and could take down a CLI audit or a cloud audit container. The parser now keeps at most 100,000 directive entries and 2 million characters of rule paths, in file order, still collects the Sitemap lines after that, and records that the rest of the file was not read, which the crawl/robots-txt syntax check shows. Page checks against robots.txt also work out each rule's pattern once instead of once per page.

• The egress guard refuses GCP's IPv6 metadata server (fd20:ce::254) and the EKS Pod Identity agent (fd00:ec2::23), as literals and as DNS answers, beside AWS's fd00:ec2::254. They sit in the unique local range, so an audit on a private network used to let a redirect reach them. Other unique local addresses stay allowed.

Docs

• The crawler page states the trusted-hostname residual. The audited host, its www twin and every subdomain of [project] domains are requested by name without an address check, so a site that controls its own DNS can point them at a metadata address. The page says how to audit untrusted sites safely; cloud audits are not affected. See follow_redirects.