v0.0.113

A safety and accuracy release. The external link check no longer requests links that are actions or carry a credential, such as one-click unsubscribe links, magic sign-in links and signed downloads, and every request it sends, redirect hops included, goes only to a public address (or a private one when the audited site is private), never to a link-local or cloud metadata address. squirrel audit's asset checks can no longer be pointed at a cloud metadata service, the GraphQL probes no longer let a compressed answer inflate past their read limit, and an audit now closes each connection once it has read what it needs, so an endless body no longer keeps downloading. Sitemap dates, bot-protection names and a dozen rules report more accurately too.

Security

• The external link check no longer requests third-party links that look like actions or carry a token. Examples are a newsletter's one-click unsubscribe link, a magic sign-in link or a signed download URL, where the URL itself is the credential and a request can use it up.
  • It uses the internal link check's rules (action words, token-shaped values, keys such as token or sig) and skips links whose text is plainly an action ("Log out", "Unsubscribe").
  • A signed-in audit does not send these links to the cloud's dead-link check either.
  • links/broken-external-links lists them in a new info check, not-requested-external-links, with tokens masked. They are never reported as working or broken, and they do not count against max_links.
  • When squirrel checks a link itself it now follows redirects by hand, so a short link that redirects into one stops at the redirect.
  • Every request and redirect hop goes only to a public address (private ones too for an audit of a private site), never to a link-local or cloud metadata address, and connects to the address it checked.
  • See What the external link check never requests.
• The internal and external link checks also skip three kinds of link they used to request.
  • A server-script extension (such as .aspx, .php, .jsp or .do) no longer counts as a word, so /MyAccountLogOut.aspx reads as the four-word sign-out it is. Any other action word is read the same way (/delete-your-account.php, like /delete-your-account).
  • Log out, log off and sign out now count in a path segment of five words (/my-account-log-out-now). A longer slug that only mentions signing out (/blog/how-to-log-out-of-every-device) is still checked.
  • A link with a credential in its query string is never requested: a key such as api_key, x-api-key, pwd, session_id, accesstoken, authorization, jwt, x-amz-credential or x-goog-credential, in any case, with -, _ and . ignored, or in brackets (config[api_key], pwd[0]).
  • A not-requested link listed in links/broken-external-links now has a path segment holding an email address masked, where /unsubscribe/user@example.com/… used to show the address. An asset name such as logo@2x.png is kept.
• External links refused for their address are now listed in links/broken-external-links as refused-external-links instead of silently left out, and never counted as working or broken. That covers a link-local or cloud metadata address on any audit, and a private or loopback one on a public site's audit.
  • A link refused on its URL alone is not counted as checked: it takes no slot under max_links, is not answered from the link cache (where an audit of a private site may have stored a verdict for the same URL), and is not sent to the cloud's dead-link check.
  • A re-audit no longer reports an old verdict for an external link this audit did not check.
• squirrel audit's asset checks can no longer be pointed at a cloud metadata service.
  • A stylesheet, script, image, font, PDF or sitemap URL on the audited site is refused at the hop, before anything is sent, when it redirects to a link-local address (169.254.0.0/16, fe80::/10) or a metadata hostname, or sits on a subdomain outside the hosts the crawl itself requests whose name points at one.
  • The hosts the crawl itself requests are requested as before, so an audit of localhost, a LAN address or a .local name still loads its own assets.
  • A sitemap URL on another host is checked without your custom request headers. They used to go to whatever host a sitemap listed, Web Bot Auth signatures and Authorization included.
• The GraphQL probes behind security/graphql-introspection and security/graphql-get-mutations no longer let a site answer with a compression bomb.
  • Their requests went out with the runtime's own decompression on, and Bun inflates a gzip or brotli body in full as it arrives, before the probe's read limit applied. A probed endpoint could cost an audit gigabytes of memory.
  • They now fetch with that decompression off, like every other request an audit sends, and decompress the answer no further than the probe's limit (1 MB for introspection, 64 KB for the mutation probe).
  • Each probe also closes its connection once its answer is read, where an endless answer used to keep downloading until the 10-second timeout.
  • Findings are unchanged. See security/graphql-introspection.
• integrity/brand-impersonation and integrity/fake-auth-overlay no longer lose a signal to how a page's markup is spaced.
  • An off-brand "Sign in with Google" link right after <h1>Welcome</h1> used to read as welcomesign in with google, so the sign-in cue was missed and a correlated overlay dropped from fail to info.
  • The cues are now also read from element-separated text, with invisible characters removed and whitespace collapsed, so a newline,  , a <br> or a zero-width space inside a link's label of up to 300 characters no longer hides it either. A longer label is read as before, lowercased only.
  • For integrity/brand-impersonation, a link whose own label says sign in or log in now counts as a credential affordance, like a password field.
  • See integrity/brand-impersonation.

Changed

• A signed-in audit's link checks are free, a firewall-blocked link included: Credits and pricing no longer lists a 2-credit charge for one. At the surface and full levels, a signed-in audit's external links go to the cloud's dead-link check, and any link it leaves unanswered is checked on your machine.
• A signed-in audit no longer waits out a busy dead-link service batch by batch. squirrel sends the service up to 200 external links at a time, and any link left out of a reply is checked locally. When a reply answers none of its links, squirrel now stops asking and checks the rest locally straight away. Before, it sent every remaining batch and waited on each, about 100 seconds on a site with 1,000 external links. See links/dead-links.

Fixed

• An audit no longer goes on downloading a response it is done with.
  • Bun keeps receiving the rest of a body it has stopped reading until the request is aborted, and most of squirrel's requests only cleared their timeout. A site answering with an endless body kept the connection open, and kept sending, after squirrel had moved on, in most cases until the audit ended.
  • That covered the reads with a size limit (robots.txt, sitemaps, llms.txt and the other root files, scripts, the external link check's look at a 403 page, the soft-404 and cloaking re-fetches, plain-HTTP page fetches in an audit that uses cloud rendering) and the status checks that drop a body unread (external link checks, asset checks, link target checks, the reliability/broken-forms probes, the reachability check of a localhost site).
  • squirrel now closes each connection as soon as it is done with it. What an audit reads is unchanged.
• A sitemap with a bare-year <lastmod>2024</lastmod>, which the W3C Datetime format allows, is now read. crawl/sitemap-valid used to report the whole file as an error starting Parse error: entry.lastmod?.trim is not a function, and none of its URLs were read.
  • Sitemap values are now kept as the text the file wrote, and a bad entry costs only itself, except that a raw < reads as an unclosed tag and can hide the entries after it.
  • An entry whose <loc> holds markup, a raw <, or appears twice is skipped and counted (N <url> entries skipped: <loc> is not a single text value), and the text around markup is never read as the URL.
  • A <lastmod>, <changefreq> or <priority> like that is dropped and its URL kept (N <lastmod> values ignored: not a single text value).
  • See crawl/sitemap-valid.
• crawl/sitemap-lastmod-drift and crawl/sitemap-lastmod-churn read a lastmod of just a year or a month (2024, 2024-09, both valid W3C Datetime values) as that whole year or month, not its first day.
  • A year-stamped sitemap used to look months behind every page dated later that year (lastmod 2024 vs dateModified 2024-09-10 (253 day(s))), and churn reported a day the sitemap never wrote.
  • Drift is now measured from the nearer end of the period, and churn leaves such values out of its sample.
  • Churn's "too small to judge" message counts them: a sitemap with only a few lastmods, all years or months, used to read Only 0 sitemap URL(s) carry a lastmod.
  • See crawl/sitemap-lastmod-drift.
• links/broken-links names the real reason in its Not checked line.
  • When the internal link check answered some targets and skipped others as actions, token links, out of the crawl scope or disallowed by robots.txt, the line said it status-checked only the N most-linked targets and told you to raise internal_links.max_targets, even with nothing over the limit.
  • The line now names the cause that applies, gives a count for each when there are several (past the limit, skipped, not trusted, no answer), and suggests raising internal_links.max_targets only when the limit left targets out. The check's details.notFetched carries the counts.
  • See links/broken-links.
• With cloud link checking (signed in), a link the checker could not verify is listed with the firewall-blocked links as status unverifiable, never as broken. That covers a timeout or failed connection from the checker's network, a firewall refusal, and a 404 waiting for its confirming second check. Before, a timeout was counted as a broken link and a refused link was left out of the report. See links/broken-external-links.
• Bot protection is named more accurately.
  • The vendor's own cookie names now count (Cloudflare's __cf_bm, DataDome's datadome, HUMAN's _px3, Kasada's KP_UIDz, Akamai Bot Manager's _abck, Imperva's incap_ses_*, AWS WAF's aws-waf-token), names only, never values.
  • A vendor behind a CDN is named over the CDN, so a DataDome block served through Cloudflare reads as DataDome.
  • A site on CloudFront, API Gateway or Lambda is no longer reported as behind AWS WAF from those services' own headers: only AWS WAF's action header, cookie or challenge script, or CloudFront's "Request blocked" error page, count.
• eeat/privacy-policy, eeat/about-page and eeat/contact-page no longer tell you to create a page your pages already link to.
  • When the crawl did not fetch the policy, because its path is excluded (/policies/** on a Shopify storefront) or the page limit came first, eeat/privacy-policy now reports Privacy Policy linked but not crawled as info. It warns Privacy Policy link returns N when the internal link probe saw that link answer a 4xx or 5xx other than 401, 403 or a rate limit, and it still warns No Privacy Policy page found when nothing links to a policy.
  • The About and Contact rules report About page linked but not crawled and Contact page linked but not crawled (or About page link returns N, Contact page link returns N), recognising the link by its path only.
  • All three count only links to the same host, so a footer link to another site's /about, or to reCAPTCHA's policies.google.com/privacy, never stands in for a missing page.
  • No extra requests are made. See eeat/privacy-policy.
• perf/critical-request-chains and perf/font-delivery count only the CSS @imports a browser fetches. An @import inside a comment (/* @import url(a.css); */), a string or a url() no longer counts as a network request, and neither does one after another rule or inside @media, which a browser ignores.
• content/placeholder-media no longer reports a 1x1 inline src as a blank pixel when the image sits in a <picture> whose <source srcset> names a real image: the browser shows the source, so the src is a stub.
• local/nap-consistency no longer reads a button label as part of a phone number's format. A tel: link showing +49 (0)6061 630 980 0 with Jetzt anrufen on German pages and Call now on English ones was reported as one number written two ways. Only the number's own characters (digits, +, parentheses, ., /, - and spaces) are compared now, and a genuinely different rendering such as 060616309800 still warns. See local/nap-consistency.
• schema/rating-scope no longer warns that a rating is not visible when the page prints it in elements with no whitespace between them. <h3>Customer rating</h3><p>5/5</p> used to read as Customer rating5/5, where no rating phrase matched. The rule now also searches the page text with a space at every element boundary, so glued and spaced markup get the same verdict. See schema/rating-scope.