v0.0.145

Rules now read the page type, so contact and legal pages are no longer reported as thin, and a site's /index.html is read as its home page. squirrel crawl --disable-discovery-probes sends no discovery probe, an audit asks for robots.txt once, and sitemap requests obey the per-host limits. --max-duration holds on very large pages and on requests that hang before the page checks, smart audits resolve old findings when a re-crawled page's rule now answers info, and query-string variants are counted with their canonical page in the entity map and the hosted report. The squirrel binary no longer autoloads a bunfig.toml or .env from the directory it runs in, a crawler proxy that owns DNS can resolve the TLS check's hosts with tls_proxy_resolves_names, and space held by audits killed while building their report comes back.

Added

• squirrel crawl --disable-discovery-probes sends no discovery probe and overrides [crawler] disable_discovery_probes, like squirrel audit. (#422)
• [crawler] tls_proxy_resolves_names lets a proxy that owns DNS resolve the TLS check's hosts. Behind HTTPS_PROXY, security/tls-certificate resolves the name itself and asks the proxy for a tunnel to the vetted address, which a proxy that only allows connections by name refuses. With tls_proxy_resolves_names = true the audited host and its www or apex counterpart are requested by name with no local lookup, the hostname stays the SNI and verification name, and a refusing, hanging or garbled proxy still gives not checked. Off by default, because where those two names resolve becomes the proxy's decision; every other host is still resolved, vetted and pinned by squirrel, and an IP literal, a metadata name or a name the guard refuses on sight never take this path. See Behind an HTTP proxy. (#1317)
• SQUIRREL_SKILLS_TEST_SOURCE, a loopback-only test hook that lets squirrel skills install read from a local fixture server. Any non-loopback value is ignored with a warning (#590)

Changed

• The squirrel binary no longer autoloads a bunfig.toml or .env from the directory it runs in. Set SQUIRRELSCAN_API_KEY and any other variable in your shell or your CI secrets, as the docs describe; a .env file in the project directory is not read. (Settings and squirrel.toml are unaffected.)
• Rules read the page type: contact and legal pages are no longer reported as thin or too short. A contact page or a privacy policy is short on purpose, and nobody searches for it by its title. When the page classifier's model labels a page contact or legal with a confidence of at least 0.9 (the bar schema/expected-type already uses), content/word-count skips it, core/meta-title and core/meta-description skip their "too short" finding, and content/keyword-stuffing skips a legal page, whose text repeats its own terms by nature. Each skipped page gets one skipped check with details.skippedByType and details.pageType, which does not appear as an issue, does not count toward the score and resolves an earlier finding on the page. "Too long", a missing title or description, and every other rule are unchanged, and a rules-fallback label, a model label below 0.9 or a page the classifier did not run on is judged as before. The page's main content (not its header, navigation, footer or aside) has to bear the label out (policy vocabulary for legal; a tel: or mailto: link, an email address, a phone number or a message box for contact), and a page with no text is never skipped. A page reused from the crawl cache is fetched again when a different classifier parsed it, so a new model or --no-classifier takes effect on unchanged pages; the first crawl after this upgrade fetches every cached page once, and each future bump of the classifier's rules version does the same. On four small sites audited before and after, findings on contact and legal pages from these four rules dropped from 15 to 1 (a "too long" description) and the 133 findings on every other page did not change. core/h1, content/reading-level and the E-E-A-T rules are unchanged, with the reasons in the rules docs. See pages short by nature. (#841)
• The crawler no longer sends a default Accept-Language. It sent en-US,en;q=0.9 on every request, so a site that redirects by language (Shopify Markets, for one) sent /de-ch and /fr-ch to /en, and the duplicate-description rule reported all three as one page. It now sends none, and [crawler] headers = { "Accept-Language" = "..." } still sets one. Pages cached by an earlier run are refetched only when the site varied on Accept-Language; use --refresh to refetch a cached language redirect. The crawl's browser User-Agent now arrives with no language, which a strict bot filter may notice. (#1355)
• perf/bad-caching names Cloudflare JavaScript Detections when it finds it. When at least half of the pages missing an ETag or Last-Modified validator are behind Cloudflare and carry the JavaScript Detections script (inline or from /cdn-cgi/challenge-platform/), the note says it was found and that Cloudflare removes the ETag where it injects it, instead of listing every feature that might. Pages behind Cloudflare without the marker keep the headers-only note. The marker is a new per-page signal, so the rule-result cache format changes: the first audit of each site after the upgrade runs every rule fresh, once. See Weak caching. (#605)

Fixed

• The entity map and the hosted report builder count query-string variants with their canonical page. entities.summary.pagesTotal could exceed meta.totalPages by the number of folded variants; the entity map in the report now lists a variant's entities under its canonical page. The report builder the hosted audits use can fold variants the same way (foldCanonicalVariants), without moving the score. A smart audit's carried pages are not folded yet. (#1350)
• --max-duration: one very large page no longer blocks the run past its budget. Parsing a multi-megabyte HTML file cannot be interrupted, so a 30 MB page held the process for 109 s against a 15 s budget. A budgeted run now stores a page over 5 MiB without parsing it, so it adds no links and its rules do not run, and counts it in maxDuration.pagesTooLarge and lists it in tooLargePages, with its own reason in the report. A run without the flag is unchanged. (#1332)
• --max-duration: a request hanging before the page checks no longer costs the page checks. The soft-404 re-fetch of suspect pages and the cloaking probe run before the page rules; one that never answered held the rules phase until the hard stop, which cut the whole phase, so the report had no page findings even though every page was fetched. These probes now stop at their own deadline inside the margin and are skipped, maxDuration.probesSkipped names them, the page checks run on the fetched pages, and siteChecksCut is true only when the site-wide checks really did not finish. See Time limit. (#1327)
• Space held by audits killed while building their report comes back. A crawl left marked building after Ctrl-C or a crash counted toward no keep_audits window and was never retired, so its rows stayed until a manual squirrel self disk --prune. A later audit of the project now retires it once its report has been building for more than six hours, no live process owns it, and at least keep_audits newer audits exist (when in doubt it is left alone). A crawl a later squirrel analyze rebuilt inside the window is never retired, and self disk --prune plans around a crawl that is still building. (#288)
• Page rows superseded before automatic retention are collected once. self disk --prune could keep a page row that a later audit then replaced, and nothing revisited it. The next audit or prune of a project removes those rows 500 at a time, a couple of seconds per run, and records that it is done; a url whose only copy is in a retired audit keeps it, so re-audits stay incremental. A removed row was the project database's only pointer to that page's stored HTML, so research corpora that keep retired audits should copy what they need first. (#289)
• A site's /index.html is read as its home page. The page classifier gave homepage only to the path /, so a site that serves its home page at /index.html had it labelled as the model's best other class when one fired, else unknown. On 628 live sites that serve an /index.html in place, the page there was the same page as / on 558 of them, 0.889 (lower 95% bound 0.862, under the 0.90 bar the issue asked for). Setting aside 38 answers that are blank or error pages rather than documents (5 error titles, 33 with fewer than 15 distinct words), it is 558 of 590, 0.946 (lower bound 0.924). This is a judgement call, recorded on #902. The set-aside is asymmetric: 73 near-empty home pages stay in the 558; treating both sides the same gives 485 of 517, 0.938 (lower bound 0.914). The page at /index.html now gets homepage from the URL rule at confidence 1, query string and fragment ignored. Nested documents (/blog/index.html) and every other document name stay as they were: index.php, index.htm, default.* and home.* did not clear the bar either (index.php 0.79, index.htm 0.52, the rest 0.0 to 0.5), and a bare locale root such as /en/ was not measured. The model still never labels a non-root page homepage (#900, already in the shipped classifier). Pages cached before this release are fetched again once so their label is refreshed, so the first crawl after upgrading re-reads every cached page. (#902)
• An audit asks for robots.txt once. The RSL licensing probe fetched it a second time; it now reuses the crawl's own read, including its Link: rel=license header. One visible difference: when that read gets a 403 or a server error, ax/rsl-license now reports "not checked" (as it does for a refused request) where its own second read used to leave it at "none declared". (#422)
• Sitemap requests obey per_host_concurrency and per_host_delay_ms. The guessed sitemap locations used to go out five at a time whatever the setting said; they now queue behind the same per-host limits as the pages and the discovery probes. (#422)
• --disable-discovery-probes false is refused. With a space, false was read as a stray argument and the probes were turned off, the opposite of what was typed. It now exits with an error that names the =false form. (#422)
• A page that redirected to another crawled page is no longer counted as a second page with that page's title and description. content/duplicate-title, content/duplicate-description and core/title-unique counted the requested URL beside the URL it ended up on. They now count the content once, at the target; when the target was never crawled, the requested URL is still evaluated. (#1355)
• perf/content-in-js-bundle no longer passes an audit where every page was browser-rendered. A rendered page's copy is in the script that rendered it, so the rule had nothing to search for and reported "No page content found", which read as clean although nothing was judged. It is now skipped with a message saying every page was rendered and that --http checks it. Audits with some rendered and some plain pages still search the plain ones. See Page content in a JavaScript bundle. (#1329)
• Smart audits resolve old findings when a re-crawled page's rule now answers info. A rule that used to fail a page (50 unlabeled form fields) and now reports No form inputs found, or reports only a different check of the same rule, left the old rows open as carried findings run after run, with the coverage line showing nothing carried. A plain info result now counts as the rule's verdict for the page (an info that says the rule could not judge, such as content too short or an unreadable page, is marked inconclusive and still carries), and a re-crawled page the rule evaluated is clean for the rule's other checks too, so their earlier findings resolve. A check explicitly skipped on the page still carries. Coverage gains pagesWithStaleCarry (re-crawled pages that still carry earlier findings), shown on the coverage line when above zero. See Output. (#1031)
• content/keyword-stuffing stops flagging German function words, menu chrome on short pages and product-card microcopy. The German list now has the formal and informal address and common prepositions (sie, für, dich, dir and more), out is an English function word, blocks of mostly link text with several different labels and a real menu signal (role="menu", menubar or navigation, or the aria-haspopup/aria-expanded of a dropdown; class names and plain lists do not count, and a block where one word is in 30% or more of the labels is a keyword cloud and still counts) are left out when a page has no main copy of its own, and rating and price text repeated once per card in a grid (an amount with a currency, a percentage, 4 of 5 or a star count that differs from card to card; bounded to 20% of the words, a 1.5% floor per word and never a word that is 10% of the page) no longer counts toward density. One anchor repeated many times and the same sentence repeated in one container still flag. See Keyword stuffing. (#1315)