v0.0.160

The integrity and secrets rules are harder to talk around and no longer break on hostile pages: security/leaked-secrets keeps a password in sample data or a value that ends in an ellipsis instead of dropping it, and its records no longer hold the page they came from; integrity/brand-impersonation and integrity/fake-auth-overlay read more look-alike spellings and more of the CSS that styles an overlay, stop sparing sign-in links that go to user content on a brand's domain, and are no longer abandoned on pages of tens of thousands of controls. Technology detection reads import maps, and its bundle samples no longer repeat text and fill their allowance (William's #1536, with our allowance fix on top). The --max-duration parse cap holds for cached pages, and the docs say that only rendering runs in the cloud.

Fixed

• The --max-duration parse cap now also holds when a cached page is reused. A page over the cap is stored but never parsed when it is fetched; reusing it from cache used to parse its HTML for links, which brought back the parse cost the cap bounds (25 stored pages of about 6 MB took a reuse pass 192 s and 2.6 GB more memory before, 1.2 s and 0.2 GB after, measured on a synthetic store of 25 pages). A cached page over the cap is now reused without discovering its links. The size is the decoded body's, as at the fetch gate. The soft-404 confirmation re-fetch gets the same guard as defense in depth: a re-fetched body over the cap leaves the candidate unconfirmed (still reported, never dropped), though that re-fetch is already cut at 1 MB, under the 5 MiB production cap, which is why the guard is bounded there. (#1502, refs #1332)
• Technology-detection bundle samples no longer repeat text, and still fill the bundle's allowance. On a page with roughly 7 to 10 scripts near 40 KB plus a late bundle, the sampler could send each small script's whole head and then send part of that same text again as its tail, spending about a third of the 512 KB page budget on duplicates. A script's source text is now sent once: its head, sampled windows and tail are separate stretches, the rest of the bundle's allowance goes on the text between them (from the front, counted in UTF-8 bytes), and stretches that touch are joined with no separator. A bundle over its cap is read up to the cap, a vendor call at the end of the first 16 KB is kept whole, and a loader in a late bundle remains detectable. The page, audit and bundle caps still hold. (#1534)
• Technology detection reads the URLs an import map maps to. A library loaded as an ES module through <script type="importmap"> (three.js through jsdelivr, unpkg, cdnjs or a self-hosted path) was not detected, because an import map is a data block and its URLs were never script URLs. The imports and scopes values of an import map (a map of up to 65,536 characters, 256 URLs a map, 512 a page; http, https and /, ./ or ../ paths without quotes, brackets, backticks, backslashes, whitespace or control characters) now count as script URLs (a map inside a <template> does not); a URL in a CSP <meta>, an anchor, JSON-LD, a JSON data script or prose still does not. (#1539)
• security/leaked-secrets no longer drops a password in sample data or a value that ends in an ellipsis. A generic secret in a seed table with reserved-domain emails, or beside puzzle or generator markers, was dropped on the structure alone; it is now kept, labelled Possible Demo Login (seed data table) at medium with a hint that says it looks like sample data. A short value that ends in ... or … (the visible start of a real secret, or a placeholder) was dropped; it is now reported at the informational tier as Partial Secret Value Visible (elided) in a new leaked-secrets-partial check, masked like the other values. A bare mask (••••, ...) is still dropped. (refs #1292, #1308)
• integrity/brand-impersonation no longer spares a sign-in control that goes to user content under google.com or microsoft.com. The brand's own hosts are now an explicit list: Google's accounts.google.com, accounts.youtube.com (its single sign-on host, reported before), myaccount.google.com and the /accounts path of google.com; Microsoft's login.microsoftonline.com, .us, .de, login.partner.microsoftonline.cn, login.chinacloudapi.cn, login.windows.net, login.live.com, login.microsoft.com, account.live.com and account.microsoft.com, and <tenant>.b2clogin.com and <tenant>.ciamlogin.com only for the audited site's own tenant, on their authorize paths. sites.google.com, docs.google.com, script.google.com, forms.google.com, drive.google.com, the google.com/url and /amp/s/ redirectors, translate.google.com, forms.microsoft.com, answers.microsoft.com and the like are reported (a lone signal is an info check). (#1418, refs #784)
• integrity/brand-impersonation is no longer abandoned on a page of tens of thousands of multi-attribute controls. A page of 5 to 10 MB of tiny links or buttons with several attributes, or with aria-labelledby ids, or thousands of distinct aria-labelledby targets, could run the rule past its 2 s budget ("Not checked"). The attributes a control is read for are now read in one pass (a control with more than 4,096 is counted in controlsUnread), the page has a work budget for the attributes it looks at when listing controls (2,000,000 units), and the text of aria-labelledby targets is read for at most 300,000 visits in all plus at most 255 (the limit is checked before each read, and a read stops at 256 visits). A page past a budget reports the controls it did not read in controlsUnread and scanTruncated instead of timing out; the budgets bound the work, not the wall-clock time. (#1462)
• integrity/brand-impersonation reads more look-alike and swapped-digit spellings of a sign-in cue. The Greek lunate sigma (U+03F2), the Latin iota (U+0269) and Cyrillic п and г fold to the Latin letter, the braille blank (U+2800) is removed like the other invisible characters, and a 0, 3 or 5 inside a word that mixes letters and digits reads as o, e or s and a 1 as i or l (S1gn 1n w1th G00g1e), in labels and in attributes, as well as as written (Office365 login still matches). The fold is bounded (16,000 characters a label; the first 300 characters of a control's attributes and of its text always fold, the rest come from a 2,000,000-character page pool) and a control not folded in full is counted in controlsUnread; the label as written is always read, and a cue past the first 300 characters can be hidden once the pool is spent. Measured: about 3.5 s for a 39 MB page of 60,000 controls. The capital lunate sigma (U+03F9) folds too. A number on its own (1995, 10) is not a cue, and Russian, Spanish and German sign-in labels do not fire. The table is a hand list, not the full UTS #39 skeleton: see the rule page. (#1410, refs #784)
• integrity/fake-auth-overlay reads more of the CSS that styles an overlay iframe. Rules inside @media, @supports, @layer, @container and @scope blocks that can apply to a screen (a print-only or speech-only @media is skipped), rules nested in a style rule, descendant, child and sibling selectors (read for their last compound) and attribute selectors now count, and the cascade follows !important, then specificity, then source order: an id rule beats a class rule written later. Each of these let a fixed full-viewport cover through before. The hidden-text scan is unchanged. The cascade is worked out twice, with and without the rules whose applicability cannot be decided (an @media that is not all or screen, @supports, @container, @scope, @layer, an ancestor part or a pseudo-class), and an iframe is a cover when either says so, so a decoy rule a browser would never apply cannot hide a cover whose own rules are decided (the protection covers decided covers only; a cover that is itself written in an undecided rule can still be hidden by a later decoy). There a class, an id and an attribute value are compared case-sensitively, as a browser does (attributes HTML reads without case, such as type and lang, and i selectors are compared without it), in standards mode (quirks mode, where class and id compare without case, is not modelled); the s flag is handled for a lone rule only (a rule that uses it is left out), and a selector list with an invalid part is not modelled (the valid parts are read, where a browser drops the whole list). A <style> with a media or type that keeps it from applying is not modelled. A rule or at-rule left open at the end of the CSS ends there. The scan stays linear: 5 MB and 10 MiB of hostile CSS were measured at about 2 s at most, and nesting is bounded (depth 8, 64 parts per list, 2,048 characters per produced selector, 50,000 nested rules and 20,000 produced selectors per sheet, and nesting stops reading once that budget is spent); anything past a bound makes the CSS cssTruncated. (#1409, refs #784)
• security/leaked-secrets records no longer hold the page they were found in. A finding's value was a slice of the page's serialized HTML or of an inline script, and a slice holds its whole parent, so the page-time collector kept about the size of every page (13 MB per 4.4 MB page in the report that found it) although its findings were a few kilobytes. A finding now holds copies of its own strings; the findings and the report are unchanged, the scan time of the typical fixture is within 5%, and a page that holds one huge value can add up to about 27%. This covers the leaked-secrets records only: other fields of the page's collected signal (scriptSrcs, fingerprint, the anchors) can still hold a page on real pages. (#1552)

Changed

• The docs and the help of --render and --render-mode say that only rendering runs in the cloud. Your machine still fetches the site: the pages (a plain request beside each render, and a plain fallback request for a page the cloud could not render or the site blocked), the start URL and TLS checks, robots.txt, sitemaps and the other discovery files, scripts and assets, internal and external link checks, and the soft-404 and cloaking probes. The new section what still comes from your machine lists them. No behaviour changes. (#1285)