2.8x performance improvement and larger audits with squirrel

A 2,500-page audit used to take 502 seconds. It now takes 177. The same audit held 3.9 GB of memory and now holds 996 MB, which is the part that matters more, because holding 3.9 GB is how an audit fails rather than how it gets slow.

We are convinced this is now the fastest way to crawl and audit a website.

The numbers

The speedup depends on size: 2.8x at 2,500 pages, 1.5x at 1,000, and nothing at all at 400. The 400-page audit takes the same 47 seconds it always did, on 265 MB instead of 685. Below about a thousand pages this release is a memory change and not a speed one. The clock only moves once the old pipeline's heap was large enough to be fighting the garbage collector for the whole run, which is a nice way of saying the old version was mostly slow at being large.

Reports are byte-identical at all three sizes, with one exception at 2,500 pages: perf/ttfb grades the test server's own response time, and it tripped on 14 pages because the old run was starving that server.

What got faster

The rules pass on script-heavy pages

security/leaked-secrets runs 70 patterns over a page and perf/js-libraries about 130, and a pass over a megabyte costs the same whether it finds anything or not. So the cost is the number of passes.

One pass now records which 4-grams the page contains, which lets a pattern be skipped when a literal that every match must contain is provably absent. That drops 48% of secret-pattern invocations and 99% of the context-keyword scans. The rest is mostly irreducible: 18 of the 70 secret patterns have no literal to prove, and they are the top of what remains.

Two script rules also stopped counting matches by building an array just to read its length, and the skip-link rule stopped serializing the whole document once per heading.

Rules that only read your chrome, run once per template

Pages of one site share their header, navigation, footer and script stack far more than their content. A real storefront's 247 pages fall into 13 chrome clusters.

Every page now carries a template cluster key, and 25 of the 198 page rules are declared safe to run once per cluster and fan their verdict to the rest. That is worth 12.9% of the streamed page-rule pass on that storefront taking the minimum of the repeats, or 10.8% by the median. Findings are identical to running every rule on every page, and SQUIRREL_TEMPLATE_FANOUT=0 turns it off.

Scope matters here. It is the page-rule pass, not total rule CPU. The theoretical ceiling if every rule could fan out is 96.5%, and quoting that would misrepresent what 25 declared rules can buy.

A warm re-crawl compiles 48 statements, not 1,482

bun:sqlite has two ways to get a statement and they are not interchangeable. db.query() caches the compiled statement by SQL text; db.prepare() compiles a new one every call. The storage layer used prepare in all 116 places.

Seven of those run per page on a warm crawl, so a re-audit was paying 12.3 compilations per page for nothing. This is a count and not a timing: the wall-clock pair behind it is inside the noise of a loaded machine, which is why the regression test asserts on compilations rather than on a clock.

The report reads each check once

Assembling a report ran two queries that differed only in their ORDER BY, and each built its own object per row. At 1,000 pages that is 203,687 rows read and materialized twice.

The rows are read once now and the ordering asked for twice. Measured on that call in isolation it is about 150 MB less. End to end it is below the noise floor of an exit measurement, with wall time and peak RSS unchanged within run-to-run spread, so it is 150 MB off that call and not off your audit's peak.

Larger audits

--max-pages was capped at 5,000 and no configuration could raise it. The cap is 10,000 now, on every plan.

On our benchmark estate, which averages 128 KB a page, a 10,000-page audit took 746 seconds and peaked at 5,447 MiB. Heavier pages cost more, so treat the cap as a real ceiling rather than a formality: past it, split the audit by section with include patterns.

Cloud audits follow the plan instead. Free stays at 500 pages and Pro at 2,000; Team goes to 10,000. Before this, no plan could exceed 2,000 no matter what it asked for.

Raising a page ceiling means raising the clock that goes with it. The run budget used to scale with page count and then clamp at one hour, and that clamp was reached at exactly 500 pages, so every audit from 500 pages upward got the same hour. A run on the paid container class now gets up to four hours. Without that, the higher ceiling would have been a trap: a run you were allowed to start and guaranteed to have killed.

The change underneath

All of that sits on one structural fix. Every phase after the crawl used to run over a single array holding every parsed page and its DOM, kept from the end of the crawl until the report was written, plus a second full read of every page to assemble the report. Retained heap grew about 1.5 MB per crawled page.

That is where 3.9 GB at 2,500 pages came from, while the operating system's resident figure read a reassuring 1.4 GB, which is the kind of number you trust right up until the process dies.

Link checks, asset fetches, the rules pass and the report are now built from batches of parsed pages. Batch size comes from a byte budget divided by the site's own average page, so a docs site with 20 KB pages and a store with 1 MB pages both fit. Peak memory tracks the batch, not the page count, which is what makes a 10,000-page audit possible at all.

Conditions

Every laptop row above is a cold squirrel audit --coverage full on a 16 GB Apple Silicon machine, heap sampled after a forced collection at exit, with a fresh content store per stage and old and new runs interleaved.

Wall time on a shared machine is the weakest column here. The same 5,000-page workload measured 359, 448 and 575 seconds across three runs in one day, so treat any wall-clock difference under roughly 60% on a loaded box as unresolved. The heap figures and the statement counts are the trustworthy part.

Try it

squirrel self update
squirrel audit https://example.com --max-pages 10000

The audit prints the requested and the effective page limit, so a run clamped by your plan is distinguishable from one that finished.

Where these come from

Every figure above is in the checked-in record at benchmarks/2026-09-perf-program.md, with the conditions, the controls, and three rows we retracted when the harness turned out to be giving each stage a fresh origin port, so the runs we had labelled warm had reused nothing at all.