Turns out detecting broken links is haaaard. Meet squirrel perch.

Checking whether a link works sounds like the easiest job on the internet. Ask for the page. If it comes back, the link works. If it doesn't, it's broken. A shell one-liner does it.

Turns out detecting broken links is haaaard.

We found out the slow way, by shipping a link checker that was wrong a lot. This post is about why, and about Perch, the thing we built to fix it.

Links rot, a lot

First, why bother. Links die all the time, and nobody tells you.

Pew Research went looking in 2024 and found:

• 38% of the web pages that existed in 2013 were gone ten years later.
• 23% of news pages and 21% of government pages had at least one broken link.
• 54% of Wikipedia pages had at least one dead link in their references, and 11% of all Wikipedia reference links were dead.
• 18% of tweets were no longer visible just months after they were posted.

Some of that is recoverable. The Internet Archive checked Pew's dead 2013 pages and found the Wayback Machine had saved copies of about 38% of them (about 15% of all the 2013 pages). Hold that thought, we'll come back to it.

Every dead link on your site is a visitor who clicked and got nothing. So you want to find them, and fix them. That's the job.

Why it's hard now

The web used to answer robots honestly. It doesn't anymore. With AI crawlers and scrapers hammering everything, the web has become extremely adversarial to anything that isn't a person in a browser. A link checker is a robot. So it gets the treatment.

Here's what stands between a link checker and the truth:

• Bot walls. A firewall answers an automated request with an error, a challenge page, or some exotic status code of its own. None of those mean the page is gone. They mean "you look like a bot", which, fair.
• Walls lie in both directions. We tried the obvious shortcut: if a site walls a link, the page must be there. Nope. On site after site, a deleted page got the exact same wall as a live one. A wall tells you nothing, so we treat it as nothing.
• CAPTCHAs for every robot. Some sites send every automated visitor to a CAPTCHA, whether the link is real or made up. That says something about the visitor, nothing about your link.
• Rate limits. Check too many links on one site too fast and it starts refusing you. Refused isn't broken.
• Pages that say 200 to everything. A video page that loads for a deleted video. A map link that opens for any code you type. A page titled "Page not found" served with a cheerful 200 OK. To a plain status check, all fine. To your visitor, not so much.
• Dead domains that don't look dead. A domain that no longer exists can come back looking like a temporary server error, the kind a checker files under "try later". Dead domains turn out to be one of the most common kinds of dead link.

A link checker can fail two ways: cry wolf on links that work, or wave through links that are dead. Ours was very good at the first one.

Our own link checker was lying

We built a hard set of tricky links our old checker had called broken, and loaded each one in a real browser. Nearly half of them (about 45%) loaded fine. Across a wider replay of real audits, it was still more than a third.

Firewalls, bot walls and rate limits were answering our robot, and we were faithfully writing that down as your problem. Not great. We are a squirrel, not a parrot.

Meet squirrel perch

Perch is our new link checking engine. A squirrel on a perch sees the whole tree, and that's the idea: check every link from the right vantage point.

We tested it against real links from real audits, and built a ton of special cases into it for the sites that like to make this hard.

What Perch does differently

• It checks from our new and improved browser-grade checking cluster, not from a robot every firewall recognises.
• Broken means confirmed. A link is only called broken once the failure is confirmed. One bad answer on a bad day doesn't make a link broken.
• Dead domains are called properly. A domain that's gone is reported as dead, not "try again later".
• Pages that say 200 to everything are caught where it can tell, so a deleted video or a page titled "Page not found" shows up as the dead link it is.
• It never clicks the dangerous ones. A third-party unsubscribe link, a magic sign-in link, a signed download: Perch is built never to request them, and never to follow a redirect into one. Checking your links should never unsubscribe you from anything.
• Answers are shared. Lots of sites link to the same pages, so a link checked for one audit is often already answered for the next.

What "unverified" means

Some sites won't let anything that isn't a person through, ever. When Perch can't get a straight answer about a link, it says so: the link is listed as unverified (your report says "status unverifiable"), never as broken.

That's deliberate. We'd rather say "we couldn't check this one" than make something up. If you see an unverified link, click it yourself. If it loads, it's fine, and the site just doesn't like robots.

The first check of a brand-new link

When Perch has never seen a link before, its first answer can take a while. If an audit can't wait for it, that audit falls back to a simpler check for that link this time, which can still call a blocked link broken. The next audit gets Perch's answer. So if a link you know is fine shows up broken on a first audit, run it again later.

The numbers

Before switching Perch on, we replayed links our old checker had already answered, and checked every one again with Perch.

• The old checker had about 7% of them down as broken, and another 15% as refused (a 403, 429, 503 and the like). Perch calls about 4% broken.
• Of the links the old checker called broken, Perch found 37% alive and well: government sites that stall robots, social profiles, live sites behind a CDN error. Another 10% are ones Perch can't verify yet, so they're listed as unverified rather than guessed.
• The scary direction is links the old checker said were fine and Perch says are broken. That happened to 0.47% of them. We re-checked every one ourselves, from a home connection. Every one was really gone.

The old answers were up to a month older than Perch's, so a few links may have changed in between. That's the honest caveat.

We also ran a hard set: mostly links our old checker had reported broken, each one labelled by loading it in a real browser from a home connection. It's a stress test, not an error rate (server errors left out):

Rows are rounded, so one adds up to 101%. Perch isn't perfect: it called 3% of the working links broken, and 2% of the dead ones fine. When it can't tell, it says so instead of guessing.

Checking links from the CLI without signing in

Signed out, there's no Perch. When squirrel audit checks external links, every request goes straight from your machine: your laptop, a CI runner, a server. Plain command-line requests are exactly the kind of traffic a lot of sites block. So expect plenty of false positives: links reported broken that load fine in your browser.

Sign in and a Surface or Full audit (Surface is the signed-in default) checks external links through our checking cluster instead, with the same confirmed-or-unverified answers as a cloud audit. Any link Perch hasn't answered in time is checked from your machine, as above.

How to use it

Get the latest squirrel, sign in, and audit:

squirrel self update
squirrel auth login
squirrel audit https://your-site.com

Cloud audits at the Surface and Full levels check external links with Perch too. The link checking guide has all the details: what's checked, what "Not checked" means, and every switch.

What's new, and what's coming

• Always-on link monitoring for Pro. Pro sites in the cloud get recurring audits. When a scheduled surface or full audit finds a link that was fine last time and is now dead, you'll get an email.
• Alerts in your agent (planned). Email is the only channel for now. We're planning to push the same alerts as MCP messages, so your coding agent hears about a dead link and can go fix it.
• A fix, not just a finding (coming soon). Remember the Internet Archive's copies? When an external link is dead and the web archive has a copy of the page, we'll suggest the archived link so you can swap it in :)
• Getting past the last walls (coming soon). For the links that still come back unverified.

Also new: broken, but looks fine

While we were in there, we added a new reliability category, and a new links check, for things that are broken but look fine:

• Broken resources: a deleted script or stylesheet behind a page that still says 200.
• Broken forms: forms that lose submissions, from unnamed fields to dead actions.
• Broken anchors: a #fragment link whose target is gone.

Now go fix some links. Or let the squirrel find them first.