pagetrace 0.13.0 → 0.14.5

package/CHANGELOG.md

@@ -6,6 +6,100 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 While the version is below 1.0.0, breaking changes ship in a minor release.
 
+## [0.14.5] - 2026-09-08
+
+### Fixed
+
+- `extractLlmsTxt` used `Buffer.byteLength`, which does not exist in a browser,
+  so any page on a site that actually has an `/llms.txt` crashed the checker
+  with "Buffer is not defined". It counts UTF-8 bytes with `TextEncoder` now —
+  the same number, available everywhere. `extract.ts` is meant to be pure and
+  bundleable, and this was the one line that was not.
+- The scan log blamed the fetch for failures that happened after it, because
+  the error handler settled a fixed step rather than the one still running.
+
+## [0.14.4] - 2026-09-08
+
+### Added
+
+- Whole-site mode in the checker. It crawls from the sitemap the way
+  `pagetrace audit --url` does and runs every rule over the result, so the
+  cross-page findings a single page cannot produce — duplicate titles, hreflang
+  reciprocity, dead internal links, sitemap health — finally appear on the site.
+  Findings are grouped by code and message and list the routes they touch, which
+  is how the CLI's own report aggregates.
+
+### Fixed
+
+- Every request now has an eight-second deadline. A proxy that is down hangs
+  rather than refusing, and the chain sat waiting on it.
+- robots.txt and sitemap.xml were fetched through a reader that renders what it
+  fetches, so they came back as HTML documents and stopped looking like
+  themselves. Those requests now ask for text, and a sitemap whose XML did not
+  survive has its URLs recovered from the body rather than being given up on.
+- Switching scope left the other scope's score standing under a heading that no
+  longer described it.
+
+## [0.14.3] - 2026-09-08
+
+### Fixed
+
+- The checker could not fetch anything: it depended on one public CORS proxy,
+  and that proxy was returning Cloudflare 520s for every URL. It now tries
+  r.jina.ai, allorigins and codetabs in order and takes the first that answers
+  with HTML, so one service having a bad day no longer takes the feature down.
+
+### Added
+
+- A scan log. The check narrates itself as it runs — each request, its size and
+  which proxy served it, then the rule pass — instead of leaving a dead page
+  during the seconds it takes. A progress bar tracks the steps, the report dims
+  while it works, the score counts up to its value, and a category holding a
+  blocker gets a coloured edge.
+- Example URLs to try, so the page can be used without owning a site.
+
+## [0.14.2] - 2026-09-08
+
+### Added
+
+- An SEO checker page on the website: enter a URL and get all twenty-three
+  page-level checks scored and grouped, passes shown alongside failures. It
+  fetches robots.txt and llms.txt too, so the site-level checks answer from
+  evidence; anything not fetched reads "not checked" rather than passing or
+  failing.
+
+### Fixed
+
+- `npm run build:site` wrote into `docs/`, which tsup cleans, so building the
+  site deleted the site. Its output now has its own directory, `docs/assets/`,
+  which is the only thing the build can remove. The Pages deploy would have
+  published a lone JavaScript file.
+
+## [0.14.1] - 2026-09-08
+
+### Added
+
+- A website at [shyamexe.github.io/pagetrace](https://shyamexe.github.io/pagetrace/), built from
+  `docs/` and deployed by GitHub Pages. Its playground compiles `extract`, `audit` and `diff` into
+  the page via a new `src/browser.ts` entry, so it runs the same rules as the CLI rather than a
+  reimplementation of them. `npm run build:site` produces the bundle, and the Pages workflow rebuilds
+  it on every deploy so it cannot drift from the rules it claims to run.
+
+## [0.14.0] - 2026-09-08
+
+### Added
+
+- `--verify-all` also checks links to assets — PDFs, images, archives. They are
+  never crawled as pages, so they were skipped entirely: a link to a deleted
+  whitepaper reported nothing. A `--dir` run checks them against the filesystem;
+  a crawl spends a request each, which is why it is opt-in.
+
+### Changed
+
+- Link verification now runs in parallel at `--concurrency`, and the cap rose
+  from 100 to 1000 unique targets. Hitting the cap now says so on stderr rather
+  than silently under-reporting, which read as a clean site.
+
 ## [0.13.0] - 2026-09-08
 
 ### Added
@@ -370,6 +464,12 @@ Initial release. `snapshot`, `check` and `audit` commands; filesystem and HTTP
 crawling; diff classified by transition; absolute, cross-page and hreflang audit
 rules; pretty, JSON, markdown, GitHub and HTML reporters.
 
+[0.14.5]: https://github.com/shyamexe/pagetrace/compare/v0.14.0...v0.14.5
+[0.14.4]: https://github.com/shyamexe/pagetrace/commit/ec2a5e2cd590
+[0.14.3]: https://github.com/shyamexe/pagetrace/commit/c45cfb49e265
+[0.14.2]: https://github.com/shyamexe/pagetrace/commit/b3aa77388ee9
+[0.14.1]: https://github.com/shyamexe/pagetrace/commit/7cc91383ed7a
+[0.14.0]: https://github.com/shyamexe/pagetrace/compare/v0.13.0...v0.14.0
 [0.13.0]: https://github.com/shyamexe/pagetrace/compare/v0.12.0...v0.13.0
 [0.12.0]: https://github.com/shyamexe/pagetrace/compare/v0.11.0...v0.12.0
 [0.11.0]: https://github.com/shyamexe/pagetrace/compare/v0.10.0...v0.11.0

package/README.md

@@ -19,6 +19,37 @@ npm install -D pagetrace
 
 Requires Node 20.19 or newer. No native modules, three small dependencies.
 
+## Playground
+
+Try the rules without installing anything: **[shyamexe.github.io/pagetrace](https://shyamexe.github.io/pagetrace/)**. The site compiles `extract`, `audit` and `diff` — the pure half of this package — into the page, so it runs the same rules the CLI does, in your browser, with nothing uploaded.
+
+## Commands
+
+| Command | Answers | Crawls | Exit 1 when |
+| --- | --- | --- | --- |
+| `init` | "get me set up" | once, to write the first lockfile | never |
+| `snapshot` | "record what the site looks like now" | whole site | never |
+| `check` | "what did this deploy change?" | whole site | findings at or above `--fail-on` (default `error`) |
+| `audit` | "what is wrong with this site?" | whole site | `--fail-on` (default `never`) |
+| `links` | "are any links dead?" | whole site, links only in the report | any broken link |
+| `page <url>` | "is this one page sound?" | that URL alone | `--fail-on` (default `error`) |
+| `update` | "am I on the latest pagetrace?" | nothing | never |
+
+Every crawling command takes `--dir <build>` or `--url <origin>`, plus:
+
+| Flag | Default | Effect |
+| --- | --- | --- |
+| `--limit <n>` | 200 | Stop after this many pages |
+| `--concurrency <n>` | 5 | Parallel requests |
+| `--external` | off (on for `page`) | Also check links that leave the site |
+| `--verify-all` | off | Also check links to assets — PDFs, images, archives |
+| `--ignore-robots` | off | Crawl paths `robots.txt` disallows |
+| `--fail-on <severity>` | varies | `error`, `warn`, `info` or `never` |
+| `--format <format>` | `pretty` | `pretty`, `json`, `markdown`; `github` and `sarif` on `check` |
+| `--config <file>` | `pagetrace.config.json` | Config file |
+
+Exit codes are the same everywhere: `0` clean, `1` findings at or above `--fail-on`, `2` the run itself failed — bad flags, an unreadable build, an unreachable origin. CI can tell "the site regressed" from "the tool broke".
+
 ## Use
 
 Set up a config file and the first baseline in one step:
@@ -56,7 +87,7 @@ npx pagetrace check --dir ./out
 5 error, 9 warning, 2 info
 ```
 
-Exit codes are `0` for clean, `1` for findings at or above `--fail-on`, and `2` when the run itself failed — bad flags, an unreadable build directory, an unreachable origin. CI can tell "the site regressed" from "the tool broke". `--fail-on` takes `error` (the default), `warn` or `info`, and rejects anything else rather than quietly letting the build pass.
+`--fail-on` rejects an unrecognised value rather than quietly letting the build pass.
 
 Note that `check` runs the absolute rules as well as the diff, so it can fail on a problem your build did not introduce. Use `--no-audit` for a pure regression gate.
 
@@ -193,7 +224,7 @@ Only `404` and `410` count as dead. A `403` from a bot wall, a `429`, a timeout
 
 Think twice before putting `--external` in `check`. A third party's bad afternoon becomes a diff in your repository and a red build you cannot fix.
 
-Internal links are checked too. Only the broken ones are stored, so a site's navigation never lands in the lockfile: `link.broken` for a link that is already dead, `link.broken.added` for one this build broke. A `--dir` crawl is authoritative — the build directory is the whole site — while a crawl confirms each candidate with a real request first, because a sitemap routinely omits pages that are live. External links are checked only with `--external`, and only a 404 or 410 counts.
+Internal links are checked too. Only the broken ones are stored, so a site's navigation never lands in the lockfile: `link.broken` for a link that is already dead, `link.broken.added` for one this build broke. A `--dir` crawl is authoritative — the build directory is the whole site — while a crawl confirms each candidate with a real request first, because a sitemap routinely omits pages that are live. External links are checked only with `--external`, and only a 404 or 410 counts. Links to assets — PDFs, images, archives — are skipped unless `--verify-all`, since each one costs a request on a crawl (a `--dir` run checks them against the filesystem instead).
 
 Redirects are recorded from the response itself, so they cost no extra requests. A redirect that only adds or drops a trailing slash is server configuration rather than drift and is not reported. `canonical.redirects` is only raised when the canonical's target was actually crawled, so a `--limit` run cannot invent it.
 

package/dist/cli.cjs

@@ -1549,7 +1549,7 @@ function extractLinks(html) {
 }
 function extractLlmsTxt(body) {
   const sections = body.split(/\r?\n/).filter((line) => line.startsWith("## ")).map((line) => line.slice(3).trim());
-  return { present: true, sections, bytes: Buffer.byteLength(body, "utf8") };
+  return { present: true, sections, bytes: new TextEncoder().encode(body).length };
 }
 var XML_ENTITIES = {
   amp: "&",
@@ -1698,12 +1698,20 @@ async function snapshotFromDir(dir, config = {}) {
   const llms = await (0, import_promises.readFile)((0, import_node_path.join)(dir, "llms.txt"), "utf8").catch(() => null);
   if (llms !== null) site.llmsTxt = extractLlmsTxt(llms);
   const origin = site.origin ?? "https://pagetrace.invalid";
-  await resolveBrokenLinks(pages, links, origin);
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: config.verifyAll,
+    // The HTML route set is authoritative here, so a missing page is simply
+    // broken. An asset is a file this crawl never walked, so it gets a stat.
+    verify: config.verifyAll ? async (target) => ASSET_PATH.test(target) ? !await (0, import_promises.access)((0, import_node_path.join)(dir, target)).then(
+      () => true,
+      () => false
+    ) : true : void 0
+  });
   if (config.checkExternal) await resolveDeadExternal(pages, links, origin, 5);
   return { schemaVersion: 1, createdAt: (/* @__PURE__ */ new Date()).toISOString(), site, pages };
 }
 var ASSET_PATH = /\.(?!html?$)[a-z0-9]+$/i;
-function linkTarget(href, from, origin) {
+function linkTarget(href, from, origin, includeAssets = false) {
   let url;
   try {
     url = new URL(href, `${origin}${from === "/" ? "" : from}/`);
@@ -1711,27 +1719,39 @@ function linkTarget(href, from, origin) {
     return null;
   }
   if (url.origin !== origin) return null;
-  if (ASSET_PATH.test(url.pathname)) return null;
-  return routeFromUrl(url.href);
+  if (!includeAssets && ASSET_PATH.test(url.pathname)) return null;
+  return ASSET_PATH.test(url.pathname) ? url.pathname : routeFromUrl(url.href);
 }
-var MAX_LINK_CHECKS = 100;
-async function resolveBrokenLinks(pages, links, origin, verify) {
+var MAX_LINK_CHECKS = 1e3;
+async function resolveBrokenLinks(pages, links, origin, options = {}) {
+  const { includeAssets = false, verify, concurrency = 5 } = options;
   const known = new Set(Object.keys(pages));
   const candidates = /* @__PURE__ */ new Map();
   for (const [route, hrefs] of Object.entries(links)) {
     const missing = [
       ...new Set(
-        hrefs.map((href) => linkTarget(href, route, origin)).filter((target) => target !== null && !known.has(target))
+        hrefs.map((href) => linkTarget(href, route, origin, includeAssets)).filter((target) => target !== null && !known.has(target))
       )
     ];
     if (missing.length > 0) candidates.set(route, missing);
   }
   const broken = /* @__PURE__ */ new Set();
   if (verify) {
-    const targets = [...new Set([...candidates.values()].flat())].slice(0, MAX_LINK_CHECKS);
-    for (const target of targets) {
-      if (await verify(target)) broken.add(target);
+    const unique = [...new Set([...candidates.values()].flat())];
+    const queue = unique.slice(0, MAX_LINK_CHECKS);
+    if (unique.length > queue.length) {
+      console.error(
+        `pagetrace: ${unique.length} link targets to confirm, checking the first ${MAX_LINK_CHECKS}.`
+      );
     }
+    await Promise.all(
+      Array.from({ length: Math.min(Math.max(1, concurrency), queue.length) }, async () => {
+        while (queue.length > 0) {
+          const target = queue.shift();
+          if (await verify(target)) broken.add(target);
+        }
+      })
+    );
   }
   for (const [route, missing] of candidates) {
     const confirmed = verify ? missing.filter((target) => broken.has(target)) : missing;
@@ -1812,11 +1832,15 @@ async function snapshotFromPage(pageUrl, options = {}) {
   const pages = { [route]: page };
   const links = { [route]: extractLinks(doc.text) };
   const origin = site.origin ?? target.origin;
-  await resolveBrokenLinks(pages, links, origin, async (candidate) => {
-    try {
-      return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: options.verifyAll,
+    concurrency: options.concurrency,
+    verify: async (candidate) => {
+      try {
+        return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {
@@ -1905,11 +1929,15 @@ async function snapshotFromOrigin(origin, options = {}) {
       }
     })
   );
-  await resolveBrokenLinks(pages, links, base.origin, async (route) => {
-    try {
-      return await fetchDoc(new URL(route, base).href, timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, base.origin, {
+    includeAssets: options.verifyAll,
+    concurrency,
+    verify: async (route) => {
+      try {
+        return await fetchDoc(new URL(route, base).href, timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {
@@ -1968,14 +1996,16 @@ async function loadConfig(path = DEFAULT_CONFIG) {
 }
 async function build(flags, config) {
   const checkExternal = flags.external ?? config.checkExternal;
-  if (flags.dir) return snapshotFromDir(flags.dir, { ...config, checkExternal });
+  const verifyAll = flags.verifyAll ?? config.verifyAll;
+  if (flags.dir) return snapshotFromDir(flags.dir, { ...config, checkExternal, verifyAll });
   if (flags.url)
     return snapshotFromOrigin(flags.url, {
       ...config,
       limit: flags.limit,
       concurrency: flags.concurrency,
       ignoreRobots: flags.ignoreRobots ?? config.ignoreRobots,
-      checkExternal
+      checkExternal,
+      verifyAll
     });
   throw new Error("Provide a source: --dir <build directory> or --url <origin>.");
 }
@@ -1994,7 +2024,7 @@ function render(findings, format) {
   }
 }
 var cli = (0, import_cac.cac)("pagetrace");
-cli.command("snapshot", "Record the current SEO/AEO surface to a lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
+cli.command("snapshot", "Record the current SEO/AEO surface to a lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
   const config = await loadConfig(flags.config);
   const snapshot = await build(flags, config);
   const written = await writeLockfile(flags.out, snapshot);
@@ -2003,7 +2033,7 @@ cli.command("snapshot", "Record the current SEO/AEO surface to a lockfile").opti
     written ? import_picocolors2.default.green(`Wrote ${flags.out} \u2014 ${count} page${count === 1 ? "" : "s"}.`) : import_picocolors2.default.dim(`${flags.out} is already up to date \u2014 ${count} page${count === 1 ? "" : "s"}.`)
   );
 });
-cli.command("check", "Compare the current surface against the lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--lockfile <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | github | sarif", { default: "pretty" }).option("--fail-on <severity>", "error | warn | info", { default: "error" }).option("--audit", "Also run absolute rules, not just the diff", { default: true }).option("--update", "Write the new state to the lockfile after reporting").option("--baseline-branch <ref>", "Read the baseline lockfile from a git ref instead of disk").action(async (flags) => {
+cli.command("check", "Compare the current surface against the lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--lockfile <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | github | sarif", { default: "pretty" }).option("--fail-on <severity>", "error | warn | info", { default: "error" }).option("--audit", "Also run absolute rules, not just the diff", { default: true }).option("--update", "Write the new state to the lockfile after reporting").option("--baseline-branch <ref>", "Read the baseline lockfile from a git ref instead of disk").action(async (flags) => {
   const failOn = parseFailOn(flags.failOn, false);
   const config = await loadConfig(flags.config);
   const next = await build(flags, config);
@@ -2033,7 +2063,7 @@ Failing: ${summary.error} error, ${summary.warn} warning (--fail-on ${failOn}).`
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("audit", "Audit a site as it stands, with explanations and fixes").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | html", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "never" }).action(async (flags) => {
+cli.command("audit", "Audit a site as it stands, with explanations and fixes").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | html", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "never" }).action(async (flags) => {
   const failOn = parseFailOn(flags.failOn, true);
   const config = await loadConfig(flags.config);
   const snapshot = await build(flags, config);
@@ -2073,13 +2103,14 @@ cli.command("audit", "Audit a site as it stands, with explanations and fixes").o
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("page <url>", "Check one page: every link on it, and its own surface").option("--external", "Check links that leave the site", { default: true }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (url, flags) => {
+cli.command("page <url>", "Check one page: every link on it, and its own surface").option("--external", "Check links that leave the site", { default: true }).option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (url, flags) => {
   const failOn = parseFailOn(flags.failOn, true);
   const config = await loadConfig(flags.config);
   const snapshot = await snapshotFromPage(url, {
     ...config,
     concurrency: flags.concurrency,
-    checkExternal: flags.external
+    checkExternal: flags.external,
+    verifyAll: flags.verifyAll ?? config.verifyAll
   });
   const [page] = Object.values(snapshot.pages);
   const platform = detectPlatform([page.generator], Object.values(page.og));
@@ -2109,7 +2140,7 @@ cli.command("page <url>", "Check one page: every link on it, and its own surface
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("links", "Find internal links that point at no page").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (flags) => {
+cli.command("links", "Find internal links that point at no page").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (flags) => {
   const failOn = parseFailOn(flags.failOn, true);
   const config = await loadConfig(flags.config);
   const snapshot = await build(flags, config);
@@ -2156,7 +2187,7 @@ cli.command("links", "Find internal links that point at no page").option("--url
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("init", "Write a config file and take the first snapshot").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
+cli.command("init", "Write a config file and take the first snapshot").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
   const existing = await (0, import_promises2.readFile)(flags.config, "utf8").catch(() => null);
   if (existing === null) {
     const config = flags.url ? { siteUrl: new URL(flags.url).origin } : {};
@@ -2179,11 +2210,11 @@ Commit both files, then run \`pagetrace check ${flags.dir ? `--dir ${flags.dir}`
 });
 cli.command("update", "Check npm for a newer pagetrace and install it").option("--check", "Only report whether an update exists").action(async (flags) => {
   const latest = await latestVersion("pagetrace");
-  if (!isNewer(latest, "0.13.0")) {
-    console.log(import_picocolors2.default.green(`pagetrace ${"0.13.0"} is the latest version.`));
+  if (!isNewer(latest, "0.14.5")) {
+    console.log(import_picocolors2.default.green(`pagetrace ${"0.14.5"} is the latest version.`));
     return;
   }
-  console.log(import_picocolors2.default.yellow(`Update available: ${"0.13.0"} \u2192 ${latest}`));
+  console.log(import_picocolors2.default.yellow(`Update available: ${"0.14.5"} \u2192 ${latest}`));
   if (flags.check) return;
   if (process.argv[1]?.startsWith(process.cwd())) {
     console.log(
@@ -2201,7 +2232,7 @@ cli.command("update", "Check npm for a newer pagetrace and install it").option("
   console.log(import_picocolors2.default.green(`Updated to pagetrace ${latest}.`));
 });
 cli.help();
-cli.version("0.13.0");
+cli.version("0.14.5");
 async function main() {
   try {
     cli.parse(process.argv, { run: false });

package/dist/cli.js

@@ -1304,7 +1304,7 @@ ${cards || "<p>No issues found.</p>"}
 
 // src/snapshot.ts
 import { execFile } from "child_process";
-import { readdir, readFile } from "fs/promises";
+import { access, readdir, readFile } from "fs/promises";
 import { join, relative, sep } from "path";
 import { promisify } from "util";
 
@@ -1526,7 +1526,7 @@ function extractLinks(html) {
 }
 function extractLlmsTxt(body) {
   const sections = body.split(/\r?\n/).filter((line) => line.startsWith("## ")).map((line) => line.slice(3).trim());
-  return { present: true, sections, bytes: Buffer.byteLength(body, "utf8") };
+  return { present: true, sections, bytes: new TextEncoder().encode(body).length };
 }
 var XML_ENTITIES = {
   amp: "&",
@@ -1675,12 +1675,20 @@ async function snapshotFromDir(dir, config = {}) {
   const llms = await readFile(join(dir, "llms.txt"), "utf8").catch(() => null);
   if (llms !== null) site.llmsTxt = extractLlmsTxt(llms);
   const origin = site.origin ?? "https://pagetrace.invalid";
-  await resolveBrokenLinks(pages, links, origin);
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: config.verifyAll,
+    // The HTML route set is authoritative here, so a missing page is simply
+    // broken. An asset is a file this crawl never walked, so it gets a stat.
+    verify: config.verifyAll ? async (target) => ASSET_PATH.test(target) ? !await access(join(dir, target)).then(
+      () => true,
+      () => false
+    ) : true : void 0
+  });
   if (config.checkExternal) await resolveDeadExternal(pages, links, origin, 5);
   return { schemaVersion: 1, createdAt: (/* @__PURE__ */ new Date()).toISOString(), site, pages };
 }
 var ASSET_PATH = /\.(?!html?$)[a-z0-9]+$/i;
-function linkTarget(href, from, origin) {
+function linkTarget(href, from, origin, includeAssets = false) {
   let url;
   try {
     url = new URL(href, `${origin}${from === "/" ? "" : from}/`);
@@ -1688,27 +1696,39 @@ function linkTarget(href, from, origin) {
     return null;
   }
   if (url.origin !== origin) return null;
-  if (ASSET_PATH.test(url.pathname)) return null;
-  return routeFromUrl(url.href);
+  if (!includeAssets && ASSET_PATH.test(url.pathname)) return null;
+  return ASSET_PATH.test(url.pathname) ? url.pathname : routeFromUrl(url.href);
 }
-var MAX_LINK_CHECKS = 100;
-async function resolveBrokenLinks(pages, links, origin, verify) {
+var MAX_LINK_CHECKS = 1e3;
+async function resolveBrokenLinks(pages, links, origin, options = {}) {
+  const { includeAssets = false, verify, concurrency = 5 } = options;
   const known = new Set(Object.keys(pages));
   const candidates = /* @__PURE__ */ new Map();
   for (const [route, hrefs] of Object.entries(links)) {
     const missing = [
       ...new Set(
-        hrefs.map((href) => linkTarget(href, route, origin)).filter((target) => target !== null && !known.has(target))
+        hrefs.map((href) => linkTarget(href, route, origin, includeAssets)).filter((target) => target !== null && !known.has(target))
       )
     ];
     if (missing.length > 0) candidates.set(route, missing);
   }
   const broken = /* @__PURE__ */ new Set();
   if (verify) {
-    const targets = [...new Set([...candidates.values()].flat())].slice(0, MAX_LINK_CHECKS);
-    for (const target of targets) {
-      if (await verify(target)) broken.add(target);
+    const unique = [...new Set([...candidates.values()].flat())];
+    const queue = unique.slice(0, MAX_LINK_CHECKS);
+    if (unique.length > queue.length) {
+      console.error(
+        `pagetrace: ${unique.length} link targets to confirm, checking the first ${MAX_LINK_CHECKS}.`
+      );
     }
+    await Promise.all(
+      Array.from({ length: Math.min(Math.max(1, concurrency), queue.length) }, async () => {
+        while (queue.length > 0) {
+          const target = queue.shift();
+          if (await verify(target)) broken.add(target);
+        }
+      })
+    );
   }
   for (const [route, missing] of candidates) {
     const confirmed = verify ? missing.filter((target) => broken.has(target)) : missing;
@@ -1789,11 +1809,15 @@ async function snapshotFromPage(pageUrl, options = {}) {
   const pages = { [route]: page };
   const links = { [route]: extractLinks(doc.text) };
   const origin = site.origin ?? target.origin;
-  await resolveBrokenLinks(pages, links, origin, async (candidate) => {
-    try {
-      return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: options.verifyAll,
+    concurrency: options.concurrency,
+    verify: async (candidate) => {
+      try {
+        return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {
@@ -1882,11 +1906,15 @@ async function snapshotFromOrigin(origin, options = {}) {
       }
     })
   );
-  await resolveBrokenLinks(pages, links, base.origin, async (route) => {
-    try {
-      return await fetchDoc(new URL(route, base).href, timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, base.origin, {
+    includeAssets: options.verifyAll,
+    concurrency,
+    verify: async (route) => {
+      try {
+        return await fetchDoc(new URL(route, base).href, timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {
@@ -1945,14 +1973,16 @@ async function loadConfig(path = DEFAULT_CONFIG) {
 }
 async function build(flags, config) {
   const checkExternal = flags.external ?? config.checkExternal;
-  if (flags.dir) return snapshotFromDir(flags.dir, { ...config, checkExternal });
+  const verifyAll = flags.verifyAll ?? config.verifyAll;
+  if (flags.dir) return snapshotFromDir(flags.dir, { ...config, checkExternal, verifyAll });
   if (flags.url)
     return snapshotFromOrigin(flags.url, {
       ...config,
       limit: flags.limit,
       concurrency: flags.concurrency,
       ignoreRobots: flags.ignoreRobots ?? config.ignoreRobots,
-      checkExternal
+      checkExternal,
+      verifyAll
     });
   throw new Error("Provide a source: --dir <build directory> or --url <origin>.");
 }
@@ -1971,7 +2001,7 @@ function render(findings, format) {
   }
 }
 var cli = cac("pagetrace");
-cli.command("snapshot", "Record the current SEO/AEO surface to a lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
+cli.command("snapshot", "Record the current SEO/AEO surface to a lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
   const config = await loadConfig(flags.config);
   const snapshot = await build(flags, config);
   const written = await writeLockfile(flags.out, snapshot);
@@ -1980,7 +2010,7 @@ cli.command("snapshot", "Record the current SEO/AEO surface to a lockfile").opti
     written ? pc2.green(`Wrote ${flags.out} \u2014 ${count} page${count === 1 ? "" : "s"}.`) : pc2.dim(`${flags.out} is already up to date \u2014 ${count} page${count === 1 ? "" : "s"}.`)
   );
 });
-cli.command("check", "Compare the current surface against the lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--lockfile <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | github | sarif", { default: "pretty" }).option("--fail-on <severity>", "error | warn | info", { default: "error" }).option("--audit", "Also run absolute rules, not just the diff", { default: true }).option("--update", "Write the new state to the lockfile after reporting").option("--baseline-branch <ref>", "Read the baseline lockfile from a git ref instead of disk").action(async (flags) => {
+cli.command("check", "Compare the current surface against the lockfile").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--lockfile <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | github | sarif", { default: "pretty" }).option("--fail-on <severity>", "error | warn | info", { default: "error" }).option("--audit", "Also run absolute rules, not just the diff", { default: true }).option("--update", "Write the new state to the lockfile after reporting").option("--baseline-branch <ref>", "Read the baseline lockfile from a git ref instead of disk").action(async (flags) => {
   const failOn = parseFailOn(flags.failOn, false);
   const config = await loadConfig(flags.config);
   const next = await build(flags, config);
@@ -2010,7 +2040,7 @@ Failing: ${summary.error} error, ${summary.warn} warning (--fail-on ${failOn}).`
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("audit", "Audit a site as it stands, with explanations and fixes").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | html", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "never" }).action(async (flags) => {
+cli.command("audit", "Audit a site as it stands, with explanations and fixes").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown | html", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "never" }).action(async (flags) => {
   const failOn = parseFailOn(flags.failOn, true);
   const config = await loadConfig(flags.config);
   const snapshot = await build(flags, config);
@@ -2050,13 +2080,14 @@ cli.command("audit", "Audit a site as it stands, with explanations and fixes").o
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("page <url>", "Check one page: every link on it, and its own surface").option("--external", "Check links that leave the site", { default: true }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (url, flags) => {
+cli.command("page <url>", "Check one page: every link on it, and its own surface").option("--external", "Check links that leave the site", { default: true }).option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (url, flags) => {
   const failOn = parseFailOn(flags.failOn, true);
   const config = await loadConfig(flags.config);
   const snapshot = await snapshotFromPage(url, {
     ...config,
     concurrency: flags.concurrency,
-    checkExternal: flags.external
+    checkExternal: flags.external,
+    verifyAll: flags.verifyAll ?? config.verifyAll
   });
   const [page] = Object.values(snapshot.pages);
   const platform = detectPlatform([page.generator], Object.values(page.og));
@@ -2086,7 +2117,7 @@ cli.command("page <url>", "Check one page: every link on it, and its own surface
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("links", "Find internal links that point at no page").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (flags) => {
+cli.command("links", "Find internal links that point at no page").option("--url <origin>", "Live origin to crawl").option("--dir <dir>", "Directory of built HTML").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).option("--format <format>", "pretty | json | markdown", { default: "pretty" }).option("--out <file>", "Write the report to a file instead of stdout").option("--fail-on <severity>", "error | warn | info | never", { default: "error" }).action(async (flags) => {
   const failOn = parseFailOn(flags.failOn, true);
   const config = await loadConfig(flags.config);
   const snapshot = await build(flags, config);
@@ -2133,7 +2164,7 @@ cli.command("links", "Find internal links that point at no page").option("--url
     process.exitCode = EXIT_FINDINGS;
   }
 });
-cli.command("init", "Write a config file and take the first snapshot").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
+cli.command("init", "Write a config file and take the first snapshot").option("--dir <dir>", "Directory of built HTML").option("--url <origin>", "Live origin to crawl").option("--limit <n>", "Max pages to crawl", { default: 200 }).option("--concurrency <n>", "Parallel requests", { default: 5 }).option("--ignore-robots", "Crawl paths that robots.txt disallows").option("--external", "Also check links that leave the site").option("--verify-all", "Also check links to assets (PDFs, images, archives)").option("--out <file>", "Lockfile path", { default: DEFAULT_LOCKFILE }).option("--config <file>", "Config file", { default: DEFAULT_CONFIG }).action(async (flags) => {
   const existing = await readFile2(flags.config, "utf8").catch(() => null);
   if (existing === null) {
     const config = flags.url ? { siteUrl: new URL(flags.url).origin } : {};
@@ -2156,11 +2187,11 @@ Commit both files, then run \`pagetrace check ${flags.dir ? `--dir ${flags.dir}`
 });
 cli.command("update", "Check npm for a newer pagetrace and install it").option("--check", "Only report whether an update exists").action(async (flags) => {
   const latest = await latestVersion("pagetrace");
-  if (!isNewer(latest, "0.13.0")) {
-    console.log(pc2.green(`pagetrace ${"0.13.0"} is the latest version.`));
+  if (!isNewer(latest, "0.14.5")) {
+    console.log(pc2.green(`pagetrace ${"0.14.5"} is the latest version.`));
     return;
   }
-  console.log(pc2.yellow(`Update available: ${"0.13.0"} \u2192 ${latest}`));
+  console.log(pc2.yellow(`Update available: ${"0.14.5"} \u2192 ${latest}`));
   if (flags.check) return;
   if (process.argv[1]?.startsWith(process.cwd())) {
     console.log(
@@ -2178,7 +2209,7 @@ cli.command("update", "Check npm for a newer pagetrace and install it").option("
   console.log(pc2.green(`Updated to pagetrace ${latest}.`));
 });
 cli.help();
-cli.version("0.13.0");
+cli.version("0.14.5");
 async function main() {
   try {
     cli.parse(process.argv, { run: false });

package/dist/index.cjs

@@ -988,7 +988,7 @@ function extractLinks(html) {
 }
 function extractLlmsTxt(body) {
   const sections = body.split(/\r?\n/).filter((line) => line.startsWith("## ")).map((line) => line.slice(3).trim());
-  return { present: true, sections, bytes: Buffer.byteLength(body, "utf8") };
+  return { present: true, sections, bytes: new TextEncoder().encode(body).length };
 }
 var XML_ENTITIES = {
   amp: "&",
@@ -1740,12 +1740,20 @@ async function snapshotFromDir(dir, config = {}) {
   const llms = await (0, import_promises.readFile)((0, import_node_path.join)(dir, "llms.txt"), "utf8").catch(() => null);
   if (llms !== null) site.llmsTxt = extractLlmsTxt(llms);
   const origin = site.origin ?? "https://pagetrace.invalid";
-  await resolveBrokenLinks(pages, links, origin);
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: config.verifyAll,
+    // The HTML route set is authoritative here, so a missing page is simply
+    // broken. An asset is a file this crawl never walked, so it gets a stat.
+    verify: config.verifyAll ? async (target) => ASSET_PATH.test(target) ? !await (0, import_promises.access)((0, import_node_path.join)(dir, target)).then(
+      () => true,
+      () => false
+    ) : true : void 0
+  });
   if (config.checkExternal) await resolveDeadExternal(pages, links, origin, 5);
   return { schemaVersion: 1, createdAt: (/* @__PURE__ */ new Date()).toISOString(), site, pages };
 }
 var ASSET_PATH = /\.(?!html?$)[a-z0-9]+$/i;
-function linkTarget(href, from, origin) {
+function linkTarget(href, from, origin, includeAssets = false) {
   let url;
   try {
     url = new URL(href, `${origin}${from === "/" ? "" : from}/`);
@@ -1753,27 +1761,39 @@ function linkTarget(href, from, origin) {
     return null;
   }
   if (url.origin !== origin) return null;
-  if (ASSET_PATH.test(url.pathname)) return null;
-  return routeFromUrl(url.href);
+  if (!includeAssets && ASSET_PATH.test(url.pathname)) return null;
+  return ASSET_PATH.test(url.pathname) ? url.pathname : routeFromUrl(url.href);
 }
-var MAX_LINK_CHECKS = 100;
-async function resolveBrokenLinks(pages, links, origin, verify) {
+var MAX_LINK_CHECKS = 1e3;
+async function resolveBrokenLinks(pages, links, origin, options = {}) {
+  const { includeAssets = false, verify, concurrency = 5 } = options;
   const known = new Set(Object.keys(pages));
   const candidates = /* @__PURE__ */ new Map();
   for (const [route, hrefs] of Object.entries(links)) {
     const missing = [
       ...new Set(
-        hrefs.map((href) => linkTarget(href, route, origin)).filter((target) => target !== null && !known.has(target))
+        hrefs.map((href) => linkTarget(href, route, origin, includeAssets)).filter((target) => target !== null && !known.has(target))
       )
     ];
     if (missing.length > 0) candidates.set(route, missing);
   }
   const broken = /* @__PURE__ */ new Set();
   if (verify) {
-    const targets = [...new Set([...candidates.values()].flat())].slice(0, MAX_LINK_CHECKS);
-    for (const target of targets) {
-      if (await verify(target)) broken.add(target);
+    const unique = [...new Set([...candidates.values()].flat())];
+    const queue = unique.slice(0, MAX_LINK_CHECKS);
+    if (unique.length > queue.length) {
+      console.error(
+        `pagetrace: ${unique.length} link targets to confirm, checking the first ${MAX_LINK_CHECKS}.`
+      );
     }
+    await Promise.all(
+      Array.from({ length: Math.min(Math.max(1, concurrency), queue.length) }, async () => {
+        while (queue.length > 0) {
+          const target = queue.shift();
+          if (await verify(target)) broken.add(target);
+        }
+      })
+    );
   }
   for (const [route, missing] of candidates) {
     const confirmed = verify ? missing.filter((target) => broken.has(target)) : missing;
@@ -1854,11 +1874,15 @@ async function snapshotFromPage(pageUrl, options = {}) {
   const pages = { [route]: page };
   const links = { [route]: extractLinks(doc.text) };
   const origin = site.origin ?? target.origin;
-  await resolveBrokenLinks(pages, links, origin, async (candidate) => {
-    try {
-      return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: options.verifyAll,
+    concurrency: options.concurrency,
+    verify: async (candidate) => {
+      try {
+        return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {
@@ -1947,11 +1971,15 @@ async function snapshotFromOrigin(origin, options = {}) {
       }
     })
   );
-  await resolveBrokenLinks(pages, links, base.origin, async (route) => {
-    try {
-      return await fetchDoc(new URL(route, base).href, timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, base.origin, {
+    includeAssets: options.verifyAll,
+    concurrency,
+    verify: async (route) => {
+      try {
+        return await fetchDoc(new URL(route, base).href, timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {

package/dist/index.d.cts

@@ -151,6 +151,11 @@ interface Config {
      * third party's bad afternoon into a diff in your repository.
      */
     checkExternal?: boolean;
+    /**
+     * Also check links to assets — PDFs, images, archives. They are never crawled
+     * as pages, so each one costs a request (or a filesystem check for --dir).
+     */
+    verifyAll?: boolean;
 }
 
 /**

package/dist/index.d.ts

@@ -151,6 +151,11 @@ interface Config {
      * third party's bad afternoon into a diff in your repository.
      */
     checkExternal?: boolean;
+    /**
+     * Also check links to assets — PDFs, images, archives. They are never crawled
+     * as pages, so each one costs a request (or a filesystem check for --dir).
+     */
+    verifyAll?: boolean;
 }
 
 /**

package/dist/index.js

@@ -913,7 +913,7 @@ function extractLinks(html) {
 }
 function extractLlmsTxt(body) {
   const sections = body.split(/\r?\n/).filter((line) => line.startsWith("## ")).map((line) => line.slice(3).trim());
-  return { present: true, sections, bytes: Buffer.byteLength(body, "utf8") };
+  return { present: true, sections, bytes: new TextEncoder().encode(body).length };
 }
 var XML_ENTITIES = {
   amp: "&",
@@ -1535,7 +1535,7 @@ ${cards || "<p>No issues found.</p>"}
 
 // src/snapshot.ts
 import { execFile } from "child_process";
-import { readdir, readFile } from "fs/promises";
+import { access, readdir, readFile } from "fs/promises";
 import { join, relative, sep } from "path";
 import { promisify } from "util";
 function routeFromFilePath(root, filePath) {
@@ -1665,12 +1665,20 @@ async function snapshotFromDir(dir, config = {}) {
   const llms = await readFile(join(dir, "llms.txt"), "utf8").catch(() => null);
   if (llms !== null) site.llmsTxt = extractLlmsTxt(llms);
   const origin = site.origin ?? "https://pagetrace.invalid";
-  await resolveBrokenLinks(pages, links, origin);
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: config.verifyAll,
+    // The HTML route set is authoritative here, so a missing page is simply
+    // broken. An asset is a file this crawl never walked, so it gets a stat.
+    verify: config.verifyAll ? async (target) => ASSET_PATH.test(target) ? !await access(join(dir, target)).then(
+      () => true,
+      () => false
+    ) : true : void 0
+  });
   if (config.checkExternal) await resolveDeadExternal(pages, links, origin, 5);
   return { schemaVersion: 1, createdAt: (/* @__PURE__ */ new Date()).toISOString(), site, pages };
 }
 var ASSET_PATH = /\.(?!html?$)[a-z0-9]+$/i;
-function linkTarget(href, from, origin) {
+function linkTarget(href, from, origin, includeAssets = false) {
   let url;
   try {
     url = new URL(href, `${origin}${from === "/" ? "" : from}/`);
@@ -1678,27 +1686,39 @@ function linkTarget(href, from, origin) {
     return null;
   }
   if (url.origin !== origin) return null;
-  if (ASSET_PATH.test(url.pathname)) return null;
-  return routeFromUrl(url.href);
+  if (!includeAssets && ASSET_PATH.test(url.pathname)) return null;
+  return ASSET_PATH.test(url.pathname) ? url.pathname : routeFromUrl(url.href);
 }
-var MAX_LINK_CHECKS = 100;
-async function resolveBrokenLinks(pages, links, origin, verify) {
+var MAX_LINK_CHECKS = 1e3;
+async function resolveBrokenLinks(pages, links, origin, options = {}) {
+  const { includeAssets = false, verify, concurrency = 5 } = options;
   const known = new Set(Object.keys(pages));
   const candidates = /* @__PURE__ */ new Map();
   for (const [route, hrefs] of Object.entries(links)) {
     const missing = [
       ...new Set(
-        hrefs.map((href) => linkTarget(href, route, origin)).filter((target) => target !== null && !known.has(target))
+        hrefs.map((href) => linkTarget(href, route, origin, includeAssets)).filter((target) => target !== null && !known.has(target))
       )
     ];
     if (missing.length > 0) candidates.set(route, missing);
   }
   const broken = /* @__PURE__ */ new Set();
   if (verify) {
-    const targets = [...new Set([...candidates.values()].flat())].slice(0, MAX_LINK_CHECKS);
-    for (const target of targets) {
-      if (await verify(target)) broken.add(target);
+    const unique = [...new Set([...candidates.values()].flat())];
+    const queue = unique.slice(0, MAX_LINK_CHECKS);
+    if (unique.length > queue.length) {
+      console.error(
+        `pagetrace: ${unique.length} link targets to confirm, checking the first ${MAX_LINK_CHECKS}.`
+      );
     }
+    await Promise.all(
+      Array.from({ length: Math.min(Math.max(1, concurrency), queue.length) }, async () => {
+        while (queue.length > 0) {
+          const target = queue.shift();
+          if (await verify(target)) broken.add(target);
+        }
+      })
+    );
   }
   for (const [route, missing] of candidates) {
     const confirmed = verify ? missing.filter((target) => broken.has(target)) : missing;
@@ -1779,11 +1799,15 @@ async function snapshotFromPage(pageUrl, options = {}) {
   const pages = { [route]: page };
   const links = { [route]: extractLinks(doc.text) };
   const origin = site.origin ?? target.origin;
-  await resolveBrokenLinks(pages, links, origin, async (candidate) => {
-    try {
-      return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, origin, {
+    includeAssets: options.verifyAll,
+    concurrency: options.concurrency,
+    verify: async (candidate) => {
+      try {
+        return await fetchDoc(new URL(candidate, target.origin).href, options.timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {
@@ -1872,11 +1896,15 @@ async function snapshotFromOrigin(origin, options = {}) {
       }
     })
   );
-  await resolveBrokenLinks(pages, links, base.origin, async (route) => {
-    try {
-      return await fetchDoc(new URL(route, base).href, timeout) === null;
-    } catch {
-      return false;
+  await resolveBrokenLinks(pages, links, base.origin, {
+    includeAssets: options.verifyAll,
+    concurrency,
+    verify: async (route) => {
+      try {
+        return await fetchDoc(new URL(route, base).href, timeout) === null;
+      } catch {
+        return false;
+      }
     }
   });
   if (options.checkExternal) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pagetrace",
-  "version": "0.13.0",
+  "version": "0.14.5",
   "description": "Baseline your site's SEO and AEO surface, diff every build against it, and fail CI on regressions.",
   "main": "./dist/index.cjs",
   "scripts": {
@@ -10,7 +10,8 @@
     "test:watch": "vitest",
     "coverage": "vitest run --coverage",
     "typecheck": "tsc --noEmit",
-    "prepublishOnly": "npm run typecheck && npm run test && npm run build"
+    "prepublishOnly": "npm run typecheck && npm run test && npm run build",
+    "build:site": "tsup src/browser.ts --format iife --global-name pagetrace --platform browser --minify --no-dts --out-dir docs/assets"
   },
   "keywords": [
     "seo",