readiness-audit 2.6.0 → 2.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,14 @@
1
1
  # Changelog
2
2
 
3
+ ## 2.6.1 (2026-11-12)
4
+
5
+ Four bugs from an outside review of 2.6.0, each reproduced first and now covered by a test. No new checks and no new warnings. D1 passes robots.txt files it wrongly failed, and the audit finishes where it used to crash.
6
+
7
+ - **The address guard reads IPv6 as bytes, and checks at connect time (security).** The URL parser writes `[::ffff:127.0.0.1]` as `[::ffff:7f00:1]`, and the guard knew only the dotted form, so a site could redirect the audit to loopback or a cloud metadata address. It now classifies IPv4-mapped, IPv4-compatible, NAT64 and 6to4 addresses by the IPv4 inside them, and refuses 198.18.0.0/15, multicast and reserved addresses. A name that doesn't resolve is no longer counted as safe. Every request outside the audited site and `--base` checks the addresses it connects to, so a name that resolves differently the second time (DNS rebinding) is still refused.
8
+ - **A URL in site content that isn't one is a finding, not a crash.** A canonical like `href="http://["` stopped the whole audit with exit code 1, the code for a failing site. W2 reports a canonical that isn't a URL, W5 a share image, W3 a redirect `Location`, S1 a skills index entry. An error in the audit itself now exits 2, as a bad option already did, and the README says so.
9
+ - **D1 lets a named robots.txt group be stricter than `*`.** `User-agent: BadBot` with `Disallow: /` failed as "badbot skips Disallow /admin". D1 now matches rules as RFC 9309 does (the longest match wins, Allow on a tie, `*` and `$`), reads Allow lines, and combines groups for the same agent, `*` included. It still fails a named group that misses a `*` Disallow, or reopens one with Allow.
10
+ - **A domain in capitals or with a trailing dot is the same site.** `MusicVenue.space` found no sitemap URLs on its own site and failed D2 and W1 to W6. The CLI and `audit()` lowercase the domain and drop a trailing dot.
11
+
3
12
  ## 2.6.0 (2026-11-04)
4
13
 
5
14
  From two projects fixing their APIs: one has no `/v1` prefix. No scoring changes, and no new warnings or failures.
package/README.md CHANGED
@@ -15,20 +15,20 @@ npx readiness-audit example.com
15
15
  It needs Node 22.23.4, 24.22.1 or 26.12.1, or later on the same line (each supported line's current security release), and has no dependencies.
16
16
 
17
17
  ```
18
- Agent and search readiness: animalhouse.ai (readiness-audit v2.6.0, 2026-11-04T22:14Z)
18
+ Agent and search readiness: animalhouse.ai (readiness-audit v2.6.1, 2026-11-13T00:15Z)
19
19
 
20
20
  Required
21
21
  PASS D1 robots.txt 1 group with consistent rules, Sitemap listed
22
- PASS D4 llms.txt and llms-full.txt llms.txt 16.2 KB, llms-full.txt 65 KB; 10 sampled links resolve without a redirect; llms-full.txt lists the 3 sampled pages
22
+ PASS D4 llms.txt and llms-full.txt llms.txt 17.5 KB, llms-full.txt 66 KB; 10 sampled links resolve without a redirect; llms-full.txt lists the 4 sampled pages
23
23
  PASS D12 No false declarations no OAuth, A2A, plugin or server-card claims without the service behind them
24
- PASS W1 Titles and descriptions 27 sampled pages (18 static, 3 templates), all with their own title and description
24
+ PASS W1 Titles and descriptions 29 sampled pages (20 static, 3 templates), all with their own title and description
25
25
  PASS M4 Hosted MCP edge both eras, JSON for every Accept, browsers redirected, aliases redirect
26
26
  ...
27
27
  Recommended
28
28
  PASS D7 JSON 404s and a JSON 405 on POST / JSON 404 for unknown well-known paths, JSON 405 for POST / (Allow: GET, HEAD, OPTIONS)
29
- WARN D8 Markdown for agents Vary: Accept missing on the HTML response
29
+ PASS D8 Markdown for agents markdown with Vary: Accept on both, x-markdown-tokens 4475; no page llms.txt links serves markdown, so no copy was compared
30
30
  ...
31
- Score on 2026-11-04: required 19 of 19, recommended 10 of 11, next level 1 of 2
31
+ Score on 2026-11-13: required 19 of 19, recommended 11 of 11, next level 2 of 2
32
32
  ```
33
33
 
34
34
  To switch a project over, give its coding agent one line:
@@ -92,7 +92,7 @@ readiness-audit --logs <access-log> [--json]
92
92
  | `--recorded file` | with `--matrix`, add your recorded T5 and T6 results (format: [examples/recorded.json](examples/recorded.json)) |
93
93
  | `--logs file` | count requests for the discovery files in an exported access log (combined log format, or JSON lines from Cloudflare, Vercel or Railway), by client family. It prints counts, never addresses |
94
94
 
95
- It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy. A warning on a required check, such as W6's heading jumps, lowers the required score ("required 14 of 15") without changing the exit code; only a failure does. The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
95
+ It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy, and 2 when it couldn't run (a bad option, an unreadable file, or an error in the audit itself), so CI can tell a failing site from a failed run. A warning on a required check, such as W6's heading jumps, lowers the required score ("required 14 of 15") without changing the exit code; only a failure does. The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
96
96
 
97
97
  To check fixes before they ship, score a local build: `npx readiness-audit@2 example.com --base http://localhost:3000`. The site's own URLs (from its sitemap, llms.txt, skills and catalog) are requested from the base and reported under the domain. From a clone, run `node bin/readiness-audit.mjs example.com`.
98
98
 
@@ -100,7 +100,7 @@ To check fixes before they ship, score a local build: `npx readiness-audit@2 exa
100
100
 
101
101
  - **It only reads:** GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`), at most six at a time. With `--matrix`, each site gets its own six.
102
102
  - **It can go slower.** A site that slows under parallel requests can be audited with `--concurrency 2`, or `--concurrency 1` for one request at a time. The same checks run, and take longer.
103
- - **It stays on public addresses.** It won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect. `--base` may be local, because you chose it.
103
+ - **It stays on public addresses.** It won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect, including an IPv4 address written inside IPv6, and it checks the address it connects to. `--base` may be local, because you chose it.
104
104
  - **A hostile site can't reach a decoder bug.** It asks for gzip and decodes it itself; a body in an encoding it didn't ask for, such as Brotli, is reported rather than read, and every body is capped at 16 MB.
105
105
  - **Run it only against sites you run,** or with the owner's permission. [SECURITY.md](SECURITY.md) has the details and how to report a problem.
106
106
 
@@ -5,12 +5,13 @@
5
5
  // readiness-audit --matrix <domain> <domain> ... [--recorded results.json] (a markdown status page)
6
6
  // readiness-audit --logs <access-log> [--json] (who requests the discovery files)
7
7
  //
8
- // Exits 1 when a required check fails, so it can gate a deploy.
8
+ // Exits 1 when a required check fails, so it can gate a deploy, and 2 when it couldn't run
9
+ // (a bad option, an unreadable file, or an error in the audit itself).
9
10
 
10
11
  import { readFileSync, createReadStream } from 'node:fs';
11
12
  import { createInterface } from 'node:readline';
12
- import { audit, matrix, score, LEVELS, REVIEW, VERSION } from '../src/audit.mjs';
13
- import { parsePrevious, compareResults, formatComparison, nodeOk, NODE_REQUIREMENT } from '../src/helpers.mjs';
13
+ import { audit, matrix, score, LEVELS, REVIEW, VERSION, REPO } from '../src/audit.mjs';
14
+ import { parsePrevious, compareResults, formatComparison, nodeOk, NODE_REQUIREMENT, siteDomain } from '../src/helpers.mjs';
14
15
  import { logTally, formatLogs } from '../src/logs.mjs';
15
16
 
16
17
  // engines in package.json only warns, so say it plainly before anything fails oddly.
@@ -33,7 +34,7 @@ const base = valueOf('--base');
33
34
  const logsPath = valueOf('--logs');
34
35
  const concurrencyOpt = valueOf('--concurrency');
35
36
  const domains = args.filter((a, i) => !a.startsWith('-') && !['--mcp', '--recorded', '--compare', '--base', '--logs', '--concurrency'].includes(args[i - 1]))
36
- .map((d) => d.replace(/^https?:\/\//, '').replace(/\/.*$/, ''));
37
+ .map((d) => siteDomain(d.replace(/^https?:\/\//i, '').replace(/[/?#].*$/, '')));
37
38
  const opts = { mcpPath: flag('--no-mcp') ? null : (mcpOpt ?? '/mcp'), base: base ?? null, concurrency: null };
38
39
  if (flag('--concurrency')) {
39
40
  const n = Number(concurrencyOpt);
@@ -88,46 +89,52 @@ an API (a service-desc link, an API catalog, an API link in llms.txt).`);
88
89
  process.exit(domains.length ? 0 : 2);
89
90
  }
90
91
 
91
- if (flag('--matrix')) {
92
- if (comparePath || base) console.error('--compare and --base work with a single domain; ignoring them for --matrix.');
93
- opts.base = null;
94
- let recorded = {};
95
- if (recordedPath) {
96
- try { recorded = JSON.parse(readFileSync(recordedPath, 'utf8')); }
97
- catch (e) { console.error(`Couldn't read ${recordedPath}: ${e.message}`); process.exit(2); }
98
- }
99
- const reports = await Promise.all(domains.map((d) => audit(d, opts)));
100
- const pinned = `readiness-audit@${VERSION.split('.')[0]}`; // the major version, as STANDARD.md §14 says to pin it
101
- console.log(matrix(reports, `npx ${pinned} --matrix ${domains.join(' ')}${recordedPath ? ` --recorded ${recordedPath}` : ''}`, recorded));
102
- } else {
103
- let previous = null;
104
- if (comparePath) {
105
- let text;
106
- try { text = readFileSync(comparePath, 'utf8'); }
107
- catch (e) { console.error(`Couldn't read ${comparePath}: ${e.message}`); process.exit(2); }
108
- previous = parsePrevious(text, domains[0]);
109
- if (!previous) { console.error(`${comparePath} has no results for ${domains[0]}`); process.exit(2); }
110
- }
111
- const rep = await audit(domains[0], opts);
112
- const since = previous ? `${previous.date ?? 'the previous run'} (${comparePath})` : null;
113
- const changes = previous ? compareResults(previous.results, rep.results) : null;
114
- if (flag('--json')) console.log(JSON.stringify(changes ? { ...rep, changes: { since: previous.date, ...changes } } : rep, null, 2));
115
- else {
116
- const label = { pass: 'PASS', warn: 'WARN', fail: 'FAIL', skip: 'skip' };
117
- console.log(`Agent and search readiness: ${rep.domain}${rep.base ? ` via ${rep.base}` : ''} (readiness-audit v${VERSION}, ${rep.checked_at.slice(0, 16)}Z)`);
118
- for (const level of LEVELS) {
119
- const rows = rep.results.filter((r) => r.level === level);
120
- if (!rows.length) continue;
121
- console.log(`\n${{ required: 'Required', recommended: 'Recommended', next: 'Next level' }[level]}`);
122
- for (const r of rows) console.log(` ${label[r.status]} ${r.id.padEnd(4)} ${r.name.padEnd(36)} ${r.detail}`);
92
+ // An error in the audit is exit 2, not 1, so CI can tell it apart from a failing site.
93
+ try {
94
+ if (flag('--matrix')) {
95
+ if (comparePath || base) console.error('--compare and --base work with a single domain; ignoring them for --matrix.');
96
+ opts.base = null;
97
+ let recorded = {};
98
+ if (recordedPath) {
99
+ try { recorded = JSON.parse(readFileSync(recordedPath, 'utf8')); }
100
+ catch (e) { console.error(`Couldn't read ${recordedPath}: ${e.message}`); process.exit(2); }
123
101
  }
124
- console.log(`\nScore on ${rep.checked_at.slice(0, 10)}: required ${score(rep.results, 'required')}, recommended ${score(rep.results, 'recommended')}, next level ${score(rep.results, 'next')}`);
125
- if (changes) console.log(`\n${formatComparison(changes, since).join('\n')}`);
126
- console.log('\nNot scored, so check these yourself:');
127
- for (const [id, what] of REVIEW) {
128
- console.log(` ${id.padEnd(6)} ${what}`);
129
- if (id === 'W4') for (const p of rep.rich_results_test ?? []) console.log(` ${p.url} (${p.features.join(', ')})\n ${p.test}`);
102
+ const reports = await Promise.all(domains.map((d) => audit(d, opts)));
103
+ const pinned = `readiness-audit@${VERSION.split('.')[0]}`; // the major version, as STANDARD.md §14 says to pin it
104
+ console.log(matrix(reports, `npx ${pinned} --matrix ${domains.join(' ')}${recordedPath ? ` --recorded ${recordedPath}` : ''}`, recorded));
105
+ } else {
106
+ let previous = null;
107
+ if (comparePath) {
108
+ let text;
109
+ try { text = readFileSync(comparePath, 'utf8'); }
110
+ catch (e) { console.error(`Couldn't read ${comparePath}: ${e.message}`); process.exit(2); }
111
+ previous = parsePrevious(text, domains[0]);
112
+ if (!previous) { console.error(`${comparePath} has no results for ${domains[0]}`); process.exit(2); }
130
113
  }
114
+ const rep = await audit(domains[0], opts);
115
+ const since = previous ? `${previous.date ?? 'the previous run'} (${comparePath})` : null;
116
+ const changes = previous ? compareResults(previous.results, rep.results) : null;
117
+ if (flag('--json')) console.log(JSON.stringify(changes ? { ...rep, changes: { since: previous.date, ...changes } } : rep, null, 2));
118
+ else {
119
+ const label = { pass: 'PASS', warn: 'WARN', fail: 'FAIL', skip: 'skip' };
120
+ console.log(`Agent and search readiness: ${rep.domain}${rep.base ? ` via ${rep.base}` : ''} (readiness-audit v${VERSION}, ${rep.checked_at.slice(0, 16)}Z)`);
121
+ for (const level of LEVELS) {
122
+ const rows = rep.results.filter((r) => r.level === level);
123
+ if (!rows.length) continue;
124
+ console.log(`\n${{ required: 'Required', recommended: 'Recommended', next: 'Next level' }[level]}`);
125
+ for (const r of rows) console.log(` ${label[r.status]} ${r.id.padEnd(4)} ${r.name.padEnd(36)} ${r.detail}`);
126
+ }
127
+ console.log(`\nScore on ${rep.checked_at.slice(0, 10)}: required ${score(rep.results, 'required')}, recommended ${score(rep.results, 'recommended')}, next level ${score(rep.results, 'next')}`);
128
+ if (changes) console.log(`\n${formatComparison(changes, since).join('\n')}`);
129
+ console.log('\nNot scored, so check these yourself:');
130
+ for (const [id, what] of REVIEW) {
131
+ console.log(` ${id.padEnd(6)} ${what}`);
132
+ if (id === 'W4') for (const p of rep.rich_results_test ?? []) console.log(` ${p.url} (${p.features.join(', ')})\n ${p.test}`);
133
+ }
134
+ }
135
+ process.exitCode = rep.results.some((r) => r.level === 'required' && r.status === 'fail') ? 1 : 0;
131
136
  }
132
- process.exitCode = rep.results.some((r) => r.level === 'required' && r.status === 'fail') ? 1 : 0;
137
+ } catch (e) {
138
+ console.error(`readiness-audit stopped on an error, so there's no score. Please report it with the domain: ${REPO}/issues\n${e?.stack ?? e}`);
139
+ process.exit(2);
133
140
  }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "readiness-audit",
3
- "version": "2.6.0",
3
+ "version": "2.6.1",
4
4
  "description": "Score a site's readiness for AI agents and search (GEO, AEO, AI SEO): llms.txt, robots.txt AI crawler rules, structured data, link previews, OpenAPI, MCP, agent skills and A2A. Read-only CLI, no account.",
5
5
  "type": "module",
6
6
  "bin": {
package/src/audit.mjs CHANGED
@@ -7,11 +7,12 @@
7
7
  // tools/list, an empty POST to /). Node 22, 24 or 26 at its security release, no dependencies.
8
8
 
9
9
  import { lookup, resolveTxt, Resolver } from 'node:dns/promises';
10
+ import { lookup as dnsLookup } from 'node:dns';
10
11
  import { isIP } from 'node:net';
11
12
  import { request } from './http.mjs';
12
- import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
13
+ import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded, safeUrl, robotsGroups, robotsLeaks, siteDomain } from './helpers.mjs';
13
14
 
14
- export const VERSION = '2.6.0';
15
+ export const VERSION = '2.6.1';
15
16
  export const REPO = 'https://github.com/geeks-accelerator/agent-and-search-readiness';
16
17
  const UA = `readiness-audit/${VERSION} (+${REPO})`;
17
18
  const AID_PROTOCOLS = ['mcp', 'a2a', 'openapi', 'grpc', 'graphql', 'websocket', 'local', 'zeroconf', 'ucp'];
@@ -45,23 +46,42 @@ export const REVIEW = [
45
46
  ['T5', 'agent usability test: a fresh agent, only the domain and a goal; report its first-try success rate'],
46
47
  ];
47
48
 
48
- // Loopback, private, shared, link-local and unique-local addresses. A site's own content (its sitemap,
49
- // share images, skills, catalog, DNS record) must not send the auditor there.
49
+ // Loopback, private, shared, link-local, unique-local, benchmarking, multicast and reserved
50
+ // addresses, and IPv4 ones carried inside IPv6. A site's own content (its sitemap, share images,
51
+ // skills, catalog, DNS record) must not send the auditor there.
50
52
  export function isPrivateAddress(ip) {
51
53
  if (isIP(ip) === 4) {
52
54
  const [a, b] = ip.split('.').map(Number);
53
55
  return a === 0 || a === 10 || a === 127 || (a === 100 && b >= 64 && b <= 127) || (a === 169 && b === 254)
54
- || (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168);
56
+ || (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168) || (a === 198 && (b === 18 || b === 19)) || a >= 224;
55
57
  }
56
58
  if (isIP(ip) === 6) {
57
- const x = ip.toLowerCase();
58
- if (x === '::' || x === '::1') return true;
59
- if (x.startsWith('::ffff:')) return isPrivateAddress(x.slice(7));
60
- return /^f[cd]/.test(x) || /^fe[89ab]/.test(x);
59
+ const x = ipv6Bytes(ip);
60
+ const v4 = (i) => x.slice(i, i + 4).join('.');
61
+ const zeros = (from, to) => x.slice(from, to).every((n) => n === 0);
62
+ if (zeros(0, 16) || (zeros(0, 15) && x[15] === 1)) return true; // :: and ::1
63
+ // The URL parser writes ::ffff:127.0.0.1 as ::ffff:7f00:1, so read the bytes, not the text.
64
+ if (zeros(0, 10) && x[10] === 0xff && x[11] === 0xff) return isPrivateAddress(v4(12)); // IPv4-mapped
65
+ if (zeros(0, 12)) return isPrivateAddress(v4(12)); // IPv4-compatible
66
+ if (x[0] === 0 && x[1] === 0x64 && x[2] === 0xff && x[3] === 0x9b && (zeros(4, 12) || x[4] === 0 && x[5] === 1)) return isPrivateAddress(v4(12)); // NAT64 (64:ff9b::/96, and 64:ff9b:1::/48 for local use)
67
+ if (x[0] === 0x20 && x[1] === 0x02) return isPrivateAddress(v4(2)); // 6to4
68
+ return (x[0] & 0xfe) === 0xfc || (x[0] === 0xfe && (x[1] & 0xc0) === 0x80) || x[0] === 0xff; // unique-local, link-local, multicast
61
69
  }
62
70
  return false;
63
71
  }
64
72
 
73
+ // An IPv6 address as its 16 bytes. Accepts :: and a dotted IPv4 tail; a zone (%eth0) is dropped.
74
+ function ipv6Bytes(ip) {
75
+ let s = ip.toLowerCase().replace(/%.*$/, '');
76
+ const tail = s.match(/(\d+\.\d+\.\d+\.\d+)$/);
77
+ if (tail) { const [a, b, c, d] = tail[1].split('.').map(Number); s = s.slice(0, -tail[1].length) + `${((a << 8) | b).toString(16)}:${((c << 8) | d).toString(16)}`; }
78
+ const [head, rest] = s.split('::');
79
+ const part = (t) => (t ? t.split(':') : []);
80
+ const left = part(head), right = rest === undefined ? [] : part(rest);
81
+ const groups = rest === undefined ? left : [...left, ...Array(8 - left.length - right.length).fill('0'), ...right];
82
+ return groups.flatMap((g) => { const n = parseInt(g, 16); return [n >> 8, n & 0xff]; });
83
+ }
84
+
65
85
  // Run fn over items with at most `limit` in flight, so the audit doesn't hammer a site.
66
86
  async function mapLimit(items, limit, fn) {
67
87
  const out = new Array(items.length);
@@ -87,7 +107,11 @@ async function resolveTxtFresh(name) {
87
107
 
88
108
  // Which URLs the audit may request. The site you chose to audit (and its www or bare variant) is
89
109
  // always allowed; anything else must be a public http(s) address, checked again at every redirect hop.
110
+ // isSafe gives a clear refusal up front; connectLookup checks the addresses the request actually
111
+ // connects to, so a name that resolves to a public address for the check and a private one for the
112
+ // connection (DNS rebinding) is still refused.
90
113
  export function makeGuard(domain) {
114
+ domain = siteDomain(domain);
91
115
  const siteHosts = new Set([domain, domain.startsWith('www.') ? domain.slice(4) : `www.${domain}`]);
92
116
  const hostCache = new Map();
93
117
  async function isSafe(url) {
@@ -99,14 +123,25 @@ export function makeGuard(domain) {
99
123
  if (host === 'localhost' || /\.(localhost|local|internal)$/.test(host)) return false;
100
124
  if (isIP(host)) return !isPrivateAddress(host);
101
125
  if (!hostCache.has(host)) {
102
- hostCache.set(host, lookup(host, { all: true }).then((addrs) => addrs.every((a) => !isPrivateAddress(a.address)), () => true));
126
+ // A name that doesn't resolve isn't known to be public, so it isn't safe.
127
+ hostCache.set(host, lookup(host, { all: true }).then((addrs) => addrs.length > 0 && addrs.every((a) => !isPrivateAddress(a.address)), () => false));
103
128
  }
104
129
  return hostCache.get(host);
105
130
  }
106
- return { siteHosts, isSafe };
131
+ // The lookup option of http(s).request: resolve, then refuse if any address is private.
132
+ function connectLookup(hostname, options, callback) {
133
+ dnsLookup(hostname, { ...options, all: true }, (err, addrs) => {
134
+ if (err) return callback(err);
135
+ if (!addrs.length || addrs.some((a) => isPrivateAddress(a.address))) return callback(new Error(`refused: ${hostname} resolves to a private address`));
136
+ if (options.all) return callback(null, addrs);
137
+ return callback(null, addrs[0].address, addrs[0].family);
138
+ });
139
+ }
140
+ return { siteHosts, isSafe, connectLookup };
107
141
  }
108
142
 
109
143
  export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency = null } = {}) {
144
+ domain = siteDomain(domain); // the package exports audit(), so normalize here as well as in the CLI
110
145
  const BASE = `https://${domain}`;
111
146
  // Requests in flight to the site at once. By default, 6 for the discovery files and 4 for
112
147
  // sitemaps and pages; `concurrency` sets one limit for all of them, for a site that slows
@@ -114,7 +149,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
114
149
  const limit = (byDefault) => concurrency ?? byDefault;
115
150
  const rand = Math.random().toString(36).slice(2, 10);
116
151
 
117
- const { siteHosts, isSafe } = makeGuard(domain);
152
+ const { siteHosts, isSafe, connectLookup } = makeGuard(domain);
118
153
  // --base scores a local or preview build as the domain: the site's URLs are requested from the
119
154
  // base (which may be a local address, because you chose it) and reported under the domain.
120
155
  const { baseOrigin, toBase, fromBase } = baseMapper(domain, base);
@@ -126,10 +161,13 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
126
161
  for (let hop = 0; ; hop++) {
127
162
  const target = toBase(url);
128
163
  if (target === url && !(await isSafe(url))) return { status: 0, headers: new Headers(), text: '', url, type: '', error: 'refused: not a public address' };
129
- const res = await request(target, { method: m, body: b, headers, userAgent: UA });
164
+ // The site's own hosts and the --base you chose may be local; anything else is checked again at connect time.
165
+ const res = await request(target, { method: m, body: b, headers, userAgent: UA, lookup: target === url && !siteHosts.has(safeUrl(url)?.hostname) ? connectLookup : undefined });
130
166
  const loc = res.headers.get('location');
131
167
  if (redirect === 'follow' && res.status >= 300 && res.status < 400 && loc && hop < 5) {
132
- url = fromBase(new URL(loc, target).href);
168
+ const next = safeUrl(loc, target);
169
+ if (!next) return { ...res, url, type: res.headers.get('content-type') ?? '', error: `redirect Location isn't a URL: ${loc.slice(0, 100)}` };
170
+ url = fromBase(next.href);
133
171
  if (res.status !== 307 && res.status !== 308) { m = 'GET'; b = undefined; }
134
172
  continue;
135
173
  }
@@ -189,35 +227,13 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
189
227
  const api = apiPresence({ openapiSpec: json(openapi), apiIndexJson: ok(apiIndex) && isJson(apiIndex), rels, catalogJson: ok(apiCatalog) ? json(apiCatalog) : null, llmsText: ok(llms) ? llms.text : '' });
190
228
 
191
229
  // ---- D1 robots.txt (required) and D15 AI crawlers and Content-Signal (recommended) ----
192
- const groups = [];
193
- let sitemapLine = false;
194
- if (ok(robots)) {
195
- let cur = null, lastUa = false;
196
- for (const raw of robots.text.split('\n')) {
197
- const line = raw.split('#')[0].trim();
198
- const i = line.indexOf(':');
199
- if (i < 0) continue;
200
- const k = line.slice(0, i).trim().toLowerCase();
201
- const v = line.slice(i + 1).trim();
202
- if (k === 'user-agent') {
203
- if (!lastUa) { cur = { agents: [], disallow: new Set(), signal: false }; groups.push(cur); }
204
- cur.agents.push(v.toLowerCase());
205
- lastUa = true;
206
- continue;
207
- }
208
- lastUa = false;
209
- if (k === 'sitemap') sitemapLine = true;
210
- if (!cur) continue;
211
- if (k === 'disallow' && v) cur.disallow.add(v);
212
- if (k === 'content-signal') cur.signal = true;
213
- }
214
- }
230
+ const groups = ok(robots) ? robotsGroups(robots.text) : [];
231
+ const sitemapLine = ok(robots) && /^\s*sitemap\s*:/im.test(robots.text);
215
232
  {
216
233
  const id = 'D1', name = 'robots.txt', level = 'required';
217
- const star = groups.find((g) => g.agents.includes('*'));
218
- const leaks = star ? groups.filter((g) => g !== star).flatMap((g) => [...star.disallow].filter((d) => !g.disallow.has(d)).map((d) => `${g.agents[0]} skips Disallow ${d}`)) : [];
234
+ const leaks = robotsLeaks(groups);
219
235
  if (!ok(robots)) miss(id, name, level, `/robots.txt answered ${robots.status}${cached(robots)}`);
220
- else if (leaks.length) miss(id, name, level, `named groups drop rules from the * group (a crawler obeys only its own group): ${list([...new Set(leaks)], '; ')}${cached(robots)}`);
236
+ else if (leaks.length) miss(id, name, level, `named groups drop rules from the * group (a crawler obeys only its own group): ${list(leaks, '; ')}${cached(robots)}`);
221
237
  else if (!sitemapLine) miss(id, name, level, `no Sitemap line${cached(robots)}`);
222
238
  else pass(id, name, level, `${groups.length} group${groups.length === 1 ? '' : 's'} with consistent rules, Sitemap listed`);
223
239
  }
@@ -497,7 +513,8 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
497
513
  // Entry URLs may be relative, and resolve against the index (RFC 3986).
498
514
  const indexUrl = `${BASE}/.well-known/agent-skills/index.json`;
499
515
  const files = await mapLimit(sample, limit(5), async (e) => {
500
- const url = new URL(e.url, indexUrl).href;
516
+ const url = safeUrl(e.url, indexUrl)?.href;
517
+ if (!url) return { url: e.url, status: 'not a URL', text: '', label: typeof e.name === 'string' && e.name ? e.name : e.url };
501
518
  const r = await get(url);
502
519
  const fm = r.text.match(/^---\r?\n([\s\S]*?)\r?\n---/)?.[1] ?? '';
503
520
  const field = (k) => fm.match(new RegExp(`^${k}:\\s*(?:"([^"]*)"|'([^']*)'|(.*))$`, 'm'))?.slice(1).find((v) => v !== undefined)?.trim() ?? '';
@@ -700,7 +717,9 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
700
717
  const id = 'W2', name = 'Canonical URLs', level = 'required';
701
718
  const none = ok200.filter((p) => !p.canonicals.length).map((p) => short(p.url));
702
719
  const many = ok200.filter((p) => new Set(p.canonicals).size > 1).map((p) => short(p.url));
703
- const elsewhere = ok200.filter((p) => p.canonicals.length && norm(new URL(p.canonicals[0], p.url).href) !== norm(p.url)).map((p) => `${short(p.url)} → ${short(new URL(p.canonicals[0], p.url).href)}`);
720
+ const canonical = (p) => safeUrl(p.canonicals[0], p.url)?.href;
721
+ const notUrl = ok200.filter((p) => p.canonicals.length && !canonical(p)).map((p) => short(p.url));
722
+ const elsewhere = ok200.filter((p) => p.canonicals.length && canonical(p) && norm(canonical(p)) !== norm(p.url)).map((p) => `${short(p.url)} → ${short(canonical(p))}`);
704
723
  const origin = new URL(BASE).origin;
705
724
  // Count only the hops it takes to reach the canonical host; a later path redirect (a language prefix, say) isn't a host hop.
706
725
  const hostHops = async (start) => {
@@ -711,7 +730,9 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
711
730
  if (!r.status) return n === 0 ? null : { n, end: url };
712
731
  const loc = r.headers.get('location');
713
732
  if (r.status < 300 || r.status >= 400 || !loc) return { n, end: url };
714
- url = new URL(loc, url).href;
733
+ const next = safeUrl(loc, url);
734
+ if (!next) return { n, end: url };
735
+ url = next.href;
715
736
  }
716
737
  return { n: 5, end: url };
717
738
  };
@@ -726,6 +747,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
726
747
  const problems = [];
727
748
  if (none.length) problems.push(`no canonical on ${list(none)}`);
728
749
  if (many.length) problems.push(`conflicting canonicals on ${list(many)}`);
750
+ if (notUrl.length) problems.push(`canonical that isn't a URL on ${list(notUrl)}`);
729
751
  if (elsewhere.length) problems.push(`sitemap URLs canonical to another URL: ${list(elsewhere)}`);
730
752
  problems.push(...hostProblems);
731
753
  if (problems.length) miss(id, name, level, problems.join('; '));
@@ -734,7 +756,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
734
756
  // W3 only indexable pages in the sitemap; real 404s
735
757
  {
736
758
  const id = 'W3', name = 'Indexable pages only', level = 'required';
737
- const broken = pages.filter((p) => p.status !== 200).map((p) => `${short(p.url)} (${p.status})`);
759
+ const broken = pages.filter((p) => p.status !== 200).map((p) => `${short(p.url)} (${p.status}${p.error ? `, ${p.error}` : ''})`);
738
760
  const moved = pages.filter((p) => p.status === 200 && norm(p.finalUrl) !== norm(p.url)).map((p) => `${short(p.url)} → ${short(p.finalUrl)}`);
739
761
  const noindex = ok200.filter((p) => /noindex/.test(p.robots)).map((p) => short(p.url));
740
762
  // A missing item under each template: a detail route is where a loading boundary or a fallback answers 200.
@@ -789,8 +811,10 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
789
811
  const images = [...new Set(ok200.map((p) => p.og.image).filter(Boolean))].slice(0, 3);
790
812
  const badImages = [];
791
813
  for (const img of images) {
792
- const r = await get(new URL(img, BASE).href);
793
- if (!(r.status === 200 && /^image\//.test(r.type))) badImages.push(`${short(new URL(img, BASE).href)} (${r.status} ${ct(r)})`);
814
+ const u = safeUrl(img, BASE)?.href;
815
+ if (!u) { badImages.push(`${img.slice(0, 100)} (not a URL)`); continue; }
816
+ const r = await get(u);
817
+ if (!(r.status === 200 && /^image\//.test(r.type))) badImages.push(`${short(u)} (${r.status} ${ct(r)})`);
794
818
  }
795
819
  const problems = [];
796
820
  if (missingOg.length) problems.push(`missing og:title, og:description or og:image on ${list(missingOg)}`);
package/src/helpers.mjs CHANGED
@@ -45,6 +45,72 @@ export function list(xs, sep = ', ') {
45
45
  return `${xs.slice(0, 3).join(sep)}${xs.length > 3 ? `, +${xs.length - 3} more` : ''}`;
46
46
  }
47
47
 
48
+ /** new URL(href, base), or null when it isn't one. Site content can hold anything (href="http://["). */
49
+ export function safeUrl(href, base) {
50
+ try { return new URL(href, base); } catch { return null; }
51
+ }
52
+
53
+ /** A domain as the audit compares hosts: lowercase, with no trailing dot (URL hostnames are both). */
54
+ export const siteDomain = (d) => String(d ?? '').trim().toLowerCase().replace(/\.+$/, '');
55
+
56
+ /**
57
+ * robots.txt as groups: { agents (lowercase), rules: [{ allow, path }], signal }, in file order.
58
+ * A run of User-agent lines starts one group; an empty Disallow allows everything, so it adds no rule.
59
+ */
60
+ export function robotsGroups(text) {
61
+ const groups = [];
62
+ let cur = null, lastUa = false;
63
+ for (const raw of String(text ?? '').split('\n')) {
64
+ const line = raw.split('#')[0].trim();
65
+ const i = line.indexOf(':');
66
+ if (i < 0) continue;
67
+ const k = line.slice(0, i).trim().toLowerCase();
68
+ const v = line.slice(i + 1).trim();
69
+ if (k === 'user-agent') {
70
+ if (!lastUa) { cur = { agents: [], rules: [], signal: false }; groups.push(cur); }
71
+ cur.agents.push(v.toLowerCase());
72
+ lastUa = true;
73
+ continue;
74
+ }
75
+ lastUa = false;
76
+ if (!cur) continue;
77
+ if ((k === 'disallow' || k === 'allow') && v) cur.rules.push({ allow: k === 'allow', path: v });
78
+ if (k === 'content-signal') cur.signal = true;
79
+ }
80
+ return groups;
81
+ }
82
+
83
+ /** Whether rules let a crawler fetch path (RFC 9309 §2.2.2): the longest match wins, Allow on a tie, * and $ as wildcards. */
84
+ export function robotsAllowed(rules, path) {
85
+ let best = null;
86
+ for (const r of rules) {
87
+ const re = new RegExp(`^${r.path.replace(/[.+?^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*').replace(/\\\$$/, '$')}`); // only a final $ anchors
88
+ if (!re.test(path)) continue;
89
+ if (!best || r.path.length > best.path.length || (r.path.length === best.path.length && r.allow)) best = r;
90
+ }
91
+ return !best || best.allow;
92
+ }
93
+
94
+ /**
95
+ * Where a named crawler's rules let it fetch what * disallows. A crawler obeys only the groups that
96
+ * name it (RFC 9309 §2.2.1, which combines groups for the same agent), so a named group must block
97
+ * everything * blocks. Stricter is fine: Disallow / covers /admin. A rule with wildcards is tested
98
+ * as its literal text (/*.pdf$ as /.pdf).
99
+ */
100
+ export function robotsLeaks(groups) {
101
+ const rulesFor = (agent) => groups.filter((g) => g.agents.includes(agent)).flatMap((g) => g.rules);
102
+ const star = rulesFor('*');
103
+ if (!groups.some((g) => g.agents.includes('*'))) return [];
104
+ const literal = (p) => p.replace(/\$$/, '').replace(/\*/g, '');
105
+ const leaks = [];
106
+ for (const agent of [...new Set(groups.flatMap((g) => g.agents))].filter((a) => a !== '*')) {
107
+ const own = rulesFor(agent);
108
+ for (const d of star.filter((r) => !r.allow)) if (robotsAllowed(own, literal(d.path))) leaks.push(`${agent} skips Disallow ${d.path}`);
109
+ for (const a of own.filter((r) => r.allow)) if (!robotsAllowed(star, literal(a.path)) && robotsAllowed(own, literal(a.path))) leaks.push(`${agent} reopens ${a.path} with Allow, which * disallows`);
110
+ }
111
+ return [...new Set(leaks)];
112
+ }
113
+
48
114
  /** A URL for comparison: lowercase host, no trailing slash, no fragment. The query stays. */
49
115
  export function norm(u) {
50
116
  try { const x = new URL(u); return `${x.protocol}//${x.host.toLowerCase()}${x.pathname.replace(/\/+$/, '') || '/'}${x.search}`; } catch { return u; }
@@ -296,6 +362,7 @@ export function llmsLinks(text, llmsUrl, hosts) {
296
362
  * fromBase() maps them back, so every check keeps seeing the domain's URLs.
297
363
  */
298
364
  export function baseMapper(domain, base) {
365
+ domain = siteDomain(domain);
299
366
  const hosts = new Set([domain, domain.startsWith('www.') ? domain.slice(4) : `www.${domain}`]);
300
367
  const baseOrigin = base ? new URL(base).origin : null;
301
368
  const toBase = (url) => {
@@ -346,7 +413,7 @@ const decode = (s) => { try { return decodeURIComponent(s); } catch { return s;
346
413
  */
347
414
  export function skillNameProblem({ name, url, entryName }) {
348
415
  if (!SKILL_NAME.test(name ?? '')) return 'not a lowercase slug';
349
- const path = new URL(url).pathname;
416
+ const path = safeUrl(url)?.pathname ?? '';
350
417
  const folder = /\/SKILL\.md$/i.test(path) ? decode(path.split('/').slice(-2, -1)[0] ?? '') : '';
351
418
  if (folder && folder !== name) return `folder is ${folder}`;
352
419
  if (typeof entryName === 'string' && entryName && entryName !== name) return `index says ${entryName}`;
package/src/http.mjs CHANGED
@@ -19,9 +19,11 @@ export const MAX_BYTES = 16 * 1024 * 1024;
19
19
  * `accept-encoding: gzip, deflate`; a header set to null isn't sent (M4's no-Accept probe).
20
20
  * Resolves, never rejects: { status, headers (a Headers object), text, error?, tooBig? }, status 0
21
21
  * on no answer. tooBig: the body passed maxBytes, raw or decoded; reading stopped and the text is empty.
22
+ * lookup: passed to http(s).request, so the caller can refuse the addresses a name resolves to.
22
23
  */
23
- export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS, maxBytes = MAX_BYTES } = {}) {
24
- const target = new URL(url);
24
+ export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS, maxBytes = MAX_BYTES, lookup } = {}) {
25
+ let target;
26
+ try { target = new URL(url); } catch { return Promise.resolve({ status: 0, headers: new Headers(), text: '', error: `not a URL: ${String(url).slice(0, 100)}` }); }
25
27
  const send = target.protocol === 'http:' ? httpRequest : httpsRequest;
26
28
  const all = { 'user-agent': userAgent, accept: '*/*', 'accept-encoding': 'gzip, deflate', ...headers };
27
29
  if (body !== undefined) all['content-length'] = Buffer.byteLength(body);
@@ -33,7 +35,12 @@ export function request(url, { method = 'GET', headers = {}, body, userAgent, ti
33
35
  return new Promise((resolve) => {
34
36
  let settled = false;
35
37
  const done = (out) => { if (!settled) { settled = true; resolve(out); } };
36
- const req = send(target, { method, headers: sent, signal: AbortSignal.timeout(timeoutMs) }, (res) => {
38
+ let req;
39
+ try { req = send(target, { method, headers: sent, signal: AbortSignal.timeout(timeoutMs), ...(lookup ? { lookup } : {}) }, onResponse); }
40
+ catch (e) { return done(failed(e)); } // a header value the site supplied that http can't send, say
41
+ req.on('error', (e) => done(failed(e)));
42
+ req.end(body);
43
+ function onResponse(res) {
37
44
  const chunks = [];
38
45
  let size = 0;
39
46
  res.on('data', (c) => {
@@ -63,8 +70,6 @@ export function request(url, { method = 'GET', headers = {}, body, userAgent, ti
63
70
  }
64
71
  done(out);
65
72
  });
66
- });
67
- req.on('error', (e) => done(failed(e)));
68
- req.end(body);
73
+ }
69
74
  });
70
75
  }