readiness-audit 2.6.0 → 2.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +9 -0
- package/README.md +7 -7
- package/bin/readiness-audit.mjs +50 -43
- package/package.json +1 -1
- package/src/audit.mjs +70 -46
- package/src/helpers.mjs +68 -1
- package/src/http.mjs +11 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,14 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.6.1 (2026-11-12)
|
|
4
|
+
|
|
5
|
+
Four bugs from an outside review of 2.6.0, each reproduced first and now covered by a test. No new checks and no new warnings. D1 passes robots.txt files it wrongly failed, and the audit finishes where it used to crash.
|
|
6
|
+
|
|
7
|
+
- **The address guard reads IPv6 as bytes, and checks at connect time (security).** The URL parser writes `[::ffff:127.0.0.1]` as `[::ffff:7f00:1]`, and the guard knew only the dotted form, so a site could redirect the audit to loopback or a cloud metadata address. It now classifies IPv4-mapped, IPv4-compatible, NAT64 and 6to4 addresses by the IPv4 inside them, and refuses 198.18.0.0/15, multicast and reserved addresses. A name that doesn't resolve is no longer counted as safe. Every request outside the audited site and `--base` checks the addresses it connects to, so a name that resolves differently the second time (DNS rebinding) is still refused.
|
|
8
|
+
- **A URL in site content that isn't one is a finding, not a crash.** A canonical like `href="http://["` stopped the whole audit with exit code 1, the code for a failing site. W2 reports a canonical that isn't a URL, W5 a share image, W3 a redirect `Location`, S1 a skills index entry. An error in the audit itself now exits 2, as a bad option already did, and the README says so.
|
|
9
|
+
- **D1 lets a named robots.txt group be stricter than `*`.** `User-agent: BadBot` with `Disallow: /` failed as "badbot skips Disallow /admin". D1 now matches rules as RFC 9309 does (the longest match wins, Allow on a tie, `*` and `$`), reads Allow lines, and combines groups for the same agent, `*` included. It still fails a named group that misses a `*` Disallow, or reopens one with Allow.
|
|
10
|
+
- **A domain in capitals or with a trailing dot is the same site.** `MusicVenue.space` found no sitemap URLs on its own site and failed D2 and W1 to W6. The CLI and `audit()` lowercase the domain and drop a trailing dot.
|
|
11
|
+
|
|
3
12
|
## 2.6.0 (2026-11-04)
|
|
4
13
|
|
|
5
14
|
From two projects fixing their APIs: one has no `/v1` prefix. No scoring changes, and no new warnings or failures.
|
package/README.md
CHANGED
|
@@ -15,20 +15,20 @@ npx readiness-audit example.com
|
|
|
15
15
|
It needs Node 22.23.4, 24.22.1 or 26.12.1, or later on the same line (each supported line's current security release), and has no dependencies.
|
|
16
16
|
|
|
17
17
|
```
|
|
18
|
-
Agent and search readiness: animalhouse.ai (readiness-audit v2.6.
|
|
18
|
+
Agent and search readiness: animalhouse.ai (readiness-audit v2.6.1, 2026-11-13T00:15Z)
|
|
19
19
|
|
|
20
20
|
Required
|
|
21
21
|
PASS D1 robots.txt 1 group with consistent rules, Sitemap listed
|
|
22
|
-
PASS D4 llms.txt and llms-full.txt llms.txt
|
|
22
|
+
PASS D4 llms.txt and llms-full.txt llms.txt 17.5 KB, llms-full.txt 66 KB; 10 sampled links resolve without a redirect; llms-full.txt lists the 4 sampled pages
|
|
23
23
|
PASS D12 No false declarations no OAuth, A2A, plugin or server-card claims without the service behind them
|
|
24
|
-
PASS W1 Titles and descriptions
|
|
24
|
+
PASS W1 Titles and descriptions 29 sampled pages (20 static, 3 templates), all with their own title and description
|
|
25
25
|
PASS M4 Hosted MCP edge both eras, JSON for every Accept, browsers redirected, aliases redirect
|
|
26
26
|
...
|
|
27
27
|
Recommended
|
|
28
28
|
PASS D7 JSON 404s and a JSON 405 on POST / JSON 404 for unknown well-known paths, JSON 405 for POST / (Allow: GET, HEAD, OPTIONS)
|
|
29
|
-
|
|
29
|
+
PASS D8 Markdown for agents markdown with Vary: Accept on both, x-markdown-tokens 4475; no page llms.txt links serves markdown, so no copy was compared
|
|
30
30
|
...
|
|
31
|
-
Score on 2026-11-
|
|
31
|
+
Score on 2026-11-13: required 19 of 19, recommended 11 of 11, next level 2 of 2
|
|
32
32
|
```
|
|
33
33
|
|
|
34
34
|
To switch a project over, give its coding agent one line:
|
|
@@ -92,7 +92,7 @@ readiness-audit --logs <access-log> [--json]
|
|
|
92
92
|
| `--recorded file` | with `--matrix`, add your recorded T5 and T6 results (format: [examples/recorded.json](examples/recorded.json)) |
|
|
93
93
|
| `--logs file` | count requests for the discovery files in an exported access log (combined log format, or JSON lines from Cloudflare, Vercel or Railway), by client family. It prints counts, never addresses |
|
|
94
94
|
|
|
95
|
-
It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy. A warning on a required check, such as W6's heading jumps, lowers the required score ("required 14 of 15") without changing the exit code; only a failure does. The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
|
|
95
|
+
It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy, and 2 when it couldn't run (a bad option, an unreadable file, or an error in the audit itself), so CI can tell a failing site from a failed run. A warning on a required check, such as W6's heading jumps, lowers the required score ("required 14 of 15") without changing the exit code; only a failure does. The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
|
|
96
96
|
|
|
97
97
|
To check fixes before they ship, score a local build: `npx readiness-audit@2 example.com --base http://localhost:3000`. The site's own URLs (from its sitemap, llms.txt, skills and catalog) are requested from the base and reported under the domain. From a clone, run `node bin/readiness-audit.mjs example.com`.
|
|
98
98
|
|
|
@@ -100,7 +100,7 @@ To check fixes before they ship, score a local build: `npx readiness-audit@2 exa
|
|
|
100
100
|
|
|
101
101
|
- **It only reads:** GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`), at most six at a time. With `--matrix`, each site gets its own six.
|
|
102
102
|
- **It can go slower.** A site that slows under parallel requests can be audited with `--concurrency 2`, or `--concurrency 1` for one request at a time. The same checks run, and take longer.
|
|
103
|
-
- **It stays on public addresses.** It won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect. `--base` may be local, because you chose it.
|
|
103
|
+
- **It stays on public addresses.** It won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect, including an IPv4 address written inside IPv6, and it checks the address it connects to. `--base` may be local, because you chose it.
|
|
104
104
|
- **A hostile site can't reach a decoder bug.** It asks for gzip and decodes it itself; a body in an encoding it didn't ask for, such as Brotli, is reported rather than read, and every body is capped at 16 MB.
|
|
105
105
|
- **Run it only against sites you run,** or with the owner's permission. [SECURITY.md](SECURITY.md) has the details and how to report a problem.
|
|
106
106
|
|
package/bin/readiness-audit.mjs
CHANGED
|
@@ -5,12 +5,13 @@
|
|
|
5
5
|
// readiness-audit --matrix <domain> <domain> ... [--recorded results.json] (a markdown status page)
|
|
6
6
|
// readiness-audit --logs <access-log> [--json] (who requests the discovery files)
|
|
7
7
|
//
|
|
8
|
-
// Exits 1 when a required check fails, so it can gate a deploy
|
|
8
|
+
// Exits 1 when a required check fails, so it can gate a deploy, and 2 when it couldn't run
|
|
9
|
+
// (a bad option, an unreadable file, or an error in the audit itself).
|
|
9
10
|
|
|
10
11
|
import { readFileSync, createReadStream } from 'node:fs';
|
|
11
12
|
import { createInterface } from 'node:readline';
|
|
12
|
-
import { audit, matrix, score, LEVELS, REVIEW, VERSION } from '../src/audit.mjs';
|
|
13
|
-
import { parsePrevious, compareResults, formatComparison, nodeOk, NODE_REQUIREMENT } from '../src/helpers.mjs';
|
|
13
|
+
import { audit, matrix, score, LEVELS, REVIEW, VERSION, REPO } from '../src/audit.mjs';
|
|
14
|
+
import { parsePrevious, compareResults, formatComparison, nodeOk, NODE_REQUIREMENT, siteDomain } from '../src/helpers.mjs';
|
|
14
15
|
import { logTally, formatLogs } from '../src/logs.mjs';
|
|
15
16
|
|
|
16
17
|
// engines in package.json only warns, so say it plainly before anything fails oddly.
|
|
@@ -33,7 +34,7 @@ const base = valueOf('--base');
|
|
|
33
34
|
const logsPath = valueOf('--logs');
|
|
34
35
|
const concurrencyOpt = valueOf('--concurrency');
|
|
35
36
|
const domains = args.filter((a, i) => !a.startsWith('-') && !['--mcp', '--recorded', '--compare', '--base', '--logs', '--concurrency'].includes(args[i - 1]))
|
|
36
|
-
.map((d) => d.replace(/^https
|
|
37
|
+
.map((d) => siteDomain(d.replace(/^https?:\/\//i, '').replace(/[/?#].*$/, '')));
|
|
37
38
|
const opts = { mcpPath: flag('--no-mcp') ? null : (mcpOpt ?? '/mcp'), base: base ?? null, concurrency: null };
|
|
38
39
|
if (flag('--concurrency')) {
|
|
39
40
|
const n = Number(concurrencyOpt);
|
|
@@ -88,46 +89,52 @@ an API (a service-desc link, an API catalog, an API link in llms.txt).`);
|
|
|
88
89
|
process.exit(domains.length ? 0 : 2);
|
|
89
90
|
}
|
|
90
91
|
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
const pinned = `readiness-audit@${VERSION.split('.')[0]}`; // the major version, as STANDARD.md §14 says to pin it
|
|
101
|
-
console.log(matrix(reports, `npx ${pinned} --matrix ${domains.join(' ')}${recordedPath ? ` --recorded ${recordedPath}` : ''}`, recorded));
|
|
102
|
-
} else {
|
|
103
|
-
let previous = null;
|
|
104
|
-
if (comparePath) {
|
|
105
|
-
let text;
|
|
106
|
-
try { text = readFileSync(comparePath, 'utf8'); }
|
|
107
|
-
catch (e) { console.error(`Couldn't read ${comparePath}: ${e.message}`); process.exit(2); }
|
|
108
|
-
previous = parsePrevious(text, domains[0]);
|
|
109
|
-
if (!previous) { console.error(`${comparePath} has no results for ${domains[0]}`); process.exit(2); }
|
|
110
|
-
}
|
|
111
|
-
const rep = await audit(domains[0], opts);
|
|
112
|
-
const since = previous ? `${previous.date ?? 'the previous run'} (${comparePath})` : null;
|
|
113
|
-
const changes = previous ? compareResults(previous.results, rep.results) : null;
|
|
114
|
-
if (flag('--json')) console.log(JSON.stringify(changes ? { ...rep, changes: { since: previous.date, ...changes } } : rep, null, 2));
|
|
115
|
-
else {
|
|
116
|
-
const label = { pass: 'PASS', warn: 'WARN', fail: 'FAIL', skip: 'skip' };
|
|
117
|
-
console.log(`Agent and search readiness: ${rep.domain}${rep.base ? ` via ${rep.base}` : ''} (readiness-audit v${VERSION}, ${rep.checked_at.slice(0, 16)}Z)`);
|
|
118
|
-
for (const level of LEVELS) {
|
|
119
|
-
const rows = rep.results.filter((r) => r.level === level);
|
|
120
|
-
if (!rows.length) continue;
|
|
121
|
-
console.log(`\n${{ required: 'Required', recommended: 'Recommended', next: 'Next level' }[level]}`);
|
|
122
|
-
for (const r of rows) console.log(` ${label[r.status]} ${r.id.padEnd(4)} ${r.name.padEnd(36)} ${r.detail}`);
|
|
92
|
+
// An error in the audit is exit 2, not 1, so CI can tell it apart from a failing site.
|
|
93
|
+
try {
|
|
94
|
+
if (flag('--matrix')) {
|
|
95
|
+
if (comparePath || base) console.error('--compare and --base work with a single domain; ignoring them for --matrix.');
|
|
96
|
+
opts.base = null;
|
|
97
|
+
let recorded = {};
|
|
98
|
+
if (recordedPath) {
|
|
99
|
+
try { recorded = JSON.parse(readFileSync(recordedPath, 'utf8')); }
|
|
100
|
+
catch (e) { console.error(`Couldn't read ${recordedPath}: ${e.message}`); process.exit(2); }
|
|
123
101
|
}
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
console.log('
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
102
|
+
const reports = await Promise.all(domains.map((d) => audit(d, opts)));
|
|
103
|
+
const pinned = `readiness-audit@${VERSION.split('.')[0]}`; // the major version, as STANDARD.md §14 says to pin it
|
|
104
|
+
console.log(matrix(reports, `npx ${pinned} --matrix ${domains.join(' ')}${recordedPath ? ` --recorded ${recordedPath}` : ''}`, recorded));
|
|
105
|
+
} else {
|
|
106
|
+
let previous = null;
|
|
107
|
+
if (comparePath) {
|
|
108
|
+
let text;
|
|
109
|
+
try { text = readFileSync(comparePath, 'utf8'); }
|
|
110
|
+
catch (e) { console.error(`Couldn't read ${comparePath}: ${e.message}`); process.exit(2); }
|
|
111
|
+
previous = parsePrevious(text, domains[0]);
|
|
112
|
+
if (!previous) { console.error(`${comparePath} has no results for ${domains[0]}`); process.exit(2); }
|
|
130
113
|
}
|
|
114
|
+
const rep = await audit(domains[0], opts);
|
|
115
|
+
const since = previous ? `${previous.date ?? 'the previous run'} (${comparePath})` : null;
|
|
116
|
+
const changes = previous ? compareResults(previous.results, rep.results) : null;
|
|
117
|
+
if (flag('--json')) console.log(JSON.stringify(changes ? { ...rep, changes: { since: previous.date, ...changes } } : rep, null, 2));
|
|
118
|
+
else {
|
|
119
|
+
const label = { pass: 'PASS', warn: 'WARN', fail: 'FAIL', skip: 'skip' };
|
|
120
|
+
console.log(`Agent and search readiness: ${rep.domain}${rep.base ? ` via ${rep.base}` : ''} (readiness-audit v${VERSION}, ${rep.checked_at.slice(0, 16)}Z)`);
|
|
121
|
+
for (const level of LEVELS) {
|
|
122
|
+
const rows = rep.results.filter((r) => r.level === level);
|
|
123
|
+
if (!rows.length) continue;
|
|
124
|
+
console.log(`\n${{ required: 'Required', recommended: 'Recommended', next: 'Next level' }[level]}`);
|
|
125
|
+
for (const r of rows) console.log(` ${label[r.status]} ${r.id.padEnd(4)} ${r.name.padEnd(36)} ${r.detail}`);
|
|
126
|
+
}
|
|
127
|
+
console.log(`\nScore on ${rep.checked_at.slice(0, 10)}: required ${score(rep.results, 'required')}, recommended ${score(rep.results, 'recommended')}, next level ${score(rep.results, 'next')}`);
|
|
128
|
+
if (changes) console.log(`\n${formatComparison(changes, since).join('\n')}`);
|
|
129
|
+
console.log('\nNot scored, so check these yourself:');
|
|
130
|
+
for (const [id, what] of REVIEW) {
|
|
131
|
+
console.log(` ${id.padEnd(6)} ${what}`);
|
|
132
|
+
if (id === 'W4') for (const p of rep.rich_results_test ?? []) console.log(` ${p.url} (${p.features.join(', ')})\n ${p.test}`);
|
|
133
|
+
}
|
|
134
|
+
}
|
|
135
|
+
process.exitCode = rep.results.some((r) => r.level === 'required' && r.status === 'fail') ? 1 : 0;
|
|
131
136
|
}
|
|
132
|
-
|
|
137
|
+
} catch (e) {
|
|
138
|
+
console.error(`readiness-audit stopped on an error, so there's no score. Please report it with the domain: ${REPO}/issues\n${e?.stack ?? e}`);
|
|
139
|
+
process.exit(2);
|
|
133
140
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "readiness-audit",
|
|
3
|
-
"version": "2.6.
|
|
3
|
+
"version": "2.6.1",
|
|
4
4
|
"description": "Score a site's readiness for AI agents and search (GEO, AEO, AI SEO): llms.txt, robots.txt AI crawler rules, structured data, link previews, OpenAPI, MCP, agent skills and A2A. Read-only CLI, no account.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
package/src/audit.mjs
CHANGED
|
@@ -7,11 +7,12 @@
|
|
|
7
7
|
// tools/list, an empty POST to /). Node 22, 24 or 26 at its security release, no dependencies.
|
|
8
8
|
|
|
9
9
|
import { lookup, resolveTxt, Resolver } from 'node:dns/promises';
|
|
10
|
+
import { lookup as dnsLookup } from 'node:dns';
|
|
10
11
|
import { isIP } from 'node:net';
|
|
11
12
|
import { request } from './http.mjs';
|
|
12
|
-
import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
|
|
13
|
+
import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded, safeUrl, robotsGroups, robotsLeaks, siteDomain } from './helpers.mjs';
|
|
13
14
|
|
|
14
|
-
export const VERSION = '2.6.
|
|
15
|
+
export const VERSION = '2.6.1';
|
|
15
16
|
export const REPO = 'https://github.com/geeks-accelerator/agent-and-search-readiness';
|
|
16
17
|
const UA = `readiness-audit/${VERSION} (+${REPO})`;
|
|
17
18
|
const AID_PROTOCOLS = ['mcp', 'a2a', 'openapi', 'grpc', 'graphql', 'websocket', 'local', 'zeroconf', 'ucp'];
|
|
@@ -45,23 +46,42 @@ export const REVIEW = [
|
|
|
45
46
|
['T5', 'agent usability test: a fresh agent, only the domain and a goal; report its first-try success rate'],
|
|
46
47
|
];
|
|
47
48
|
|
|
48
|
-
// Loopback, private, shared, link-local
|
|
49
|
-
//
|
|
49
|
+
// Loopback, private, shared, link-local, unique-local, benchmarking, multicast and reserved
|
|
50
|
+
// addresses, and IPv4 ones carried inside IPv6. A site's own content (its sitemap, share images,
|
|
51
|
+
// skills, catalog, DNS record) must not send the auditor there.
|
|
50
52
|
export function isPrivateAddress(ip) {
|
|
51
53
|
if (isIP(ip) === 4) {
|
|
52
54
|
const [a, b] = ip.split('.').map(Number);
|
|
53
55
|
return a === 0 || a === 10 || a === 127 || (a === 100 && b >= 64 && b <= 127) || (a === 169 && b === 254)
|
|
54
|
-
|| (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168);
|
|
56
|
+
|| (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168) || (a === 198 && (b === 18 || b === 19)) || a >= 224;
|
|
55
57
|
}
|
|
56
58
|
if (isIP(ip) === 6) {
|
|
57
|
-
const x = ip
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
59
|
+
const x = ipv6Bytes(ip);
|
|
60
|
+
const v4 = (i) => x.slice(i, i + 4).join('.');
|
|
61
|
+
const zeros = (from, to) => x.slice(from, to).every((n) => n === 0);
|
|
62
|
+
if (zeros(0, 16) || (zeros(0, 15) && x[15] === 1)) return true; // :: and ::1
|
|
63
|
+
// The URL parser writes ::ffff:127.0.0.1 as ::ffff:7f00:1, so read the bytes, not the text.
|
|
64
|
+
if (zeros(0, 10) && x[10] === 0xff && x[11] === 0xff) return isPrivateAddress(v4(12)); // IPv4-mapped
|
|
65
|
+
if (zeros(0, 12)) return isPrivateAddress(v4(12)); // IPv4-compatible
|
|
66
|
+
if (x[0] === 0 && x[1] === 0x64 && x[2] === 0xff && x[3] === 0x9b && (zeros(4, 12) || x[4] === 0 && x[5] === 1)) return isPrivateAddress(v4(12)); // NAT64 (64:ff9b::/96, and 64:ff9b:1::/48 for local use)
|
|
67
|
+
if (x[0] === 0x20 && x[1] === 0x02) return isPrivateAddress(v4(2)); // 6to4
|
|
68
|
+
return (x[0] & 0xfe) === 0xfc || (x[0] === 0xfe && (x[1] & 0xc0) === 0x80) || x[0] === 0xff; // unique-local, link-local, multicast
|
|
61
69
|
}
|
|
62
70
|
return false;
|
|
63
71
|
}
|
|
64
72
|
|
|
73
|
+
// An IPv6 address as its 16 bytes. Accepts :: and a dotted IPv4 tail; a zone (%eth0) is dropped.
|
|
74
|
+
function ipv6Bytes(ip) {
|
|
75
|
+
let s = ip.toLowerCase().replace(/%.*$/, '');
|
|
76
|
+
const tail = s.match(/(\d+\.\d+\.\d+\.\d+)$/);
|
|
77
|
+
if (tail) { const [a, b, c, d] = tail[1].split('.').map(Number); s = s.slice(0, -tail[1].length) + `${((a << 8) | b).toString(16)}:${((c << 8) | d).toString(16)}`; }
|
|
78
|
+
const [head, rest] = s.split('::');
|
|
79
|
+
const part = (t) => (t ? t.split(':') : []);
|
|
80
|
+
const left = part(head), right = rest === undefined ? [] : part(rest);
|
|
81
|
+
const groups = rest === undefined ? left : [...left, ...Array(8 - left.length - right.length).fill('0'), ...right];
|
|
82
|
+
return groups.flatMap((g) => { const n = parseInt(g, 16); return [n >> 8, n & 0xff]; });
|
|
83
|
+
}
|
|
84
|
+
|
|
65
85
|
// Run fn over items with at most `limit` in flight, so the audit doesn't hammer a site.
|
|
66
86
|
async function mapLimit(items, limit, fn) {
|
|
67
87
|
const out = new Array(items.length);
|
|
@@ -87,7 +107,11 @@ async function resolveTxtFresh(name) {
|
|
|
87
107
|
|
|
88
108
|
// Which URLs the audit may request. The site you chose to audit (and its www or bare variant) is
|
|
89
109
|
// always allowed; anything else must be a public http(s) address, checked again at every redirect hop.
|
|
110
|
+
// isSafe gives a clear refusal up front; connectLookup checks the addresses the request actually
|
|
111
|
+
// connects to, so a name that resolves to a public address for the check and a private one for the
|
|
112
|
+
// connection (DNS rebinding) is still refused.
|
|
90
113
|
export function makeGuard(domain) {
|
|
114
|
+
domain = siteDomain(domain);
|
|
91
115
|
const siteHosts = new Set([domain, domain.startsWith('www.') ? domain.slice(4) : `www.${domain}`]);
|
|
92
116
|
const hostCache = new Map();
|
|
93
117
|
async function isSafe(url) {
|
|
@@ -99,14 +123,25 @@ export function makeGuard(domain) {
|
|
|
99
123
|
if (host === 'localhost' || /\.(localhost|local|internal)$/.test(host)) return false;
|
|
100
124
|
if (isIP(host)) return !isPrivateAddress(host);
|
|
101
125
|
if (!hostCache.has(host)) {
|
|
102
|
-
|
|
126
|
+
// A name that doesn't resolve isn't known to be public, so it isn't safe.
|
|
127
|
+
hostCache.set(host, lookup(host, { all: true }).then((addrs) => addrs.length > 0 && addrs.every((a) => !isPrivateAddress(a.address)), () => false));
|
|
103
128
|
}
|
|
104
129
|
return hostCache.get(host);
|
|
105
130
|
}
|
|
106
|
-
|
|
131
|
+
// The lookup option of http(s).request: resolve, then refuse if any address is private.
|
|
132
|
+
function connectLookup(hostname, options, callback) {
|
|
133
|
+
dnsLookup(hostname, { ...options, all: true }, (err, addrs) => {
|
|
134
|
+
if (err) return callback(err);
|
|
135
|
+
if (!addrs.length || addrs.some((a) => isPrivateAddress(a.address))) return callback(new Error(`refused: ${hostname} resolves to a private address`));
|
|
136
|
+
if (options.all) return callback(null, addrs);
|
|
137
|
+
return callback(null, addrs[0].address, addrs[0].family);
|
|
138
|
+
});
|
|
139
|
+
}
|
|
140
|
+
return { siteHosts, isSafe, connectLookup };
|
|
107
141
|
}
|
|
108
142
|
|
|
109
143
|
export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency = null } = {}) {
|
|
144
|
+
domain = siteDomain(domain); // the package exports audit(), so normalize here as well as in the CLI
|
|
110
145
|
const BASE = `https://${domain}`;
|
|
111
146
|
// Requests in flight to the site at once. By default, 6 for the discovery files and 4 for
|
|
112
147
|
// sitemaps and pages; `concurrency` sets one limit for all of them, for a site that slows
|
|
@@ -114,7 +149,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
114
149
|
const limit = (byDefault) => concurrency ?? byDefault;
|
|
115
150
|
const rand = Math.random().toString(36).slice(2, 10);
|
|
116
151
|
|
|
117
|
-
const { siteHosts, isSafe } = makeGuard(domain);
|
|
152
|
+
const { siteHosts, isSafe, connectLookup } = makeGuard(domain);
|
|
118
153
|
// --base scores a local or preview build as the domain: the site's URLs are requested from the
|
|
119
154
|
// base (which may be a local address, because you chose it) and reported under the domain.
|
|
120
155
|
const { baseOrigin, toBase, fromBase } = baseMapper(domain, base);
|
|
@@ -126,10 +161,13 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
126
161
|
for (let hop = 0; ; hop++) {
|
|
127
162
|
const target = toBase(url);
|
|
128
163
|
if (target === url && !(await isSafe(url))) return { status: 0, headers: new Headers(), text: '', url, type: '', error: 'refused: not a public address' };
|
|
129
|
-
|
|
164
|
+
// The site's own hosts and the --base you chose may be local; anything else is checked again at connect time.
|
|
165
|
+
const res = await request(target, { method: m, body: b, headers, userAgent: UA, lookup: target === url && !siteHosts.has(safeUrl(url)?.hostname) ? connectLookup : undefined });
|
|
130
166
|
const loc = res.headers.get('location');
|
|
131
167
|
if (redirect === 'follow' && res.status >= 300 && res.status < 400 && loc && hop < 5) {
|
|
132
|
-
|
|
168
|
+
const next = safeUrl(loc, target);
|
|
169
|
+
if (!next) return { ...res, url, type: res.headers.get('content-type') ?? '', error: `redirect Location isn't a URL: ${loc.slice(0, 100)}` };
|
|
170
|
+
url = fromBase(next.href);
|
|
133
171
|
if (res.status !== 307 && res.status !== 308) { m = 'GET'; b = undefined; }
|
|
134
172
|
continue;
|
|
135
173
|
}
|
|
@@ -189,35 +227,13 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
189
227
|
const api = apiPresence({ openapiSpec: json(openapi), apiIndexJson: ok(apiIndex) && isJson(apiIndex), rels, catalogJson: ok(apiCatalog) ? json(apiCatalog) : null, llmsText: ok(llms) ? llms.text : '' });
|
|
190
228
|
|
|
191
229
|
// ---- D1 robots.txt (required) and D15 AI crawlers and Content-Signal (recommended) ----
|
|
192
|
-
const groups = [];
|
|
193
|
-
|
|
194
|
-
if (ok(robots)) {
|
|
195
|
-
let cur = null, lastUa = false;
|
|
196
|
-
for (const raw of robots.text.split('\n')) {
|
|
197
|
-
const line = raw.split('#')[0].trim();
|
|
198
|
-
const i = line.indexOf(':');
|
|
199
|
-
if (i < 0) continue;
|
|
200
|
-
const k = line.slice(0, i).trim().toLowerCase();
|
|
201
|
-
const v = line.slice(i + 1).trim();
|
|
202
|
-
if (k === 'user-agent') {
|
|
203
|
-
if (!lastUa) { cur = { agents: [], disallow: new Set(), signal: false }; groups.push(cur); }
|
|
204
|
-
cur.agents.push(v.toLowerCase());
|
|
205
|
-
lastUa = true;
|
|
206
|
-
continue;
|
|
207
|
-
}
|
|
208
|
-
lastUa = false;
|
|
209
|
-
if (k === 'sitemap') sitemapLine = true;
|
|
210
|
-
if (!cur) continue;
|
|
211
|
-
if (k === 'disallow' && v) cur.disallow.add(v);
|
|
212
|
-
if (k === 'content-signal') cur.signal = true;
|
|
213
|
-
}
|
|
214
|
-
}
|
|
230
|
+
const groups = ok(robots) ? robotsGroups(robots.text) : [];
|
|
231
|
+
const sitemapLine = ok(robots) && /^\s*sitemap\s*:/im.test(robots.text);
|
|
215
232
|
{
|
|
216
233
|
const id = 'D1', name = 'robots.txt', level = 'required';
|
|
217
|
-
const
|
|
218
|
-
const leaks = star ? groups.filter((g) => g !== star).flatMap((g) => [...star.disallow].filter((d) => !g.disallow.has(d)).map((d) => `${g.agents[0]} skips Disallow ${d}`)) : [];
|
|
234
|
+
const leaks = robotsLeaks(groups);
|
|
219
235
|
if (!ok(robots)) miss(id, name, level, `/robots.txt answered ${robots.status}${cached(robots)}`);
|
|
220
|
-
else if (leaks.length) miss(id, name, level, `named groups drop rules from the * group (a crawler obeys only its own group): ${list(
|
|
236
|
+
else if (leaks.length) miss(id, name, level, `named groups drop rules from the * group (a crawler obeys only its own group): ${list(leaks, '; ')}${cached(robots)}`);
|
|
221
237
|
else if (!sitemapLine) miss(id, name, level, `no Sitemap line${cached(robots)}`);
|
|
222
238
|
else pass(id, name, level, `${groups.length} group${groups.length === 1 ? '' : 's'} with consistent rules, Sitemap listed`);
|
|
223
239
|
}
|
|
@@ -497,7 +513,8 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
497
513
|
// Entry URLs may be relative, and resolve against the index (RFC 3986).
|
|
498
514
|
const indexUrl = `${BASE}/.well-known/agent-skills/index.json`;
|
|
499
515
|
const files = await mapLimit(sample, limit(5), async (e) => {
|
|
500
|
-
const url =
|
|
516
|
+
const url = safeUrl(e.url, indexUrl)?.href;
|
|
517
|
+
if (!url) return { url: e.url, status: 'not a URL', text: '', label: typeof e.name === 'string' && e.name ? e.name : e.url };
|
|
501
518
|
const r = await get(url);
|
|
502
519
|
const fm = r.text.match(/^---\r?\n([\s\S]*?)\r?\n---/)?.[1] ?? '';
|
|
503
520
|
const field = (k) => fm.match(new RegExp(`^${k}:\\s*(?:"([^"]*)"|'([^']*)'|(.*))$`, 'm'))?.slice(1).find((v) => v !== undefined)?.trim() ?? '';
|
|
@@ -700,7 +717,9 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
700
717
|
const id = 'W2', name = 'Canonical URLs', level = 'required';
|
|
701
718
|
const none = ok200.filter((p) => !p.canonicals.length).map((p) => short(p.url));
|
|
702
719
|
const many = ok200.filter((p) => new Set(p.canonicals).size > 1).map((p) => short(p.url));
|
|
703
|
-
const
|
|
720
|
+
const canonical = (p) => safeUrl(p.canonicals[0], p.url)?.href;
|
|
721
|
+
const notUrl = ok200.filter((p) => p.canonicals.length && !canonical(p)).map((p) => short(p.url));
|
|
722
|
+
const elsewhere = ok200.filter((p) => p.canonicals.length && canonical(p) && norm(canonical(p)) !== norm(p.url)).map((p) => `${short(p.url)} → ${short(canonical(p))}`);
|
|
704
723
|
const origin = new URL(BASE).origin;
|
|
705
724
|
// Count only the hops it takes to reach the canonical host; a later path redirect (a language prefix, say) isn't a host hop.
|
|
706
725
|
const hostHops = async (start) => {
|
|
@@ -711,7 +730,9 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
711
730
|
if (!r.status) return n === 0 ? null : { n, end: url };
|
|
712
731
|
const loc = r.headers.get('location');
|
|
713
732
|
if (r.status < 300 || r.status >= 400 || !loc) return { n, end: url };
|
|
714
|
-
|
|
733
|
+
const next = safeUrl(loc, url);
|
|
734
|
+
if (!next) return { n, end: url };
|
|
735
|
+
url = next.href;
|
|
715
736
|
}
|
|
716
737
|
return { n: 5, end: url };
|
|
717
738
|
};
|
|
@@ -726,6 +747,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
726
747
|
const problems = [];
|
|
727
748
|
if (none.length) problems.push(`no canonical on ${list(none)}`);
|
|
728
749
|
if (many.length) problems.push(`conflicting canonicals on ${list(many)}`);
|
|
750
|
+
if (notUrl.length) problems.push(`canonical that isn't a URL on ${list(notUrl)}`);
|
|
729
751
|
if (elsewhere.length) problems.push(`sitemap URLs canonical to another URL: ${list(elsewhere)}`);
|
|
730
752
|
problems.push(...hostProblems);
|
|
731
753
|
if (problems.length) miss(id, name, level, problems.join('; '));
|
|
@@ -734,7 +756,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
734
756
|
// W3 only indexable pages in the sitemap; real 404s
|
|
735
757
|
{
|
|
736
758
|
const id = 'W3', name = 'Indexable pages only', level = 'required';
|
|
737
|
-
const broken = pages.filter((p) => p.status !== 200).map((p) => `${short(p.url)} (${p.status})`);
|
|
759
|
+
const broken = pages.filter((p) => p.status !== 200).map((p) => `${short(p.url)} (${p.status}${p.error ? `, ${p.error}` : ''})`);
|
|
738
760
|
const moved = pages.filter((p) => p.status === 200 && norm(p.finalUrl) !== norm(p.url)).map((p) => `${short(p.url)} → ${short(p.finalUrl)}`);
|
|
739
761
|
const noindex = ok200.filter((p) => /noindex/.test(p.robots)).map((p) => short(p.url));
|
|
740
762
|
// A missing item under each template: a detail route is where a loading boundary or a fallback answers 200.
|
|
@@ -789,8 +811,10 @@ export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency
|
|
|
789
811
|
const images = [...new Set(ok200.map((p) => p.og.image).filter(Boolean))].slice(0, 3);
|
|
790
812
|
const badImages = [];
|
|
791
813
|
for (const img of images) {
|
|
792
|
-
const
|
|
793
|
-
if (!
|
|
814
|
+
const u = safeUrl(img, BASE)?.href;
|
|
815
|
+
if (!u) { badImages.push(`${img.slice(0, 100)} (not a URL)`); continue; }
|
|
816
|
+
const r = await get(u);
|
|
817
|
+
if (!(r.status === 200 && /^image\//.test(r.type))) badImages.push(`${short(u)} (${r.status} ${ct(r)})`);
|
|
794
818
|
}
|
|
795
819
|
const problems = [];
|
|
796
820
|
if (missingOg.length) problems.push(`missing og:title, og:description or og:image on ${list(missingOg)}`);
|
package/src/helpers.mjs
CHANGED
|
@@ -45,6 +45,72 @@ export function list(xs, sep = ', ') {
|
|
|
45
45
|
return `${xs.slice(0, 3).join(sep)}${xs.length > 3 ? `, +${xs.length - 3} more` : ''}`;
|
|
46
46
|
}
|
|
47
47
|
|
|
48
|
+
/** new URL(href, base), or null when it isn't one. Site content can hold anything (href="http://["). */
|
|
49
|
+
export function safeUrl(href, base) {
|
|
50
|
+
try { return new URL(href, base); } catch { return null; }
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
/** A domain as the audit compares hosts: lowercase, with no trailing dot (URL hostnames are both). */
|
|
54
|
+
export const siteDomain = (d) => String(d ?? '').trim().toLowerCase().replace(/\.+$/, '');
|
|
55
|
+
|
|
56
|
+
/**
|
|
57
|
+
* robots.txt as groups: { agents (lowercase), rules: [{ allow, path }], signal }, in file order.
|
|
58
|
+
* A run of User-agent lines starts one group; an empty Disallow allows everything, so it adds no rule.
|
|
59
|
+
*/
|
|
60
|
+
export function robotsGroups(text) {
|
|
61
|
+
const groups = [];
|
|
62
|
+
let cur = null, lastUa = false;
|
|
63
|
+
for (const raw of String(text ?? '').split('\n')) {
|
|
64
|
+
const line = raw.split('#')[0].trim();
|
|
65
|
+
const i = line.indexOf(':');
|
|
66
|
+
if (i < 0) continue;
|
|
67
|
+
const k = line.slice(0, i).trim().toLowerCase();
|
|
68
|
+
const v = line.slice(i + 1).trim();
|
|
69
|
+
if (k === 'user-agent') {
|
|
70
|
+
if (!lastUa) { cur = { agents: [], rules: [], signal: false }; groups.push(cur); }
|
|
71
|
+
cur.agents.push(v.toLowerCase());
|
|
72
|
+
lastUa = true;
|
|
73
|
+
continue;
|
|
74
|
+
}
|
|
75
|
+
lastUa = false;
|
|
76
|
+
if (!cur) continue;
|
|
77
|
+
if ((k === 'disallow' || k === 'allow') && v) cur.rules.push({ allow: k === 'allow', path: v });
|
|
78
|
+
if (k === 'content-signal') cur.signal = true;
|
|
79
|
+
}
|
|
80
|
+
return groups;
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
/** Whether rules let a crawler fetch path (RFC 9309 §2.2.2): the longest match wins, Allow on a tie, * and $ as wildcards. */
|
|
84
|
+
export function robotsAllowed(rules, path) {
|
|
85
|
+
let best = null;
|
|
86
|
+
for (const r of rules) {
|
|
87
|
+
const re = new RegExp(`^${r.path.replace(/[.+?^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*').replace(/\\\$$/, '$')}`); // only a final $ anchors
|
|
88
|
+
if (!re.test(path)) continue;
|
|
89
|
+
if (!best || r.path.length > best.path.length || (r.path.length === best.path.length && r.allow)) best = r;
|
|
90
|
+
}
|
|
91
|
+
return !best || best.allow;
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
/**
|
|
95
|
+
* Where a named crawler's rules let it fetch what * disallows. A crawler obeys only the groups that
|
|
96
|
+
* name it (RFC 9309 §2.2.1, which combines groups for the same agent), so a named group must block
|
|
97
|
+
* everything * blocks. Stricter is fine: Disallow / covers /admin. A rule with wildcards is tested
|
|
98
|
+
* as its literal text (/*.pdf$ as /.pdf).
|
|
99
|
+
*/
|
|
100
|
+
export function robotsLeaks(groups) {
|
|
101
|
+
const rulesFor = (agent) => groups.filter((g) => g.agents.includes(agent)).flatMap((g) => g.rules);
|
|
102
|
+
const star = rulesFor('*');
|
|
103
|
+
if (!groups.some((g) => g.agents.includes('*'))) return [];
|
|
104
|
+
const literal = (p) => p.replace(/\$$/, '').replace(/\*/g, '');
|
|
105
|
+
const leaks = [];
|
|
106
|
+
for (const agent of [...new Set(groups.flatMap((g) => g.agents))].filter((a) => a !== '*')) {
|
|
107
|
+
const own = rulesFor(agent);
|
|
108
|
+
for (const d of star.filter((r) => !r.allow)) if (robotsAllowed(own, literal(d.path))) leaks.push(`${agent} skips Disallow ${d.path}`);
|
|
109
|
+
for (const a of own.filter((r) => r.allow)) if (!robotsAllowed(star, literal(a.path)) && robotsAllowed(own, literal(a.path))) leaks.push(`${agent} reopens ${a.path} with Allow, which * disallows`);
|
|
110
|
+
}
|
|
111
|
+
return [...new Set(leaks)];
|
|
112
|
+
}
|
|
113
|
+
|
|
48
114
|
/** A URL for comparison: lowercase host, no trailing slash, no fragment. The query stays. */
|
|
49
115
|
export function norm(u) {
|
|
50
116
|
try { const x = new URL(u); return `${x.protocol}//${x.host.toLowerCase()}${x.pathname.replace(/\/+$/, '') || '/'}${x.search}`; } catch { return u; }
|
|
@@ -296,6 +362,7 @@ export function llmsLinks(text, llmsUrl, hosts) {
|
|
|
296
362
|
* fromBase() maps them back, so every check keeps seeing the domain's URLs.
|
|
297
363
|
*/
|
|
298
364
|
export function baseMapper(domain, base) {
|
|
365
|
+
domain = siteDomain(domain);
|
|
299
366
|
const hosts = new Set([domain, domain.startsWith('www.') ? domain.slice(4) : `www.${domain}`]);
|
|
300
367
|
const baseOrigin = base ? new URL(base).origin : null;
|
|
301
368
|
const toBase = (url) => {
|
|
@@ -346,7 +413,7 @@ const decode = (s) => { try { return decodeURIComponent(s); } catch { return s;
|
|
|
346
413
|
*/
|
|
347
414
|
export function skillNameProblem({ name, url, entryName }) {
|
|
348
415
|
if (!SKILL_NAME.test(name ?? '')) return 'not a lowercase slug';
|
|
349
|
-
const path =
|
|
416
|
+
const path = safeUrl(url)?.pathname ?? '';
|
|
350
417
|
const folder = /\/SKILL\.md$/i.test(path) ? decode(path.split('/').slice(-2, -1)[0] ?? '') : '';
|
|
351
418
|
if (folder && folder !== name) return `folder is ${folder}`;
|
|
352
419
|
if (typeof entryName === 'string' && entryName && entryName !== name) return `index says ${entryName}`;
|
package/src/http.mjs
CHANGED
|
@@ -19,9 +19,11 @@ export const MAX_BYTES = 16 * 1024 * 1024;
|
|
|
19
19
|
* `accept-encoding: gzip, deflate`; a header set to null isn't sent (M4's no-Accept probe).
|
|
20
20
|
* Resolves, never rejects: { status, headers (a Headers object), text, error?, tooBig? }, status 0
|
|
21
21
|
* on no answer. tooBig: the body passed maxBytes, raw or decoded; reading stopped and the text is empty.
|
|
22
|
+
* lookup: passed to http(s).request, so the caller can refuse the addresses a name resolves to.
|
|
22
23
|
*/
|
|
23
|
-
export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS, maxBytes = MAX_BYTES } = {}) {
|
|
24
|
-
|
|
24
|
+
export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS, maxBytes = MAX_BYTES, lookup } = {}) {
|
|
25
|
+
let target;
|
|
26
|
+
try { target = new URL(url); } catch { return Promise.resolve({ status: 0, headers: new Headers(), text: '', error: `not a URL: ${String(url).slice(0, 100)}` }); }
|
|
25
27
|
const send = target.protocol === 'http:' ? httpRequest : httpsRequest;
|
|
26
28
|
const all = { 'user-agent': userAgent, accept: '*/*', 'accept-encoding': 'gzip, deflate', ...headers };
|
|
27
29
|
if (body !== undefined) all['content-length'] = Buffer.byteLength(body);
|
|
@@ -33,7 +35,12 @@ export function request(url, { method = 'GET', headers = {}, body, userAgent, ti
|
|
|
33
35
|
return new Promise((resolve) => {
|
|
34
36
|
let settled = false;
|
|
35
37
|
const done = (out) => { if (!settled) { settled = true; resolve(out); } };
|
|
36
|
-
|
|
38
|
+
let req;
|
|
39
|
+
try { req = send(target, { method, headers: sent, signal: AbortSignal.timeout(timeoutMs), ...(lookup ? { lookup } : {}) }, onResponse); }
|
|
40
|
+
catch (e) { return done(failed(e)); } // a header value the site supplied that http can't send, say
|
|
41
|
+
req.on('error', (e) => done(failed(e)));
|
|
42
|
+
req.end(body);
|
|
43
|
+
function onResponse(res) {
|
|
37
44
|
const chunks = [];
|
|
38
45
|
let size = 0;
|
|
39
46
|
res.on('data', (c) => {
|
|
@@ -63,8 +70,6 @@ export function request(url, { method = 'GET', headers = {}, body, userAgent, ti
|
|
|
63
70
|
}
|
|
64
71
|
done(out);
|
|
65
72
|
});
|
|
66
|
-
}
|
|
67
|
-
req.on('error', (e) => done(failed(e)));
|
|
68
|
-
req.end(body);
|
|
73
|
+
}
|
|
69
74
|
});
|
|
70
75
|
}
|