readiness-audit 2.3.0 → 2.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/README.md +14 -13
- package/STANDARD.md +11 -4
- package/bin/readiness-audit.mjs +14 -4
- package/package.json +2 -2
- package/src/audit.mjs +25 -11
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,27 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.5.0 (2026-11-03)
|
|
4
|
+
|
|
5
|
+
From a project's agent that was asked why llms-full.txt should hold a site's whole text. D4 said "the whole corpus" without saying what that meant, and the scorecard only checked that the file existed. One new warning, no new failures: a site whose llms-full.txt leaves out pages llms.txt links now warns in D4, which lowers its required score until fixed but doesn't change the exit code.
|
|
6
|
+
|
|
7
|
+
- **D4 warns about pages llms-full.txt doesn't list.** Of the llms.txt links it samples, each one that answers HTML (besides the homepage) must appear as a link in llms-full.txt. Files like `/openapi.json` are surfaces, not pages, and aren't counted. A passing D4 says how many sampled pages llms-full.txt lists.
|
|
8
|
+
- **STANDARD.md:** D4 describes three depths. llms.txt is the map, llms-full.txt is the catalog an agent works from in one fetch (every page, with what it is and the facts an agent needs to choose or use it), and each page's markdown copy has the rest. For an API or docs site, the catalog holds the reference itself; for things people read, watch or listen to, each item's summary, facts and link. The harm on each side is named, with the documented read limits of three agent fetch tools. D4 says llms-full.txt isn't the control for what AI may use (that's D15's `Content-Signal`, or auth), and that llms-full.txt is a convention the llms.txt proposal doesn't define. §16 cites the three tools' limits.
|
|
9
|
+
|
|
10
|
+
From an outside review of 2.4.0:
|
|
11
|
+
|
|
12
|
+
- **The §14 stub links a release tag, not `main`.** Agents read the stub every session and act on it, and a link to `main` gives them whatever was pushed last, before any release, tag or test. The stub says `blob/v[version]/STANDARD.md`, and §14 and docs/adopting.md say to fill in the latest tag and change it after reading the CHANGELOG. The status page that `--matrix` writes links the standard at the tag of the scorecard version that wrote it.
|
|
13
|
+
- **"Last verified" is 2026-10-25,** the newest date a source in §16 was read. A test fails when it's earlier.
|
|
14
|
+
- **The README's sample output is fresh** (animalhouse.ai, 2026-11-04), and a test fails when its version isn't the package's.
|
|
15
|
+
- **"Score a site's readiness",** not "any site's", in the README and package.json, and SECURITY.md says why to audit only sites you own or may test: one run sends dozens to a few hundred requests, several at a time, and POSTs to any MCP endpoint it finds.
|
|
16
|
+
|
|
17
|
+
## 2.4.0 (2026-11-02)
|
|
18
|
+
|
|
19
|
+
From a review by the agents that run the scorecard daily for two projects. No scoring changes, and no new warnings or failures.
|
|
20
|
+
|
|
21
|
+
- **`--concurrency <n>` sets how many requests the audit sends to a site at once,** for a site that slows under parallel requests. Without it, nothing changes: 6 at a time for the discovery files, 4 for sitemaps and pages. The limit now also covers the sampled skill files and the MCP alias probes, which went out together before. At achurch.ai, a sweep of ten requests at a time slowed the site for minutes (2026-10-12); its own checks now send at most two. The library takes it as `audit(domain, { concurrency })`.
|
|
22
|
+
- **README:** documents `--concurrency` in the options and in "Safe to run", and says that a warning on a required check lowers the required score without changing the exit code (a site can show "required 14 of 15" with no FAIL line, because W6 warns). Two links to the research pointed at a placeholder; they now open [Who requests agent discovery files?](docs/research/who-reads-discovery-files.md).
|
|
23
|
+
- **Plans:** the status lines of the sample-by-template, proving-the-score and dependency-updates plans say 2.0.0 is published, and keep what's still open.
|
|
24
|
+
|
|
3
25
|
## 2.3.0 (2026-10-24)
|
|
4
26
|
|
|
5
27
|
From the sister project's agent, after it reviewed this one. No scoring changes.
|
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
[](https://www.npmjs.com/package/readiness-audit)
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
|
|
7
|
-
**Score
|
|
7
|
+
**Score a site's readiness for AI agents and search:** llms.txt, robots.txt rules for AI crawlers, the sitemap, structured data, link previews, OpenAPI, MCP servers, agent skills and the A2A path. A standard built from production request logs, and a command-line audit that checks a live site against it. Read-only, no account, no API key.
|
|
8
8
|
|
|
9
9
|
## Quick start
|
|
10
10
|
|
|
@@ -15,21 +15,20 @@ npx readiness-audit example.com
|
|
|
15
15
|
It needs Node 22.23.4, 24.22.1 or 26.12.1, or later on the same line (each supported line's current security release), and has no dependencies.
|
|
16
16
|
|
|
17
17
|
```
|
|
18
|
-
Agent and search readiness: animalhouse.ai (readiness-audit v2.
|
|
18
|
+
Agent and search readiness: animalhouse.ai (readiness-audit v2.5.0, 2026-11-04T02:41Z)
|
|
19
19
|
|
|
20
20
|
Required
|
|
21
21
|
PASS D1 robots.txt 1 group with consistent rules, Sitemap listed
|
|
22
|
-
PASS D4 llms.txt and llms-full.txt llms.txt 16.2 KB, llms-full.txt 65 KB; 10 sampled links resolve
|
|
23
|
-
PASS D12 No false declarations no OAuth, A2A, plugin or server-card claims without the service
|
|
24
|
-
PASS W1 Titles and descriptions 27 sampled pages (18 static, 3 templates), all with their own
|
|
25
|
-
|
|
26
|
-
PASS M4 Hosted MCP edge both eras, JSON for every Accept, browsers redirected
|
|
22
|
+
PASS D4 llms.txt and llms-full.txt llms.txt 16.2 KB, llms-full.txt 65 KB; 10 sampled links resolve without a redirect; llms-full.txt lists the 3 sampled pages
|
|
23
|
+
PASS D12 No false declarations no OAuth, A2A, plugin or server-card claims without the service behind them
|
|
24
|
+
PASS W1 Titles and descriptions 27 sampled pages (18 static, 3 templates), all with their own title and description
|
|
25
|
+
PASS M4 Hosted MCP edge both eras, JSON for every Accept, browsers redirected, aliases redirect
|
|
27
26
|
...
|
|
28
27
|
Recommended
|
|
29
28
|
PASS D7 JSON 404s and a JSON 405 on POST / JSON 404 for unknown well-known paths, JSON 405 for POST / (Allow: GET, HEAD, OPTIONS)
|
|
30
29
|
WARN D8 Markdown for agents Vary: Accept missing on the HTML response
|
|
31
30
|
...
|
|
32
|
-
Score on 2026-
|
|
31
|
+
Score on 2026-11-04: required 19 of 19, recommended 10 of 11, next level 1 of 2
|
|
33
32
|
```
|
|
34
33
|
|
|
35
34
|
To switch a project over, give its coding agent one line:
|
|
@@ -42,7 +41,7 @@ The agent reads the page when it runs. To have it follow the text you reviewed,
|
|
|
42
41
|
|
|
43
42
|
What agents and crawlers actually ask a site for, from production logs:
|
|
44
43
|
|
|
45
|
-
- **The AI crawlers read robots.txt and the sitemap, not llms.txt.** In a week across 56 domains on Cloudflare, bots in Cloudflare's AI categories requested robots.txt 3,644 times and the sitemap 2,232 times, and llms.txt 49 times. llms.txt still matters to other readers: directories, tools, scripts and search bots requested it on every site ([research](
|
|
44
|
+
- **The AI crawlers read robots.txt and the sitemap, not llms.txt.** In a week across 56 domains on Cloudflare, bots in Cloudflare's AI categories requested robots.txt 3,644 times and the sitemap 2,232 times, and llms.txt 49 times. llms.txt still matters to other readers: directories, tools, scripts and search bots requested it on every site ([research](docs/research/who-reads-discovery-files.md)).
|
|
46
45
|
- **Directories crawl the well-known files; working agents call the API.** At animalhouse.ai, 4 of the 28 addresses that read llms.txt called the API ([§2](STANDARD.md#2-what-the-traffic-says)). Across the 16 sites with an API, 13% of llms.txt readers also got a successful API answer that week.
|
|
47
46
|
- **Bots ask for markdown.** 12,601 requests asked for markdown at ordinary page URLs that week, 92% of them from one search engine's crawler (Exa's).
|
|
48
47
|
- **A hosted MCP endpoint gets found within hours.** Seven MCP directory crawlers connected to animalhouse.ai's in its first two hours, and 14 browser visits reached it in the same two hours ([§2](STANDARD.md#2-what-the-traffic-says)).
|
|
@@ -51,7 +50,7 @@ What agents and crawlers actually ask a site for, from production logs:
|
|
|
51
50
|
## This repo holds
|
|
52
51
|
|
|
53
52
|
- **[STANDARD.md](STANDARD.md):** what to build, item by item, each with an ID, a level and the reason it's at that level. It covers discovery files, search (SEO, and the AI answer engines that GEO and AEO are about), the API, MCP servers, skills, engagement, measurement and tests.
|
|
54
|
-
- **[Who requests agent discovery files?](
|
|
53
|
+
- **[Who requests agent discovery files?](docs/research/who-reads-discovery-files.md):** the week across 56 domains, with charts, its queries and its classifier.
|
|
55
54
|
- **`readiness-audit`:** the scorecard. It checks a live site against everything the standard can verify from outside, about 30 checks in one pass, and lists what it can't see for you to review.
|
|
56
55
|
|
|
57
56
|
## Why this one
|
|
@@ -76,7 +75,7 @@ What agents and crawlers actually ask a site for, from production logs:
|
|
|
76
75
|
## Score a site
|
|
77
76
|
|
|
78
77
|
```
|
|
79
|
-
readiness-audit <domain> [--mcp /path] [--no-mcp] [--json] [--compare previous] [--base url]
|
|
78
|
+
readiness-audit <domain> [--mcp /path] [--no-mcp] [--json] [--compare previous] [--base url] [--concurrency n]
|
|
80
79
|
readiness-audit --matrix <domain> <domain> ... [--recorded results.json]
|
|
81
80
|
readiness-audit --logs <access-log> [--json]
|
|
82
81
|
```
|
|
@@ -88,17 +87,19 @@ readiness-audit --logs <access-log> [--json]
|
|
|
88
87
|
| `--json` | print the results as JSON |
|
|
89
88
|
| `--compare file` | list what changed since a previous run: a status page from `--matrix` or a saved `--json` report |
|
|
90
89
|
| `--base url` | score a local or preview build as the domain before you deploy (`--base http://localhost:3000`); DNS and host redirects are skipped |
|
|
90
|
+
| `--concurrency n` | send at most n requests to the site at once (default 6 for the discovery files, 4 for sitemaps and pages). Lower it for a site that slows under parallel requests; the checks are the same |
|
|
91
91
|
| `--matrix` | score several sites and print a markdown status page |
|
|
92
92
|
| `--recorded file` | with `--matrix`, add your recorded T5 and T6 results (format: [examples/recorded.json](examples/recorded.json)) |
|
|
93
93
|
| `--logs file` | count requests for the discovery files in an exported access log (combined log format, or JSON lines from Cloudflare, Vercel or Railway), by client family. It prints counts, never addresses |
|
|
94
94
|
|
|
95
|
-
It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy. The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
|
|
95
|
+
It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy. A warning on a required check, such as W6's heading jumps, lowers the required score ("required 14 of 15") without changing the exit code; only a failure does. The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
|
|
96
96
|
|
|
97
97
|
To check fixes before they ship, score a local build: `npx readiness-audit@2 example.com --base http://localhost:3000`. The site's own URLs (from its sitemap, llms.txt, skills and catalog) are requested from the base and reported under the domain. From a clone, run `node bin/readiness-audit.mjs example.com`.
|
|
98
98
|
|
|
99
99
|
### Safe to run
|
|
100
100
|
|
|
101
|
-
- **It only reads:** GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`),
|
|
101
|
+
- **It only reads:** GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`), at most six at a time. With `--matrix`, each site gets its own six.
|
|
102
|
+
- **It can go slower.** A site that slows under parallel requests can be audited with `--concurrency 2`, or `--concurrency 1` for one request at a time. The same checks run, and take longer.
|
|
102
103
|
- **It stays on public addresses.** It won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect. `--base` may be local, because you chose it.
|
|
103
104
|
- **A hostile site can't reach a decoder bug.** It asks for gzip and decodes it itself; a body in an encoding it didn't ask for, such as Brotli, is reported rather than read, and every body is capped at 16 MB.
|
|
104
105
|
- **Run it only against sites you run,** or with the owner's permission. [SECURITY.md](SECURITY.md) has the details and how to report a problem.
|
package/STANDARD.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**What it is:** a common standard for making a site findable and usable by AI agents and search engines, built from six projects' own playbooks (animalhouse.ai, inbed.ai, magnifica.family, drifts.bot, achurch.ai, botsmatter.live), checked against the specs ([§16](#16-specs-and-sources)), scored against each live site, and revised after each project's agent reviewed it.
|
|
4
4
|
**Scorecard:** `npx readiness-audit <domain>` (see the [README](README.md))
|
|
5
|
-
**Last verified:** 2026-10-
|
|
5
|
+
**Last verified:** 2026-10-25
|
|
6
6
|
|
|
7
7
|
> **Start here**
|
|
8
8
|
> 1. Score your site: `npx readiness-audit <domain>`. It needs Node 22.23.4, 24.22.1 or 26.12.1, or later on the same line, and no dependencies.
|
|
@@ -134,7 +134,10 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
134
134
|
|
|
135
135
|
**D3 Server-rendered pages.** Every page's content is in the HTML the server sends, not loaded later by script, with a real logo through the framework's icon convention and a favicon set (missing icons show up as steady 404s). *Why: crawlers, link-preview bots and agents that fetch pages read the HTML, and many don't run JavaScript.* inbed's chat pages once shipped none of their 48 messages in the HTML. The page-level details (titles, canonicals, structured data, previews, headings, links) are W1 to W7 in [§5](#5-search-people-and-answer-engines).
|
|
136
136
|
|
|
137
|
-
**D4 `/llms.txt` and `/llms-full.txt`**:
|
|
137
|
+
**D4 `/llms.txt` and `/llms-full.txt`**: three depths. llms.txt is the map: what this is, how to start, links to every other surface. llms-full.txt is the catalog an agent works from in one fetch: every page llms.txt links, with its URL, what it is, and the facts an agent needs to choose it or use it. Each page's markdown copy (D8) has the rest. Generate both files and test them against the sitemap: agents follow the links, so every one must open. The scorecard samples llms.txt's links, warns about missing pages (an endpoint that answers 401 for want of a key is fine; templates like `{id}` are skipped), and warns about sampled pages llms-full.txt doesn't list. Both files are markdown; serve them as `text/markdown` or `text/plain`, and use the same type wherever you link them.
|
|
138
|
+
- **Put in what agents act on.** For an API or docs site, that's the reference itself: an agent writes a call from the parameters, not from a summary of them. For things people read, watch or listen to (articles, episodes, recordings), it's each item's summary, its facts (length, date, credits) and its link, not the body. A test: from this file alone, can an agent answer your site's common questions and pick the right link? *Harm prevented:* a file as thin as llms.txt sends the agent to open every page, and a file of every page's full text gets read in part. Agent fetch tools return the first part of a long file: 5,000 characters by default in the MCP reference fetch server, 20,000 in OpenClaw's `web_fetch`, 100,000 in Claude Code's WebFetch. Some tell the agent where to resume, but each further part is another fetch.
|
|
139
|
+
- **It isn't the control for what AI may use.** Everything in llms-full.txt is already on your pages. To say what AI may do with your content, use D15's `Content-Signal` (`ai-train=no` keeps search and AI answers while declining training), or keep the content behind auth. Leaving a page's body out of llms-full.txt doesn't keep it from crawlers.
|
|
140
|
+
- llms-full.txt is a convention: the llms.txt proposal defines only llms.txt and per-page markdown.
|
|
138
141
|
- *Why required: agents doing tasks use it, though the AI crawlers mostly don't.* Across 56 domains in a week of October 2026, bots in Cloudflare's legacy AI categories made 49 of 1,406 llms.txt requests (GPTBot 5, ClaudeBot none in the sample) and requested robots.txt and the sitemap instead. On the 16 sites with an API, 13% of the addresses that requested llms.txt also got a successful API answer that week, 9% with a user agent that wasn't a browser's ([research](docs/research/who-reads-discovery-files.md)). In a controlled benchmark across 20 documentation sites, markdown with a linked llms.txt cut agents' wasted fetches from 2.23 per task with HTML to 0.11 ([Mintlify](https://www.mintlify.com/blog/llms-txt-agent-benchmark), 2026-07-17). Google Search ignores llms.txt (it said so in June 2026). At animalhouse, 79 requests in four and a half days came from:
|
|
139
142
|
- agent directories (AgenstryBot, BrickBlueBot, agentprobe, MCPHarbor, AgentTrustBot);
|
|
140
143
|
- llms.txt directories and validators;
|
|
@@ -799,7 +802,7 @@ Step-by-step instructions for a project's coding agent: [docs/adopting.md](docs/
|
|
|
799
802
|
# Agent and Search Readiness
|
|
800
803
|
|
|
801
804
|
This project follows the Agent and Search Readiness Standard:
|
|
802
|
-
https://github.com/geeks-accelerator/agent-and-search-readiness/blob/
|
|
805
|
+
https://github.com/geeks-accelerator/agent-and-search-readiness/blob/v[version]/STANDARD.md
|
|
803
806
|
|
|
804
807
|
Score this site: `npx readiness-audit@2 <domain>`. Current status: docs/readiness-status.md
|
|
805
808
|
|
|
@@ -809,6 +812,7 @@ Step-by-step instructions for a project's coding agent: [docs/adopting.md](docs/
|
|
|
809
812
|
Project-specific notes:
|
|
810
813
|
- (anything the standard doesn't cover)
|
|
811
814
|
```
|
|
815
|
+
- **Pin the standard to a release tag.** Put the latest tag from the [releases page](https://github.com/geeks-accelerator/agent-and-search-readiness/releases) in place of `v[version]`. Your agents read the stub every session and act on what it links. A link to `main` gives them whatever was pushed last, before any release, tag or test; a tag gives them the text you reviewed. To take a newer standard, read the CHANGELOG entries since your tag, then change the tag.
|
|
812
816
|
- **Pin the scorecard to a major version** (`npx readiness-audit@2 <domain>`), so a new release can't fail your build without warning.
|
|
813
817
|
- **Run it in CI** after each deploy or on a schedule, and fail on a required regression. `examples/github-action.yml` is a starting point.
|
|
814
818
|
- **Record declined items** (principle 9) in your CLAUDE.md or AGENTS.md, with the reason.
|
|
@@ -825,7 +829,7 @@ Level: **R** required where it applies, **Rec** recommended, **N** next level. W
|
|
|
825
829
|
| D1 | robots.txt: the same rules for every crawler, Sitemap, side effects disallowed | R | agents, honesty | score |
|
|
826
830
|
| D2 | sitemap.xml with real lastmod dates | R | traffic | score |
|
|
827
831
|
| D3 | Server-rendered pages, real logo and favicons | R | agents, discovery | review |
|
|
828
|
-
| D4 | llms.txt and llms-full.txt, generated, linked from skills and README | R | traffic | score, test |
|
|
832
|
+
| D4 | llms.txt and llms-full.txt, generated, linked from skills and README; llms-full.txt lists every page llms.txt links | R | traffic | score, test |
|
|
829
833
|
| D5 | `Link` headers: service-desc, service-doc, describedby, api-catalog | Rec | discovery | score |
|
|
830
834
|
| D6 | security.txt with Expires once | R | hygiene | score |
|
|
831
835
|
| D7 | JSON 404s for unknown well-known paths (`agent.json` included), JSON 405 with `Allow` on `POST /` | Rec | agents | score |
|
|
@@ -954,6 +958,9 @@ Status as of 2026-10-05. Several are drafts, so re-check before relying on a det
|
|
|
954
958
|
| Discovery-file requests ("Who requests agent discovery files?") | this repo's research, 56 domains on Cloudflare, 2026-10-17 to 10-24, corrected 2026-10-24 | docs/research/who-reads-discovery-files.md |
|
|
955
959
|
| Negotiated markdown requests | GoodBarber's own logs, 2026-09-04 to 09-18, read 2026-10-24 | https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/ |
|
|
956
960
|
| Markdown and llms.txt in agent tasks | Mintlify benchmark, 2026-07-17, read 2026-10-24 | https://www.mintlify.com/blog/llms-txt-agent-benchmark |
|
|
961
|
+
| MCP reference fetch server's `max_length` (default 5,000 characters) | source, read 2026-10-25 | https://github.com/modelcontextprotocol/servers/blob/main/src/fetch/src/mcp_server_fetch/server.py |
|
|
962
|
+
| OpenClaw `web_fetch` limits (`maxChars` default 20,000) | docs and source, read 2026-10-25 | https://docs.openclaw.ai/tools/web-fetch |
|
|
963
|
+
| Claude Code WebFetch (100,000 characters per call) | Claude Code docs, read 2026-10-25 | https://code.claude.com/docs/en/tools-reference |
|
|
957
964
|
| Crawler impersonation ("Bots in AI clothing") | agent-security-pranks research, 56 domains on Cloudflare, 2026-10-16 to 10-23, read 2026-10-24 | https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md |
|
|
958
965
|
| 405 and `Allow` | RFC 9110 §15.5.6 | https://www.rfc-editor.org/rfc/rfc9110.html#section-15.5.6 |
|
|
959
966
|
| Headings | W3C WAI tutorial, read 2026-10-21 | https://www.w3.org/WAI/tutorials/page-structure/headings/ |
|
package/bin/readiness-audit.mjs
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
// readiness-audit: score a live site against the Agent and Search Readiness Standard.
|
|
3
3
|
//
|
|
4
|
-
// readiness-audit <domain> [--mcp /path] [--no-mcp] [--json] [--compare previous] [--base url]
|
|
4
|
+
// readiness-audit <domain> [--mcp /path] [--no-mcp] [--json] [--compare previous] [--base url] [--concurrency n]
|
|
5
5
|
// readiness-audit --matrix <domain> <domain> ... [--recorded results.json] (a markdown status page)
|
|
6
6
|
// readiness-audit --logs <access-log> [--json] (who requests the discovery files)
|
|
7
7
|
//
|
|
@@ -31,9 +31,15 @@ const recordedPath = valueOf('--recorded');
|
|
|
31
31
|
const comparePath = valueOf('--compare');
|
|
32
32
|
const base = valueOf('--base');
|
|
33
33
|
const logsPath = valueOf('--logs');
|
|
34
|
-
const
|
|
34
|
+
const concurrencyOpt = valueOf('--concurrency');
|
|
35
|
+
const domains = args.filter((a, i) => !a.startsWith('-') && !['--mcp', '--recorded', '--compare', '--base', '--logs', '--concurrency'].includes(args[i - 1]))
|
|
35
36
|
.map((d) => d.replace(/^https?:\/\//, '').replace(/\/.*$/, ''));
|
|
36
|
-
const opts = { mcpPath: flag('--no-mcp') ? null : (mcpOpt ?? '/mcp'), base: base ?? null };
|
|
37
|
+
const opts = { mcpPath: flag('--no-mcp') ? null : (mcpOpt ?? '/mcp'), base: base ?? null, concurrency: null };
|
|
38
|
+
if (flag('--concurrency')) {
|
|
39
|
+
const n = Number(concurrencyOpt);
|
|
40
|
+
if (!Number.isInteger(n) || n < 1) { console.error(`--concurrency needs a whole number of 1 or more, like --concurrency 2 (got ${concurrencyOpt})`); process.exit(2); }
|
|
41
|
+
opts.concurrency = n;
|
|
42
|
+
}
|
|
37
43
|
if (base) {
|
|
38
44
|
let u;
|
|
39
45
|
try { u = new URL(base); } catch { u = null; }
|
|
@@ -54,7 +60,7 @@ if (!domains.length || flag('--help') || flag('-h')) {
|
|
|
54
60
|
console.error(`readiness-audit ${VERSION}
|
|
55
61
|
|
|
56
62
|
Usage:
|
|
57
|
-
readiness-audit <domain> [--mcp /path] [--no-mcp] [--json] [--compare previous] [--base url]
|
|
63
|
+
readiness-audit <domain> [--mcp /path] [--no-mcp] [--json] [--compare previous] [--base url] [--concurrency n]
|
|
58
64
|
readiness-audit --matrix <domain> <domain> ... [--recorded results.json]
|
|
59
65
|
readiness-audit --logs <access-log> [--json]
|
|
60
66
|
|
|
@@ -69,6 +75,10 @@ Options:
|
|
|
69
75
|
--base url score a local or preview build as the domain, before you
|
|
70
76
|
deploy (http://localhost:3000); DNS and host redirects are
|
|
71
77
|
skipped
|
|
78
|
+
--concurrency n
|
|
79
|
+
at most n requests to the site at once (default 6 for
|
|
80
|
+
the discovery files, 4 for sitemaps and pages); lower it
|
|
81
|
+
for a site that slows under parallel requests
|
|
72
82
|
--logs f count requests for the discovery files in an exported access
|
|
73
83
|
log (combined log format or JSON lines), by client family;
|
|
74
84
|
prints counts, never addresses
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "readiness-audit",
|
|
3
|
-
"version": "2.
|
|
4
|
-
"description": "Score
|
|
3
|
+
"version": "2.5.0",
|
|
4
|
+
"description": "Score a site's readiness for AI agents and search (GEO, AEO, AI SEO): llms.txt, robots.txt AI crawler rules, structured data, link previews, OpenAPI, MCP, agent skills and A2A. Read-only CLI, no account.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"readiness-audit": "bin/readiness-audit.mjs"
|
package/src/audit.mjs
CHANGED
|
@@ -11,7 +11,7 @@ import { isIP } from 'node:net';
|
|
|
11
11
|
import { request } from './http.mjs';
|
|
12
12
|
import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
|
|
13
13
|
|
|
14
|
-
export const VERSION = '2.
|
|
14
|
+
export const VERSION = '2.5.0';
|
|
15
15
|
export const REPO = 'https://github.com/geeks-accelerator/agent-and-search-readiness';
|
|
16
16
|
const UA = `readiness-audit/${VERSION} (+${REPO})`;
|
|
17
17
|
const AID_PROTOCOLS = ['mcp', 'a2a', 'openapi', 'grpc', 'graphql', 'websocket', 'local', 'zeroconf', 'ucp'];
|
|
@@ -105,8 +105,12 @@ export function makeGuard(domain) {
|
|
|
105
105
|
return { siteHosts, isSafe };
|
|
106
106
|
}
|
|
107
107
|
|
|
108
|
-
export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
108
|
+
export async function audit(domain, { mcpPath = '/mcp', base = null, concurrency = null } = {}) {
|
|
109
109
|
const BASE = `https://${domain}`;
|
|
110
|
+
// Requests in flight to the site at once. By default, 6 for the discovery files and 4 for
|
|
111
|
+
// sitemaps and pages; `concurrency` sets one limit for all of them, for a site that slows
|
|
112
|
+
// under parallel requests.
|
|
113
|
+
const limit = (byDefault) => concurrency ?? byDefault;
|
|
110
114
|
const rand = Math.random().toString(36).slice(2, 10);
|
|
111
115
|
|
|
112
116
|
const { siteHosts, isSafe } = makeGuard(domain);
|
|
@@ -178,7 +182,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
178
182
|
() => get('/', { method: 'POST', redirect: 'manual' }),
|
|
179
183
|
() => get(D.legacyCard),
|
|
180
184
|
() => get(D.wkCard),
|
|
181
|
-
], 6, (job) => job());
|
|
185
|
+
], limit(6), (job) => job());
|
|
182
186
|
const catalogEntries = [...(json(ard)?.entries ?? []), ...(json(aiCatalog)?.entries ?? [])];
|
|
183
187
|
const rels = linkRels(home.headers.get('link'));
|
|
184
188
|
const api = apiPresence({ openapiSpec: json(openapi), apiIndexJson: ok(apiIndex) && isJson(apiIndex), rels, catalogJson: ok(apiCatalog) ? json(apiCatalog) : null, llmsText: ok(llms) ? llms.text : '' });
|
|
@@ -231,7 +235,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
231
235
|
// ---- The sitemap, read once for D2 and the W checks: an index is expanded to its pages ----
|
|
232
236
|
const sitemapIndex = /<sitemapindex/.test(sitemap.text);
|
|
233
237
|
const sitemapLists = sitemapIndex
|
|
234
|
-
? await mapLimit(sitemapEntries(sitemap.text).slice(0, 10), 4, async (e) => sitemapEntries((await get(e.loc)).text))
|
|
238
|
+
? await mapLimit(sitemapEntries(sitemap.text).slice(0, 10), limit(4), async (e) => sitemapEntries((await get(e.loc)).text))
|
|
235
239
|
: [sitemapEntries(sitemap.text)];
|
|
236
240
|
const seenLocs = new Set();
|
|
237
241
|
const entries = sitemapLists.flat().filter((e) => {
|
|
@@ -258,7 +262,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
258
262
|
// Agents follow the links in llms.txt, so a sample of the site's own links must answer 200.
|
|
259
263
|
const links = ok(llms) ? llmsLinks(llms.text, `${BASE}/llms.txt`, siteHosts) : [];
|
|
260
264
|
const sample = spread(links, 10);
|
|
261
|
-
const answers = await mapLimit(sample, 4, async (u) => ({ u, r: await get(u) }));
|
|
265
|
+
const answers = await mapLimit(sample, limit(4), async (u) => ({ u, r: await get(u) }));
|
|
262
266
|
llmsAnswers = answers;
|
|
263
267
|
// Missing pages, server errors and no answer are broken. A 401, 403 or 405 is an endpoint that
|
|
264
268
|
// exists but wants a key or another method, which is fine to list.
|
|
@@ -268,12 +272,22 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
268
272
|
const linkProblems = [];
|
|
269
273
|
if (broken.length) linkProblems.push(`broken links in llms.txt: ${list(broken)} (${broken.length} of ${sample.length} sampled)`);
|
|
270
274
|
if (redirected.length) linkProblems.push(`llms.txt links that redirect (link the destination): ${list(redirected)}`);
|
|
275
|
+
// llms-full.txt is the catalog: every page llms.txt links appears in it, so an agent can choose from one fetch.
|
|
276
|
+
// Pages are the sampled links that answer HTML, besides the homepage; files like /openapi.json are surfaces, not pages.
|
|
277
|
+
let pagesListed = 0;
|
|
278
|
+
if (ok(llmsFull) && !llmsFull.tooBig) {
|
|
279
|
+
const covered = new Set(llmsLinks(llmsFull.text, `${BASE}/llms-full.txt`, siteHosts).map(norm));
|
|
280
|
+
const pagesLinked = answers.filter(({ u, r }) => ok(r) && /text\/html/.test(r.type) && norm(r.url) === norm(u) && new URL(u).pathname !== '/');
|
|
281
|
+
const uncovered = pagesLinked.filter(({ u }) => !covered.has(norm(u))).map(({ u }) => u.replace(BASE, '') || '/');
|
|
282
|
+
if (uncovered.length) linkProblems.push(`pages llms.txt links that llms-full.txt doesn't list: ${list(uncovered)} (${uncovered.length} of ${pagesLinked.length} sampled pages)`);
|
|
283
|
+
else pagesListed = pagesLinked.length;
|
|
284
|
+
}
|
|
271
285
|
const kb = (r, digits) => (r.tooBig ? r.error.replace(/^.*?over /, 'over ') : `${(r.text.length / 1024).toFixed(digits)} KB`);
|
|
272
286
|
const sizes = `llms.txt ${kb(llms, 1)}, llms-full.txt ${kb(llmsFull, 0)}`;
|
|
273
287
|
if (!ok(llms) || !/^\s*# \S/.test(llms.text)) miss(id, name, level, (ok(llms) ? 'llms.txt has no H1 title on its first line' : `/llms.txt answered ${llms.status}`) + cached(llms));
|
|
274
288
|
else if (!ok(llmsFull)) miss(id, name, level, `/llms-full.txt missing${cached(llmsFull)}`);
|
|
275
289
|
else if (linkProblems.length) record(id, name, level, 'warn', `${linkProblems.join('; ')}${cached(llms)}`);
|
|
276
|
-
else pass(id, name, level, `${sizes}${sample.length ? `; ${sample.length} sampled link${sample.length === 1 ? '' : 's'} resolve without a redirect` : ''}`);
|
|
290
|
+
else pass(id, name, level, `${sizes}${sample.length ? `; ${sample.length} sampled link${sample.length === 1 ? '' : 's'} resolve without a redirect` : ''}${pagesListed ? `; llms-full.txt lists the ${pagesListed} sampled page${pagesListed === 1 ? '' : 's'}` : ''}`);
|
|
277
291
|
}
|
|
278
292
|
|
|
279
293
|
// ---- D5 Link headers (recommended; a wrong service-desc is a false declaration) ----
|
|
@@ -481,14 +495,14 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
481
495
|
} else {
|
|
482
496
|
// Entry URLs may be relative, and resolve against the index (RFC 3986).
|
|
483
497
|
const indexUrl = `${BASE}/.well-known/agent-skills/index.json`;
|
|
484
|
-
const files = await
|
|
498
|
+
const files = await mapLimit(sample, limit(5), async (e) => {
|
|
485
499
|
const url = new URL(e.url, indexUrl).href;
|
|
486
500
|
const r = await get(url);
|
|
487
501
|
const fm = r.text.match(/^---\r?\n([\s\S]*?)\r?\n---/)?.[1] ?? '';
|
|
488
502
|
const field = (k) => fm.match(new RegExp(`^${k}:\\s*(?:"([^"]*)"|'([^']*)'|(.*))$`, 'm'))?.slice(1).find((v) => v !== undefined)?.trim() ?? '';
|
|
489
503
|
const label = typeof e.name === 'string' && e.name ? e.name : url;
|
|
490
504
|
return { url, status: r.status, text: r.text, name: field('name'), description: field('description'), entryName: e.name, label, lines: r.text.split('\n').length };
|
|
491
|
-
})
|
|
505
|
+
});
|
|
492
506
|
const unreachable = files.filter((f) => f.status !== 200).map((f) => `${f.label} (${f.status})`);
|
|
493
507
|
const read = files.filter((f) => f.status === 200);
|
|
494
508
|
const badName = read.map((f) => [f, skillNameProblem(f)]).filter(([, why]) => why).map(([f, why]) => `${f.label} (${why})`);
|
|
@@ -553,7 +567,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
553
567
|
variants.push(['(none)', await get(mcpPath, { method: 'POST', headers: { 'content-type': 'application/json', accept: null }, body: INIT })]);
|
|
554
568
|
const discover = await modern('server/discover', 2);
|
|
555
569
|
const browser = await get(mcpPath, { headers: { accept: 'text/html' }, redirect: 'manual' });
|
|
556
|
-
const aliases = await
|
|
570
|
+
const aliases = await mapLimit(['/sse', '/api/mcp'], limit(2), (p) => get(p, { method: 'POST', redirect: 'manual', headers: { 'content-type': 'application/json' }, body: INIT }));
|
|
557
571
|
const problems = [];
|
|
558
572
|
const refused = variants.filter(([, r]) => r.status !== 200).map(([a, r]) => `${a} → ${r.status}`);
|
|
559
573
|
if (refused.length) problems.push(`2025-era initialize refused for Accept ${refused.join(', ')}`);
|
|
@@ -651,7 +665,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
651
665
|
ldBroken: ld.filter((x) => x === null).length, types,
|
|
652
666
|
};
|
|
653
667
|
};
|
|
654
|
-
const pages = await mapLimit(pick, 4, async (u) => parsePage(u, await get(u, { headers: { accept: 'text/html' } })));
|
|
668
|
+
const pages = await mapLimit(pick, limit(4), async (u) => parsePage(u, await get(u, { headers: { accept: 'text/html' } })));
|
|
655
669
|
// A body the audit refused (an encoding it doesn't decode) is reported once, in W3, not as missing tags everywhere.
|
|
656
670
|
const ok200 = pages.filter((p) => p.status === 200 && p.html && !p.error);
|
|
657
671
|
if (ok200.length) { const sizes = ok200.map((p) => p.bytes).sort((a, b) => a - b); htmlKb = Math.round(sizes[Math.floor(sizes.length / 2)] / 1024); }
|
|
@@ -884,7 +898,7 @@ export function matrix(reports, command, recorded = {}) {
|
|
|
884
898
|
const ids = reports[0].results.map((r) => [r.id, r.name, r.level]);
|
|
885
899
|
const out = [
|
|
886
900
|
'# Agent and Search Readiness Status', '',
|
|
887
|
-
`Generated ${new Date().toISOString().slice(0, 10)} by \`${command}\` (readiness-audit v${VERSION}). Don't edit it by hand: regenerate it. What each ID means: [the standard](${REPO}/blob/
|
|
901
|
+
`Generated ${new Date().toISOString().slice(0, 10)} by \`${command}\` (readiness-audit v${VERSION}). Don't edit it by hand: regenerate it. What each ID means: [the standard](${REPO}/blob/v${VERSION}/STANDARD.md).`, '',
|
|
888
902
|
'The scorecard sees only what is visible from outside a site. Items checked in review and tests (listed at the end) aren\'t in these numbers, so a high score can hide real gaps.', '',
|
|
889
903
|
'✓ pass, ✗ fail, ! warning, – not applicable.', '',
|
|
890
904
|
`| ID | Check | Level | ${domains.join(' | ')} |`, `|---|---|---|${domains.map(() => '---').join('|')}|`,
|