readiness-audit 2.0.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/README.md +46 -8
- package/STANDARD.md +31 -19
- package/bin/readiness-audit.mjs +4 -1
- package/package.json +14 -3
- package/src/audit.mjs +22 -8
- package/src/helpers.mjs +38 -0
- package/src/http.mjs +33 -10
- package/src/logs.mjs +20 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,27 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.2.0 (2026-10-24)
|
|
4
|
+
|
|
5
|
+
From a review of the sister project, [agent-security-pranks](https://github.com/geeks-accelerator/agent-security-pranks), and its research. Two new warnings (D7's `Allow`, and D8 at the recommended level), no new failures. A warning lowers the recommended score until fixed, but doesn't change the exit code.
|
|
6
|
+
|
|
7
|
+
- **D7 warns about a 405 with no `Allow` header.** RFC 9110 requires it on every 405, and it tells a client what does work. All six sites the standard came from already send it.
|
|
8
|
+
- **Response bodies are capped at 16 MB,** before and after decoding, so a site that answers with a gzip bomb or an endless stream can't exhaust the memory of the machine running the audit. Over the cap, the check sees the status and no body, as with a refused encoding, and D4 reports an oversized llms-full.txt by its size.
|
|
9
|
+
- **`--logs` counts requests under crawler names that asked for secret files** (`.env`, `.git`, keys, dumps), per family, and says the families are the names clients send, not verified. The JSON report has `crawler_names_on_secret_probes`.
|
|
10
|
+
- **D8 (markdown for agents) is recommended,** up from next level, so a site without it now warns at the recommended level and its recommended score can drop. The evidence: in a week across 56 domains, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler.
|
|
11
|
+
- **`--logs` names other audit tools** (Semrush, Ahrefs, agentprobe, agent-tools.cloud, Lumar, SEOJuice and the sister project's audit) as their own family.
|
|
12
|
+
- **New research: [Who reads the discovery files](docs/research/who-reads-discovery-files.md),** a week across 56 domains on Cloudflare: who requested each discovery file, verified or not, which files sites are asked for without serving them, and how many readers went on to call the API. STANDARD.md §2 cites it; D4's reason now names who reads llms.txt; D7 and D11 cover the pre-1.0 path `/.well-known/agent.json`; T3 counts audits apart from agents.
|
|
13
|
+
- **STANDARD.md:** §2 adds that a crawler's name is a claim: in a week across one owner's 56 domains, none of about 448k vulnerability probes came from a verified bot, and one scanner used about 20 AI-crawler names. T3 counts a crawler by its verified status. D7 asks for `Allow`. §16 cites the research and RFC 9110 §15.5.6. docs/recipes.md shows `Allow` on the 405.
|
|
14
|
+
- **README:** a row for the sister project in "How it fits with other tools".
|
|
15
|
+
- **README:** opens with what the audit checks, in the words people search for, a real run's output, "Why this one", and how it fits with other tools. Badges for the npm version and license.
|
|
16
|
+
- **package.json:** the description leads with what the audit scores; keywords add aeo, ai-seo, ai-visibility, generative-engine-optimization, robots-txt, ai-crawlers, seo-audit, website-audit, agent-ready, llms-txt-validator and cli. Both reach npm with the next release.
|
|
17
|
+
|
|
18
|
+
## 2.1.0 (2026-10-23)
|
|
19
|
+
|
|
20
|
+
From a project's agent that checked pages in Google's Rich Results Test by hand. The standard asked for it ("accepts a sample") without saying which pages, and the scorecard never mentioned it. No scoring changes.
|
|
21
|
+
|
|
22
|
+
- **The scorecard lists the pages to run through Google's Rich Results Test,** each with a link that opens it in the test: the homepage, one page of each template with rich-result markup, and any other page that adds a rich-result feature. Pages from one template share their markup, so one tests them all. The CLI prints them under W4 in "check these yourself", the JSON report has `rich_results_test`, and the status page has a section for them. The rich-result types are those in Google's search gallery (updated 2026-10-16).
|
|
23
|
+
- **STANDARD.md:** W4 says which pages to test and when (after a change to a template's markup), when to use the Schema Markup Validator instead, and that valid means eligible, not shown. Its done-when is no errors on the listed pages. T6's monthly review checks Search Console's rich result reports, which cover every indexed page. §16 cites the search gallery, both testing tools, the reports and the URL Inspection API.
|
|
24
|
+
|
|
3
25
|
## 2.0.0 (2026-10-21)
|
|
4
26
|
|
|
5
27
|
From an adopting agent's review: a full score showed a site was set up correctly, not that agents did better on it, and the default run graded content sites on an API they don't have. This release makes the score fit the site, checks more of what agents and link previews actually read, and gives T5 and the traffic numbers a recorded form. Pin `readiness-audit@2`.
|
package/README.md
CHANGED
|
@@ -1,20 +1,56 @@
|
|
|
1
1
|
# Agent and Search Readiness
|
|
2
2
|
|
|
3
3
|
[](https://github.com/geeks-accelerator/agent-and-search-readiness/actions/workflows/ci.yml)
|
|
4
|
+
[](https://www.npmjs.com/package/readiness-audit)
|
|
5
|
+
[](LICENSE)
|
|
4
6
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
AI agents find a site through discovery files, MCP registries, skills and search. People find it through Google, Bing and the AI answer engines built on them. Both then need the first call to work, and a reason to come back. This repo holds:
|
|
8
|
-
|
|
9
|
-
- **[STANDARD.md](STANDARD.md):** what to build, item by item, each with an ID, a level and the reason it's at that level. It covers discovery files, search, the API, MCP servers, skills, engagement, measurement and tests.
|
|
10
|
-
- **`readiness-audit`:** a command-line scorecard that checks a live site against everything the standard can verify from outside, about 30 checks in one pass.
|
|
11
|
-
|
|
12
|
-
## Score a site
|
|
7
|
+
**Score any site's readiness for AI agents and search:** llms.txt, robots.txt rules for AI crawlers, the sitemap, structured data, link previews, OpenAPI, MCP servers, agent skills and the A2A path. A standard built from production request logs, and a command-line audit that checks a live site against it. Read-only, no account, no API key.
|
|
13
8
|
|
|
14
9
|
```
|
|
15
10
|
npx readiness-audit example.com
|
|
16
11
|
```
|
|
17
12
|
|
|
13
|
+
```
|
|
14
|
+
Agent and search readiness: animalhouse.ai (readiness-audit v2.1.0)
|
|
15
|
+
|
|
16
|
+
Required
|
|
17
|
+
PASS D1 robots.txt 1 group with consistent rules, Sitemap listed
|
|
18
|
+
PASS D4 llms.txt and llms-full.txt llms.txt 16.2 KB, llms-full.txt 65 KB; 10 sampled links resolve
|
|
19
|
+
PASS D12 No false declarations no OAuth, A2A, plugin or server-card claims without the service
|
|
20
|
+
PASS W1 Titles and descriptions 27 sampled pages (18 static, 3 templates), all with their own
|
|
21
|
+
WARN W5 Link previews og:image size or alt text not declared on /hall, /about, +11 more
|
|
22
|
+
WARN W6 Page structure heading levels skipped on /agent/[name] (h1 to h3)
|
|
23
|
+
PASS A1 OpenAPI 3.1 OpenAPI 3.1.0; all 17 operations and their request fields described
|
|
24
|
+
PASS M4 Hosted MCP edge both eras, JSON for every Accept, browsers redirected
|
|
25
|
+
...
|
|
26
|
+
Score on 2026-10-23: required 17 of 19, recommended 10 of 10, next level 1 of 3
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This repo holds:
|
|
30
|
+
|
|
31
|
+
- **[STANDARD.md](STANDARD.md):** what to build, item by item, each with an ID, a level and the reason it's at that level. It covers discovery files, search (SEO, and the AI answer engines that GEO and AEO are about), the API, MCP servers, skills, engagement, measurement and tests.
|
|
32
|
+
- **[Who reads the discovery files](docs/research/who-reads-discovery-files.md):** a week of traffic across 56 domains on Cloudflare, with charts. The AI crawlers read robots.txt and the sitemap, not llms.txt; bots asked for markdown 12,601 times; directories read the agent cards; and on sites with an API, some readers of llms.txt and openapi.json went on to call it.
|
|
33
|
+
- **`readiness-audit`:** the scorecard. It checks a live site against everything the standard can verify from outside, about 30 checks in one pass, and lists what it can't see for you to review.
|
|
34
|
+
|
|
35
|
+
## Why this one
|
|
36
|
+
|
|
37
|
+
- **Built from evidence.** Each item says why it's there: production request logs (which agents fetch which files, and which ones go on to call the API), the specs, and Google's and Bing's own documentation, cited with dates. Items that only scanners look for are recommended at most.
|
|
38
|
+
- **It checks that declarations are true.** An A2A card that isn't a valid A2A card fails D11; OAuth metadata with no token endpoint, or a catalog entry typed as something it isn't, fails D12. Declining an item in writing is fine; faking a pass isn't.
|
|
39
|
+
- **It fits the site.** The API checks run only when a site has or declares an API, and the MCP checks only when one answers. Pages are sampled by type: every static page, and three pages of each generated template.
|
|
40
|
+
- **Before and after you ship.** `--base http://localhost:3000` scores a local or preview build; `--compare` lists what changed since the last run; the exit code gates a deploy.
|
|
41
|
+
- **The real test is an agent.** The standard's T5 gives a fresh agent only your domain and a goal, and records whether it succeeds. The scorecard is the setup for that, not the goal.
|
|
42
|
+
|
|
43
|
+
## How it fits with other tools
|
|
44
|
+
|
|
45
|
+
| Tool | What it does | Use it with this |
|
|
46
|
+
|---|---|---|
|
|
47
|
+
| [Is Your Site Agent-Ready?](https://isitagentready.com/) (Cloudflare) | A web scan for llms.txt, MCP, agent skills and other agent-facing standards | For a quick look in the browser. This runs in your terminal and CI, scores local builds, and checks search as well as agents |
|
|
48
|
+
| llms.txt validators | Check that one file is well formed | This checks llms.txt and samples its links, as one of about 30 checks |
|
|
49
|
+
| [security-pranks-audit](https://github.com/geeks-accelerator/agent-security-pranks) | The sister project: checks that a site answers `.env`, `.git` and other secret-hunting probes safely | This welcomes agents and checks what they read; that checks what the scanners get. Same shape, run both |
|
|
50
|
+
| [specification.website](https://specification.website) | A sourced website specification: HTML, accessibility, security, SEO, agent readiness | A spec for the whole site, accessibility and security included. This one focuses on agents and search, and comes with a scorer |
|
|
51
|
+
|
|
52
|
+
## Score a site
|
|
53
|
+
|
|
18
54
|
It needs Node 26 or later, with current security releases, and has no dependencies. It only reads: GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`). It keeps to about six requests at a time, and it won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect. It asks for gzip and decodes it itself; a body in an encoding it didn't ask for, such as Brotli, is reported rather than read, so a site can't reach a decoder bug on the machine running the audit. It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy.
|
|
19
55
|
|
|
20
56
|
```
|
|
@@ -52,6 +88,8 @@ The scorecard covers what's visible from outside. It checks:
|
|
|
52
88
|
- a sample of skill files: spec names and descriptions, and links to llms.txt and the API reference;
|
|
53
89
|
- markdown negotiation, the AI catalog, the DNS AID record, the API catalog, auth.md and the skills index.
|
|
54
90
|
|
|
91
|
+
It also lists the pages to run through Google's Rich Results Test (W4), with a link that opens each one in the test: the homepage, one page of each template with rich-result markup, and any page that adds a rich-result feature. The scorecard checks that markup parses; Google's test checks each rich result's required properties.
|
|
92
|
+
|
|
55
93
|
It can't see whether `next_steps` guide well, whether input is forgiving, whether an agent can follow your skills through the core loop, or whether you have tests. The standard marks those items "review" or "test". The agent usability test (T5) measures what all of it is for: whether a fresh agent, given only your domain and a goal, succeeds. [docs/usability-test.md](docs/usability-test.md) has the procedure and a runner. A high score is not the goal.
|
|
56
94
|
|
|
57
95
|
## Use it in your project
|
package/STANDARD.md
CHANGED
|
@@ -85,6 +85,8 @@ What it means:
|
|
|
85
85
|
| AI search indexes | Meta-WebIndexer, OAI-SearchBot, Claude-SearchBot, PerplexityBot | ordinary pages, like any search crawler (§5) |
|
|
86
86
|
|
|
87
87
|
So no single file reaches every agent. Directories want the well-known files, scripts want llms.txt and the API, tools want clean pages and docs, and search indexes want everything in §5.
|
|
88
|
+
- **A crawler's name is a claim.** Across one owner's 56 domains on Cloudflare, October 16 to 23, 2026, none of about 448k vulnerability probes, most of them hunting `.env` files, came from a bot Cloudflare verified, and one scanner sent 308k requests under about 20 AI-crawler names, GPTBot and ClaudeBot among them ([Bots in AI clothing](https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md), from the sister project agent-security-pranks). The crawlers in the table above are the verified ones; a name in a log line alone doesn't show which requests are theirs (T3).
|
|
89
|
+
- **A week across the whole network says the same, at scale.** [Who reads the discovery files](docs/research/who-reads-discovery-files.md) covers 56 domains, October 17 to 24, 2026. Verified AI crawlers read robots.txt (3,644 requests) and the sitemap (2,232, 40% of its reads) far more than llms.txt (49 of 1,406). Verified bots, nearly all Exa's search crawler, asked for markdown 12,601 times. Directories, not AI crawlers, read agent-card.json (2 of 991 requests were verified AI). On the 16 sites with an API, 13% of the addresses that read llms.txt, and 23% of those that read openapi.json, made a successful API call the same week.
|
|
88
90
|
- **Check your own traffic before building a next-level item.** botsmatter looked: 2 of its 1,940 Grounds exactly repeat an earlier one, so retries aren't a problem there and N2 isn't worth building.
|
|
89
91
|
|
|
90
92
|
---
|
|
@@ -133,7 +135,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
133
135
|
**D3 Server-rendered pages.** Every page's content is in the HTML the server sends, not loaded later by script, with a real logo through the framework's icon convention and a favicon set (missing icons show up as steady 404s). *Why: crawlers, link-preview bots and agents that fetch pages read the HTML, and many don't run JavaScript.* inbed's chat pages once shipped none of their 48 messages in the HTML. The page-level details (titles, canonicals, structured data, previews, headings, links) are W1 to W7 in [§5](#5-search-people-and-answer-engines).
|
|
134
136
|
|
|
135
137
|
**D4 `/llms.txt` and `/llms-full.txt`**: a short map (what this is, how to start, links to every other surface) and the whole corpus in one fetch. Generate both and test them against the sitemap: agents follow the links, so every one must open. The scorecard samples them and warns about missing pages (an endpoint that answers 401 for want of a key is fine; templates like `{id}` are skipped). The file is markdown; serve it as `text/markdown` or `text/plain`, and use the same type wherever you link it.
|
|
136
|
-
- *Why required:
|
|
138
|
+
- *Why required: agents that go on to use the site read it, though the AI crawlers mostly don't.* Across 56 domains in a week of October 2026, verified AI crawlers made 49 of 1,406 llms.txt requests (GPTBot 5, ClaudeBot none) and read robots.txt and the sitemap instead. On the 16 sites with an API, 13% of the addresses that read llms.txt made a successful API call that week, 9% with a user agent that wasn't a browser's ([research](docs/research/who-reads-discovery-files.md)). Google Search ignores llms.txt (it said so in June 2026). At animalhouse, 79 requests in four and a half days came from:
|
|
137
139
|
- agent directories (AgenstryBot, BrickBlueBot, agentprobe, MCPHarbor, AgentTrustBot);
|
|
138
140
|
- llms.txt directories and validators;
|
|
139
141
|
- SEO tools and Amazonbot;
|
|
@@ -147,6 +149,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
147
149
|
|
|
148
150
|
**D11 The A2A path is honest.** `/.well-known/agent-card.json` is the most-requested agent path. Serve a valid A2A v1.0 card only if you run an A2A endpoint ([Only if true](#only-if-true)). Otherwise answer a JSON 404 that names your MCP endpoint, OpenAPI document and llms.txt (achurch.ai). A custom document or an "A2A-shaped" card at that path fails. *Why: honesty. A2A §8.3.1 requires every declared interface to be real.*
|
|
149
151
|
- **The cost:** directory crawlers drive most requests to this path, and some may stop listing a site that answers 404.
|
|
152
|
+
- **The pre-1.0 path still gets asked.** Directories request `/.well-known/agent.json` too: 445 times at 19 sites in a week of October 2026, none of which serve it ([research](docs/research/who-reads-discovery-files.md)). Answer it the same way; D7's catch-all does.
|
|
150
153
|
- **Roll it out on one site first.** inbed's agent said it would accept the risk. Watch for two weeks whether AgenstryBot and the other directories keep listing it, then change the rest.
|
|
151
154
|
|
|
152
155
|
**D12 No false declarations.** *Why: honesty.*
|
|
@@ -189,7 +192,18 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
189
192
|
- `service-desc` is the machine-readable description (your OpenAPI). `service-doc` is the human documentation (RFC 8631). Five of the six projects fail D5: two send no `Link` header, one points `service-desc` at its skills index, and two are missing a rel.
|
|
190
193
|
- Registered relations are `describedby`, `service-desc`, `service-doc`, `service-meta` and `api-catalog`. `ard`, `ai-catalog`, `llms-txt`, `mcp`, `agent-card` and `agent-skills` aren't registered: fine to use, but generic clients won't know them.
|
|
191
194
|
|
|
192
|
-
**D7 JSON 404s for unknown `/.well-known/*` paths, and a JSON 405 on `POST /`** naming the MCP endpoint and the API (achurch.ai, magnifica). An HTML 404 page is noise to an agent. *Why: cheap, and it helps the agents and crawlers that guess. No project passes all of it yet.* One catch-all route can answer D7 and D11 together ([recipe](docs/recipes.md#one-catch-all-for-unknown-well-known-paths-d7-d11)).
|
|
195
|
+
**D7 JSON 404s for unknown `/.well-known/*` paths, including the pre-1.0 A2A path `/.well-known/agent.json`, and a JSON 405 on `POST /`** naming the MCP endpoint and the API (achurch.ai, magnifica), with an `Allow` header naming the methods that work (`Allow: GET, HEAD`): RFC 9110 requires it on every 405. An HTML 404 page is noise to an agent. *Why: cheap, and it helps the agents and crawlers that guess. No project passes all of it yet.* One catch-all route can answer D7 and D11 together ([recipe](docs/recipes.md#one-catch-all-for-unknown-well-known-paths-d7-d11)).
|
|
196
|
+
|
|
197
|
+
**D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and agents ask for it with `Accept: text/markdown`. *Why recommended: traffic.* Across 56 domains in a week of October 2026, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler and the rest ShapBot, Exa's user-triggered fetcher and KeenableBot ([research](docs/research/who-reads-discovery-files.md)). That's one search engine more than all the others, so check your own logs: the case rests on who asks you. Many hosts' logs don't keep the `Accept` header (Railway's HTTP logs and Cloudflare's Free-plan analytics don't); a markdown answer at a page URL shows that one was sent. To see it on your own site, log the header in middleware ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)).
|
|
198
|
+
- Start with the homepage and your docs, not every page. `Accept: text/markdown` and a `.md` URL both return the page as markdown. animalhouse serves llms.txt as the homepage's markdown, and its API reference at `/docs/api.md`.
|
|
199
|
+
- No CDN plan is needed: a `.md` route and a few lines of middleware do it ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)). Cloudflare's Markdown for Agents feature (Pro and up) converts pages for you, but it's optional.
|
|
200
|
+
- Send `Vary: Accept` on both representations (RFC 9110 §12.5.5). Otherwise a cache can hand HTML readers the markdown, or the reverse.
|
|
201
|
+
- Next.js drops a `Vary` set in middleware, proxy or `next.config` on App Router pages, static or dynamic, while it keeps the other headers (tested on 14.2 and 16.3). So `Accept` has to reach the HTML from the edge (a CDN response header rule), or decline that half in writing. Send it from the markdown route yourself (animalhouse.ai).
|
|
202
|
+
- A page's markdown copy has the page's sections. The homepage's can be llms.txt instead. *Harm prevented:* agents reading the markdown get a different site than people do. The scorecard compares the headings of one page llms.txt links, when that page serves markdown, and a guard test can compare them all (§11).
|
|
203
|
+
- Make links absolute, and link back to the HTML page as canonical.
|
|
204
|
+
- Add `x-markdown-tokens` so an agent can budget before reading. It's Cloudflare's header, with no spec behind it.
|
|
205
|
+
- Negotiate in front of prerendered pages too, or they ignore it (magnifica).
|
|
206
|
+
- Whole-corpus indexes help: `/docs/index.md` and `/docs/index.json` (achurch.ai).
|
|
193
207
|
|
|
194
208
|
**D9 An AI catalog (ARD)**
|
|
195
209
|
- One list of what an agent can use here, such as the MCP server card, the OpenAPI document, skills and llms.txt, with identifiers in the form `urn:air:<publisher-domain>:<namespace>:<name>`.
|
|
@@ -329,7 +343,9 @@ The evidence that this work pays: achurch.ai's August batch rewrote titles and s
|
|
|
329
343
|
- **Honest types for agents.** schema.org has no type for AI agents. Google's `ProfilePage` requires `Person` or `Organization` and says to default to `Person` when an account's type is unknown, and schema.org's `Person` isn't limited to humans. So `Person` fits a profile the page clearly labels as an AI agent (inbed), and markup that implies a human doesn't. botsmatter uses `Thing` and gives up profile rich results. Either is honest; say on the page what the agent is.
|
|
330
344
|
- **Don't chase retired rich results.** Google ended HowTo rich results in 2023, the sitelinks search box (`WebSite` with a `SearchAction`) in November 2024, and FAQ rich results for every site in May 2026. Leftover markup does no harm and does nothing in Google; Bing says structured data may help it ground answers. Keep `FAQPage` or `HowTo` only where the page really is one.
|
|
331
345
|
- **Escape JSON-LD.** User content containing `<`, `>`, U+2028 or U+2029 can break out of the script tag (achurch, magnifica).
|
|
332
|
-
- **
|
|
346
|
+
- **Test in Google's tools, one page per kind of markup.** The scorecard checks that markup parses and has the expected types; Google's Rich Results Test checks each rich result's required properties. The scorecard lists the pages to test, each with a link that opens it in the test: the homepage, one page of each template with rich-result markup (pages from one template share it, and each template is its own code), and any other page that adds a rich-result feature. Retest after deploying a change to a template's markup. For types Google shows no rich result for, use the Schema Markup Validator. A valid result means a page is eligible, not that Google will show it.
|
|
347
|
+
- **Search Console's rich result reports cover every indexed page,** one report per feature, with valid and invalid items. Check them in the monthly review (T6).
|
|
348
|
+
- **Done when** the scorecard passes W4 and the Rich Results Test finds no errors on the pages the scorecard lists.
|
|
333
349
|
|
|
334
350
|
**W5 Link previews.** *Why: a shared link is how many people, and some agents, first see a page.*
|
|
335
351
|
- Open Graph (`og:type`, `og:url`, `og:title`, `og:description`, `og:image`) and `twitter:card` set to `summary_large_image` on every page.
|
|
@@ -630,17 +646,6 @@ The test is whether most of the six projects should do it. These came from one o
|
|
|
630
646
|
|
|
631
647
|
Higher-value features that most of our projects don't have yet. Each one fixes a failure we've seen or can predict. Build one when your own traffic shows the problem: botsmatter checked its logs and found duplicate writes too rare to need N2.
|
|
632
648
|
|
|
633
|
-
**D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and some agent fetchers ask for it with `Accept: text/markdown`. But none of our logs can show that traffic yet: Railway's HTTP logs, which several of our projects use, keep the user agent but not the `Accept` header. And a scanner check alone doesn't make an item recommended (principle 3). *Build it when* your logs show agents asking: log the `Accept` header in middleware to find out ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)).
|
|
634
|
-
- Start with the homepage and your docs, not every page. `Accept: text/markdown` and a `.md` URL both return the page as markdown. animalhouse serves llms.txt as the homepage's markdown, and its API reference at `/docs/api.md`.
|
|
635
|
-
- No CDN plan is needed: a `.md` route and a few lines of middleware do it ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)). Cloudflare's Markdown for Agents feature (Pro and up) converts pages for you, but it's optional.
|
|
636
|
-
- Send `Vary: Accept` on both representations (RFC 9110 §12.5.5). Otherwise a cache can hand HTML readers the markdown, or the reverse.
|
|
637
|
-
- Next.js drops a `Vary` set in middleware, proxy or `next.config` on App Router pages, static or dynamic, while it keeps the other headers (tested on 14.2 and 16.3). So `Accept` has to reach the HTML from the edge (a CDN response header rule), or decline that half in writing. Send it from the markdown route yourself (animalhouse.ai).
|
|
638
|
-
- A page's markdown copy has the page's sections. The homepage's can be llms.txt instead. *Harm prevented:* agents reading the markdown get a different site than people do. The scorecard compares the headings of one page llms.txt links, when that page serves markdown, and a guard test can compare them all (§11).
|
|
639
|
-
- Make links absolute, and link back to the HTML page as canonical.
|
|
640
|
-
- Add `x-markdown-tokens` so an agent can budget before reading. It's Cloudflare's header, with no spec behind it.
|
|
641
|
-
- Negotiate in front of prerendered pages too, or they ignore it (magnifica).
|
|
642
|
-
- Whole-corpus indexes help: `/docs/index.md` and `/docs/index.json` (achurch.ai).
|
|
643
|
-
|
|
644
649
|
**N2 `Idempotency-Key` on writes** (IETF draft). Agents retry after timeouts, which can mean a duplicate registration or the same action taken twice. Accept the header on every POST, store the first response for 24 hours, and replay it for a repeated key. *Build it when* your logs show the same write repeated within minutes.
|
|
645
650
|
|
|
646
651
|
**N3 Conditional requests on polled reads.** Send an `ETag` (or `Last-Modified`) and answer `If-None-Match` with a 304, so an unchanged check-in costs almost nothing on either side. *Build it when* scheduled polling is a large share of your traffic.
|
|
@@ -676,7 +681,8 @@ Higher-value features that most of our projects don't have yet. Each one fixes a
|
|
|
676
681
|
- **T2 Attribution and one come-back measure** (required; per-agent tracking may be declined). Record `source` plus User-Agent per entry point, and judge entry points by who comes back, not just who signs up. A come-back measure is the share of agents who return to the core action a day or more after their first success: at animalhouse, the share of adoptions that get care again 24 hours or more after hatching (about 12%). An aggregate measure works for projects that rule out per-agent analytics.
|
|
677
682
|
- **T3 Reviews** (required):
|
|
678
683
|
- A monthly log review, paged so no window is truncated (platform log commands often return only the newest window): requested well-known paths, 404s, refusals, crawler user agents. `readiness-audit --logs <file>` counts the discovery requests in an exported log by client family.
|
|
679
|
-
- **
|
|
684
|
+
- **Count audits apart from agents.** Audit tools, this one included, are among the largest readers of discovery files: in a week across 56 domains, this audit and its sister made 3,564 discovery requests, more than all the week's llms.txt requests, and other audit tools were the largest readers of ard.json and ai-catalog.json ([research](docs/research/who-reads-discovery-files.md)). `--logs` names them as their own families. *Harm prevented:* mistaking scanners of your setup for agents that use it.
|
|
685
|
+
- **Classify traffic before counting it.** Set known junk crawlers aside first, and count a crawler by its verified status, not its name: the CDN's verified-bot flag, the vendor's published addresses, or reverse DNS. Scanners send crawler names: one sent 308k requests under about 20 AI-crawler names in a week (§2), and `--logs` counts requests under crawler names that asked for secret files, which no real crawler does. Count a browser, headless or unidentified client as an agent only when something backs it up, such as the same address calling the API: agent tools often send a browser's user agent (§2), so their user agent alone proves nothing either way. *Harm prevented:* readiness decisions made on noise.
|
|
680
686
|
- After each deploy, run the scorecard, lint the OpenAPI spec, and diff generated files against the previous production output. About 24 hours later, review the logs and database query performance.
|
|
681
687
|
- **T5 The agent usability test** (required).
|
|
682
688
|
- **Why:** it's the only check that measures what the whole standard is for, and without it the scorecard becomes the target.
|
|
@@ -693,6 +699,7 @@ Higher-value features that most of our projects don't have yet. Each one fixes a
|
|
|
693
699
|
- Verify the site in Google Search Console and Bing Webmaster Tools, and submit the sitemap to both.
|
|
694
700
|
- Review them monthly: coverage (including "crawled, not indexed"), queries, clicks and click-through rate. Re-check about two weeks after an SEO change (inbed).
|
|
695
701
|
- Record the numbers next to the score as fields, not prose, so months compare: the period, clicks, impressions, click-through rate, average position, pages indexed and not indexed, and "crawled, not indexed", for Google and Bing (format: [examples/recorded.json](examples/recorded.json)). They're the outcome the W items are for.
|
|
702
|
+
- Check the rich result reports: invalid items first, since a critical issue keeps an item out of rich results (W4). The URL Inspection API returns the same findings for one URL (`richResultsResult`), for scripts with the owner's access.
|
|
696
703
|
- Record the top queries with the page each lands on (`queries`), and ask in the review whether each page answers its query in its title, `h1` and opening text. animalhouse's homepage ranked for "tamagotchi ai" without those words on the page. It isn't scored: the data is entered by hand, and matching query words against a title invites keyword stuffing.
|
|
697
704
|
- It takes a person, or a browser session, with access to Search Console and Bing Webmaster Tools. There's no unattended way to read them.
|
|
698
705
|
- In analytics, watch referrals from AI answer engines (chatgpt.com, perplexity.ai, copilot.microsoft.com, gemini.google.com, claude.ai).
|
|
@@ -772,7 +779,7 @@ Score each site monthly and after big releases, and keep the results where the p
|
|
|
772
779
|
npx readiness-audit@2 --matrix site-one.com site-two.com --recorded docs/readiness-recorded.json > docs/readiness-status.md
|
|
773
780
|
```
|
|
774
781
|
|
|
775
|
-
The page lists every check for every site, the scores at each level,
|
|
782
|
+
The page lists every check for every site, the scores at each level, each failure or warning with its reason, and the pages to run through Google's Rich Results Test (W4), each with a link that opens it in the test. Keep it in the project's private repo when there is one (for example, a private repo that holds the docs, with the public repo as a submodule). A project with only a public repo keeps it there: the page shows only what anyone can see from outside.
|
|
776
783
|
|
|
777
784
|
Score a local or preview build before you deploy: `npx readiness-audit@2 example.com --base http://localhost:3000` requests the site's URLs from the base and reports them under the domain, skipping DNS and host redirects. To see what changed since the last run, compare against that page: `npx readiness-audit@2 <domain> --compare docs/readiness-status.md` lists what was fixed, what newly fails and what newly applies. Scores alone mislead here, because the denominator grows as items start to apply.
|
|
778
785
|
|
|
@@ -821,8 +828,8 @@ Level: **R** required where it applies, **Rec** recommended, **N** next level. W
|
|
|
821
828
|
| D4 | llms.txt and llms-full.txt, generated, linked from skills and README | R | traffic | score, test |
|
|
822
829
|
| D5 | `Link` headers: service-desc, service-doc, describedby, api-catalog | Rec | discovery | score |
|
|
823
830
|
| D6 | security.txt with Expires once | R | hygiene | score |
|
|
824
|
-
| D7 | JSON 404s for unknown well-known paths, JSON 405 on `POST /` | Rec | agents | score |
|
|
825
|
-
| D8 | Markdown for agents with `Vary: Accept` and x-markdown-tokens
|
|
831
|
+
| D7 | JSON 404s for unknown well-known paths (`agent.json` included), JSON 405 with `Allow` on `POST /` | Rec | agents | score |
|
|
832
|
+
| D8 | Markdown for agents with `Vary: Accept` and x-markdown-tokens | Rec | traffic | score |
|
|
826
833
|
| D9 | AI catalog at ard.json and ai-catalog.json, with `rel` links | Rec | discovery | score |
|
|
827
834
|
| D10 | One DNS AID record naming a service you run | Rec | discovery | score |
|
|
828
835
|
| D11 | The A2A path: a valid card or a JSON 404 | R | honesty | score |
|
|
@@ -833,7 +840,7 @@ Level: **R** required where it applies, **Rec** recommended, **N** next level. W
|
|
|
833
840
|
| W1 | Titles and descriptions: one helper, unique, searched-for words, 140 to 160 character descriptions | R | traffic | score |
|
|
834
841
|
| W2 | Self-referencing canonicals, one host in one redirect | R | discovery | score |
|
|
835
842
|
| W3 | One indexability rule; sitemap of 200 canonical URLs; real 404s | R | discovery, honesty | score |
|
|
836
|
-
| W4 | True structured data: Organization and WebSite, breadcrumbs, honest types, escaped | R | honesty | score |
|
|
843
|
+
| W4 | True structured data: Organization and WebSite, breadcrumbs, honest types, escaped, tested in Google's Rich Results Test | R | honesty | score, review |
|
|
837
844
|
| W5 | Link previews: Open Graph, `twitter:card`, each page's own card text, declared images, versioned cards | R | discovery | score |
|
|
838
845
|
| W6 | One h1, `html lang`, alt text, unique paragraph on generated pages | R | discovery | score |
|
|
839
846
|
| W7 | Internal links with descriptive, searched-for anchors true to the target, no orphan pages | R | discovery | review |
|
|
@@ -941,6 +948,11 @@ Status as of 2026-10-05. Several are drafts, so re-check before relying on a det
|
|
|
941
948
|
| Google's AI optimization guide | Search Central, May 2026 | https://developers.google.com/search/docs/fundamentals/ai-optimization-guide |
|
|
942
949
|
| Title links and snippets | Search Central | https://developers.google.com/search/docs/appearance/title-link · https://developers.google.com/search/docs/appearance/snippet |
|
|
943
950
|
| Redirects | Search Central, read 2026-10-21 | https://developers.google.com/search/docs/crawling-indexing/301-redirects |
|
|
951
|
+
| Google rich results | Search Central search gallery (updated 2026-10-16), read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data/search-gallery |
|
|
952
|
+
| Testing structured data | Search Central: the Rich Results Test and the Schema Markup Validator, read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data · https://search.google.com/test/rich-results · https://validator.schema.org/ |
|
|
953
|
+
| Rich result reports, URL Inspection API | Search Console Help and API reference, read 2026-10-21 | https://support.google.com/webmasters/answer/7552505 · https://developers.google.com/webmaster-tools/v1/urlInspection.index/UrlInspectionResult |
|
|
954
|
+
| Crawler impersonation ("Bots in AI clothing") | agent-security-pranks research, 56 domains on Cloudflare, 2026-10-16 to 10-23, read 2026-10-24 | https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md |
|
|
955
|
+
| 405 and `Allow` | RFC 9110 §15.5.6 | https://www.rfc-editor.org/rfc/rfc9110.html#section-15.5.6 |
|
|
944
956
|
| Headings | W3C WAI tutorial, read 2026-10-21 | https://www.w3.org/WAI/tutorials/page-structure/headings/ |
|
|
945
957
|
| Publication dates | Search Central, read 2026-10-21 | https://developers.google.com/search/docs/appearance/publication-dates |
|
|
946
958
|
| Canonicalization | Search Central | https://developers.google.com/search/docs/crawling-indexing/canonicalization |
|
package/bin/readiness-audit.mjs
CHANGED
|
@@ -115,7 +115,10 @@ if (flag('--matrix')) {
|
|
|
115
115
|
console.log(`\nScore on ${rep.checked_at.slice(0, 10)}: required ${score(rep.results, 'required')}, recommended ${score(rep.results, 'recommended')}, next level ${score(rep.results, 'next')}`);
|
|
116
116
|
if (changes) console.log(`\n${formatComparison(changes, since).join('\n')}`);
|
|
117
117
|
console.log('\nNot scored, so check these yourself:');
|
|
118
|
-
for (const [id, what] of REVIEW)
|
|
118
|
+
for (const [id, what] of REVIEW) {
|
|
119
|
+
console.log(` ${id.padEnd(6)} ${what}`);
|
|
120
|
+
if (id === 'W4') for (const p of rep.rich_results_test ?? []) console.log(` ${p.url} (${p.features.join(', ')})\n ${p.test}`);
|
|
121
|
+
}
|
|
119
122
|
}
|
|
120
123
|
process.exitCode = rep.results.some((r) => r.level === 'required' && r.status === 'fail') ? 1 : 0;
|
|
121
124
|
}
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "readiness-audit",
|
|
3
|
-
"version": "2.
|
|
4
|
-
"description": "Score
|
|
3
|
+
"version": "2.2.0",
|
|
4
|
+
"description": "Score any site's readiness for AI agents and search (GEO, AEO, AI SEO): llms.txt, robots.txt AI crawler rules, structured data, link previews, OpenAPI, MCP, agent skills and A2A. Read-only CLI, no account.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"readiness-audit": "bin/readiness-audit.mjs"
|
|
@@ -32,7 +32,18 @@
|
|
|
32
32
|
"openapi",
|
|
33
33
|
"structured-data",
|
|
34
34
|
"audit",
|
|
35
|
-
"scorecard"
|
|
35
|
+
"scorecard",
|
|
36
|
+
"aeo",
|
|
37
|
+
"ai-seo",
|
|
38
|
+
"ai-visibility",
|
|
39
|
+
"generative-engine-optimization",
|
|
40
|
+
"robots-txt",
|
|
41
|
+
"ai-crawlers",
|
|
42
|
+
"seo-audit",
|
|
43
|
+
"website-audit",
|
|
44
|
+
"agent-ready",
|
|
45
|
+
"llms-txt-validator",
|
|
46
|
+
"cli"
|
|
36
47
|
],
|
|
37
48
|
"license": "MIT",
|
|
38
49
|
"repository": {
|
package/src/audit.mjs
CHANGED
|
@@ -9,19 +9,20 @@
|
|
|
9
9
|
import { lookup, resolveTxt, Resolver } from 'node:dns/promises';
|
|
10
10
|
import { isIP } from 'node:net';
|
|
11
11
|
import { request } from './http.mjs';
|
|
12
|
-
import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
|
|
12
|
+
import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
|
|
13
13
|
|
|
14
|
-
export const VERSION = '2.
|
|
14
|
+
export const VERSION = '2.2.0';
|
|
15
15
|
export const REPO = 'https://github.com/geeks-accelerator/agent-and-search-readiness';
|
|
16
16
|
const UA = `readiness-audit/${VERSION} (+${REPO})`;
|
|
17
17
|
const AID_PROTOCOLS = ['mcp', 'a2a', 'openapi', 'grpc', 'graphql', 'websocket', 'local', 'zeroconf', 'ucp'];
|
|
18
18
|
const SKILLS_SCHEMA = 'https://schemas.agentskills.io/discovery/0.2.0/schema.json';
|
|
19
19
|
const CARD_SCHEMA = 'https://static.modelcontextprotocol.io/schemas/v1/server-card.schema.json';
|
|
20
20
|
export const LEVELS = ['required', 'recommended', 'next'];
|
|
21
|
-
const ORDER = ['D1', 'D2', 'D4', 'D6', 'D11', 'D12', 'W1', 'W2', 'W3', 'W4', 'W5', 'W6', 'A1', 'A2', 'A6', 'A7', 'M1', 'M4', 'S1', 'D5', 'D7', 'D9', 'D10', 'D15', 'A3', 'A4', 'A5', 'M5', 'S2', '
|
|
21
|
+
const ORDER = ['D1', 'D2', 'D4', 'D6', 'D11', 'D12', 'W1', 'W2', 'W3', 'W4', 'W5', 'W6', 'A1', 'A2', 'A6', 'A7', 'M1', 'M4', 'S1', 'D5', 'D7', 'D9', 'D10', 'D15', 'D8', 'A3', 'A4', 'A5', 'M5', 'S2', 'N5', 'N6'];
|
|
22
22
|
// Items the scorecard can't see from outside. Each project checks these in review and tests.
|
|
23
23
|
export const REVIEW = [
|
|
24
24
|
['D3', 'pages read well: server-rendered, unique prose on generated pages (the scorecard checks W1 to W6 on a sample)'],
|
|
25
|
+
['W4', 'Google\'s Rich Results Test on one page per kind of markup (listed below: the scorecard checks the markup parses, Google checks its required properties), after deploying a change to a template\'s markup; Search Console\'s rich result reports cover every page (T6)'],
|
|
25
26
|
['W7', 'internal links: entity names link to their pages, anchors that say what the target is (not the brand alone), no orphan pages; sibling links in context, never sitewide (E6)'],
|
|
26
27
|
['W8', 'Core Web Vitals on the main pages (Lighthouse or PageSpeed Insights)'],
|
|
27
28
|
['W9', 'listed in the directories your audience uses'],
|
|
@@ -267,7 +268,8 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
267
268
|
const linkProblems = [];
|
|
268
269
|
if (broken.length) linkProblems.push(`broken links in llms.txt: ${list(broken)} (${broken.length} of ${sample.length} sampled)`);
|
|
269
270
|
if (redirected.length) linkProblems.push(`llms.txt links that redirect (link the destination): ${list(redirected)}`);
|
|
270
|
-
const
|
|
271
|
+
const kb = (r, digits) => (r.tooBig ? r.error.replace(/^.*?over /, 'over ') : `${(r.text.length / 1024).toFixed(digits)} KB`);
|
|
272
|
+
const sizes = `llms.txt ${kb(llms, 1)}, llms-full.txt ${kb(llmsFull, 0)}`;
|
|
271
273
|
if (!ok(llms) || !/^\s*# \S/.test(llms.text)) miss(id, name, level, (ok(llms) ? 'llms.txt has no H1 title on its first line' : `/llms.txt answered ${llms.status}`) + cached(llms));
|
|
272
274
|
else if (!ok(llmsFull)) miss(id, name, level, `/llms-full.txt missing${cached(llmsFull)}`);
|
|
273
275
|
else if (linkProblems.length) record(id, name, level, 'warn', `${linkProblems.join('; ')}${cached(llms)}`);
|
|
@@ -304,13 +306,14 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
304
306
|
const problems = [];
|
|
305
307
|
if (!isJson(wkMiss)) problems.push(`an unknown /.well-known path answers ${wkMiss.status} ${ct(wkMiss)}`);
|
|
306
308
|
if (!(postRoot.status === 405 && isJson(postRoot))) problems.push(`POST / answers ${postRoot.status} ${ct(postRoot)}`);
|
|
309
|
+
else if (!postRoot.headers.get('allow')) problems.push('POST / answers a JSON 405 with no Allow header (RFC 9110 requires one, naming the methods that work)');
|
|
307
310
|
if (problems.length) miss(id, name, level, problems.join('; '));
|
|
308
|
-
else pass(id, name, level,
|
|
311
|
+
else pass(id, name, level, `JSON 404 for unknown well-known paths, JSON 405 for POST / (Allow: ${postRoot.headers.get('allow')})`);
|
|
309
312
|
}
|
|
310
313
|
|
|
311
314
|
// ---- D8 markdown for agents ----
|
|
312
315
|
{
|
|
313
|
-
const id = 'D8', name = 'Markdown for agents', level = '
|
|
316
|
+
const id = 'D8', name = 'Markdown for agents', level = 'recommended';
|
|
314
317
|
const varyAccept = (r) => /(^|,)\s*accept\s*(,|$)/i.test(r.headers.get('vary') ?? '');
|
|
315
318
|
const tokens = homeMd.headers.get('x-markdown-tokens');
|
|
316
319
|
// The homepage's markdown may be llms.txt, so compare a page llms.txt links instead, when it serves markdown.
|
|
@@ -611,11 +614,13 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
611
614
|
|
|
612
615
|
// ---- W: search, on a sample of sitemap pages ----
|
|
613
616
|
let htmlKb = null; // median HTML size of the sampled pages, reported, not scored
|
|
617
|
+
let richTest = []; // the pages to run through Google's Rich Results Test (W4, in review)
|
|
614
618
|
{
|
|
615
619
|
const urls = entries.map((e) => e.loc);
|
|
616
620
|
const sample = samplePages(urls);
|
|
617
621
|
const pick = sample.pages;
|
|
618
|
-
const
|
|
622
|
+
const templateOf = new Map(sample.templates.flatMap((t) => t.urls.map((u) => [norm(u), t.pattern])));
|
|
623
|
+
const templatePages = new Set(templateOf.keys());
|
|
619
624
|
const sampled = `${pick.length} sampled pages (${sample.static} static, ${sample.templates.length ? `${sample.templates.length} template${sample.templates.length === 1 ? '' : 's'}` : 'no templates'}${sample.skipped ? `; ${sample.skipped} static pages over the cap not read` : ''})`;
|
|
620
625
|
const attrs = (tag) => Object.fromEntries([...tag.matchAll(/([\w:-]+)\s*=\s*("([^"]*)"|'([^']*)')/g)].map((m) => [m[1].toLowerCase(), m[3] ?? m[4]]));
|
|
621
626
|
const tags = (html, name) => [...html.matchAll(new RegExp(`<${name}\\b[^>]*>`, 'gi'))].map((m) => attrs(m[0]));
|
|
@@ -746,6 +751,10 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
746
751
|
const deepPages = ok200.filter((p) => templatePages.has(norm(p.url)) && pathDepth(p.url) >= 2);
|
|
747
752
|
const noCrumbs = deepPages.filter((p) => !p.types.includes('BreadcrumbList')).map((p) => short(p.url));
|
|
748
753
|
const all = new Set(ok200.flatMap((p) => p.types));
|
|
754
|
+
// One page per kind of markup: the homepage, a page of each template, then the rest.
|
|
755
|
+
const homePage = ok200.find((p) => short(p.url) === '/') ?? (home.status === 200 ? { url: `${BASE}/`, types: homeTypes } : null);
|
|
756
|
+
const isTemplate = (p) => templatePages.has(norm(p.url));
|
|
757
|
+
richTest = richResultPages([homePage, ...ok200.filter(isTemplate).map((p) => ({ ...p, template: templateOf.get(norm(p.url)) })), ...ok200.filter((p) => !isTemplate(p))].filter(Boolean));
|
|
749
758
|
const notes = [];
|
|
750
759
|
if (all.has('FAQPage') || all.has('HowTo')) notes.push(`${['FAQPage', 'HowTo'].filter((t) => all.has(t)).join(' and ')} markup earns no rich results in Google anymore (HowTo ended in 2023, FAQ in May 2026); keep it only where the page really is one`);
|
|
751
760
|
if (all.has('Person')) notes.push('Person markup found: fine on a profile the page labels as an AI agent; don\'t imply a human');
|
|
@@ -859,7 +868,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
859
868
|
}
|
|
860
869
|
|
|
861
870
|
results.sort((a, b) => LEVELS.indexOf(a.level) - LEVELS.indexOf(b.level) || ORDER.indexOf(a.id) - ORDER.indexOf(b.id));
|
|
862
|
-
return { domain, base: baseOrigin, version: VERSION, checked_at: new Date().toISOString(), hosted_mcp: hosted, has_api: api.present, html_kb_median: htmlKb, results };
|
|
871
|
+
return { domain, base: baseOrigin, version: VERSION, checked_at: new Date().toISOString(), hosted_mcp: hosted, has_api: api.present, html_kb_median: htmlKb, rich_results_test: richTest, results };
|
|
863
872
|
}
|
|
864
873
|
|
|
865
874
|
export function score(results, level) {
|
|
@@ -900,5 +909,10 @@ export function matrix(reports, command, recorded = {}) {
|
|
|
900
909
|
out.push('');
|
|
901
910
|
}
|
|
902
911
|
out.push('## Checked in review and tests, not scored', '', ...REVIEW.map(([id, what]) => `- **${id}**: ${what}`), '');
|
|
912
|
+
const rich = reports.filter((rep) => rep.rich_results_test?.length);
|
|
913
|
+
if (rich.length) {
|
|
914
|
+
out.push('## Pages for Google\'s Rich Results Test (W4)', '', 'One page per kind of markup. Pages built from one template share it, so one tests them all.', '');
|
|
915
|
+
for (const rep of rich) out.push(`**${rep.domain}**`, '', ...rep.rich_results_test.map((p) => `- [${p.url}](${p.test}): ${p.features.join(', ')}`), '');
|
|
916
|
+
}
|
|
903
917
|
return out.join('\n');
|
|
904
918
|
}
|
package/src/helpers.mjs
CHANGED
|
@@ -68,6 +68,44 @@ export function headingSkips(hs) {
|
|
|
68
68
|
return skips;
|
|
69
69
|
}
|
|
70
70
|
|
|
71
|
+
/**
|
|
72
|
+
* schema.org types that can earn a Google rich result, by the feature Google's search gallery
|
|
73
|
+
* names (read 2026-10-21; the gallery was updated 2026-10-16). FAQ and HowTo aren't there anymore.
|
|
74
|
+
*/
|
|
75
|
+
export const GOOGLE_RICH_RESULTS = {
|
|
76
|
+
Article: ['Article', 'NewsArticle', 'BlogPosting'], Breadcrumb: ['BreadcrumbList'], 'Course list': ['Course'], Dataset: ['Dataset'],
|
|
77
|
+
'Discussion forum': ['DiscussionForumPosting'], 'Education Q&A': ['Quiz'], 'Employer aggregate rating': ['EmployerAggregateRating'],
|
|
78
|
+
Event: ['Event'], 'Image metadata': ['ImageObject'], 'Job posting': ['JobPosting'], 'Local business': ['LocalBusiness'],
|
|
79
|
+
'Math solver': ['MathSolver'], Movie: ['Movie'], Organization: ['Organization'], Product: ['Product'], 'Profile page': ['ProfilePage'],
|
|
80
|
+
'Q&A': ['QAPage'], Recipe: ['Recipe'], 'Review snippet': ['Review', 'AggregateRating'],
|
|
81
|
+
'Software app': ['SoftwareApplication', 'MobileApplication', 'WebApplication', 'VideoGame'], 'Vacation rental': ['VacationRental'], Video: ['VideoObject'],
|
|
82
|
+
};
|
|
83
|
+
const RICH_FEATURE = new Map(Object.entries(GOOGLE_RICH_RESULTS).flatMap(([feature, types]) => types.map((t) => [t, feature])));
|
|
84
|
+
|
|
85
|
+
/**
|
|
86
|
+
* The pages to run through Google's Rich Results Test. Pages from one template share their markup,
|
|
87
|
+
* so one page tests them all; each template is its own code, so each one with rich-result markup is
|
|
88
|
+
* listed once. Candidates come in order (the homepage, the template pages, then the rest), each
|
|
89
|
+
* { url, types, template } with template the page's template pattern, if any; a page outside a
|
|
90
|
+
* template is kept only when it adds a feature no kept page has.
|
|
91
|
+
* Returns up to `max` { url, features, test }.
|
|
92
|
+
*/
|
|
93
|
+
export function richResultPages(pages, max = 8) {
|
|
94
|
+
const seen = new Set();
|
|
95
|
+
const listedTemplates = new Set();
|
|
96
|
+
const out = [];
|
|
97
|
+
for (const p of pages) {
|
|
98
|
+
const features = [...new Set((p.types ?? []).map((t) => RICH_FEATURE.get(t)).filter(Boolean))].sort();
|
|
99
|
+
if (!features.length || listedTemplates.has(p.template)) continue;
|
|
100
|
+
if (!p.template && !features.some((f) => !seen.has(f))) continue;
|
|
101
|
+
if (p.template) listedTemplates.add(p.template);
|
|
102
|
+
features.forEach((f) => seen.add(f));
|
|
103
|
+
out.push({ url: p.url, features, test: `https://search.google.com/test/rich-results?url=${encodeURIComponent(p.url)}` });
|
|
104
|
+
if (out.length === max) break;
|
|
105
|
+
}
|
|
106
|
+
return out;
|
|
107
|
+
}
|
|
108
|
+
|
|
71
109
|
/** A sitemap's entries, urlset or index, as { loc, lastmod } (lastmod undefined when absent). */
|
|
72
110
|
export function sitemapEntries(xml) {
|
|
73
111
|
return [...String(xml ?? '').matchAll(/<(url|sitemap)>([\s\S]*?)<\/\1>/g)].map((m) => ({
|
package/src/http.mjs
CHANGED
|
@@ -3,45 +3,68 @@
|
|
|
3
3
|
// Not fetch(): fetch decodes a Brotli body even when the request didn't ask for one, so a
|
|
4
4
|
// hostile site could hand any body to the Brotli decoder (CVE-2026-111497 was an out-of-bounds
|
|
5
5
|
// write there). node:http decodes nothing; this decodes gzip and deflate and refuses the rest.
|
|
6
|
+
// Every body is capped, before and after decoding, so a site that answers with a zip bomb or an
|
|
7
|
+
// endless stream can't exhaust the memory of the machine running the audit.
|
|
6
8
|
|
|
7
9
|
import { request as httpRequest } from 'node:http';
|
|
8
10
|
import { request as httpsRequest } from 'node:https';
|
|
9
11
|
import { gunzipSync, inflateSync, inflateRawSync } from 'node:zlib';
|
|
10
12
|
|
|
11
13
|
export const TIMEOUT_MS = 15000;
|
|
14
|
+
// Large enough for a big llms-full.txt or OpenAPI document, small enough to be safe.
|
|
15
|
+
export const MAX_BYTES = 16 * 1024 * 1024;
|
|
12
16
|
|
|
13
17
|
/**
|
|
14
18
|
* One request, no redirects followed. Headers default to `accept: *\/*` and
|
|
15
19
|
* `accept-encoding: gzip, deflate`; a header set to null isn't sent (M4's no-Accept probe).
|
|
16
|
-
* Resolves, never rejects: { status, headers (a Headers object), text, error? }, status 0
|
|
20
|
+
* Resolves, never rejects: { status, headers (a Headers object), text, error?, tooBig? }, status 0
|
|
21
|
+
* on no answer. tooBig: the body passed maxBytes, raw or decoded; reading stopped and the text is empty.
|
|
17
22
|
*/
|
|
18
|
-
export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS } = {}) {
|
|
23
|
+
export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS, maxBytes = MAX_BYTES } = {}) {
|
|
19
24
|
const target = new URL(url);
|
|
20
25
|
const send = target.protocol === 'http:' ? httpRequest : httpsRequest;
|
|
21
26
|
const all = { 'user-agent': userAgent, accept: '*/*', 'accept-encoding': 'gzip, deflate', ...headers };
|
|
22
27
|
if (body !== undefined) all['content-length'] = Buffer.byteLength(body);
|
|
23
28
|
const sent = Object.fromEntries(Object.entries(all).filter(([, v]) => v !== null && v !== undefined));
|
|
24
29
|
const none = (error) => ({ status: 0, headers: new Headers(), text: '', error });
|
|
30
|
+
const failed = (e) => none(e.name === 'AbortError' || e.name === 'TimeoutError' ? `no answer in ${timeoutMs / 1000} seconds` : e.message);
|
|
31
|
+
const headersOf = (res) => new Headers(Object.entries(res.headers).flatMap(([k, v]) => [].concat(v).map((x) => [k, String(x)])));
|
|
32
|
+
const mb = (n) => (n >= 1024 * 1024 ? `${Math.round(n / 1024 / 1024)} MB` : `${Math.round(n / 1024)} KB`);
|
|
25
33
|
return new Promise((resolve) => {
|
|
34
|
+
let settled = false;
|
|
35
|
+
const done = (out) => { if (!settled) { settled = true; resolve(out); } };
|
|
26
36
|
const req = send(target, { method, headers: sent, signal: AbortSignal.timeout(timeoutMs) }, (res) => {
|
|
27
37
|
const chunks = [];
|
|
28
|
-
|
|
29
|
-
res.on('
|
|
38
|
+
let size = 0;
|
|
39
|
+
res.on('data', (c) => {
|
|
40
|
+
size += c.length;
|
|
41
|
+
if (size > maxBytes) {
|
|
42
|
+
done({ status: res.statusCode, headers: headersOf(res), text: '', tooBig: true, error: `body over ${mb(maxBytes)}` });
|
|
43
|
+
req.destroy();
|
|
44
|
+
return;
|
|
45
|
+
}
|
|
46
|
+
chunks.push(c);
|
|
47
|
+
});
|
|
48
|
+
res.on('error', (e) => done(failed(e)));
|
|
30
49
|
res.on('end', () => {
|
|
31
|
-
const headers =
|
|
50
|
+
const headers = headersOf(res);
|
|
32
51
|
const raw = Buffer.concat(chunks);
|
|
33
52
|
const encoding = (headers.get('content-encoding') ?? 'identity').trim().toLowerCase();
|
|
34
53
|
const out = { status: res.statusCode, headers, text: '' };
|
|
54
|
+
const opts = { maxOutputLength: maxBytes };
|
|
35
55
|
try {
|
|
36
56
|
if (encoding === 'identity' || encoding === '') out.text = raw.toString('utf8');
|
|
37
|
-
else if (encoding === 'gzip' || encoding === 'x-gzip') out.text = gunzipSync(raw).toString('utf8');
|
|
38
|
-
else if (encoding === 'deflate') { try { out.text = inflateSync(raw).toString('utf8'); } catch { out.text = inflateRawSync(raw).toString('utf8'); } } // zlib-wrapped, or raw as some servers send it
|
|
57
|
+
else if (encoding === 'gzip' || encoding === 'x-gzip') out.text = gunzipSync(raw, opts).toString('utf8');
|
|
58
|
+
else if (encoding === 'deflate') { try { out.text = inflateSync(raw, opts).toString('utf8'); } catch (e) { if (e.code === 'ERR_BUFFER_TOO_LARGE') throw e; out.text = inflateRawSync(raw, opts).toString('utf8'); } } // zlib-wrapped, or raw as some servers send it
|
|
39
59
|
else out.error = `refused: ${encoding} body`;
|
|
40
|
-
} catch (e) {
|
|
41
|
-
|
|
60
|
+
} catch (e) {
|
|
61
|
+
if (e.code === 'ERR_BUFFER_TOO_LARGE') { out.tooBig = true; out.error = `${encoding} body decodes to over ${mb(maxBytes)}`; }
|
|
62
|
+
else out.error = `undecodable ${encoding} body: ${e.message}`;
|
|
63
|
+
}
|
|
64
|
+
done(out);
|
|
42
65
|
});
|
|
43
66
|
});
|
|
44
|
-
req.on('error', (e) =>
|
|
67
|
+
req.on('error', (e) => done(failed(e)));
|
|
45
68
|
req.end(body);
|
|
46
69
|
});
|
|
47
70
|
}
|
package/src/logs.mjs
CHANGED
|
@@ -7,6 +7,8 @@ import { DISCOVERY_FILES, AI_BOTS } from './helpers.mjs';
|
|
|
7
7
|
// Directories come before the generic "bot" match, assistants before AI crawlers.
|
|
8
8
|
export const FAMILIES = [
|
|
9
9
|
['This audit (readiness-audit)', /^readiness-audit\//i],
|
|
10
|
+
// Other audit tools read discovery files at least as often as agents do (docs/research/who-reads-discovery-files.md).
|
|
11
|
+
['Other audit tools', /security-pranks-audit|isitagentready|agentprobe|agent-tools\.cloud|semrushbot|ahrefsbot|lumar|seojuice|sitebulb|screaming frog/i],
|
|
10
12
|
['Agent and MCP directories', /agenstrybot|brickbluebot|agentprobe|mcpharbor|agenttrustbot|taifoonharvester|mcphub-probe|invokerankbot|proofbench|thepluginstorebot|protogrid-probe|glidemcpindex|agent-tools\.cloud/i],
|
|
11
13
|
['llms.txt directories', /llms\.?txt/i],
|
|
12
14
|
['Assistants fetching for a person', /chatgpt-user|claude-user|perplexity-user|meta-externalfetcher/i],
|
|
@@ -26,6 +28,16 @@ export function family(ua) {
|
|
|
26
28
|
// Families whose user agent can't show an agent: count them as one only when the same address called the API.
|
|
27
29
|
const UNVERIFIED = new Set(['Browsers', 'Headless browsers', 'Unidentified', 'No user agent']);
|
|
28
30
|
|
|
31
|
+
// Families named after a crawler or directory. A name is a claim: across one owner's 56 domains in a week of
|
|
32
|
+
// October 2026, none of 448k probes came from a verified bot, and one scanner used about 20 AI-crawler names
|
|
33
|
+
// (agent-security-pranks, "Bots in AI clothing"). A request under one of these names that asks for a secret file
|
|
34
|
+
// isn't the crawler it names.
|
|
35
|
+
const CRAWLER_FAMILIES = new Set(['Agent and MCP directories', 'llms.txt directories', 'Assistants fetching for a person', 'AI crawlers', 'Search crawlers']);
|
|
36
|
+
// Paths that only secret hunters ask for, from that research's list, minus the WordPress and PHP paths a real
|
|
37
|
+
// crawler fetches on a site that runs them.
|
|
38
|
+
const SECRET_PROBE = /\.env\b|\/\.git\/|\/\.svn\/|\.gitconfig|\/@fs\/|__vite|wp-config|actuator|heapdump|\/\.aws\b|\/\.ssh\b|id_rsa|\.sql\b|\.ds_store|phpinfo|\/proc\/self\/environ/i;
|
|
39
|
+
export const isSecretProbe = (path) => SECRET_PROBE.test(path);
|
|
40
|
+
|
|
29
41
|
const COMBINED = /^(\S+) \S+ \S+ \[[^\]]*\] "(\S+) (\S+)[^"]*" (\d{3}) \S+(?: "[^"]*" "([^"]*)")?/;
|
|
30
42
|
|
|
31
43
|
// Field names per host's JSON lines, from each host's docs (read 2026-10-21): Cloudflare Logpush, Vercel log
|
|
@@ -76,6 +88,7 @@ export function logTally() {
|
|
|
76
88
|
const readers = new Map(); // bucket -> Set(addr)
|
|
77
89
|
const apiAddrs = new Set();
|
|
78
90
|
const unverifiedAddrs = new Set();
|
|
91
|
+
const impostors = new Map(); // crawler family -> requests for secret files under its name
|
|
79
92
|
return {
|
|
80
93
|
add(line) {
|
|
81
94
|
lines++;
|
|
@@ -84,6 +97,7 @@ export function logTally() {
|
|
|
84
97
|
parsed++;
|
|
85
98
|
const fam = family(r.ua);
|
|
86
99
|
if (isApiCall(r)) { if (r.addr) apiAddrs.add(r.addr); if (UNVERIFIED.has(fam) && r.addr) unverifiedAddrs.add(r.addr); }
|
|
100
|
+
if (CRAWLER_FAMILIES.has(fam) && isSecretProbe(r.path)) impostors.set(fam, (impostors.get(fam) ?? 0) + 1);
|
|
87
101
|
const b = bucket(r.path);
|
|
88
102
|
if (!b) return;
|
|
89
103
|
if (!hits.has(b)) hits.set(b, new Map());
|
|
@@ -101,14 +115,15 @@ export function logTally() {
|
|
|
101
115
|
also_called_api: [...addrs].filter((a) => apiAddrs.has(a)).length,
|
|
102
116
|
};
|
|
103
117
|
}).sort((a, b) => b.requests - a.requests);
|
|
104
|
-
|
|
118
|
+
const crawlerNames = { requests: [...impostors.values()].reduce((a, b) => a + b, 0), families: Object.fromEntries([...impostors].sort((a, b) => b[1] - a[1])) };
|
|
119
|
+
return { lines, parsed, api_addresses: apiAddrs.size, unverified_api_addresses: unverifiedAddrs.size, crawler_names_on_secret_probes: crawlerNames, paths };
|
|
105
120
|
},
|
|
106
121
|
};
|
|
107
122
|
}
|
|
108
123
|
|
|
109
124
|
/** The report as text. */
|
|
110
125
|
export function formatLogs(rep) {
|
|
111
|
-
const out = [`Read ${rep.parsed} of ${rep.lines} log lines (${rep.lines - rep.parsed} not understood)
|
|
126
|
+
const out = [`Read ${rep.parsed} of ${rep.lines} log lines (${rep.lines - rep.parsed} not understood). Families are the names clients send, not verified.`, ''];
|
|
112
127
|
if (!rep.paths.length) out.push('No requests for discovery files.');
|
|
113
128
|
for (const p of rep.paths) {
|
|
114
129
|
out.push(`${p.path} ${p.requests} request${p.requests === 1 ? '' : 's'}${p.addresses ? `, ${p.addresses} address${p.addresses === 1 ? '' : 'es'}, ${p.also_called_api} of them also called the API` : ''}`);
|
|
@@ -116,5 +131,8 @@ export function formatLogs(rep) {
|
|
|
116
131
|
}
|
|
117
132
|
out.push('', `API calls came from ${rep.api_addresses} address${rep.api_addresses === 1 ? '' : 'es'}. ${rep.unverified_api_addresses} of them sent a browser, headless or missing user agent: count those as agents, since they called the API.`);
|
|
118
133
|
out.push('Browsers, headless browsers and unidentified clients above count as agents only where they also called the API (STANDARD.md, T3).');
|
|
134
|
+
const named = rep.crawler_names_on_secret_probes;
|
|
135
|
+
if (named.requests) out.push('', `${named.requests} request${named.requests === 1 ? '' : 's'} under crawler names asked for secret files (.env, .git, keys, dumps): ${Object.entries(named.families).map(([f, n]) => `${f} ${n}`).join(', ')}. Real crawlers don't ask for these, so treat those names as claims: count a crawler by the CDN's verified flag, the vendor's published addresses or reverse DNS, not its name (T3).`);
|
|
136
|
+
else out.push('', 'No request under a crawler name asked for secret files.');
|
|
119
137
|
return out;
|
|
120
138
|
}
|