readiness-audit 2.2.0 → 2.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,14 @@
1
1
  # Changelog
2
2
 
3
+ ## 2.2.1 (2026-10-24)
4
+
5
+ Corrections after an outside review of 2.2.0 and its research. No new checks; D8 is easier to pass.
6
+
7
+ - **D8 no longer requires `x-markdown-tokens`.** Cloudflare stopped sending the header on 2026-10-09, so a site using Cloudflare's own Markdown for Agents couldn't pass. It's optional, and shown when present.
8
+ - **`--logs` counts only successful API answers** (a 2xx from `/api/*` or a POST to `/mcp`). A 404 or a 5xx no longer counts as a call. It no longer tells you to count browser, headless or unidentified addresses as agents because they called the API: a site's own pages call its API too. The JSON fields are `also_got_api_answer`, `api_answer_addresses` and `browser_like_api_answer_addresses`. agentprobe is counted once, as an audit tool.
9
+ - **The research is corrected and reproducible.** [Who requests agent discovery files?](docs/research/who-reads-discovery-files.md) now says its window overlaps the sister study's for 161 of 168 hours; names Cloudflare's legacy AI categories, which leave out ExaSearchBot; calls the API measure same-week address overlap, not a sequence; says which sites were tested to show the markdown answers came from requests that asked; corrects three "mostly" claims; states Exa's 92% share of markdown requests; cites GoodBarber's earlier count and Mintlify's benchmark; and lists every correction. [method.md](docs/research/who-reads-discovery-files/method.md) publishes the queries and the classifier.
10
+ - **STANDARD.md:** §2, D4, D8, D7, D11 and T3 use the corrected wording; D4 cites Mintlify's benchmark and D8 GoodBarber's logs; T3 says a successful API call doesn't make a client an agent; §16 cites both and the research.
11
+
3
12
  ## 2.2.0 (2026-10-24)
4
13
 
5
14
  From a review of the sister project, [agent-security-pranks](https://github.com/geeks-accelerator/agent-security-pranks), and its research. Two new warnings (D7's `Allow`, and D8 at the recommended level), no new failures. A warning lowers the recommended score until fixed, but doesn't change the exit code.
package/README.md CHANGED
@@ -29,7 +29,7 @@ Score on 2026-10-23: required 17 of 19, recommended 10 of 10, next level 1 of 3
29
29
  This repo holds:
30
30
 
31
31
  - **[STANDARD.md](STANDARD.md):** what to build, item by item, each with an ID, a level and the reason it's at that level. It covers discovery files, search (SEO, and the AI answer engines that GEO and AEO are about), the API, MCP servers, skills, engagement, measurement and tests.
32
- - **[Who reads the discovery files](docs/research/who-reads-discovery-files.md):** a week of traffic across 56 domains on Cloudflare, with charts. The AI crawlers read robots.txt and the sitemap, not llms.txt; bots asked for markdown 12,601 times; directories read the agent cards; and on sites with an API, some readers of llms.txt and openapi.json went on to call it.
32
+ - **[Who requests agent discovery files?](docs/research/who-reads-discovery-files.md):** a week of traffic across 56 domains on Cloudflare, with charts, its queries and its classifier. Bots in Cloudflare's AI categories requested robots.txt and the sitemap, not llms.txt; bots, mostly one search engine's, asked for markdown 12,601 times; directories and audit tools made most agent-card requests; and on sites with an API, some readers of llms.txt and openapi.json also got API answers the same week.
33
33
  - **`readiness-audit`:** the scorecard. It checks a live site against everything the standard can verify from outside, about 30 checks in one pass, and lists what it can't see for you to review.
34
34
 
35
35
  ## Why this one
package/STANDARD.md CHANGED
@@ -86,7 +86,7 @@ What it means:
86
86
 
87
87
  So no single file reaches every agent. Directories want the well-known files, scripts want llms.txt and the API, tools want clean pages and docs, and search indexes want everything in §5.
88
88
  - **A crawler's name is a claim.** Across one owner's 56 domains on Cloudflare, October 16 to 23, 2026, none of about 448k vulnerability probes, most of them hunting `.env` files, came from a bot Cloudflare verified, and one scanner sent 308k requests under about 20 AI-crawler names, GPTBot and ClaudeBot among them ([Bots in AI clothing](https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md), from the sister project agent-security-pranks). The crawlers in the table above are the verified ones; a name in a log line alone doesn't show which requests are theirs (T3).
89
- - **A week across the whole network says the same, at scale.** [Who reads the discovery files](docs/research/who-reads-discovery-files.md) covers 56 domains, October 17 to 24, 2026. Verified AI crawlers read robots.txt (3,644 requests) and the sitemap (2,232, 40% of its reads) far more than llms.txt (49 of 1,406). Verified bots, nearly all Exa's search crawler, asked for markdown 12,601 times. Directories, not AI crawlers, read agent-card.json (2 of 991 requests were verified AI). On the 16 sites with an API, 13% of the addresses that read llms.txt, and 23% of those that read openapi.json, made a successful API call the same week.
89
+ - **A week across the whole network says the same, at scale.** [Who requests agent discovery files?](docs/research/who-reads-discovery-files.md) covers 56 domains, October 17 to 24, 2026. Bots in Cloudflare's legacy AI categories requested robots.txt (3,644 times) and the sitemap (2,232, 40% of its requests) far more than llms.txt (49 of 1,406; 187 counting every verified bot). Verified bots, 92% of them Exa's search crawler, asked for markdown 12,601 times. Directories and audit tools made most agent-card.json requests (556 of 991). On the 16 sites with an API, 13% of the addresses that requested llms.txt, and 23% of those that requested openapi.json, also got a successful API answer that week: overlap, not a measured sequence.
90
90
  - **Check your own traffic before building a next-level item.** botsmatter looked: 2 of its 1,940 Grounds exactly repeat an earlier one, so retries aren't a problem there and N2 isn't worth building.
91
91
 
92
92
  ---
@@ -135,7 +135,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
135
135
  **D3 Server-rendered pages.** Every page's content is in the HTML the server sends, not loaded later by script, with a real logo through the framework's icon convention and a favicon set (missing icons show up as steady 404s). *Why: crawlers, link-preview bots and agents that fetch pages read the HTML, and many don't run JavaScript.* inbed's chat pages once shipped none of their 48 messages in the HTML. The page-level details (titles, canonicals, structured data, previews, headings, links) are W1 to W7 in [§5](#5-search-people-and-answer-engines).
136
136
 
137
137
  **D4 `/llms.txt` and `/llms-full.txt`**: a short map (what this is, how to start, links to every other surface) and the whole corpus in one fetch. Generate both and test them against the sitemap: agents follow the links, so every one must open. The scorecard samples them and warns about missing pages (an endpoint that answers 401 for want of a key is fine; templates like `{id}` are skipped). The file is markdown; serve it as `text/markdown` or `text/plain`, and use the same type wherever you link it.
138
- - *Why required: agents that go on to use the site read it, though the AI crawlers mostly don't.* Across 56 domains in a week of October 2026, verified AI crawlers made 49 of 1,406 llms.txt requests (GPTBot 5, ClaudeBot none) and read robots.txt and the sitemap instead. On the 16 sites with an API, 13% of the addresses that read llms.txt made a successful API call that week, 9% with a user agent that wasn't a browser's ([research](docs/research/who-reads-discovery-files.md)). Google Search ignores llms.txt (it said so in June 2026). At animalhouse, 79 requests in four and a half days came from:
138
+ - *Why required: agents doing tasks use it, though the AI crawlers mostly don't.* Across 56 domains in a week of October 2026, bots in Cloudflare's legacy AI categories made 49 of 1,406 llms.txt requests (GPTBot 5, ClaudeBot none in the sample) and requested robots.txt and the sitemap instead. On the 16 sites with an API, 13% of the addresses that requested llms.txt also got a successful API answer that week, 9% with a user agent that wasn't a browser's ([research](docs/research/who-reads-discovery-files.md)). In a controlled benchmark across 20 documentation sites, markdown with a linked llms.txt cut agents' wasted fetches from 2.23 per task with HTML to 0.11 ([Mintlify](https://www.mintlify.com/blog/llms-txt-agent-benchmark), 2026-07-17). Google Search ignores llms.txt (it said so in June 2026). At animalhouse, 79 requests in four and a half days came from:
139
139
  - agent directories (AgenstryBot, BrickBlueBot, agentprobe, MCPHarbor, AgentTrustBot);
140
140
  - llms.txt directories and validators;
141
141
  - SEO tools and Amazonbot;
@@ -149,7 +149,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
149
149
 
150
150
  **D11 The A2A path is honest.** `/.well-known/agent-card.json` is the most-requested agent path. Serve a valid A2A v1.0 card only if you run an A2A endpoint ([Only if true](#only-if-true)). Otherwise answer a JSON 404 that names your MCP endpoint, OpenAPI document and llms.txt (achurch.ai). A custom document or an "A2A-shaped" card at that path fails. *Why: honesty. A2A §8.3.1 requires every declared interface to be real.*
151
151
  - **The cost:** directory crawlers drive most requests to this path, and some may stop listing a site that answers 404.
152
- - **The pre-1.0 path still gets asked.** Directories request `/.well-known/agent.json` too: 445 times at 19 sites in a week of October 2026, none of which serve it ([research](docs/research/who-reads-discovery-files.md)). Answer it the same way; D7's catch-all does.
152
+ - **The older path still gets asked.** Directories request `/.well-known/agent.json` too: 445 times at 19 sites in a week of October 2026, none of which serve it ([research](docs/research/who-reads-discovery-files.md)). Answer it the same way; D7's catch-all does.
153
153
  - **Roll it out on one site first.** inbed's agent said it would accept the risk. Watch for two weeks whether AgenstryBot and the other directories keep listing it, then change the rest.
154
154
 
155
155
  **D12 No false declarations.** *Why: honesty.*
@@ -192,16 +192,16 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
192
192
  - `service-desc` is the machine-readable description (your OpenAPI). `service-doc` is the human documentation (RFC 8631). Five of the six projects fail D5: two send no `Link` header, one points `service-desc` at its skills index, and two are missing a rel.
193
193
  - Registered relations are `describedby`, `service-desc`, `service-doc`, `service-meta` and `api-catalog`. `ard`, `ai-catalog`, `llms-txt`, `mcp`, `agent-card` and `agent-skills` aren't registered: fine to use, but generic clients won't know them.
194
194
 
195
- **D7 JSON 404s for unknown `/.well-known/*` paths, including the pre-1.0 A2A path `/.well-known/agent.json`, and a JSON 405 on `POST /`** naming the MCP endpoint and the API (achurch.ai, magnifica), with an `Allow` header naming the methods that work (`Allow: GET, HEAD`): RFC 9110 requires it on every 405. An HTML 404 page is noise to an agent. *Why: cheap, and it helps the agents and crawlers that guess. No project passes all of it yet.* One catch-all route can answer D7 and D11 together ([recipe](docs/recipes.md#one-catch-all-for-unknown-well-known-paths-d7-d11)).
195
+ **D7 JSON 404s for unknown `/.well-known/*` paths, including the older A2A path `/.well-known/agent.json` (agent-card.json replaced it by A2A 0.3), and a JSON 405 on `POST /`** naming the MCP endpoint and the API (achurch.ai, magnifica), with an `Allow` header naming the methods that work (`Allow: GET, HEAD`): RFC 9110 requires it on every 405. An HTML 404 page is noise to an agent. *Why: cheap, and it helps the agents and crawlers that guess. No project passes all of it yet.* One catch-all route can answer D7 and D11 together ([recipe](docs/recipes.md#one-catch-all-for-unknown-well-known-paths-d7-d11)).
196
196
 
197
- **D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and agents ask for it with `Accept: text/markdown`. *Why recommended: traffic.* Across 56 domains in a week of October 2026, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler and the rest ShapBot, Exa's user-triggered fetcher and KeenableBot ([research](docs/research/who-reads-discovery-files.md)). That's one search engine more than all the others, so check your own logs: the case rests on who asks you. Many hosts' logs don't keep the `Accept` header (Railway's HTTP logs and Cloudflare's Free-plan analytics don't); a markdown answer at a page URL shows that one was sent. To see it on your own site, log the header in middleware ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)).
197
+ **D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and agents ask for it with `Accept: text/markdown`. *Why recommended: traffic.* Across 56 domains in a week of October 2026, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler and the rest ShapBot, Exa's user-triggered fetcher and KeenableBot ([research](docs/research/who-reads-discovery-files.md)). That's one search engine more than all the others; GoodBarber's logs show the same clients, with Claude Code too ([GoodBarber](https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/), September 2026). Check your own logs: the case rests on who asks you. Many hosts' logs don't keep the `Accept` header (Railway's HTTP logs and Cloudflare's Free-plan analytics don't); a markdown answer at a page URL shows that one was sent. To see it on your own site, log the header in middleware ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)).
198
198
  - Start with the homepage and your docs, not every page. `Accept: text/markdown` and a `.md` URL both return the page as markdown. animalhouse serves llms.txt as the homepage's markdown, and its API reference at `/docs/api.md`.
199
199
  - No CDN plan is needed: a `.md` route and a few lines of middleware do it ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)). Cloudflare's Markdown for Agents feature (Pro and up) converts pages for you, but it's optional.
200
200
  - Send `Vary: Accept` on both representations (RFC 9110 §12.5.5). Otherwise a cache can hand HTML readers the markdown, or the reverse.
201
201
  - Next.js drops a `Vary` set in middleware, proxy or `next.config` on App Router pages, static or dynamic, while it keeps the other headers (tested on 14.2 and 16.3). So `Accept` has to reach the HTML from the edge (a CDN response header rule), or decline that half in writing. Send it from the markdown route yourself (animalhouse.ai).
202
202
  - A page's markdown copy has the page's sections. The homepage's can be llms.txt instead. *Harm prevented:* agents reading the markdown get a different site than people do. The scorecard compares the headings of one page llms.txt links, when that page serves markdown, and a guard test can compare them all (§11).
203
203
  - Make links absolute, and link back to the HTML page as canonical.
204
- - Add `x-markdown-tokens` so an agent can budget before reading. It's Cloudflare's header, with no spec behind it.
204
+ - `x-markdown-tokens` is optional. It was Cloudflare's header, with no spec behind it, and Cloudflare stopped sending it on 2026-10-09: agents count tokens themselves.
205
205
  - Negotiate in front of prerendered pages too, or they ignore it (magnifica).
206
206
  - Whole-corpus indexes help: `/docs/index.md` and `/docs/index.json` (achurch.ai).
207
207
 
@@ -682,7 +682,7 @@ Higher-value features that most of our projects don't have yet. Each one fixes a
682
682
  - **T3 Reviews** (required):
683
683
  - A monthly log review, paged so no window is truncated (platform log commands often return only the newest window): requested well-known paths, 404s, refusals, crawler user agents. `readiness-audit --logs <file>` counts the discovery requests in an exported log by client family.
684
684
  - **Count audits apart from agents.** Audit tools, this one included, are among the largest readers of discovery files: in a week across 56 domains, this audit and its sister made 3,564 discovery requests, more than all the week's llms.txt requests, and other audit tools were the largest readers of ard.json and ai-catalog.json ([research](docs/research/who-reads-discovery-files.md)). `--logs` names them as their own families. *Harm prevented:* mistaking scanners of your setup for agents that use it.
685
- - **Classify traffic before counting it.** Set known junk crawlers aside first, and count a crawler by its verified status, not its name: the CDN's verified-bot flag, the vendor's published addresses, or reverse DNS. Scanners send crawler names: one sent 308k requests under about 20 AI-crawler names in a week (§2), and `--logs` counts requests under crawler names that asked for secret files, which no real crawler does. Count a browser, headless or unidentified client as an agent only when something backs it up, such as the same address calling the API: agent tools often send a browser's user agent (§2), so their user agent alone proves nothing either way. *Harm prevented:* readiness decisions made on noise.
685
+ - **Classify traffic before counting it.** Set known junk crawlers aside first, and count a crawler by its verified status, not its name: the CDN's verified-bot flag, the vendor's published addresses, or reverse DNS. Scanners send crawler names: one sent 308k requests under about 20 AI-crawler names in a week (§2), and `--logs` counts requests under crawler names that asked for secret files, which no real crawler does. Count a browser, headless or unidentified client as an agent only when something backs it up, such as an API key it registered: agent tools often send a browser's user agent (§2), so the user agent alone proves nothing either way, and a successful API call doesn't either, since a site's own pages call its API too. *Harm prevented:* readiness decisions made on noise.
686
686
  - After each deploy, run the scorecard, lint the OpenAPI spec, and diff generated files against the previous production output. About 24 hours later, review the logs and database query performance.
687
687
  - **T5 The agent usability test** (required).
688
688
  - **Why:** it's the only check that measures what the whole standard is for, and without it the scorecard becomes the target.
@@ -829,7 +829,7 @@ Level: **R** required where it applies, **Rec** recommended, **N** next level. W
829
829
  | D5 | `Link` headers: service-desc, service-doc, describedby, api-catalog | Rec | discovery | score |
830
830
  | D6 | security.txt with Expires once | R | hygiene | score |
831
831
  | D7 | JSON 404s for unknown well-known paths (`agent.json` included), JSON 405 with `Allow` on `POST /` | Rec | agents | score |
832
- | D8 | Markdown for agents with `Vary: Accept` and x-markdown-tokens | Rec | traffic | score |
832
+ | D8 | Markdown for agents with `Vary: Accept` | Rec | traffic | score |
833
833
  | D9 | AI catalog at ard.json and ai-catalog.json, with `rel` links | Rec | discovery | score |
834
834
  | D10 | One DNS AID record naming a service you run | Rec | discovery | score |
835
835
  | D11 | The A2A path: a valid card or a JSON 404 | R | honesty | score |
@@ -911,7 +911,7 @@ Status as of 2026-10-05. Several are drafts, so re-check before relying on a det
911
911
  | robots.txt | RFC 9309 | https://www.rfc-editor.org/rfc/rfc9309.html |
912
912
  | Content Signals | Cloudflare policy, 2025-09-24 | https://contentsignals.org/ |
913
913
  | AIPREF (`Content-Usage`) | IETF working-group drafts | https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/ · https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/ |
914
- | Markdown for Agents, `x-markdown-tokens` | Cloudflare feature | https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ |
914
+ | Markdown for Agents | Cloudflare feature; `x-markdown-tokens` removed 2026-10-09 | https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ · https://developers.cloudflare.com/changelog/post/2026-10-09-markdown-for-agents-in-process-conversion/ |
915
915
  | `Vary` | RFC 9110 §12.5.5 | https://www.rfc-editor.org/rfc/rfc9110.html#section-12.5.5 |
916
916
  | Agent Skills discovery index | Cloudflare RFC v0.2.0, draft | https://github.com/cloudflare/agent-skills-discovery-rfc |
917
917
  | SKILL.md | agentskills.io specification | https://agentskills.io/specification |
@@ -951,6 +951,9 @@ Status as of 2026-10-05. Several are drafts, so re-check before relying on a det
951
951
  | Google rich results | Search Central search gallery (updated 2026-10-16), read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data/search-gallery |
952
952
  | Testing structured data | Search Central: the Rich Results Test and the Schema Markup Validator, read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data · https://search.google.com/test/rich-results · https://validator.schema.org/ |
953
953
  | Rich result reports, URL Inspection API | Search Console Help and API reference, read 2026-10-21 | https://support.google.com/webmasters/answer/7552505 · https://developers.google.com/webmaster-tools/v1/urlInspection.index/UrlInspectionResult |
954
+ | Discovery-file requests ("Who requests agent discovery files?") | this repo's research, 56 domains on Cloudflare, 2026-10-17 to 10-24, corrected 2026-10-24 | docs/research/who-reads-discovery-files.md |
955
+ | Negotiated markdown requests | GoodBarber's own logs, 2026-09-04 to 09-18, read 2026-10-24 | https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/ |
956
+ | Markdown and llms.txt in agent tasks | Mintlify benchmark, 2026-07-17, read 2026-10-24 | https://www.mintlify.com/blog/llms-txt-agent-benchmark |
954
957
  | Crawler impersonation ("Bots in AI clothing") | agent-security-pranks research, 56 domains on Cloudflare, 2026-10-16 to 10-23, read 2026-10-24 | https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md |
955
958
  | 405 and `Allow` | RFC 9110 §15.5.6 | https://www.rfc-editor.org/rfc/rfc9110.html#section-15.5.6 |
956
959
  | Headings | W3C WAI tutorial, read 2026-10-21 | https://www.w3.org/WAI/tutorials/page-structure/headings/ |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "readiness-audit",
3
- "version": "2.2.0",
3
+ "version": "2.2.1",
4
4
  "description": "Score any site's readiness for AI agents and search (GEO, AEO, AI SEO): llms.txt, robots.txt AI crawler rules, structured data, link previews, OpenAPI, MCP, agent skills and A2A. Read-only CLI, no account.",
5
5
  "type": "module",
6
6
  "bin": {
package/src/audit.mjs CHANGED
@@ -11,7 +11,7 @@ import { isIP } from 'node:net';
11
11
  import { request } from './http.mjs';
12
12
  import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
13
13
 
14
- export const VERSION = '2.2.0';
14
+ export const VERSION = '2.2.1';
15
15
  export const REPO = 'https://github.com/geeks-accelerator/agent-and-search-readiness';
16
16
  const UA = `readiness-audit/${VERSION} (+${REPO})`;
17
17
  const AID_PROTOCOLS = ['mcp', 'a2a', 'openapi', 'grpc', 'graphql', 'websocket', 'local', 'zeroconf', 'ucp'];
@@ -326,8 +326,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
326
326
  if (!/text\/markdown/.test(homeMd.type)) miss(id, name, level, `Accept: text/markdown on / returns ${ct(homeMd)}`);
327
327
  else if (!varyAccept(homeMd) || !varyAccept(home)) miss(id, name, level, `Vary: Accept missing on the ${!varyAccept(home) ? 'HTML' : 'markdown'} response`);
328
328
  else if (copy.gone.length) miss(id, name, level, `the markdown copy of ${copy.path} lacks ${copy.gone.length} section${copy.gone.length === 1 ? '' : 's'} the page has: ${list(copy.gone.map((h) => `"${h}"`))}`);
329
- else if (!tokens) miss(id, name, level, 'markdown and Vary are right; add x-markdown-tokens');
330
- else pass(id, name, level, `markdown with Vary: Accept on both, x-markdown-tokens ${tokens}${copy.path ? `; ${copy.path}'s markdown copy has every section` : '; no page llms.txt links serves markdown, so no copy was compared'}`);
329
+ else pass(id, name, level, `markdown with Vary: Accept on both${tokens ? `, x-markdown-tokens ${tokens}` : ''}${copy.path ? `; ${copy.path}'s markdown copy has every section` : '; no page llms.txt links serves markdown, so no copy was compared'}`);
331
330
  }
332
331
 
333
332
  // ---- D9 AI catalog (ARD) ----
package/src/logs.mjs CHANGED
@@ -9,7 +9,7 @@ export const FAMILIES = [
9
9
  ['This audit (readiness-audit)', /^readiness-audit\//i],
10
10
  // Other audit tools read discovery files at least as often as agents do (docs/research/who-reads-discovery-files.md).
11
11
  ['Other audit tools', /security-pranks-audit|isitagentready|agentprobe|agent-tools\.cloud|semrushbot|ahrefsbot|lumar|seojuice|sitebulb|screaming frog/i],
12
- ['Agent and MCP directories', /agenstrybot|brickbluebot|agentprobe|mcpharbor|agenttrustbot|taifoonharvester|mcphub-probe|invokerankbot|proofbench|thepluginstorebot|protogrid-probe|glidemcpindex|agent-tools\.cloud/i],
12
+ ['Agent and MCP directories', /agenstrybot|brickbluebot|mcpharbor|agenttrustbot|taifoonharvester|mcphub-probe|invokerankbot|proofbench|thepluginstorebot|protogrid-probe|glidemcpindex|agent-tools\.cloud/i],
13
13
  ['llms.txt directories', /llms\.?txt/i],
14
14
  ['Assistants fetching for a person', /chatgpt-user|claude-user|perplexity-user|meta-externalfetcher/i],
15
15
  ['AI crawlers', new RegExp([...AI_BOTS, 'meta-webindexer', 'meta-externalagent', 'bytespider', 'ccbot'].map((b) => b.replace(/[-.]/g, '\\$&')).join('|'), 'i')],
@@ -25,8 +25,9 @@ export function family(ua) {
25
25
  return FAMILIES.find(([, re]) => re.test(ua))?.[0] ?? 'Unidentified';
26
26
  }
27
27
 
28
- // Families whose user agent can't show an agent: count them as one only when the same address called the API.
29
- const UNVERIFIED = new Set(['Browsers', 'Headless browsers', 'Unidentified', 'No user agent']);
28
+ // Families whose user agent says nothing about what the client is. A successful API answer to one of them is
29
+ // activity, not proof of an agent: a site's own pages call its API too.
30
+ const BROWSER_LIKE = new Set(['Browsers', 'Headless browsers', 'Unidentified', 'No user agent']);
30
31
 
31
32
  // Families named after a crawler or directory. A name is a claim: across one owner's 56 domains in a week of
32
33
  // October 2026, none of 448k probes came from a verified bot, and one scanner used about 20 AI-crawler names
@@ -79,7 +80,8 @@ export function bucket(path) {
79
80
  if (/\.md$/i.test(p)) return 'markdown pages (*.md)';
80
81
  return null;
81
82
  }
82
- const isApiCall = (r) => r.path.startsWith('/api/') || (r.path === '/mcp' && r.method === 'POST');
83
+ // A call that worked: a 2xx from /api/* or a POST to /mcp. A 404 or a 5xx there is a guess or a probe, not use.
84
+ const isApiSuccess = (r) => (r.path.startsWith('/api/') || (r.path === '/mcp' && r.method === 'POST')) && r.status >= 200 && r.status < 300;
83
85
 
84
86
  /** Tally lines one at a time; result() gives the report. */
85
87
  export function logTally() {
@@ -87,7 +89,7 @@ export function logTally() {
87
89
  const hits = new Map(); // bucket -> Map(family -> count)
88
90
  const readers = new Map(); // bucket -> Set(addr)
89
91
  const apiAddrs = new Set();
90
- const unverifiedAddrs = new Set();
92
+ const browserLikeAddrs = new Set();
91
93
  const impostors = new Map(); // crawler family -> requests for secret files under its name
92
94
  return {
93
95
  add(line) {
@@ -96,7 +98,7 @@ export function logTally() {
96
98
  if (!r) return;
97
99
  parsed++;
98
100
  const fam = family(r.ua);
99
- if (isApiCall(r)) { if (r.addr) apiAddrs.add(r.addr); if (UNVERIFIED.has(fam) && r.addr) unverifiedAddrs.add(r.addr); }
101
+ if (isApiSuccess(r) && r.addr) { apiAddrs.add(r.addr); if (BROWSER_LIKE.has(fam)) browserLikeAddrs.add(r.addr); }
100
102
  if (CRAWLER_FAMILIES.has(fam) && isSecretProbe(r.path)) impostors.set(fam, (impostors.get(fam) ?? 0) + 1);
101
103
  const b = bucket(r.path);
102
104
  if (!b) return;
@@ -112,11 +114,11 @@ export function logTally() {
112
114
  requests: [...fams.values()].reduce((a, b) => a + b, 0),
113
115
  families: Object.fromEntries([...fams].sort((a, b) => b[1] - a[1])),
114
116
  addresses: addrs.size,
115
- also_called_api: [...addrs].filter((a) => apiAddrs.has(a)).length,
117
+ also_got_api_answer: [...addrs].filter((a) => apiAddrs.has(a)).length,
116
118
  };
117
119
  }).sort((a, b) => b.requests - a.requests);
118
120
  const crawlerNames = { requests: [...impostors.values()].reduce((a, b) => a + b, 0), families: Object.fromEntries([...impostors].sort((a, b) => b[1] - a[1])) };
119
- return { lines, parsed, api_addresses: apiAddrs.size, unverified_api_addresses: unverifiedAddrs.size, crawler_names_on_secret_probes: crawlerNames, paths };
121
+ return { lines, parsed, api_answer_addresses: apiAddrs.size, browser_like_api_answer_addresses: browserLikeAddrs.size, crawler_names_on_secret_probes: crawlerNames, paths };
120
122
  },
121
123
  };
122
124
  }
@@ -126,11 +128,11 @@ export function formatLogs(rep) {
126
128
  const out = [`Read ${rep.parsed} of ${rep.lines} log lines (${rep.lines - rep.parsed} not understood). Families are the names clients send, not verified.`, ''];
127
129
  if (!rep.paths.length) out.push('No requests for discovery files.');
128
130
  for (const p of rep.paths) {
129
- out.push(`${p.path} ${p.requests} request${p.requests === 1 ? '' : 's'}${p.addresses ? `, ${p.addresses} address${p.addresses === 1 ? '' : 'es'}, ${p.also_called_api} of them also called the API` : ''}`);
131
+ out.push(`${p.path} ${p.requests} request${p.requests === 1 ? '' : 's'}${p.addresses ? `, ${p.addresses} address${p.addresses === 1 ? '' : 'es'}, ${p.also_got_api_answer} of them also got a successful API answer` : ''}`);
130
132
  for (const [fam, n] of Object.entries(p.families)) out.push(` ${String(n).padStart(6)} ${fam}`);
131
133
  }
132
- out.push('', `API calls came from ${rep.api_addresses} address${rep.api_addresses === 1 ? '' : 'es'}. ${rep.unverified_api_addresses} of them sent a browser, headless or missing user agent: count those as agents, since they called the API.`);
133
- out.push('Browsers, headless browsers and unidentified clients above count as agents only where they also called the API (STANDARD.md, T3).');
134
+ out.push('', `Successful API answers (a 2xx from /api/* or a POST to /mcp) went to ${rep.api_answer_addresses} address${rep.api_answer_addresses === 1 ? '' : 'es'}, ${rep.browser_like_api_answer_addresses} of them with a browser, headless or missing user agent.`);
135
+ out.push('An address that read a file and also got an API answer shows activity in the same log, not order, and not that the client is an agent: a site\'s own pages call its API too (STANDARD.md, T3).');
134
136
  const named = rep.crawler_names_on_secret_probes;
135
137
  if (named.requests) out.push('', `${named.requests} request${named.requests === 1 ? '' : 's'} under crawler names asked for secret files (.env, .git, keys, dumps): ${Object.entries(named.families).map(([f, n]) => `${f} ${n}`).join(', ')}. Real crawlers don't ask for these, so treat those names as claims: count a crawler by the CDN's verified flag, the vendor's published addresses or reverse DNS, not its name (T3).`);
136
138
  else out.push('', 'No request under a crawler name asked for secret files.');