readiness-audit 2.2.0 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +44 -13
- package/STANDARD.md +13 -10
- package/bin/readiness-audit.mjs +3 -4
- package/package.json +2 -2
- package/src/audit.mjs +3 -4
- package/src/helpers.mjs +20 -0
- package/src/logs.mjs +13 -11
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.3.0 (2026-10-24)
|
|
4
|
+
|
|
5
|
+
From the sister project's agent, after it reviewed this one. No scoring changes.
|
|
6
|
+
|
|
7
|
+
- **Node 22 and 24 work again,** each at its current security release: the CLI accepts 22.23.4, 24.22.1 or 26.12.1, or later on the same line, and refuses older releases and end-of-life or odd-numbered lines, with advice on updating. 2.1.0 to 2.2.1 needed Node 26, so `npx readiness-audit` failed on the many machines and CI images that run 22 or 24. The audit's own defenses (refusing Brotli bodies, capping body sizes) don't depend on the Node line, and each line's security release has the October 21 fixes. CI tests all three lines with `check-latest`, and a test keeps `engines`, the docs and the CI matrix in step with `NODE_LINES`.
|
|
8
|
+
- **README:** a Quick start with fresh output and the one-line prompt to adopt the standard; "Why this exists", with the evidence up front; a "Safe to run" section for the mechanics; Lighthouse and Google's Rich Results Test in "How it fits"; and a "Next step".
|
|
9
|
+
|
|
10
|
+
## 2.2.1 (2026-10-24)
|
|
11
|
+
|
|
12
|
+
Corrections after an outside review of 2.2.0 and its research. No new checks; D8 is easier to pass.
|
|
13
|
+
|
|
14
|
+
- **D8 no longer requires `x-markdown-tokens`.** Cloudflare stopped sending the header on 2026-10-09, so a site using Cloudflare's own Markdown for Agents couldn't pass. It's optional, and shown when present.
|
|
15
|
+
- **`--logs` counts only successful API answers** (a 2xx from `/api/*` or a POST to `/mcp`). A 404 or a 5xx no longer counts as a call. It no longer tells you to count browser, headless or unidentified addresses as agents because they called the API: a site's own pages call its API too. The JSON fields are `also_got_api_answer`, `api_answer_addresses` and `browser_like_api_answer_addresses`. agentprobe is counted once, as an audit tool.
|
|
16
|
+
- **The research is corrected and reproducible.** [Who requests agent discovery files?](docs/research/who-reads-discovery-files.md) now says its window overlaps the sister study's for 161 of 168 hours; names Cloudflare's legacy AI categories, which leave out ExaSearchBot; calls the API measure same-week address overlap, not a sequence; says which sites were tested to show the markdown answers came from requests that asked; corrects three "mostly" claims; states Exa's 92% share of markdown requests; cites GoodBarber's earlier count and Mintlify's benchmark; and lists every correction. [method.md](docs/research/who-reads-discovery-files/method.md) publishes the queries and the classifier.
|
|
17
|
+
- **STANDARD.md:** §2, D4, D8, D7, D11 and T3 use the corrected wording; D4 cites Mintlify's benchmark and D8 GoodBarber's logs; T3 says a successful API call doesn't make a client an agent; §16 cites both and the research.
|
|
18
|
+
|
|
3
19
|
## 2.2.0 (2026-10-24)
|
|
4
20
|
|
|
5
21
|
From a review of the sister project, [agent-security-pranks](https://github.com/geeks-accelerator/agent-security-pranks), and its research. Two new warnings (D7's `Allow`, and D8 at the recommended level), no new failures. A warning lowers the recommended score until fixed, but doesn't change the exit code.
|
package/README.md
CHANGED
|
@@ -6,35 +6,57 @@
|
|
|
6
6
|
|
|
7
7
|
**Score any site's readiness for AI agents and search:** llms.txt, robots.txt rules for AI crawlers, the sitemap, structured data, link previews, OpenAPI, MCP servers, agent skills and the A2A path. A standard built from production request logs, and a command-line audit that checks a live site against it. Read-only, no account, no API key.
|
|
8
8
|
|
|
9
|
+
## Quick start
|
|
10
|
+
|
|
9
11
|
```
|
|
10
12
|
npx readiness-audit example.com
|
|
11
13
|
```
|
|
12
14
|
|
|
15
|
+
It needs Node 22.23.4, 24.22.1 or 26.12.1, or later on the same line (each supported line's current security release), and has no dependencies.
|
|
16
|
+
|
|
13
17
|
```
|
|
14
|
-
Agent and search readiness: animalhouse.ai (readiness-audit v2.
|
|
18
|
+
Agent and search readiness: animalhouse.ai (readiness-audit v2.3.0)
|
|
15
19
|
|
|
16
20
|
Required
|
|
17
21
|
PASS D1 robots.txt 1 group with consistent rules, Sitemap listed
|
|
18
22
|
PASS D4 llms.txt and llms-full.txt llms.txt 16.2 KB, llms-full.txt 65 KB; 10 sampled links resolve
|
|
19
23
|
PASS D12 No false declarations no OAuth, A2A, plugin or server-card claims without the service
|
|
20
24
|
PASS W1 Titles and descriptions 27 sampled pages (18 static, 3 templates), all with their own
|
|
21
|
-
WARN W5 Link previews og:image size or alt text not declared on /hall, /about, +11 more
|
|
22
25
|
WARN W6 Page structure heading levels skipped on /agent/[name] (h1 to h3)
|
|
23
|
-
PASS A1 OpenAPI 3.1 OpenAPI 3.1.0; all 17 operations and their request fields described
|
|
24
26
|
PASS M4 Hosted MCP edge both eras, JSON for every Accept, browsers redirected
|
|
25
27
|
...
|
|
26
|
-
|
|
28
|
+
Recommended
|
|
29
|
+
PASS D7 JSON 404s and a JSON 405 on POST / JSON 404 for unknown well-known paths, JSON 405 for POST / (Allow: GET, HEAD, OPTIONS)
|
|
30
|
+
WARN D8 Markdown for agents Vary: Accept missing on the HTML response
|
|
31
|
+
...
|
|
32
|
+
Score on 2026-10-24: required 17 of 19, recommended 10 of 11, next level 1 of 2
|
|
27
33
|
```
|
|
28
34
|
|
|
29
|
-
|
|
35
|
+
To switch a project over, give its coding agent one line:
|
|
36
|
+
|
|
37
|
+
> Adopt the Agent and Search Readiness Standard in this repo by following https://github.com/geeks-accelerator/agent-and-search-readiness/blob/main/docs/adopting.md
|
|
38
|
+
|
|
39
|
+
The agent reads the page when it runs. To have it follow the text you reviewed, put the latest release tag in place of `main` (`blob/v<version>/docs/adopting.md`, from the [releases page](https://github.com/geeks-accelerator/agent-and-search-readiness/releases)).
|
|
40
|
+
|
|
41
|
+
## Why this exists
|
|
42
|
+
|
|
43
|
+
What agents and crawlers actually ask a site for, from production logs:
|
|
44
|
+
|
|
45
|
+
- **The AI crawlers read robots.txt and the sitemap, not llms.txt.** In a week across 56 domains on Cloudflare, bots in Cloudflare's AI categories requested robots.txt 3,644 times and the sitemap 2,232 times, and llms.txt 49 times. llms.txt still matters to other readers: directories, tools, scripts and search bots requested it on every site ([research]({R})).
|
|
46
|
+
- **Directories crawl the well-known files; working agents call the API.** At animalhouse.ai, 4 of the 28 addresses that read llms.txt called the API ([§2](STANDARD.md#2-what-the-traffic-says)). Across the 16 sites with an API, 13% of llms.txt readers also got a successful API answer that week.
|
|
47
|
+
- **Bots ask for markdown.** 12,601 requests asked for markdown at ordinary page URLs that week, 92% of them from one search engine's crawler (Exa's).
|
|
48
|
+
- **A hosted MCP endpoint gets found within hours.** Seven MCP directory crawlers connected to animalhouse.ai's in its first two hours, and 14 browser visits reached it in the same two hours ([§2](STANDARD.md#2-what-the-traffic-says)).
|
|
49
|
+
- **No single file reaches every kind of agent.** Directories want the well-known files, scripts want llms.txt and the API, agent tools want clean pages, and search indexes want everything a search engine wants. The standard covers each, at a level its evidence supports.
|
|
50
|
+
|
|
51
|
+
## This repo holds
|
|
30
52
|
|
|
31
53
|
- **[STANDARD.md](STANDARD.md):** what to build, item by item, each with an ID, a level and the reason it's at that level. It covers discovery files, search (SEO, and the AI answer engines that GEO and AEO are about), the API, MCP servers, skills, engagement, measurement and tests.
|
|
32
|
-
- **[Who
|
|
54
|
+
- **[Who requests agent discovery files?]({R}):** the week across 56 domains, with charts, its queries and its classifier.
|
|
33
55
|
- **`readiness-audit`:** the scorecard. It checks a live site against everything the standard can verify from outside, about 30 checks in one pass, and lists what it can't see for you to review.
|
|
34
56
|
|
|
35
57
|
## Why this one
|
|
36
58
|
|
|
37
|
-
- **Built from evidence.** Each item says why it's there: production request logs
|
|
59
|
+
- **Built from evidence.** Each item says why it's there: production request logs, the specs, and Google's and Bing's own documentation, cited with dates. Items that only scanners look for are recommended at most.
|
|
38
60
|
- **It checks that declarations are true.** An A2A card that isn't a valid A2A card fails D11; OAuth metadata with no token endpoint, or a catalog entry typed as something it isn't, fails D12. Declining an item in writing is fine; faking a pass isn't.
|
|
39
61
|
- **It fits the site.** The API checks run only when a site has or declares an API, and the MCP checks only when one answers. Pages are sampled by type: every static page, and three pages of each generated template.
|
|
40
62
|
- **Before and after you ship.** `--base http://localhost:3000` scores a local or preview build; `--compare` lists what changed since the last run; the exit code gates a deploy.
|
|
@@ -45,14 +67,14 @@ This repo holds:
|
|
|
45
67
|
| Tool | What it does | Use it with this |
|
|
46
68
|
|---|---|---|
|
|
47
69
|
| [Is Your Site Agent-Ready?](https://isitagentready.com/) (Cloudflare) | A web scan for llms.txt, MCP, agent skills and other agent-facing standards | For a quick look in the browser. This runs in your terminal and CI, scores local builds, and checks search as well as agents |
|
|
70
|
+
| [Lighthouse](https://developer.chrome.com/docs/lighthouse/overview) | Audits one page's performance, accessibility, best practices and SEO in a browser | Use it page by page, for Core Web Vitals (W8). This checks a site's discovery files and samples its pages |
|
|
71
|
+
| [Rich Results Test](https://search.google.com/test/rich-results) (Google) | Checks a page's structured data against Google's rich result rules | This lists which pages to run through it, with a link for each (W4) |
|
|
48
72
|
| llms.txt validators | Check that one file is well formed | This checks llms.txt and samples its links, as one of about 30 checks |
|
|
49
73
|
| [security-pranks-audit](https://github.com/geeks-accelerator/agent-security-pranks) | The sister project: checks that a site answers `.env`, `.git` and other secret-hunting probes safely | This welcomes agents and checks what they read; that checks what the scanners get. Same shape, run both |
|
|
50
74
|
| [specification.website](https://specification.website) | A sourced website specification: HTML, accessibility, security, SEO, agent readiness | A spec for the whole site, accessibility and security included. This one focuses on agents and search, and comes with a scorer |
|
|
51
75
|
|
|
52
76
|
## Score a site
|
|
53
77
|
|
|
54
|
-
It needs Node 26 or later, with current security releases, and has no dependencies. It only reads: GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`). It keeps to about six requests at a time, and it won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect. It asks for gzip and decodes it itself; a body in an encoding it didn't ask for, such as Brotli, is reported rather than read, so a site can't reach a decoder bug on the machine running the audit. It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy.
|
|
55
|
-
|
|
56
78
|
```
|
|
57
79
|
readiness-audit <domain> [--mcp /path] [--no-mcp] [--json] [--compare previous] [--base url]
|
|
58
80
|
readiness-audit --matrix <domain> <domain> ... [--recorded results.json]
|
|
@@ -70,11 +92,16 @@ readiness-audit --logs <access-log> [--json]
|
|
|
70
92
|
| `--recorded file` | with `--matrix`, add your recorded T5 and T6 results (format: [examples/recorded.json](examples/recorded.json)) |
|
|
71
93
|
| `--logs file` | count requests for the discovery files in an exported access log (combined log format, or JSON lines from Cloudflare, Vercel or Railway), by client family. It prints counts, never addresses |
|
|
72
94
|
|
|
73
|
-
The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
|
|
95
|
+
It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy. The API checks run when `/openapi.json` or `/api` answers, or when the site declares an API (a `service-desc` link, an API catalog, an API link in llms.txt). A declared API that doesn't answer fails them. A site with neither skips them, the way a site with no hosted MCP endpoint skips the MCP checks.
|
|
74
96
|
|
|
75
|
-
From a clone, run `node bin/readiness-audit.mjs example.com`.
|
|
97
|
+
To check fixes before they ship, score a local build: `npx readiness-audit@2 example.com --base http://localhost:3000`. The site's own URLs (from its sitemap, llms.txt, skills and catalog) are requested from the base and reported under the domain. From a clone, run `node bin/readiness-audit.mjs example.com`.
|
|
76
98
|
|
|
77
|
-
|
|
99
|
+
### Safe to run
|
|
100
|
+
|
|
101
|
+
- **It only reads:** GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`), about six at a time.
|
|
102
|
+
- **It stays on public addresses.** It won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect. `--base` may be local, because you chose it.
|
|
103
|
+
- **A hostile site can't reach a decoder bug.** It asks for gzip and decodes it itself; a body in an encoding it didn't ask for, such as Brotli, is reported rather than read, and every body is capped at 16 MB.
|
|
104
|
+
- **Run it only against sites you run,** or with the owner's permission. [SECURITY.md](SECURITY.md) has the details and how to report a problem.
|
|
78
105
|
|
|
79
106
|
## What it checks, and what it can't
|
|
80
107
|
|
|
@@ -96,7 +123,7 @@ It can't see whether `next_steps` guide well, whether input is forgiving, whethe
|
|
|
96
123
|
|
|
97
124
|
Point to [STANDARD.md](STANDARD.md) from your repo instead of copying it (§14 has a short stub), and pin the scorecard to a major version: `npx readiness-audit@2 <domain>`. To run it on a schedule in GitHub Actions, see [examples/github-action.yml](examples/github-action.yml).
|
|
98
125
|
|
|
99
|
-
To switch a project over,
|
|
126
|
+
To switch a project over, give its coding agent the one-line prompt in [Quick start](#quick-start). [docs/adopting.md](docs/adopting.md) has the step-by-step instructions it follows, with lessons from the first projects that did it.
|
|
100
127
|
|
|
101
128
|
## Where it comes from
|
|
102
129
|
|
|
@@ -104,6 +131,10 @@ Six projects wrote down what they do for agents and search: [animalhouse.ai](htt
|
|
|
104
131
|
|
|
105
132
|
An item is in the standard when most projects should do it, not because one project happens to. Specs that are still drafts are marked as such, with dates. If you find something wrong or out of date, open an issue or a pull request.
|
|
106
133
|
|
|
134
|
+
## Next step
|
|
135
|
+
|
|
136
|
+
Score your site with `npx readiness-audit your-domain`, read what it reports against [STANDARD.md](STANDARD.md), and when you're ready to switch a project over, give its coding agent the one-line prompt in [Quick start](#quick-start).
|
|
137
|
+
|
|
107
138
|
## Contributing
|
|
108
139
|
|
|
109
140
|
See [CONTRIBUTING.md](CONTRIBUTING.md). To report a security problem, see [SECURITY.md](SECURITY.md).
|
package/STANDARD.md
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
**Last verified:** 2026-10-05
|
|
6
6
|
|
|
7
7
|
> **Start here**
|
|
8
|
-
> 1. Score your site: `npx readiness-audit <domain>`. It needs Node 26 or later
|
|
8
|
+
> 1. Score your site: `npx readiness-audit <domain>`. It needs Node 22.23.4, 24.22.1 or 26.12.1, or later on the same line, and no dependencies.
|
|
9
9
|
> 2. Check [§3](#3-decide-what-applies) for the items that apply to your project.
|
|
10
10
|
> 3. Fix required failures first. Each line the scorecard prints names an ID and the reason.
|
|
11
11
|
> 4. Then the recommended items. Build a next-level item when your own traffic shows the problem it solves.
|
|
@@ -86,7 +86,7 @@ What it means:
|
|
|
86
86
|
|
|
87
87
|
So no single file reaches every agent. Directories want the well-known files, scripts want llms.txt and the API, tools want clean pages and docs, and search indexes want everything in §5.
|
|
88
88
|
- **A crawler's name is a claim.** Across one owner's 56 domains on Cloudflare, October 16 to 23, 2026, none of about 448k vulnerability probes, most of them hunting `.env` files, came from a bot Cloudflare verified, and one scanner sent 308k requests under about 20 AI-crawler names, GPTBot and ClaudeBot among them ([Bots in AI clothing](https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md), from the sister project agent-security-pranks). The crawlers in the table above are the verified ones; a name in a log line alone doesn't show which requests are theirs (T3).
|
|
89
|
-
- **A week across the whole network says the same, at scale.** [Who
|
|
89
|
+
- **A week across the whole network says the same, at scale.** [Who requests agent discovery files?](docs/research/who-reads-discovery-files.md) covers 56 domains, October 17 to 24, 2026. Bots in Cloudflare's legacy AI categories requested robots.txt (3,644 times) and the sitemap (2,232, 40% of its requests) far more than llms.txt (49 of 1,406; 187 counting every verified bot). Verified bots, 92% of them Exa's search crawler, asked for markdown 12,601 times. Directories and audit tools made most agent-card.json requests (556 of 991). On the 16 sites with an API, 13% of the addresses that requested llms.txt, and 23% of those that requested openapi.json, also got a successful API answer that week: overlap, not a measured sequence.
|
|
90
90
|
- **Check your own traffic before building a next-level item.** botsmatter looked: 2 of its 1,940 Grounds exactly repeat an earlier one, so retries aren't a problem there and N2 isn't worth building.
|
|
91
91
|
|
|
92
92
|
---
|
|
@@ -135,7 +135,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
135
135
|
**D3 Server-rendered pages.** Every page's content is in the HTML the server sends, not loaded later by script, with a real logo through the framework's icon convention and a favicon set (missing icons show up as steady 404s). *Why: crawlers, link-preview bots and agents that fetch pages read the HTML, and many don't run JavaScript.* inbed's chat pages once shipped none of their 48 messages in the HTML. The page-level details (titles, canonicals, structured data, previews, headings, links) are W1 to W7 in [§5](#5-search-people-and-answer-engines).
|
|
136
136
|
|
|
137
137
|
**D4 `/llms.txt` and `/llms-full.txt`**: a short map (what this is, how to start, links to every other surface) and the whole corpus in one fetch. Generate both and test them against the sitemap: agents follow the links, so every one must open. The scorecard samples them and warns about missing pages (an endpoint that answers 401 for want of a key is fine; templates like `{id}` are skipped). The file is markdown; serve it as `text/markdown` or `text/plain`, and use the same type wherever you link it.
|
|
138
|
-
- *Why required: agents
|
|
138
|
+
- *Why required: agents doing tasks use it, though the AI crawlers mostly don't.* Across 56 domains in a week of October 2026, bots in Cloudflare's legacy AI categories made 49 of 1,406 llms.txt requests (GPTBot 5, ClaudeBot none in the sample) and requested robots.txt and the sitemap instead. On the 16 sites with an API, 13% of the addresses that requested llms.txt also got a successful API answer that week, 9% with a user agent that wasn't a browser's ([research](docs/research/who-reads-discovery-files.md)). In a controlled benchmark across 20 documentation sites, markdown with a linked llms.txt cut agents' wasted fetches from 2.23 per task with HTML to 0.11 ([Mintlify](https://www.mintlify.com/blog/llms-txt-agent-benchmark), 2026-07-17). Google Search ignores llms.txt (it said so in June 2026). At animalhouse, 79 requests in four and a half days came from:
|
|
139
139
|
- agent directories (AgenstryBot, BrickBlueBot, agentprobe, MCPHarbor, AgentTrustBot);
|
|
140
140
|
- llms.txt directories and validators;
|
|
141
141
|
- SEO tools and Amazonbot;
|
|
@@ -149,7 +149,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
149
149
|
|
|
150
150
|
**D11 The A2A path is honest.** `/.well-known/agent-card.json` is the most-requested agent path. Serve a valid A2A v1.0 card only if you run an A2A endpoint ([Only if true](#only-if-true)). Otherwise answer a JSON 404 that names your MCP endpoint, OpenAPI document and llms.txt (achurch.ai). A custom document or an "A2A-shaped" card at that path fails. *Why: honesty. A2A §8.3.1 requires every declared interface to be real.*
|
|
151
151
|
- **The cost:** directory crawlers drive most requests to this path, and some may stop listing a site that answers 404.
|
|
152
|
-
- **The
|
|
152
|
+
- **The older path still gets asked.** Directories request `/.well-known/agent.json` too: 445 times at 19 sites in a week of October 2026, none of which serve it ([research](docs/research/who-reads-discovery-files.md)). Answer it the same way; D7's catch-all does.
|
|
153
153
|
- **Roll it out on one site first.** inbed's agent said it would accept the risk. Watch for two weeks whether AgenstryBot and the other directories keep listing it, then change the rest.
|
|
154
154
|
|
|
155
155
|
**D12 No false declarations.** *Why: honesty.*
|
|
@@ -192,16 +192,16 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
192
192
|
- `service-desc` is the machine-readable description (your OpenAPI). `service-doc` is the human documentation (RFC 8631). Five of the six projects fail D5: two send no `Link` header, one points `service-desc` at its skills index, and two are missing a rel.
|
|
193
193
|
- Registered relations are `describedby`, `service-desc`, `service-doc`, `service-meta` and `api-catalog`. `ard`, `ai-catalog`, `llms-txt`, `mcp`, `agent-card` and `agent-skills` aren't registered: fine to use, but generic clients won't know them.
|
|
194
194
|
|
|
195
|
-
**D7 JSON 404s for unknown `/.well-known/*` paths, including the
|
|
195
|
+
**D7 JSON 404s for unknown `/.well-known/*` paths, including the older A2A path `/.well-known/agent.json` (agent-card.json replaced it by A2A 0.3), and a JSON 405 on `POST /`** naming the MCP endpoint and the API (achurch.ai, magnifica), with an `Allow` header naming the methods that work (`Allow: GET, HEAD`): RFC 9110 requires it on every 405. An HTML 404 page is noise to an agent. *Why: cheap, and it helps the agents and crawlers that guess. No project passes all of it yet.* One catch-all route can answer D7 and D11 together ([recipe](docs/recipes.md#one-catch-all-for-unknown-well-known-paths-d7-d11)).
|
|
196
196
|
|
|
197
|
-
**D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and agents ask for it with `Accept: text/markdown`. *Why recommended: traffic.* Across 56 domains in a week of October 2026, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler and the rest ShapBot, Exa's user-triggered fetcher and KeenableBot ([research](docs/research/who-reads-discovery-files.md)). That's one search engine more than all the others,
|
|
197
|
+
**D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and agents ask for it with `Accept: text/markdown`. *Why recommended: traffic.* Across 56 domains in a week of October 2026, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler and the rest ShapBot, Exa's user-triggered fetcher and KeenableBot ([research](docs/research/who-reads-discovery-files.md)). That's one search engine more than all the others; GoodBarber's logs show the same clients, with Claude Code too ([GoodBarber](https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/), September 2026). Check your own logs: the case rests on who asks you. Many hosts' logs don't keep the `Accept` header (Railway's HTTP logs and Cloudflare's Free-plan analytics don't); a markdown answer at a page URL shows that one was sent. To see it on your own site, log the header in middleware ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)).
|
|
198
198
|
- Start with the homepage and your docs, not every page. `Accept: text/markdown` and a `.md` URL both return the page as markdown. animalhouse serves llms.txt as the homepage's markdown, and its API reference at `/docs/api.md`.
|
|
199
199
|
- No CDN plan is needed: a `.md` route and a few lines of middleware do it ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)). Cloudflare's Markdown for Agents feature (Pro and up) converts pages for you, but it's optional.
|
|
200
200
|
- Send `Vary: Accept` on both representations (RFC 9110 §12.5.5). Otherwise a cache can hand HTML readers the markdown, or the reverse.
|
|
201
201
|
- Next.js drops a `Vary` set in middleware, proxy or `next.config` on App Router pages, static or dynamic, while it keeps the other headers (tested on 14.2 and 16.3). So `Accept` has to reach the HTML from the edge (a CDN response header rule), or decline that half in writing. Send it from the markdown route yourself (animalhouse.ai).
|
|
202
202
|
- A page's markdown copy has the page's sections. The homepage's can be llms.txt instead. *Harm prevented:* agents reading the markdown get a different site than people do. The scorecard compares the headings of one page llms.txt links, when that page serves markdown, and a guard test can compare them all (§11).
|
|
203
203
|
- Make links absolute, and link back to the HTML page as canonical.
|
|
204
|
-
-
|
|
204
|
+
- `x-markdown-tokens` is optional. It was Cloudflare's header, with no spec behind it, and Cloudflare stopped sending it on 2026-10-09: agents count tokens themselves.
|
|
205
205
|
- Negotiate in front of prerendered pages too, or they ignore it (magnifica).
|
|
206
206
|
- Whole-corpus indexes help: `/docs/index.md` and `/docs/index.json` (achurch.ai).
|
|
207
207
|
|
|
@@ -682,7 +682,7 @@ Higher-value features that most of our projects don't have yet. Each one fixes a
|
|
|
682
682
|
- **T3 Reviews** (required):
|
|
683
683
|
- A monthly log review, paged so no window is truncated (platform log commands often return only the newest window): requested well-known paths, 404s, refusals, crawler user agents. `readiness-audit --logs <file>` counts the discovery requests in an exported log by client family.
|
|
684
684
|
- **Count audits apart from agents.** Audit tools, this one included, are among the largest readers of discovery files: in a week across 56 domains, this audit and its sister made 3,564 discovery requests, more than all the week's llms.txt requests, and other audit tools were the largest readers of ard.json and ai-catalog.json ([research](docs/research/who-reads-discovery-files.md)). `--logs` names them as their own families. *Harm prevented:* mistaking scanners of your setup for agents that use it.
|
|
685
|
-
- **Classify traffic before counting it.** Set known junk crawlers aside first, and count a crawler by its verified status, not its name: the CDN's verified-bot flag, the vendor's published addresses, or reverse DNS. Scanners send crawler names: one sent 308k requests under about 20 AI-crawler names in a week (§2), and `--logs` counts requests under crawler names that asked for secret files, which no real crawler does. Count a browser, headless or unidentified client as an agent only when something backs it up, such as
|
|
685
|
+
- **Classify traffic before counting it.** Set known junk crawlers aside first, and count a crawler by its verified status, not its name: the CDN's verified-bot flag, the vendor's published addresses, or reverse DNS. Scanners send crawler names: one sent 308k requests under about 20 AI-crawler names in a week (§2), and `--logs` counts requests under crawler names that asked for secret files, which no real crawler does. Count a browser, headless or unidentified client as an agent only when something backs it up, such as an API key it registered: agent tools often send a browser's user agent (§2), so the user agent alone proves nothing either way, and a successful API call doesn't either, since a site's own pages call its API too. *Harm prevented:* readiness decisions made on noise.
|
|
686
686
|
- After each deploy, run the scorecard, lint the OpenAPI spec, and diff generated files against the previous production output. About 24 hours later, review the logs and database query performance.
|
|
687
687
|
- **T5 The agent usability test** (required).
|
|
688
688
|
- **Why:** it's the only check that measures what the whole standard is for, and without it the scorecard becomes the target.
|
|
@@ -829,7 +829,7 @@ Level: **R** required where it applies, **Rec** recommended, **N** next level. W
|
|
|
829
829
|
| D5 | `Link` headers: service-desc, service-doc, describedby, api-catalog | Rec | discovery | score |
|
|
830
830
|
| D6 | security.txt with Expires once | R | hygiene | score |
|
|
831
831
|
| D7 | JSON 404s for unknown well-known paths (`agent.json` included), JSON 405 with `Allow` on `POST /` | Rec | agents | score |
|
|
832
|
-
| D8 | Markdown for agents with `Vary: Accept`
|
|
832
|
+
| D8 | Markdown for agents with `Vary: Accept` | Rec | traffic | score |
|
|
833
833
|
| D9 | AI catalog at ard.json and ai-catalog.json, with `rel` links | Rec | discovery | score |
|
|
834
834
|
| D10 | One DNS AID record naming a service you run | Rec | discovery | score |
|
|
835
835
|
| D11 | The A2A path: a valid card or a JSON 404 | R | honesty | score |
|
|
@@ -911,7 +911,7 @@ Status as of 2026-10-05. Several are drafts, so re-check before relying on a det
|
|
|
911
911
|
| robots.txt | RFC 9309 | https://www.rfc-editor.org/rfc/rfc9309.html |
|
|
912
912
|
| Content Signals | Cloudflare policy, 2025-09-24 | https://contentsignals.org/ |
|
|
913
913
|
| AIPREF (`Content-Usage`) | IETF working-group drafts | https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/ · https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/ |
|
|
914
|
-
| Markdown for Agents
|
|
914
|
+
| Markdown for Agents | Cloudflare feature; `x-markdown-tokens` removed 2026-10-09 | https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ · https://developers.cloudflare.com/changelog/post/2026-10-09-markdown-for-agents-in-process-conversion/ |
|
|
915
915
|
| `Vary` | RFC 9110 §12.5.5 | https://www.rfc-editor.org/rfc/rfc9110.html#section-12.5.5 |
|
|
916
916
|
| Agent Skills discovery index | Cloudflare RFC v0.2.0, draft | https://github.com/cloudflare/agent-skills-discovery-rfc |
|
|
917
917
|
| SKILL.md | agentskills.io specification | https://agentskills.io/specification |
|
|
@@ -951,6 +951,9 @@ Status as of 2026-10-05. Several are drafts, so re-check before relying on a det
|
|
|
951
951
|
| Google rich results | Search Central search gallery (updated 2026-10-16), read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data/search-gallery |
|
|
952
952
|
| Testing structured data | Search Central: the Rich Results Test and the Schema Markup Validator, read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data · https://search.google.com/test/rich-results · https://validator.schema.org/ |
|
|
953
953
|
| Rich result reports, URL Inspection API | Search Console Help and API reference, read 2026-10-21 | https://support.google.com/webmasters/answer/7552505 · https://developers.google.com/webmaster-tools/v1/urlInspection.index/UrlInspectionResult |
|
|
954
|
+
| Discovery-file requests ("Who requests agent discovery files?") | this repo's research, 56 domains on Cloudflare, 2026-10-17 to 10-24, corrected 2026-10-24 | docs/research/who-reads-discovery-files.md |
|
|
955
|
+
| Negotiated markdown requests | GoodBarber's own logs, 2026-09-04 to 09-18, read 2026-10-24 | https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/ |
|
|
956
|
+
| Markdown and llms.txt in agent tasks | Mintlify benchmark, 2026-07-17, read 2026-10-24 | https://www.mintlify.com/blog/llms-txt-agent-benchmark |
|
|
954
957
|
| Crawler impersonation ("Bots in AI clothing") | agent-security-pranks research, 56 domains on Cloudflare, 2026-10-16 to 10-23, read 2026-10-24 | https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md |
|
|
955
958
|
| 405 and `Allow` | RFC 9110 §15.5.6 | https://www.rfc-editor.org/rfc/rfc9110.html#section-15.5.6 |
|
|
956
959
|
| Headings | W3C WAI tutorial, read 2026-10-21 | https://www.w3.org/WAI/tutorials/page-structure/headings/ |
|
package/bin/readiness-audit.mjs
CHANGED
|
@@ -10,13 +10,12 @@
|
|
|
10
10
|
import { readFileSync, createReadStream } from 'node:fs';
|
|
11
11
|
import { createInterface } from 'node:readline';
|
|
12
12
|
import { audit, matrix, score, LEVELS, REVIEW, VERSION } from '../src/audit.mjs';
|
|
13
|
-
import { parsePrevious, compareResults, formatComparison } from '../src/helpers.mjs';
|
|
13
|
+
import { parsePrevious, compareResults, formatComparison, nodeOk, NODE_REQUIREMENT } from '../src/helpers.mjs';
|
|
14
14
|
import { logTally, formatLogs } from '../src/logs.mjs';
|
|
15
15
|
|
|
16
16
|
// engines in package.json only warns, so say it plainly before anything fails oddly.
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
console.error(`readiness-audit needs Node ${NODE_MAJOR} or later, with current security releases (this is ${process.versions.node}).`);
|
|
17
|
+
if (!nodeOk(process.versions.node)) {
|
|
18
|
+
console.error(`readiness-audit needs ${NODE_REQUIREMENT} (each supported line's current security release); this is ${process.versions.node}. Update with your version manager (nvm install 24), or set node-version: 24 with check-latest: true in CI.`);
|
|
20
19
|
process.exit(2);
|
|
21
20
|
}
|
|
22
21
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "readiness-audit",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.3.0",
|
|
4
4
|
"description": "Score any site's readiness for AI agents and search (GEO, AEO, AI SEO): llms.txt, robots.txt AI crawler rules, structured data, link previews, OpenAPI, MCP, agent skills and A2A. Read-only CLI, no account.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -19,7 +19,7 @@
|
|
|
19
19
|
"test": "node --test"
|
|
20
20
|
},
|
|
21
21
|
"engines": {
|
|
22
|
-
"node": ">=26"
|
|
22
|
+
"node": "^22.23.4 || ^24.22.1 || >=26.12.1"
|
|
23
23
|
},
|
|
24
24
|
"keywords": [
|
|
25
25
|
"ai-agents",
|
package/src/audit.mjs
CHANGED
|
@@ -4,14 +4,14 @@
|
|
|
4
4
|
// The standard and what each ID means: STANDARD.md in
|
|
5
5
|
// https://github.com/geeks-accelerator/agent-and-search-readiness
|
|
6
6
|
// Read-only: GETs plus a few POSTs that create nothing (an MCP initialize, server/discover and
|
|
7
|
-
// tools/list, an empty POST to /). Node
|
|
7
|
+
// tools/list, an empty POST to /). Node 22, 24 or 26 at its security release, no dependencies.
|
|
8
8
|
|
|
9
9
|
import { lookup, resolveTxt, Resolver } from 'node:dns/promises';
|
|
10
10
|
import { isIP } from 'node:net';
|
|
11
11
|
import { request } from './http.mjs';
|
|
12
12
|
import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
|
|
13
13
|
|
|
14
|
-
export const VERSION = '2.
|
|
14
|
+
export const VERSION = '2.3.0';
|
|
15
15
|
export const REPO = 'https://github.com/geeks-accelerator/agent-and-search-readiness';
|
|
16
16
|
const UA = `readiness-audit/${VERSION} (+${REPO})`;
|
|
17
17
|
const AID_PROTOCOLS = ['mcp', 'a2a', 'openapi', 'grpc', 'graphql', 'websocket', 'local', 'zeroconf', 'ucp'];
|
|
@@ -326,8 +326,7 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
326
326
|
if (!/text\/markdown/.test(homeMd.type)) miss(id, name, level, `Accept: text/markdown on / returns ${ct(homeMd)}`);
|
|
327
327
|
else if (!varyAccept(homeMd) || !varyAccept(home)) miss(id, name, level, `Vary: Accept missing on the ${!varyAccept(home) ? 'HTML' : 'markdown'} response`);
|
|
328
328
|
else if (copy.gone.length) miss(id, name, level, `the markdown copy of ${copy.path} lacks ${copy.gone.length} section${copy.gone.length === 1 ? '' : 's'} the page has: ${list(copy.gone.map((h) => `"${h}"`))}`);
|
|
329
|
-
else
|
|
330
|
-
else pass(id, name, level, `markdown with Vary: Accept on both, x-markdown-tokens ${tokens}${copy.path ? `; ${copy.path}'s markdown copy has every section` : '; no page llms.txt links serves markdown, so no copy was compared'}`);
|
|
329
|
+
else pass(id, name, level, `markdown with Vary: Accept on both${tokens ? `, x-markdown-tokens ${tokens}` : ''}${copy.path ? `; ${copy.path}'s markdown copy has every section` : '; no page llms.txt links serves markdown, so no copy was compared'}`);
|
|
331
330
|
}
|
|
332
331
|
|
|
333
332
|
// ---- D9 AI catalog (ARD) ----
|
package/src/helpers.mjs
CHANGED
|
@@ -20,6 +20,26 @@ export function spread(list, n) {
|
|
|
20
20
|
return list.length <= n ? list : Array.from({ length: n }, (_, i) => list[Math.floor((i * list.length) / n)]);
|
|
21
21
|
}
|
|
22
22
|
|
|
23
|
+
/**
|
|
24
|
+
* Each supported Node line at its current security release (the 2026-10-21 release fixed a
|
|
25
|
+
* Brotli decoder bug and TLS hostname bypasses in all three). TLS checks and the decoders are
|
|
26
|
+
* Node's, so being on a line's latest security release matters more than which line. End-of-life
|
|
27
|
+
* and odd-numbered lines get no security releases and are refused. Raise these after each Node
|
|
28
|
+
* security release, with package.json's engines and the docs (a test checks they agree).
|
|
29
|
+
*/
|
|
30
|
+
export const NODE_LINES = { 22: '22.23.4', 24: '24.22.1', 26: '26.12.1' };
|
|
31
|
+
export const NODE_REQUIREMENT = `Node ${Object.values(NODE_LINES).slice(0, -1).join(', ')} or ${Object.values(NODE_LINES).at(-1)}, or later on the same line`;
|
|
32
|
+
|
|
33
|
+
/** Whether a Node version is a supported line at its security release, or an even line newer than these. */
|
|
34
|
+
export function nodeOk(version) {
|
|
35
|
+
const parts = (v) => v.split('.').map(Number);
|
|
36
|
+
const atLeast = (v, min) => { const [a, b] = [parts(v), parts(min)]; const i = a.findIndex((n, j) => n !== b[j]); return i < 0 || a[i] > b[i]; };
|
|
37
|
+
const major = parts(version)[0];
|
|
38
|
+
const newest = Math.max(...Object.keys(NODE_LINES).map(Number));
|
|
39
|
+
if (major > newest) return major % 2 === 0; // an even line newer than this release knows; odd lines get no long-term support
|
|
40
|
+
return NODE_LINES[major] !== undefined && atLeast(version, NODE_LINES[major]);
|
|
41
|
+
}
|
|
42
|
+
|
|
23
43
|
/** Up to three items, then how many more: "a, b, c, +4 more". */
|
|
24
44
|
export function list(xs, sep = ', ') {
|
|
25
45
|
return `${xs.slice(0, 3).join(sep)}${xs.length > 3 ? `, +${xs.length - 3} more` : ''}`;
|
package/src/logs.mjs
CHANGED
|
@@ -9,7 +9,7 @@ export const FAMILIES = [
|
|
|
9
9
|
['This audit (readiness-audit)', /^readiness-audit\//i],
|
|
10
10
|
// Other audit tools read discovery files at least as often as agents do (docs/research/who-reads-discovery-files.md).
|
|
11
11
|
['Other audit tools', /security-pranks-audit|isitagentready|agentprobe|agent-tools\.cloud|semrushbot|ahrefsbot|lumar|seojuice|sitebulb|screaming frog/i],
|
|
12
|
-
['Agent and MCP directories', /agenstrybot|brickbluebot|
|
|
12
|
+
['Agent and MCP directories', /agenstrybot|brickbluebot|mcpharbor|agenttrustbot|taifoonharvester|mcphub-probe|invokerankbot|proofbench|thepluginstorebot|protogrid-probe|glidemcpindex|agent-tools\.cloud/i],
|
|
13
13
|
['llms.txt directories', /llms\.?txt/i],
|
|
14
14
|
['Assistants fetching for a person', /chatgpt-user|claude-user|perplexity-user|meta-externalfetcher/i],
|
|
15
15
|
['AI crawlers', new RegExp([...AI_BOTS, 'meta-webindexer', 'meta-externalagent', 'bytespider', 'ccbot'].map((b) => b.replace(/[-.]/g, '\\$&')).join('|'), 'i')],
|
|
@@ -25,8 +25,9 @@ export function family(ua) {
|
|
|
25
25
|
return FAMILIES.find(([, re]) => re.test(ua))?.[0] ?? 'Unidentified';
|
|
26
26
|
}
|
|
27
27
|
|
|
28
|
-
// Families whose user agent
|
|
29
|
-
|
|
28
|
+
// Families whose user agent says nothing about what the client is. A successful API answer to one of them is
|
|
29
|
+
// activity, not proof of an agent: a site's own pages call its API too.
|
|
30
|
+
const BROWSER_LIKE = new Set(['Browsers', 'Headless browsers', 'Unidentified', 'No user agent']);
|
|
30
31
|
|
|
31
32
|
// Families named after a crawler or directory. A name is a claim: across one owner's 56 domains in a week of
|
|
32
33
|
// October 2026, none of 448k probes came from a verified bot, and one scanner used about 20 AI-crawler names
|
|
@@ -79,7 +80,8 @@ export function bucket(path) {
|
|
|
79
80
|
if (/\.md$/i.test(p)) return 'markdown pages (*.md)';
|
|
80
81
|
return null;
|
|
81
82
|
}
|
|
82
|
-
|
|
83
|
+
// A call that worked: a 2xx from /api/* or a POST to /mcp. A 404 or a 5xx there is a guess or a probe, not use.
|
|
84
|
+
const isApiSuccess = (r) => (r.path.startsWith('/api/') || (r.path === '/mcp' && r.method === 'POST')) && r.status >= 200 && r.status < 300;
|
|
83
85
|
|
|
84
86
|
/** Tally lines one at a time; result() gives the report. */
|
|
85
87
|
export function logTally() {
|
|
@@ -87,7 +89,7 @@ export function logTally() {
|
|
|
87
89
|
const hits = new Map(); // bucket -> Map(family -> count)
|
|
88
90
|
const readers = new Map(); // bucket -> Set(addr)
|
|
89
91
|
const apiAddrs = new Set();
|
|
90
|
-
const
|
|
92
|
+
const browserLikeAddrs = new Set();
|
|
91
93
|
const impostors = new Map(); // crawler family -> requests for secret files under its name
|
|
92
94
|
return {
|
|
93
95
|
add(line) {
|
|
@@ -96,7 +98,7 @@ export function logTally() {
|
|
|
96
98
|
if (!r) return;
|
|
97
99
|
parsed++;
|
|
98
100
|
const fam = family(r.ua);
|
|
99
|
-
if (
|
|
101
|
+
if (isApiSuccess(r) && r.addr) { apiAddrs.add(r.addr); if (BROWSER_LIKE.has(fam)) browserLikeAddrs.add(r.addr); }
|
|
100
102
|
if (CRAWLER_FAMILIES.has(fam) && isSecretProbe(r.path)) impostors.set(fam, (impostors.get(fam) ?? 0) + 1);
|
|
101
103
|
const b = bucket(r.path);
|
|
102
104
|
if (!b) return;
|
|
@@ -112,11 +114,11 @@ export function logTally() {
|
|
|
112
114
|
requests: [...fams.values()].reduce((a, b) => a + b, 0),
|
|
113
115
|
families: Object.fromEntries([...fams].sort((a, b) => b[1] - a[1])),
|
|
114
116
|
addresses: addrs.size,
|
|
115
|
-
|
|
117
|
+
also_got_api_answer: [...addrs].filter((a) => apiAddrs.has(a)).length,
|
|
116
118
|
};
|
|
117
119
|
}).sort((a, b) => b.requests - a.requests);
|
|
118
120
|
const crawlerNames = { requests: [...impostors.values()].reduce((a, b) => a + b, 0), families: Object.fromEntries([...impostors].sort((a, b) => b[1] - a[1])) };
|
|
119
|
-
return { lines, parsed,
|
|
121
|
+
return { lines, parsed, api_answer_addresses: apiAddrs.size, browser_like_api_answer_addresses: browserLikeAddrs.size, crawler_names_on_secret_probes: crawlerNames, paths };
|
|
120
122
|
},
|
|
121
123
|
};
|
|
122
124
|
}
|
|
@@ -126,11 +128,11 @@ export function formatLogs(rep) {
|
|
|
126
128
|
const out = [`Read ${rep.parsed} of ${rep.lines} log lines (${rep.lines - rep.parsed} not understood). Families are the names clients send, not verified.`, ''];
|
|
127
129
|
if (!rep.paths.length) out.push('No requests for discovery files.');
|
|
128
130
|
for (const p of rep.paths) {
|
|
129
|
-
out.push(`${p.path} ${p.requests} request${p.requests === 1 ? '' : 's'}${p.addresses ? `, ${p.addresses} address${p.addresses === 1 ? '' : 'es'}, ${p.
|
|
131
|
+
out.push(`${p.path} ${p.requests} request${p.requests === 1 ? '' : 's'}${p.addresses ? `, ${p.addresses} address${p.addresses === 1 ? '' : 'es'}, ${p.also_got_api_answer} of them also got a successful API answer` : ''}`);
|
|
130
132
|
for (const [fam, n] of Object.entries(p.families)) out.push(` ${String(n).padStart(6)} ${fam}`);
|
|
131
133
|
}
|
|
132
|
-
out.push('', `API
|
|
133
|
-
out.push('
|
|
134
|
+
out.push('', `Successful API answers (a 2xx from /api/* or a POST to /mcp) went to ${rep.api_answer_addresses} address${rep.api_answer_addresses === 1 ? '' : 'es'}, ${rep.browser_like_api_answer_addresses} of them with a browser, headless or missing user agent.`);
|
|
135
|
+
out.push('An address that read a file and also got an API answer shows activity in the same log, not order, and not that the client is an agent: a site\'s own pages call its API too (STANDARD.md, T3).');
|
|
134
136
|
const named = rep.crawler_names_on_secret_probes;
|
|
135
137
|
if (named.requests) out.push('', `${named.requests} request${named.requests === 1 ? '' : 's'} under crawler names asked for secret files (.env, .git, keys, dumps): ${Object.entries(named.families).map(([f, n]) => `${f} ${n}`).join(', ')}. Real crawlers don't ask for these, so treat those names as claims: count a crawler by the CDN's verified flag, the vendor's published addresses or reverse DNS, not its name (T3).`);
|
|
136
138
|
else out.push('', 'No request under a crawler name asked for secret files.');
|