readiness-audit 2.1.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -0
- package/README.md +44 -8
- package/STANDARD.md +23 -17
- package/package.json +14 -3
- package/src/audit.mjs +7 -5
- package/src/http.mjs +33 -10
- package/src/logs.mjs +20 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,20 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.2.0 (2026-10-24)
|
|
4
|
+
|
|
5
|
+
From a review of the sister project, [agent-security-pranks](https://github.com/geeks-accelerator/agent-security-pranks), and its research. Two new warnings (D7's `Allow`, and D8 at the recommended level), no new failures. A warning lowers the recommended score until fixed, but doesn't change the exit code.
|
|
6
|
+
|
|
7
|
+
- **D7 warns about a 405 with no `Allow` header.** RFC 9110 requires it on every 405, and it tells a client what does work. All six sites the standard came from already send it.
|
|
8
|
+
- **Response bodies are capped at 16 MB,** before and after decoding, so a site that answers with a gzip bomb or an endless stream can't exhaust the memory of the machine running the audit. Over the cap, the check sees the status and no body, as with a refused encoding, and D4 reports an oversized llms-full.txt by its size.
|
|
9
|
+
- **`--logs` counts requests under crawler names that asked for secret files** (`.env`, `.git`, keys, dumps), per family, and says the families are the names clients send, not verified. The JSON report has `crawler_names_on_secret_probes`.
|
|
10
|
+
- **D8 (markdown for agents) is recommended,** up from next level, so a site without it now warns at the recommended level and its recommended score can drop. The evidence: in a week across 56 domains, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler.
|
|
11
|
+
- **`--logs` names other audit tools** (Semrush, Ahrefs, agentprobe, agent-tools.cloud, Lumar, SEOJuice and the sister project's audit) as their own family.
|
|
12
|
+
- **New research: [Who reads the discovery files](docs/research/who-reads-discovery-files.md),** a week across 56 domains on Cloudflare: who requested each discovery file, verified or not, which files sites are asked for without serving them, and how many readers went on to call the API. STANDARD.md §2 cites it; D4's reason now names who reads llms.txt; D7 and D11 cover the pre-1.0 path `/.well-known/agent.json`; T3 counts audits apart from agents.
|
|
13
|
+
- **STANDARD.md:** §2 adds that a crawler's name is a claim: in a week across one owner's 56 domains, none of about 448k vulnerability probes came from a verified bot, and one scanner used about 20 AI-crawler names. T3 counts a crawler by its verified status. D7 asks for `Allow`. §16 cites the research and RFC 9110 §15.5.6. docs/recipes.md shows `Allow` on the 405.
|
|
14
|
+
- **README:** a row for the sister project in "How it fits with other tools".
|
|
15
|
+
- **README:** opens with what the audit checks, in the words people search for, a real run's output, "Why this one", and how it fits with other tools. Badges for the npm version and license.
|
|
16
|
+
- **package.json:** the description leads with what the audit scores; keywords add aeo, ai-seo, ai-visibility, generative-engine-optimization, robots-txt, ai-crawlers, seo-audit, website-audit, agent-ready, llms-txt-validator and cli. Both reach npm with the next release.
|
|
17
|
+
|
|
3
18
|
## 2.1.0 (2026-10-23)
|
|
4
19
|
|
|
5
20
|
From a project's agent that checked pages in Google's Rich Results Test by hand. The standard asked for it ("accepts a sample") without saying which pages, and the scorecard never mentioned it. No scoring changes.
|
package/README.md
CHANGED
|
@@ -1,20 +1,56 @@
|
|
|
1
1
|
# Agent and Search Readiness
|
|
2
2
|
|
|
3
3
|
[](https://github.com/geeks-accelerator/agent-and-search-readiness/actions/workflows/ci.yml)
|
|
4
|
+
[](https://www.npmjs.com/package/readiness-audit)
|
|
5
|
+
[](LICENSE)
|
|
4
6
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
AI agents find a site through discovery files, MCP registries, skills and search. People find it through Google, Bing and the AI answer engines built on them. Both then need the first call to work, and a reason to come back. This repo holds:
|
|
8
|
-
|
|
9
|
-
- **[STANDARD.md](STANDARD.md):** what to build, item by item, each with an ID, a level and the reason it's at that level. It covers discovery files, search, the API, MCP servers, skills, engagement, measurement and tests.
|
|
10
|
-
- **`readiness-audit`:** a command-line scorecard that checks a live site against everything the standard can verify from outside, about 30 checks in one pass.
|
|
11
|
-
|
|
12
|
-
## Score a site
|
|
7
|
+
**Score any site's readiness for AI agents and search:** llms.txt, robots.txt rules for AI crawlers, the sitemap, structured data, link previews, OpenAPI, MCP servers, agent skills and the A2A path. A standard built from production request logs, and a command-line audit that checks a live site against it. Read-only, no account, no API key.
|
|
13
8
|
|
|
14
9
|
```
|
|
15
10
|
npx readiness-audit example.com
|
|
16
11
|
```
|
|
17
12
|
|
|
13
|
+
```
|
|
14
|
+
Agent and search readiness: animalhouse.ai (readiness-audit v2.1.0)
|
|
15
|
+
|
|
16
|
+
Required
|
|
17
|
+
PASS D1 robots.txt 1 group with consistent rules, Sitemap listed
|
|
18
|
+
PASS D4 llms.txt and llms-full.txt llms.txt 16.2 KB, llms-full.txt 65 KB; 10 sampled links resolve
|
|
19
|
+
PASS D12 No false declarations no OAuth, A2A, plugin or server-card claims without the service
|
|
20
|
+
PASS W1 Titles and descriptions 27 sampled pages (18 static, 3 templates), all with their own
|
|
21
|
+
WARN W5 Link previews og:image size or alt text not declared on /hall, /about, +11 more
|
|
22
|
+
WARN W6 Page structure heading levels skipped on /agent/[name] (h1 to h3)
|
|
23
|
+
PASS A1 OpenAPI 3.1 OpenAPI 3.1.0; all 17 operations and their request fields described
|
|
24
|
+
PASS M4 Hosted MCP edge both eras, JSON for every Accept, browsers redirected
|
|
25
|
+
...
|
|
26
|
+
Score on 2026-10-23: required 17 of 19, recommended 10 of 10, next level 1 of 3
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This repo holds:
|
|
30
|
+
|
|
31
|
+
- **[STANDARD.md](STANDARD.md):** what to build, item by item, each with an ID, a level and the reason it's at that level. It covers discovery files, search (SEO, and the AI answer engines that GEO and AEO are about), the API, MCP servers, skills, engagement, measurement and tests.
|
|
32
|
+
- **[Who reads the discovery files](docs/research/who-reads-discovery-files.md):** a week of traffic across 56 domains on Cloudflare, with charts. The AI crawlers read robots.txt and the sitemap, not llms.txt; bots asked for markdown 12,601 times; directories read the agent cards; and on sites with an API, some readers of llms.txt and openapi.json went on to call it.
|
|
33
|
+
- **`readiness-audit`:** the scorecard. It checks a live site against everything the standard can verify from outside, about 30 checks in one pass, and lists what it can't see for you to review.
|
|
34
|
+
|
|
35
|
+
## Why this one
|
|
36
|
+
|
|
37
|
+
- **Built from evidence.** Each item says why it's there: production request logs (which agents fetch which files, and which ones go on to call the API), the specs, and Google's and Bing's own documentation, cited with dates. Items that only scanners look for are recommended at most.
|
|
38
|
+
- **It checks that declarations are true.** An A2A card that isn't a valid A2A card fails D11; OAuth metadata with no token endpoint, or a catalog entry typed as something it isn't, fails D12. Declining an item in writing is fine; faking a pass isn't.
|
|
39
|
+
- **It fits the site.** The API checks run only when a site has or declares an API, and the MCP checks only when one answers. Pages are sampled by type: every static page, and three pages of each generated template.
|
|
40
|
+
- **Before and after you ship.** `--base http://localhost:3000` scores a local or preview build; `--compare` lists what changed since the last run; the exit code gates a deploy.
|
|
41
|
+
- **The real test is an agent.** The standard's T5 gives a fresh agent only your domain and a goal, and records whether it succeeds. The scorecard is the setup for that, not the goal.
|
|
42
|
+
|
|
43
|
+
## How it fits with other tools
|
|
44
|
+
|
|
45
|
+
| Tool | What it does | Use it with this |
|
|
46
|
+
|---|---|---|
|
|
47
|
+
| [Is Your Site Agent-Ready?](https://isitagentready.com/) (Cloudflare) | A web scan for llms.txt, MCP, agent skills and other agent-facing standards | For a quick look in the browser. This runs in your terminal and CI, scores local builds, and checks search as well as agents |
|
|
48
|
+
| llms.txt validators | Check that one file is well formed | This checks llms.txt and samples its links, as one of about 30 checks |
|
|
49
|
+
| [security-pranks-audit](https://github.com/geeks-accelerator/agent-security-pranks) | The sister project: checks that a site answers `.env`, `.git` and other secret-hunting probes safely | This welcomes agents and checks what they read; that checks what the scanners get. Same shape, run both |
|
|
50
|
+
| [specification.website](https://specification.website) | A sourced website specification: HTML, accessibility, security, SEO, agent readiness | A spec for the whole site, accessibility and security included. This one focuses on agents and search, and comes with a scorer |
|
|
51
|
+
|
|
52
|
+
## Score a site
|
|
53
|
+
|
|
18
54
|
It needs Node 26 or later, with current security releases, and has no dependencies. It only reads: GET requests, plus a few POSTs that create nothing (an MCP `initialize`, `server/discover` and `tools/list`, and an empty POST to `/`). It keeps to about six requests at a time, and it won't follow a link the site supplies (in its sitemap, share images, skills, catalog or DNS record) to a private or local address, at any redirect. It asks for gzip and decodes it itself; a body in an encoding it didn't ask for, such as Brotli, is reported rather than read, so a site can't reach a decoder bug on the machine running the audit. It prints a pass, warning or failure for each check, with the reason, and exits with code 1 when a required check fails, so it can gate a deploy.
|
|
19
55
|
|
|
20
56
|
```
|
package/STANDARD.md
CHANGED
|
@@ -85,6 +85,8 @@ What it means:
|
|
|
85
85
|
| AI search indexes | Meta-WebIndexer, OAI-SearchBot, Claude-SearchBot, PerplexityBot | ordinary pages, like any search crawler (§5) |
|
|
86
86
|
|
|
87
87
|
So no single file reaches every agent. Directories want the well-known files, scripts want llms.txt and the API, tools want clean pages and docs, and search indexes want everything in §5.
|
|
88
|
+
- **A crawler's name is a claim.** Across one owner's 56 domains on Cloudflare, October 16 to 23, 2026, none of about 448k vulnerability probes, most of them hunting `.env` files, came from a bot Cloudflare verified, and one scanner sent 308k requests under about 20 AI-crawler names, GPTBot and ClaudeBot among them ([Bots in AI clothing](https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md), from the sister project agent-security-pranks). The crawlers in the table above are the verified ones; a name in a log line alone doesn't show which requests are theirs (T3).
|
|
89
|
+
- **A week across the whole network says the same, at scale.** [Who reads the discovery files](docs/research/who-reads-discovery-files.md) covers 56 domains, October 17 to 24, 2026. Verified AI crawlers read robots.txt (3,644 requests) and the sitemap (2,232, 40% of its reads) far more than llms.txt (49 of 1,406). Verified bots, nearly all Exa's search crawler, asked for markdown 12,601 times. Directories, not AI crawlers, read agent-card.json (2 of 991 requests were verified AI). On the 16 sites with an API, 13% of the addresses that read llms.txt, and 23% of those that read openapi.json, made a successful API call the same week.
|
|
88
90
|
- **Check your own traffic before building a next-level item.** botsmatter looked: 2 of its 1,940 Grounds exactly repeat an earlier one, so retries aren't a problem there and N2 isn't worth building.
|
|
89
91
|
|
|
90
92
|
---
|
|
@@ -133,7 +135,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
133
135
|
**D3 Server-rendered pages.** Every page's content is in the HTML the server sends, not loaded later by script, with a real logo through the framework's icon convention and a favicon set (missing icons show up as steady 404s). *Why: crawlers, link-preview bots and agents that fetch pages read the HTML, and many don't run JavaScript.* inbed's chat pages once shipped none of their 48 messages in the HTML. The page-level details (titles, canonicals, structured data, previews, headings, links) are W1 to W7 in [§5](#5-search-people-and-answer-engines).
|
|
134
136
|
|
|
135
137
|
**D4 `/llms.txt` and `/llms-full.txt`**: a short map (what this is, how to start, links to every other surface) and the whole corpus in one fetch. Generate both and test them against the sitemap: agents follow the links, so every one must open. The scorecard samples them and warns about missing pages (an endpoint that answers 401 for want of a key is fine; templates like `{id}` are skipped). The file is markdown; serve it as `text/markdown` or `text/plain`, and use the same type wherever you link it.
|
|
136
|
-
- *Why required:
|
|
138
|
+
- *Why required: agents that go on to use the site read it, though the AI crawlers mostly don't.* Across 56 domains in a week of October 2026, verified AI crawlers made 49 of 1,406 llms.txt requests (GPTBot 5, ClaudeBot none) and read robots.txt and the sitemap instead. On the 16 sites with an API, 13% of the addresses that read llms.txt made a successful API call that week, 9% with a user agent that wasn't a browser's ([research](docs/research/who-reads-discovery-files.md)). Google Search ignores llms.txt (it said so in June 2026). At animalhouse, 79 requests in four and a half days came from:
|
|
137
139
|
- agent directories (AgenstryBot, BrickBlueBot, agentprobe, MCPHarbor, AgentTrustBot);
|
|
138
140
|
- llms.txt directories and validators;
|
|
139
141
|
- SEO tools and Amazonbot;
|
|
@@ -147,6 +149,7 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
147
149
|
|
|
148
150
|
**D11 The A2A path is honest.** `/.well-known/agent-card.json` is the most-requested agent path. Serve a valid A2A v1.0 card only if you run an A2A endpoint ([Only if true](#only-if-true)). Otherwise answer a JSON 404 that names your MCP endpoint, OpenAPI document and llms.txt (achurch.ai). A custom document or an "A2A-shaped" card at that path fails. *Why: honesty. A2A §8.3.1 requires every declared interface to be real.*
|
|
149
151
|
- **The cost:** directory crawlers drive most requests to this path, and some may stop listing a site that answers 404.
|
|
152
|
+
- **The pre-1.0 path still gets asked.** Directories request `/.well-known/agent.json` too: 445 times at 19 sites in a week of October 2026, none of which serve it ([research](docs/research/who-reads-discovery-files.md)). Answer it the same way; D7's catch-all does.
|
|
150
153
|
- **Roll it out on one site first.** inbed's agent said it would accept the risk. Watch for two weeks whether AgenstryBot and the other directories keep listing it, then change the rest.
|
|
151
154
|
|
|
152
155
|
**D12 No false declarations.** *Why: honesty.*
|
|
@@ -189,7 +192,18 @@ Each item says what it is, why it's at its level, and how you know it's done. Se
|
|
|
189
192
|
- `service-desc` is the machine-readable description (your OpenAPI). `service-doc` is the human documentation (RFC 8631). Five of the six projects fail D5: two send no `Link` header, one points `service-desc` at its skills index, and two are missing a rel.
|
|
190
193
|
- Registered relations are `describedby`, `service-desc`, `service-doc`, `service-meta` and `api-catalog`. `ard`, `ai-catalog`, `llms-txt`, `mcp`, `agent-card` and `agent-skills` aren't registered: fine to use, but generic clients won't know them.
|
|
191
194
|
|
|
192
|
-
**D7 JSON 404s for unknown `/.well-known/*` paths, and a JSON 405 on `POST /`** naming the MCP endpoint and the API (achurch.ai, magnifica). An HTML 404 page is noise to an agent. *Why: cheap, and it helps the agents and crawlers that guess. No project passes all of it yet.* One catch-all route can answer D7 and D11 together ([recipe](docs/recipes.md#one-catch-all-for-unknown-well-known-paths-d7-d11)).
|
|
195
|
+
**D7 JSON 404s for unknown `/.well-known/*` paths, including the pre-1.0 A2A path `/.well-known/agent.json`, and a JSON 405 on `POST /`** naming the MCP endpoint and the API (achurch.ai, magnifica), with an `Allow` header naming the methods that work (`Allow: GET, HEAD`): RFC 9110 requires it on every 405. An HTML 404 page is noise to an agent. *Why: cheap, and it helps the agents and crawlers that guess. No project passes all of it yet.* One catch-all route can answer D7 and D11 together ([recipe](docs/recipes.md#one-catch-all-for-unknown-well-known-paths-d7-d11)).
|
|
196
|
+
|
|
197
|
+
**D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and agents ask for it with `Accept: text/markdown`. *Why recommended: traffic.* Across 56 domains in a week of October 2026, Cloudflare-verified bots asked for markdown 12,601 times on the sites that serve it, 92% of them Exa's search crawler and the rest ShapBot, Exa's user-triggered fetcher and KeenableBot ([research](docs/research/who-reads-discovery-files.md)). That's one search engine more than all the others, so check your own logs: the case rests on who asks you. Many hosts' logs don't keep the `Accept` header (Railway's HTTP logs and Cloudflare's Free-plan analytics don't); a markdown answer at a page URL shows that one was sent. To see it on your own site, log the header in middleware ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)).
|
|
198
|
+
- Start with the homepage and your docs, not every page. `Accept: text/markdown` and a `.md` URL both return the page as markdown. animalhouse serves llms.txt as the homepage's markdown, and its API reference at `/docs/api.md`.
|
|
199
|
+
- No CDN plan is needed: a `.md` route and a few lines of middleware do it ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)). Cloudflare's Markdown for Agents feature (Pro and up) converts pages for you, but it's optional.
|
|
200
|
+
- Send `Vary: Accept` on both representations (RFC 9110 §12.5.5). Otherwise a cache can hand HTML readers the markdown, or the reverse.
|
|
201
|
+
- Next.js drops a `Vary` set in middleware, proxy or `next.config` on App Router pages, static or dynamic, while it keeps the other headers (tested on 14.2 and 16.3). So `Accept` has to reach the HTML from the edge (a CDN response header rule), or decline that half in writing. Send it from the markdown route yourself (animalhouse.ai).
|
|
202
|
+
- A page's markdown copy has the page's sections. The homepage's can be llms.txt instead. *Harm prevented:* agents reading the markdown get a different site than people do. The scorecard compares the headings of one page llms.txt links, when that page serves markdown, and a guard test can compare them all (§11).
|
|
203
|
+
- Make links absolute, and link back to the HTML page as canonical.
|
|
204
|
+
- Add `x-markdown-tokens` so an agent can budget before reading. It's Cloudflare's header, with no spec behind it.
|
|
205
|
+
- Negotiate in front of prerendered pages too, or they ignore it (magnifica).
|
|
206
|
+
- Whole-corpus indexes help: `/docs/index.md` and `/docs/index.json` (achurch.ai).
|
|
193
207
|
|
|
194
208
|
**D9 An AI catalog (ARD)**
|
|
195
209
|
- One list of what an agent can use here, such as the MCP server card, the OpenAPI document, skills and llms.txt, with identifiers in the form `urn:air:<publisher-domain>:<namespace>:<name>`.
|
|
@@ -632,17 +646,6 @@ The test is whether most of the six projects should do it. These came from one o
|
|
|
632
646
|
|
|
633
647
|
Higher-value features that most of our projects don't have yet. Each one fixes a failure we've seen or can predict. Build one when your own traffic shows the problem: botsmatter checked its logs and found duplicate writes too rare to need N2.
|
|
634
648
|
|
|
635
|
-
**D8 Markdown for agents.** Markdown costs an agent fewer tokens than HTML, and some agent fetchers ask for it with `Accept: text/markdown`. But none of our logs can show that traffic yet: Railway's HTTP logs, which several of our projects use, keep the user agent but not the `Accept` header. And a scanner check alone doesn't make an item recommended (principle 3). *Build it when* your logs show agents asking: log the `Accept` header in middleware to find out ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)).
|
|
636
|
-
- Start with the homepage and your docs, not every page. `Accept: text/markdown` and a `.md` URL both return the page as markdown. animalhouse serves llms.txt as the homepage's markdown, and its API reference at `/docs/api.md`.
|
|
637
|
-
- No CDN plan is needed: a `.md` route and a few lines of middleware do it ([recipe](docs/recipes.md#markdown-for-agents-without-a-cdn-plan-d8)). Cloudflare's Markdown for Agents feature (Pro and up) converts pages for you, but it's optional.
|
|
638
|
-
- Send `Vary: Accept` on both representations (RFC 9110 §12.5.5). Otherwise a cache can hand HTML readers the markdown, or the reverse.
|
|
639
|
-
- Next.js drops a `Vary` set in middleware, proxy or `next.config` on App Router pages, static or dynamic, while it keeps the other headers (tested on 14.2 and 16.3). So `Accept` has to reach the HTML from the edge (a CDN response header rule), or decline that half in writing. Send it from the markdown route yourself (animalhouse.ai).
|
|
640
|
-
- A page's markdown copy has the page's sections. The homepage's can be llms.txt instead. *Harm prevented:* agents reading the markdown get a different site than people do. The scorecard compares the headings of one page llms.txt links, when that page serves markdown, and a guard test can compare them all (§11).
|
|
641
|
-
- Make links absolute, and link back to the HTML page as canonical.
|
|
642
|
-
- Add `x-markdown-tokens` so an agent can budget before reading. It's Cloudflare's header, with no spec behind it.
|
|
643
|
-
- Negotiate in front of prerendered pages too, or they ignore it (magnifica).
|
|
644
|
-
- Whole-corpus indexes help: `/docs/index.md` and `/docs/index.json` (achurch.ai).
|
|
645
|
-
|
|
646
649
|
**N2 `Idempotency-Key` on writes** (IETF draft). Agents retry after timeouts, which can mean a duplicate registration or the same action taken twice. Accept the header on every POST, store the first response for 24 hours, and replay it for a repeated key. *Build it when* your logs show the same write repeated within minutes.
|
|
647
650
|
|
|
648
651
|
**N3 Conditional requests on polled reads.** Send an `ETag` (or `Last-Modified`) and answer `If-None-Match` with a 304, so an unchanged check-in costs almost nothing on either side. *Build it when* scheduled polling is a large share of your traffic.
|
|
@@ -678,7 +681,8 @@ Higher-value features that most of our projects don't have yet. Each one fixes a
|
|
|
678
681
|
- **T2 Attribution and one come-back measure** (required; per-agent tracking may be declined). Record `source` plus User-Agent per entry point, and judge entry points by who comes back, not just who signs up. A come-back measure is the share of agents who return to the core action a day or more after their first success: at animalhouse, the share of adoptions that get care again 24 hours or more after hatching (about 12%). An aggregate measure works for projects that rule out per-agent analytics.
|
|
679
682
|
- **T3 Reviews** (required):
|
|
680
683
|
- A monthly log review, paged so no window is truncated (platform log commands often return only the newest window): requested well-known paths, 404s, refusals, crawler user agents. `readiness-audit --logs <file>` counts the discovery requests in an exported log by client family.
|
|
681
|
-
- **
|
|
684
|
+
- **Count audits apart from agents.** Audit tools, this one included, are among the largest readers of discovery files: in a week across 56 domains, this audit and its sister made 3,564 discovery requests, more than all the week's llms.txt requests, and other audit tools were the largest readers of ard.json and ai-catalog.json ([research](docs/research/who-reads-discovery-files.md)). `--logs` names them as their own families. *Harm prevented:* mistaking scanners of your setup for agents that use it.
|
|
685
|
+
- **Classify traffic before counting it.** Set known junk crawlers aside first, and count a crawler by its verified status, not its name: the CDN's verified-bot flag, the vendor's published addresses, or reverse DNS. Scanners send crawler names: one sent 308k requests under about 20 AI-crawler names in a week (§2), and `--logs` counts requests under crawler names that asked for secret files, which no real crawler does. Count a browser, headless or unidentified client as an agent only when something backs it up, such as the same address calling the API: agent tools often send a browser's user agent (§2), so their user agent alone proves nothing either way. *Harm prevented:* readiness decisions made on noise.
|
|
682
686
|
- After each deploy, run the scorecard, lint the OpenAPI spec, and diff generated files against the previous production output. About 24 hours later, review the logs and database query performance.
|
|
683
687
|
- **T5 The agent usability test** (required).
|
|
684
688
|
- **Why:** it's the only check that measures what the whole standard is for, and without it the scorecard becomes the target.
|
|
@@ -775,7 +779,7 @@ Score each site monthly and after big releases, and keep the results where the p
|
|
|
775
779
|
npx readiness-audit@2 --matrix site-one.com site-two.com --recorded docs/readiness-recorded.json > docs/readiness-status.md
|
|
776
780
|
```
|
|
777
781
|
|
|
778
|
-
The page lists every check for every site, the scores at each level,
|
|
782
|
+
The page lists every check for every site, the scores at each level, each failure or warning with its reason, and the pages to run through Google's Rich Results Test (W4), each with a link that opens it in the test. Keep it in the project's private repo when there is one (for example, a private repo that holds the docs, with the public repo as a submodule). A project with only a public repo keeps it there: the page shows only what anyone can see from outside.
|
|
779
783
|
|
|
780
784
|
Score a local or preview build before you deploy: `npx readiness-audit@2 example.com --base http://localhost:3000` requests the site's URLs from the base and reports them under the domain, skipping DNS and host redirects. To see what changed since the last run, compare against that page: `npx readiness-audit@2 <domain> --compare docs/readiness-status.md` lists what was fixed, what newly fails and what newly applies. Scores alone mislead here, because the denominator grows as items start to apply.
|
|
781
785
|
|
|
@@ -824,8 +828,8 @@ Level: **R** required where it applies, **Rec** recommended, **N** next level. W
|
|
|
824
828
|
| D4 | llms.txt and llms-full.txt, generated, linked from skills and README | R | traffic | score, test |
|
|
825
829
|
| D5 | `Link` headers: service-desc, service-doc, describedby, api-catalog | Rec | discovery | score |
|
|
826
830
|
| D6 | security.txt with Expires once | R | hygiene | score |
|
|
827
|
-
| D7 | JSON 404s for unknown well-known paths, JSON 405 on `POST /` | Rec | agents | score |
|
|
828
|
-
| D8 | Markdown for agents with `Vary: Accept` and x-markdown-tokens
|
|
831
|
+
| D7 | JSON 404s for unknown well-known paths (`agent.json` included), JSON 405 with `Allow` on `POST /` | Rec | agents | score |
|
|
832
|
+
| D8 | Markdown for agents with `Vary: Accept` and x-markdown-tokens | Rec | traffic | score |
|
|
829
833
|
| D9 | AI catalog at ard.json and ai-catalog.json, with `rel` links | Rec | discovery | score |
|
|
830
834
|
| D10 | One DNS AID record naming a service you run | Rec | discovery | score |
|
|
831
835
|
| D11 | The A2A path: a valid card or a JSON 404 | R | honesty | score |
|
|
@@ -947,6 +951,8 @@ Status as of 2026-10-05. Several are drafts, so re-check before relying on a det
|
|
|
947
951
|
| Google rich results | Search Central search gallery (updated 2026-10-16), read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data/search-gallery |
|
|
948
952
|
| Testing structured data | Search Central: the Rich Results Test and the Schema Markup Validator, read 2026-10-21 | https://developers.google.com/search/docs/appearance/structured-data · https://search.google.com/test/rich-results · https://validator.schema.org/ |
|
|
949
953
|
| Rich result reports, URL Inspection API | Search Console Help and API reference, read 2026-10-21 | https://support.google.com/webmasters/answer/7552505 · https://developers.google.com/webmaster-tools/v1/urlInspection.index/UrlInspectionResult |
|
|
954
|
+
| Crawler impersonation ("Bots in AI clothing") | agent-security-pranks research, 56 domains on Cloudflare, 2026-10-16 to 10-23, read 2026-10-24 | https://github.com/geeks-accelerator/agent-security-pranks/blob/main/docs/research/bots-in-ai-clothing.md |
|
|
955
|
+
| 405 and `Allow` | RFC 9110 §15.5.6 | https://www.rfc-editor.org/rfc/rfc9110.html#section-15.5.6 |
|
|
950
956
|
| Headings | W3C WAI tutorial, read 2026-10-21 | https://www.w3.org/WAI/tutorials/page-structure/headings/ |
|
|
951
957
|
| Publication dates | Search Central, read 2026-10-21 | https://developers.google.com/search/docs/appearance/publication-dates |
|
|
952
958
|
| Canonicalization | Search Central | https://developers.google.com/search/docs/crawling-indexing/canonicalization |
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "readiness-audit",
|
|
3
|
-
"version": "2.
|
|
4
|
-
"description": "Score
|
|
3
|
+
"version": "2.2.0",
|
|
4
|
+
"description": "Score any site's readiness for AI agents and search (GEO, AEO, AI SEO): llms.txt, robots.txt AI crawler rules, structured data, link previews, OpenAPI, MCP, agent skills and A2A. Read-only CLI, no account.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"readiness-audit": "bin/readiness-audit.mjs"
|
|
@@ -32,7 +32,18 @@
|
|
|
32
32
|
"openapi",
|
|
33
33
|
"structured-data",
|
|
34
34
|
"audit",
|
|
35
|
-
"scorecard"
|
|
35
|
+
"scorecard",
|
|
36
|
+
"aeo",
|
|
37
|
+
"ai-seo",
|
|
38
|
+
"ai-visibility",
|
|
39
|
+
"generative-engine-optimization",
|
|
40
|
+
"robots-txt",
|
|
41
|
+
"ai-crawlers",
|
|
42
|
+
"seo-audit",
|
|
43
|
+
"website-audit",
|
|
44
|
+
"agent-ready",
|
|
45
|
+
"llms-txt-validator",
|
|
46
|
+
"cli"
|
|
36
47
|
],
|
|
37
48
|
"license": "MIT",
|
|
38
49
|
"repository": {
|
package/src/audit.mjs
CHANGED
|
@@ -11,14 +11,14 @@ import { isIP } from 'node:net';
|
|
|
11
11
|
import { request } from './http.mjs';
|
|
12
12
|
import { SKILL_NAME, sampleSkillEntries, skillNameProblem, cdnCacheNote, repeatsSummary, hasVariantDescription, spread, samplePages, sharedCardText, API_LINK, MCP_LINK, linkRels, apiPresence, DISCOVERY_FILES, AI_BOTS, missingSections, richResultPages, list, norm, pathDepth, decodeText, headings, headingSkips, sitemapEntries, llmsLinks, baseMapper, formatRecorded } from './helpers.mjs';
|
|
13
13
|
|
|
14
|
-
export const VERSION = '2.
|
|
14
|
+
export const VERSION = '2.2.0';
|
|
15
15
|
export const REPO = 'https://github.com/geeks-accelerator/agent-and-search-readiness';
|
|
16
16
|
const UA = `readiness-audit/${VERSION} (+${REPO})`;
|
|
17
17
|
const AID_PROTOCOLS = ['mcp', 'a2a', 'openapi', 'grpc', 'graphql', 'websocket', 'local', 'zeroconf', 'ucp'];
|
|
18
18
|
const SKILLS_SCHEMA = 'https://schemas.agentskills.io/discovery/0.2.0/schema.json';
|
|
19
19
|
const CARD_SCHEMA = 'https://static.modelcontextprotocol.io/schemas/v1/server-card.schema.json';
|
|
20
20
|
export const LEVELS = ['required', 'recommended', 'next'];
|
|
21
|
-
const ORDER = ['D1', 'D2', 'D4', 'D6', 'D11', 'D12', 'W1', 'W2', 'W3', 'W4', 'W5', 'W6', 'A1', 'A2', 'A6', 'A7', 'M1', 'M4', 'S1', 'D5', 'D7', 'D9', 'D10', 'D15', 'A3', 'A4', 'A5', 'M5', 'S2', '
|
|
21
|
+
const ORDER = ['D1', 'D2', 'D4', 'D6', 'D11', 'D12', 'W1', 'W2', 'W3', 'W4', 'W5', 'W6', 'A1', 'A2', 'A6', 'A7', 'M1', 'M4', 'S1', 'D5', 'D7', 'D9', 'D10', 'D15', 'D8', 'A3', 'A4', 'A5', 'M5', 'S2', 'N5', 'N6'];
|
|
22
22
|
// Items the scorecard can't see from outside. Each project checks these in review and tests.
|
|
23
23
|
export const REVIEW = [
|
|
24
24
|
['D3', 'pages read well: server-rendered, unique prose on generated pages (the scorecard checks W1 to W6 on a sample)'],
|
|
@@ -268,7 +268,8 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
268
268
|
const linkProblems = [];
|
|
269
269
|
if (broken.length) linkProblems.push(`broken links in llms.txt: ${list(broken)} (${broken.length} of ${sample.length} sampled)`);
|
|
270
270
|
if (redirected.length) linkProblems.push(`llms.txt links that redirect (link the destination): ${list(redirected)}`);
|
|
271
|
-
const
|
|
271
|
+
const kb = (r, digits) => (r.tooBig ? r.error.replace(/^.*?over /, 'over ') : `${(r.text.length / 1024).toFixed(digits)} KB`);
|
|
272
|
+
const sizes = `llms.txt ${kb(llms, 1)}, llms-full.txt ${kb(llmsFull, 0)}`;
|
|
272
273
|
if (!ok(llms) || !/^\s*# \S/.test(llms.text)) miss(id, name, level, (ok(llms) ? 'llms.txt has no H1 title on its first line' : `/llms.txt answered ${llms.status}`) + cached(llms));
|
|
273
274
|
else if (!ok(llmsFull)) miss(id, name, level, `/llms-full.txt missing${cached(llmsFull)}`);
|
|
274
275
|
else if (linkProblems.length) record(id, name, level, 'warn', `${linkProblems.join('; ')}${cached(llms)}`);
|
|
@@ -305,13 +306,14 @@ export async function audit(domain, { mcpPath = '/mcp', base = null } = {}) {
|
|
|
305
306
|
const problems = [];
|
|
306
307
|
if (!isJson(wkMiss)) problems.push(`an unknown /.well-known path answers ${wkMiss.status} ${ct(wkMiss)}`);
|
|
307
308
|
if (!(postRoot.status === 405 && isJson(postRoot))) problems.push(`POST / answers ${postRoot.status} ${ct(postRoot)}`);
|
|
309
|
+
else if (!postRoot.headers.get('allow')) problems.push('POST / answers a JSON 405 with no Allow header (RFC 9110 requires one, naming the methods that work)');
|
|
308
310
|
if (problems.length) miss(id, name, level, problems.join('; '));
|
|
309
|
-
else pass(id, name, level,
|
|
311
|
+
else pass(id, name, level, `JSON 404 for unknown well-known paths, JSON 405 for POST / (Allow: ${postRoot.headers.get('allow')})`);
|
|
310
312
|
}
|
|
311
313
|
|
|
312
314
|
// ---- D8 markdown for agents ----
|
|
313
315
|
{
|
|
314
|
-
const id = 'D8', name = 'Markdown for agents', level = '
|
|
316
|
+
const id = 'D8', name = 'Markdown for agents', level = 'recommended';
|
|
315
317
|
const varyAccept = (r) => /(^|,)\s*accept\s*(,|$)/i.test(r.headers.get('vary') ?? '');
|
|
316
318
|
const tokens = homeMd.headers.get('x-markdown-tokens');
|
|
317
319
|
// The homepage's markdown may be llms.txt, so compare a page llms.txt links instead, when it serves markdown.
|
package/src/http.mjs
CHANGED
|
@@ -3,45 +3,68 @@
|
|
|
3
3
|
// Not fetch(): fetch decodes a Brotli body even when the request didn't ask for one, so a
|
|
4
4
|
// hostile site could hand any body to the Brotli decoder (CVE-2026-111497 was an out-of-bounds
|
|
5
5
|
// write there). node:http decodes nothing; this decodes gzip and deflate and refuses the rest.
|
|
6
|
+
// Every body is capped, before and after decoding, so a site that answers with a zip bomb or an
|
|
7
|
+
// endless stream can't exhaust the memory of the machine running the audit.
|
|
6
8
|
|
|
7
9
|
import { request as httpRequest } from 'node:http';
|
|
8
10
|
import { request as httpsRequest } from 'node:https';
|
|
9
11
|
import { gunzipSync, inflateSync, inflateRawSync } from 'node:zlib';
|
|
10
12
|
|
|
11
13
|
export const TIMEOUT_MS = 15000;
|
|
14
|
+
// Large enough for a big llms-full.txt or OpenAPI document, small enough to be safe.
|
|
15
|
+
export const MAX_BYTES = 16 * 1024 * 1024;
|
|
12
16
|
|
|
13
17
|
/**
|
|
14
18
|
* One request, no redirects followed. Headers default to `accept: *\/*` and
|
|
15
19
|
* `accept-encoding: gzip, deflate`; a header set to null isn't sent (M4's no-Accept probe).
|
|
16
|
-
* Resolves, never rejects: { status, headers (a Headers object), text, error? }, status 0
|
|
20
|
+
* Resolves, never rejects: { status, headers (a Headers object), text, error?, tooBig? }, status 0
|
|
21
|
+
* on no answer. tooBig: the body passed maxBytes, raw or decoded; reading stopped and the text is empty.
|
|
17
22
|
*/
|
|
18
|
-
export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS } = {}) {
|
|
23
|
+
export function request(url, { method = 'GET', headers = {}, body, userAgent, timeoutMs = TIMEOUT_MS, maxBytes = MAX_BYTES } = {}) {
|
|
19
24
|
const target = new URL(url);
|
|
20
25
|
const send = target.protocol === 'http:' ? httpRequest : httpsRequest;
|
|
21
26
|
const all = { 'user-agent': userAgent, accept: '*/*', 'accept-encoding': 'gzip, deflate', ...headers };
|
|
22
27
|
if (body !== undefined) all['content-length'] = Buffer.byteLength(body);
|
|
23
28
|
const sent = Object.fromEntries(Object.entries(all).filter(([, v]) => v !== null && v !== undefined));
|
|
24
29
|
const none = (error) => ({ status: 0, headers: new Headers(), text: '', error });
|
|
30
|
+
const failed = (e) => none(e.name === 'AbortError' || e.name === 'TimeoutError' ? `no answer in ${timeoutMs / 1000} seconds` : e.message);
|
|
31
|
+
const headersOf = (res) => new Headers(Object.entries(res.headers).flatMap(([k, v]) => [].concat(v).map((x) => [k, String(x)])));
|
|
32
|
+
const mb = (n) => (n >= 1024 * 1024 ? `${Math.round(n / 1024 / 1024)} MB` : `${Math.round(n / 1024)} KB`);
|
|
25
33
|
return new Promise((resolve) => {
|
|
34
|
+
let settled = false;
|
|
35
|
+
const done = (out) => { if (!settled) { settled = true; resolve(out); } };
|
|
26
36
|
const req = send(target, { method, headers: sent, signal: AbortSignal.timeout(timeoutMs) }, (res) => {
|
|
27
37
|
const chunks = [];
|
|
28
|
-
|
|
29
|
-
res.on('
|
|
38
|
+
let size = 0;
|
|
39
|
+
res.on('data', (c) => {
|
|
40
|
+
size += c.length;
|
|
41
|
+
if (size > maxBytes) {
|
|
42
|
+
done({ status: res.statusCode, headers: headersOf(res), text: '', tooBig: true, error: `body over ${mb(maxBytes)}` });
|
|
43
|
+
req.destroy();
|
|
44
|
+
return;
|
|
45
|
+
}
|
|
46
|
+
chunks.push(c);
|
|
47
|
+
});
|
|
48
|
+
res.on('error', (e) => done(failed(e)));
|
|
30
49
|
res.on('end', () => {
|
|
31
|
-
const headers =
|
|
50
|
+
const headers = headersOf(res);
|
|
32
51
|
const raw = Buffer.concat(chunks);
|
|
33
52
|
const encoding = (headers.get('content-encoding') ?? 'identity').trim().toLowerCase();
|
|
34
53
|
const out = { status: res.statusCode, headers, text: '' };
|
|
54
|
+
const opts = { maxOutputLength: maxBytes };
|
|
35
55
|
try {
|
|
36
56
|
if (encoding === 'identity' || encoding === '') out.text = raw.toString('utf8');
|
|
37
|
-
else if (encoding === 'gzip' || encoding === 'x-gzip') out.text = gunzipSync(raw).toString('utf8');
|
|
38
|
-
else if (encoding === 'deflate') { try { out.text = inflateSync(raw).toString('utf8'); } catch { out.text = inflateRawSync(raw).toString('utf8'); } } // zlib-wrapped, or raw as some servers send it
|
|
57
|
+
else if (encoding === 'gzip' || encoding === 'x-gzip') out.text = gunzipSync(raw, opts).toString('utf8');
|
|
58
|
+
else if (encoding === 'deflate') { try { out.text = inflateSync(raw, opts).toString('utf8'); } catch (e) { if (e.code === 'ERR_BUFFER_TOO_LARGE') throw e; out.text = inflateRawSync(raw, opts).toString('utf8'); } } // zlib-wrapped, or raw as some servers send it
|
|
39
59
|
else out.error = `refused: ${encoding} body`;
|
|
40
|
-
} catch (e) {
|
|
41
|
-
|
|
60
|
+
} catch (e) {
|
|
61
|
+
if (e.code === 'ERR_BUFFER_TOO_LARGE') { out.tooBig = true; out.error = `${encoding} body decodes to over ${mb(maxBytes)}`; }
|
|
62
|
+
else out.error = `undecodable ${encoding} body: ${e.message}`;
|
|
63
|
+
}
|
|
64
|
+
done(out);
|
|
42
65
|
});
|
|
43
66
|
});
|
|
44
|
-
req.on('error', (e) =>
|
|
67
|
+
req.on('error', (e) => done(failed(e)));
|
|
45
68
|
req.end(body);
|
|
46
69
|
});
|
|
47
70
|
}
|
package/src/logs.mjs
CHANGED
|
@@ -7,6 +7,8 @@ import { DISCOVERY_FILES, AI_BOTS } from './helpers.mjs';
|
|
|
7
7
|
// Directories come before the generic "bot" match, assistants before AI crawlers.
|
|
8
8
|
export const FAMILIES = [
|
|
9
9
|
['This audit (readiness-audit)', /^readiness-audit\//i],
|
|
10
|
+
// Other audit tools read discovery files at least as often as agents do (docs/research/who-reads-discovery-files.md).
|
|
11
|
+
['Other audit tools', /security-pranks-audit|isitagentready|agentprobe|agent-tools\.cloud|semrushbot|ahrefsbot|lumar|seojuice|sitebulb|screaming frog/i],
|
|
10
12
|
['Agent and MCP directories', /agenstrybot|brickbluebot|agentprobe|mcpharbor|agenttrustbot|taifoonharvester|mcphub-probe|invokerankbot|proofbench|thepluginstorebot|protogrid-probe|glidemcpindex|agent-tools\.cloud/i],
|
|
11
13
|
['llms.txt directories', /llms\.?txt/i],
|
|
12
14
|
['Assistants fetching for a person', /chatgpt-user|claude-user|perplexity-user|meta-externalfetcher/i],
|
|
@@ -26,6 +28,16 @@ export function family(ua) {
|
|
|
26
28
|
// Families whose user agent can't show an agent: count them as one only when the same address called the API.
|
|
27
29
|
const UNVERIFIED = new Set(['Browsers', 'Headless browsers', 'Unidentified', 'No user agent']);
|
|
28
30
|
|
|
31
|
+
// Families named after a crawler or directory. A name is a claim: across one owner's 56 domains in a week of
|
|
32
|
+
// October 2026, none of 448k probes came from a verified bot, and one scanner used about 20 AI-crawler names
|
|
33
|
+
// (agent-security-pranks, "Bots in AI clothing"). A request under one of these names that asks for a secret file
|
|
34
|
+
// isn't the crawler it names.
|
|
35
|
+
const CRAWLER_FAMILIES = new Set(['Agent and MCP directories', 'llms.txt directories', 'Assistants fetching for a person', 'AI crawlers', 'Search crawlers']);
|
|
36
|
+
// Paths that only secret hunters ask for, from that research's list, minus the WordPress and PHP paths a real
|
|
37
|
+
// crawler fetches on a site that runs them.
|
|
38
|
+
const SECRET_PROBE = /\.env\b|\/\.git\/|\/\.svn\/|\.gitconfig|\/@fs\/|__vite|wp-config|actuator|heapdump|\/\.aws\b|\/\.ssh\b|id_rsa|\.sql\b|\.ds_store|phpinfo|\/proc\/self\/environ/i;
|
|
39
|
+
export const isSecretProbe = (path) => SECRET_PROBE.test(path);
|
|
40
|
+
|
|
29
41
|
const COMBINED = /^(\S+) \S+ \S+ \[[^\]]*\] "(\S+) (\S+)[^"]*" (\d{3}) \S+(?: "[^"]*" "([^"]*)")?/;
|
|
30
42
|
|
|
31
43
|
// Field names per host's JSON lines, from each host's docs (read 2026-10-21): Cloudflare Logpush, Vercel log
|
|
@@ -76,6 +88,7 @@ export function logTally() {
|
|
|
76
88
|
const readers = new Map(); // bucket -> Set(addr)
|
|
77
89
|
const apiAddrs = new Set();
|
|
78
90
|
const unverifiedAddrs = new Set();
|
|
91
|
+
const impostors = new Map(); // crawler family -> requests for secret files under its name
|
|
79
92
|
return {
|
|
80
93
|
add(line) {
|
|
81
94
|
lines++;
|
|
@@ -84,6 +97,7 @@ export function logTally() {
|
|
|
84
97
|
parsed++;
|
|
85
98
|
const fam = family(r.ua);
|
|
86
99
|
if (isApiCall(r)) { if (r.addr) apiAddrs.add(r.addr); if (UNVERIFIED.has(fam) && r.addr) unverifiedAddrs.add(r.addr); }
|
|
100
|
+
if (CRAWLER_FAMILIES.has(fam) && isSecretProbe(r.path)) impostors.set(fam, (impostors.get(fam) ?? 0) + 1);
|
|
87
101
|
const b = bucket(r.path);
|
|
88
102
|
if (!b) return;
|
|
89
103
|
if (!hits.has(b)) hits.set(b, new Map());
|
|
@@ -101,14 +115,15 @@ export function logTally() {
|
|
|
101
115
|
also_called_api: [...addrs].filter((a) => apiAddrs.has(a)).length,
|
|
102
116
|
};
|
|
103
117
|
}).sort((a, b) => b.requests - a.requests);
|
|
104
|
-
|
|
118
|
+
const crawlerNames = { requests: [...impostors.values()].reduce((a, b) => a + b, 0), families: Object.fromEntries([...impostors].sort((a, b) => b[1] - a[1])) };
|
|
119
|
+
return { lines, parsed, api_addresses: apiAddrs.size, unverified_api_addresses: unverifiedAddrs.size, crawler_names_on_secret_probes: crawlerNames, paths };
|
|
105
120
|
},
|
|
106
121
|
};
|
|
107
122
|
}
|
|
108
123
|
|
|
109
124
|
/** The report as text. */
|
|
110
125
|
export function formatLogs(rep) {
|
|
111
|
-
const out = [`Read ${rep.parsed} of ${rep.lines} log lines (${rep.lines - rep.parsed} not understood)
|
|
126
|
+
const out = [`Read ${rep.parsed} of ${rep.lines} log lines (${rep.lines - rep.parsed} not understood). Families are the names clients send, not verified.`, ''];
|
|
112
127
|
if (!rep.paths.length) out.push('No requests for discovery files.');
|
|
113
128
|
for (const p of rep.paths) {
|
|
114
129
|
out.push(`${p.path} ${p.requests} request${p.requests === 1 ? '' : 's'}${p.addresses ? `, ${p.addresses} address${p.addresses === 1 ? '' : 'es'}, ${p.also_called_api} of them also called the API` : ''}`);
|
|
@@ -116,5 +131,8 @@ export function formatLogs(rep) {
|
|
|
116
131
|
}
|
|
117
132
|
out.push('', `API calls came from ${rep.api_addresses} address${rep.api_addresses === 1 ? '' : 'es'}. ${rep.unverified_api_addresses} of them sent a browser, headless or missing user agent: count those as agents, since they called the API.`);
|
|
118
133
|
out.push('Browsers, headless browsers and unidentified clients above count as agents only where they also called the API (STANDARD.md, T3).');
|
|
134
|
+
const named = rep.crawler_names_on_secret_probes;
|
|
135
|
+
if (named.requests) out.push('', `${named.requests} request${named.requests === 1 ? '' : 's'} under crawler names asked for secret files (.env, .git, keys, dumps): ${Object.entries(named.families).map(([f, n]) => `${f} ${n}`).join(', ')}. Real crawlers don't ask for these, so treat those names as claims: count a crawler by the CDN's verified flag, the vendor's published addresses or reverse DNS, not its name (T3).`);
|
|
136
|
+
else out.push('', 'No request under a crawler name asked for secret files.');
|
|
119
137
|
return out;
|
|
120
138
|
}
|