ax-audit 3.6.0 → 4.0.0

package/docs/checks.md

@@ -1,8 +1,29 @@
 # Checks Reference
 
-ax-audit runs 18 checks. Fourteen are **weighted** (summing to 100% of the overall score); four are **informational** in 3.x — they run and report findings but carry weight 0 until v4.0, because score-affecting changes are treated as breaking (see [CHANGELOG 3.0.0](../CHANGELOG.md)).
+ax-audit runs 26 checks. Twenty-three are **weighted**, summing to 100. Three rest on draft specifications and stay at weight 0: scoring a site against a specification that may be renamed next quarter would make the number less trustworthy, not more.
 
-This page documents the **exact scoring** of every check: each deduction, bonus, and formula, extracted from the source. Every finding links to a step-by-step remediation guide at `lucioduran.com/projects/ax-audit/guides/<check-id>`.
+Checks are grouped into five areas, and reports are ordered by them: **content** (is there substance an agent can read?), **discovery** (can an agent find your machine-readable files?), **access** (can it actually retrieve them?), **policy** (what usage rights do you declare?), and **protocols** (what can an agent call?).
+
+Some checks are **conditional**. A blog has no API to describe, no MCP server to advertise and nothing to authorize, so those checks report **n/a** and are excluded from the score rather than counted as failures. Everything counted against a site is something the site could have done. `--profile api|mcp|agent|docs|commerce|all` forces them applicable, for auditing against what a site intends to become.
+
+## Weights
+
+| Area | Total | Checks |
+| --- | --- | --- |
+| Content | 33 | html-rendering 11 · agent-operability 7 · structured-data 6 · seo-basics 5 · content-negotiation 4 |
+| Access | 24 | agent-access 9 · ai-directives 6 · http-hygiene 4 · tls-https 3 · crawl-efficiency 2 |
+| Discovery | 21 | robots-txt 9 · llms-txt 5 · http-headers 4 · sitemap 2 · meta-tags 1 |
+| Protocols | 13 | api-discovery 4 · agent-card 3 · mcp-discovery 3 · agent-skills 2 · auth-discovery 1 |
+| Policy | 9 | usage-policy 4 · security-txt 3 · rsl 2 |
+| Draft specifications | 0 | ai-catalog · webmcp · commerce-discovery |
+
+Content leads because the failure that breaks the most agents is a page with nothing in its HTML: most crawlers do not run JavaScript, and a site whose content appears only after hydration is invisible no matter how many discovery files it publishes. Access is second because a firewall rule or a `nosnippet` directive silently undoes everything else, and those are the failures operators are least likely to know about. llms.txt sits at 5 rather than the 11 it carried in 3.x, because most published files are never fetched by an AI search crawler.
+
+Weights live in `CHECK_WEIGHTS` in `src/constants.ts`, and only there. Checks used to declare their own alongside it, and the two drifted.
+
+Every probed path is labelled by standing — **IANA-registered**, **vendor convention**, **draft**, or **legacy** — because the agent web mixes registered URIs with drafts that get renamed. A missing draft file is not the same kind of finding as a missing registered one, and reports say which is which.
+
+This page documents the **exact scoring** of every check: each deduction, bonus, and formula, extracted from the source. Every finding links to a step-by-step remediation guide at `axrush.com/guides/<check-id>`.
 
 **Reading the tables:** each check starts at 100 unless noted. Deductions stack additively; `buildResult` clamps the final score to [0, 100]. "Hard fail" rows short-circuit the check.
 
@@ -24,10 +45,19 @@ This page documents the **exact scoring** of every check: each deduction, bonus,
 | No Markdown links | −10 |
 | Content under 100 characters | −10 |
 | `/llms-full.txt` also available | **+10** (capped at 100) |
+| Broken links among a sample of 15 | informational, 0 in 3.x |
+| Redirecting or duplicated links | informational, 0 |
+| No `rel="describedby"` pointing at the file (llms.txt v2) | informational, 0 |
+| No per-page Markdown mirror (`/index.md`, `/index.html.md`) | informational, 0 |
+| File over 50 KB | informational, 0 |
+
+Every report states plainly that Google says Search ignores llms.txt and that most published files are never fetched by an AI search crawler, while Claude Code, Cursor and OpenCode do read it. It is a developer-tooling signal, not a search-visibility one.
 
 ### `robots-txt` — 11%
 
-AI-crawler configuration. Core crawlers: GPTBot, ClaudeBot, ChatGPT-User, Claude-SearchBot, Google-Extended, PerplexityBot, OAI-SearchBot, CCBot.
+AI-crawler configuration. Scoring runs against the frozen 3.x core set (GPTBot, ClaudeBot, ChatGPT-User, Claude-SearchBot, Google-Extended, PerplexityBot, OAI-SearchBot, CCBot); the wider September-2026 core set adds Meta-ExternalAgent, Applebot-Extended, Amazonbot and Bytespider, reported but not scored until 4.0.
+
+Findings are tiered by what a client does with a page, because that determines the cost of blocking it. Blocking a **training** crawler is a policy choice and is reported as such. Blocking a **search** crawler removes the site from that assistant's answers. Blocking a **user-triggered fetcher** often does nothing, because most vendors document that robots.txt may not apply to them.
 
 | Condition | Points |
 | --- | --- |
@@ -38,7 +68,10 @@ AI-crawler configuration. Core crawlers: GPTBot, ClaudeBot, ChatGPT-User, Claude
 | Known AI crawler(s) explicitly blocked (`Disallow: /`) | −3 per crawler |
 | No `Sitemap:` directive | −5 |
 | Partial path restrictions on AI crawlers | warn only, 0 |
-| [Content Signals](https://contentsignals.org) findings (declared / malformed / unknown / missing) | informational, 0 in 3.x |
+| Blocking a crawler token added in 3.7 (`meta-webindexer`, `Amzn-SearchBot`, …) | informational, 0 in 3.x |
+| Rules targeting a retired or fictional token (`GeminiBot`, `Claude-Web`, `NeevaBot`, …) | informational, 0 |
+| [Content Signals](https://contentsignals.org) findings, including the `use=immediate\|reference\|full` field | informational, 0 in 3.x |
+| [IETF AIPREF](https://datatracker.ietf.org/wg/aipref/documents/) `Content-Usage:` findings, including vocabulary mix-ups | informational, 0 in 3.x |
 
 ### `html-rendering` — 9%
 
@@ -72,54 +105,84 @@ JSON-LD on the homepage. Key entity types: Person, Organization, WebSite, WebPag
 | Only one key entity type | −10 |
 | No `@graph` array | −5 |
 | No `BreadcrumbList` | −5 |
+| No `author`, no `sameAs`, or an author with no `publisher` | informational, 0 in 3.x |
+| No `dateModified` / `datePublished`, a future date, or content over 730 days old | informational, 0 |
+| `headline` or `name` not present in the visible text | informational, 0 |
+
+The visible-text comparison is Google's one explicit requirement for structured data and AI features. It reads static HTML, so text rendered by script reads as missing here too, and the finding says so.
 
 ### `http-headers` — 9%
 
-Security headers, AI discovery `Link` headers (RFC 5988-parsed), CORS on `.well-known`.
+Security headers, AI discovery `Link` headers (RFC 5988-parsed), CORS on `.well-known`. Either Agent Card path satisfies the discovery-link requirement.
 
 | Condition | Points |
 | --- | --- |
 | No headers retrievable | **hard fail → 0** |
 | Missing critical security header (HSTS, X-Content-Type-Options) | −10 each |
 | Only 1–3 of the 7 tracked security headers present | −5 |
-| `Link` header missing both llms.txt and agent.json references | −15 |
+| `Link` header missing both llms.txt and the Agent Card | −15 |
 | `Link` header missing one of the two | −5 |
-| No CORS on `/.well-known/agent.json` | −10 |
+| No CORS on the Agent Card | −10 |
+| Additional discovery relations (`describedby`, `api-catalog`, `service-desc`, `service-doc`, `ai-catalog`, `c2pa-manifest`, `license`, markdown `alternate`) and the `X-Llms-Txt` header | informational, 0 in 3.x |
 
-### `agent-json` — 7%
+### `agent-card` — 7%
 
-`/.well-known/agent.json` [A2A Agent Card](https://a2a-protocol.org). Required fields: `name`, `description`, `url`, `skills`.
+The [A2A Agent Card](https://a2a-protocol.org), probed at `/.well-known/agent-card.json` (IANA-registered since A2A v0.3.0, 2025-07-30) and then at the pre-0.3 path `/.well-known/agent.json`. *Former id: `agent-json`, still accepted in `--checks` and in saved baselines.*
+
+Two spec generations are in the wild, and the check detects which one a card follows from its own structure rather than from a version field:
+
+- **A2A 1.0** (2026-03-12) declares every endpoint inside `supportedInterfaces[]`. Required: `name`, `description`, `version`, `capabilities`, `supportedInterfaces`, `defaultInputModes`, `defaultOutputModes`, `skills`.
+- **A2A 0.3** declares a top-level `url` and `protocolVersion`. Required: those two plus `name`, `description`, `version`, `capabilities`, `defaultInputModes`, `defaultOutputModes`, `skills`.
 
 | Condition | Points |
 | --- | --- |
-| Not found | **hard fail → 0** |
+| Not found at either path | **hard fail → 0** |
 | Invalid JSON | **→ 10** |
-| Wrong Content-Type (expected `application/json`) | −5 |
-| Missing required field | −15 per field |
-| `url` on a different origin | −5 |
-| `url` not an absolute URL | −5 |
+| Served only from the pre-0.3 `agent.json` path | warn only, 0 |
+| Wrong Content-Type (expected `application/json` or `application/a2a+json`) | −5 |
+| Card shape matches neither generation | −30 |
+| Missing required field for the detected generation | −15 per field |
+| `supportedInterfaces[]` empty (1.0) | −15 |
+| Interface missing `url`, `protocolBinding` or `protocolVersion` (1.0) | −10 |
+| Unrecognised `protocolBinding` (not JSONRPC / GRPC / HTTP+JSON) | −5 |
+| Interface or `url` on a different origin | −5 |
+| `url` not an absolute URL (0.3) | −5 |
 | `skills` empty | −10 |
 | `skills` entries missing `id` or `description` | −5 |
-| No `protocolVersion` | −5 |
-| No optional fields (`capabilities`, `authentication`, `documentationUrl`) | −5 |
+| Uses `authentication`, removed from the spec in 0.2.x | −5 |
+| No optional descriptive fields (`provider`, `documentationUrl`, `iconUrl`) | −5 |
+
+### `mcp-discovery` — 7%
+
+How an agent finds this site's [Model Context Protocol](https://modelcontextprotocol.io) server. *Former id: `mcp`.*
 
-### `mcp` — 7%
+`/.well-known/mcp.json` was never part of the MCP specification. What emerged instead is the **server card**, which deliberately carries no `tools[]` — tool lists come from a live `tools/list` call, and a static copy drifts the day it is written. Discovery is probed in this order:
 
-`/.well-known/mcp.json` [Model Context Protocol](https://modelcontextprotocol.io) server configuration.
+1. `/.well-known/ai-catalog.json` entries of type `application/mcp-server-card+json` *(draft)*
+2. `/.well-known/mcp/server-card.json`, `/.well-known/mcp/server-cards.json` *(vendor convention: Cloudflare, Mintlify)*
+3. `<endpoint>/server-card` for `/mcp`, `/api/mcp`, `/sse` *(the MCP extension's own recommendation)*
+4. `/.well-known/mcp.json` *(legacy)*
+
+An HTML response counts as absence, not a malformed card: SPA catch-alls answer every unknown path with the index shell.
+
+**When a server card is found:**
 
 | Condition | Points |
 | --- | --- |
-| Not found | **hard fail → 0** |
+| Wrong Content-Type (expected `application/json` or `application/mcp-server-card+json`) | −5 |
 | Invalid JSON | **→ 10** |
-| Wrong Content-Type | −5 |
-| Missing `name` | −10 |
-| Missing `description` | −5 |
-| No `tools` array, or empty | −15 |
-| No tool has a description | −10 |
-| Some tools missing descriptions | −5 |
-| No `resources` | −5 |
-| No protocol version | −5 |
+| Missing `$schema`, `name`, `version` or `description` | −15 each |
+| `name` not in reverse-DNS form | −5 |
+| No `remotes[]` | −15 |
+| Remote with an unrecognised transport (not `streamable-http` / `sse`) | −5 |
+| Remote missing a `url` | −10 |
+| No `supportedProtocolVersions` | −10 |
+| Only pre-2025-06 protocol revisions | −10 |
+| Unrecognised protocol version | −5 |
 | No CORS headers | −10 |
+| Declares `tools[]`, which the schema omits by design | warn only, 0 |
+
+**When only `/.well-known/mcp.json` is found**, the pre-3.7 rules are applied unchanged so the score is exactly what 3.6 produced (missing `name` −10, missing `description` −5, no `tools` −15, no tool descriptions −10 / −5, no `resources` −5, no version −5, no CORS −10, wrong Content-Type −5). The path itself is reported, not penalised.
 
 ### `seo-basics` — 7%
 
@@ -161,7 +224,7 @@ AI meta tags (`ai:summary`, `ai:content_type`, `ai:author`, `ai:api`, `ai:agent_
 | 0 AI meta tags | −18 |
 | Only 1–2 AI meta tags | −12 |
 | No `rel="alternate"` → llms.txt | −12 |
-| No `rel="alternate"` → agent.json | −8 |
+| No `rel="alternate"` → the Agent Card | −8 |
 | No `rel="me"` identity links | −8 |
 | No Open Graph tags at all | −12 |
 | OG required incomplete (`og:title`, `og:description`, `og:url`, `og:type`) | −8 |
@@ -170,21 +233,33 @@ AI meta tags (`ai:summary`, `ai:content_type`, `ai:author`, `ai:api`, `ai:agent_
 | Twitter required incomplete (`twitter:card`, `twitter:title`, `twitter:description`) | −5 |
 | Twitter recommended incomplete (`twitter:image`) | −2 |
 
-### `openapi` — 6%
+### `api-discovery` — 6%
+
+Whether an agent can find, and read, a machine-readable API description. *Former id: `openapi`.*
 
-`/.well-known/openapi.json`.
+`/.well-known/openapi.json` is a folk convention — unregistered, and not prescribed by the OpenAPI specification, which recommends the file name `openapi.json` without a location. Discovery is probed in order of authority:
+
+1. `/.well-known/api-catalog` *(RFC 9727, IANA-registered)* → its `service-desc` links
+2. `Link: rel="service-desc"` on the homepage *(RFC 8631)*
+3. `<link rel="service-desc">` in the HTML head
+4. Conventional paths: `/.well-known/openapi.json`, `/openapi.json`, `/openapi.yaml`, `/.well-known/openapi.yaml`, `/api/openapi.json`, `/v1/openapi.json`, `/swagger.json`, `/api-docs`, `/asyncapi.json`, `/arazzo.json`
 
 | Condition | Points |
 | --- | --- |
-| Not found | **hard fail → 0** |
+| No description found by any mechanism | **hard fail → 0** |
 | Invalid JSON | **→ 10** |
-| Wrong Content-Type | −5 |
+| Wrong Content-Type on a JSON document | −5 |
 | No `openapi`/`swagger` version field | −20 |
 | Swagger 2.x instead of OpenAPI 3.x | −10 |
 | Missing `info.title` | −10 |
 | Missing `info.description` | −5 |
 | No `paths` documented | −15 |
 | No `servers` | −5 |
+| Found only by guessing a path (nothing links to it) | warn only, 0 |
+| `operationId` coverage below 100% | informational, 0 in 3.x |
+| API catalog present but empty, or entries missing `anchor` / `service-doc` | warn only, 0 |
+
+YAML descriptions are recognised and reported, but only surface-validated: ax-audit ships no YAML parser, and the finding says so rather than pretending otherwise.
 
 ### `tls-https` — 5%
 
@@ -220,21 +295,13 @@ Located via robots.txt `Sitemap:` or `/sitemap.xml`. Limits: 50,000 URLs / 50 MB
 | `<lastmod>` coverage < 50% | −5 |
 | Newest `<lastmod>` older than 365 days | −5 |
 
-### `well-known-ai` — 3%
-
-Emerging AI discovery files. **Purely proportional** — no deductions:
-
-```
-score = round(present / 5 × 100)
-```
-
-over `/.well-known/ai.txt` (Spawning), `/.well-known/genai.txt`, `/ai-plugin.json`, `/agents.json`, `/.well-known/nlweb.json`. Files with invalid content produce warnings without counting as present.
-
 ---
 
-## Informational checks (weight 0 in 3.x)
+## Checks on draft specifications (weight 0)
+
+`ai-catalog`, `webmcp` and `commerce-discovery` run on every audit and report full findings, but never affect the score. Each rests on a specification that is still a draft and may be renamed; scoring a site against one would make the number less trustworthy, not more.
 
-These run on every audit and report full findings, but do not affect the overall score or baselines. They gain weight in v4.0.
+`well-known-ai` was removed in 4.0. Re-verification found three of its five scored files had no consumer: `/.well-known/nlweb.json` appears in no NLWeb release, `genai.txt` has no specification, and `/ai-plugin.json` described a product shut down in 2024. Its live probes moved into the checks that own them — TDMRep into `usage-policy`, the Web Bot Auth key directory and `AGENTS.md` into the reporting they belong to.
 
 ### `content-negotiation` — Markdown for Agents
 
@@ -250,6 +317,11 @@ Probes the homepage with `Accept: text/markdown` — the pattern served by Cloud
 | Body is a relabeled HTML document | −25 |
 | `Vary` does not include `Accept` | −15 |
 | Markdown not smaller than HTML | warn only, 0 |
+| Origin-reported token counts (`x-markdown-tokens` / `x-original-tokens`) | informational, 0 |
+| No frontmatter, or frontmatter with no title / canonical URL / date | informational, 0 |
+| User-agent negotiation or a `.md` suffix URL, when Accept negotiation fails | informational, 0 |
+
+The probe sends the Accept header a real agent sends (`text/markdown, text/html;q=0.9, */*;q=0.1`). A bare `text/markdown` would pass against an implementation that fails every real request.
 
 ### `rsl` — Really Simple Licensing
 
@@ -270,23 +342,27 @@ Probes the homepage with `Accept: text/markdown` — the pattern served by Cloud
 | Tokens outside the RSL 1.0 vocabulary (incl. pre-1.0 draft tokens) | −5 |
 | Invalid `payment` type | −5 |
 
-### `agent-access` — Cloaking detection
+### `agent-access` — blocking and cloaking detection
 
-Probes the homepage with realistic UAs for the 8 core AI crawlers and compares status + visible text against the default-UA baseline. **Credit-ratio formula:**
+Probes the homepage with realistic user agents for the 10 core AI crawlers that actually issue requests (`Google-Extended` and `Applebot-Extended` are robots.txt control tokens, so probing with them tests nothing) and compares each response against the default-UA baseline. **Credit-ratio formula:**
 
 ```
-score = round(credit / 8 × 100)
+score = round(credit / 10 × 100)
 ```
 
+Responses are classified by *how* a request was turned away, because the remedies differ completely:
+
 | Outcome per crawler | Credit |
 | --- | --- |
-| Equivalent response | 1 |
-| Blocked, consistent with robots.txt `Disallow` (explicit or wildcard) | 1 |
-| 200 but < 50% of baseline visible text (baseline ≥ 200 chars) | 0.5 |
-| Blocked while robots.txt allows (or doesn't restrict) it | 0 |
+| Same page as a regular client | 1 |
+| Refused, consistent with an explicit robots.txt `Disallow` | 1 |
+| Priced access (`402` + `crawler-price`) or an RSL licence challenge | 1 |
+| JavaScript challenge (`cf-mitigated: challenge`, `x-vercel-mitigated`, AWS WAF's `202`), Web Bot Auth demand, rate limit, or a refusal from a bot-verifying CDN | 0.75, **inconclusive** |
+| Different page than the baseline: less text, or a changed title / h1 / JSON-LD block count | 0.5 |
+| Refused by a plain origin while robots.txt permits it | 0 |
 | Baseline request itself fails | **hard fail → 0** |
 
-Caveat: WAFs using Web Bot Auth / IP verification may pass the real crawler while rejecting this unverified probe — confirm against WAF logs before changing rules.
+The probe is unsigned and comes from the auditor's own network, so an edge that verifies crawlers by IP range or Web Bot Auth signature will reject it while admitting the genuine crawler. Those outcomes are reported as inconclusive with the exact header observed, never as "blocks AI crawlers". Confirm against WAF logs before changing a rule.
 
 ### `crawl-efficiency`
 
@@ -299,6 +375,161 @@ Caveat: WAFs using Web Bot Auth / IP verification may pass the real crawler whil
 | Validator present but conditional request not answered with `304` | −15 |
 | Page > 2 MB decompressed | −10 |
 | Page > 500 KB decompressed | −5 |
+| Content tokens, wire tokens and markup share (estimated at 4 chars per token) | informational, 0 |
+| Response over 2s | informational, 0 |
+
+---
+
+### `ai-directives` — page-level AI controls
+
+The controls Google and Microsoft document that they honor, read from robots meta tags and `X-Robots-Tag` (including the user-agent-scoped header form).
+
+| Condition | Points |
+| --- | --- |
+| Homepage HTML unavailable | **hard fail → 0** |
+| `noindex` or `none` | **hard fail → 0** — invisible to every search-grounded assistant |
+| `nosnippet`, or `max-snippet:0` | −30 — excluded as a direct input to Google AI Overviews and AI Mode |
+| `noarchive` | −30 — excluded from Microsoft Copilot grounding |
+| `nocache` | −10 — Copilot may use only the URL, title and snippet |
+| `data-nosnippet` wrapping `<main>`, `<article>` or `<body>` | −20 |
+| `noimageindex` | −5 |
+| `max-snippet:[n]` under 160 | warn only, 0 |
+| `noai` / `noimageai` | reported, 0 — no major operator documents honoring them |
+| robots.txt disallows `Google-Extended` with no snippet directive set | warn only, 0 |
+
+That last row is the finding this check exists for. `Google-Extended` governs Gemini training and grounding in Gemini Apps and Vertex AI, **not** AI Overviews, which follow Googlebot and the snippet directives. A site that disallows it expecting to leave AI Overviews has opted out of the thing it probably did not mind.
+
+### `usage-policy` — do your usage signals agree?
+
+Normalises every machine-readable usage declaration onto three questions — may you train on it, ground an answer in it, index it — and reports where they disagree.
+
+| Mechanism | Training | Grounding | Search |
+| --- | --- | --- | --- |
+| Content Signals (robots.txt or header) | `ai-train=yes\|no` | `ai-input=yes\|no` | `search=yes\|no` |
+| IETF AIPREF (robots.txt or header) | `train-ai=y\|n` | *(no category yet)* | `search=y\|n` |
+| RSL licence | `ai-train` | `ai-input` | `ai-index`, `search` |
+| TDMRep (meta > header > well-known) | `tdm-reservation: 0\|1` | — | — |
+| robots meta | `noai` | — | — |
+
+| Condition | Points |
+| --- | --- |
+| No declaration of any kind | **→ 40** |
+| Two mechanisms give opposite answers on one dimension | −25 per dimension |
+| `Content-Usage` header outside the AIPREF vocabulary | warn only, 0 |
+| A dimension no declaration covers | warn only, 0 |
+
+Every report states that only robots.txt access rules are documented as honored by major AI operators. The rest are declarations whose weight is legal rather than technical.
+
+### `http-hygiene` — status-code honesty
+
+| Condition | Points |
+| --- | --- |
+| A nonexistent path returns 200 | −30 |
+| A nonexistent path redirects | −20 |
+| A 429 with no `Retry-After` | −20 |
+| A 429 with `Retry-After` on the second request | −10 |
+| `HEAD` refused (405/501) | −10 |
+| Over one redirect hop to the homepage | −10 |
+| No `Content-Type` header | −10 |
+| No charset in the header or the document | −10 |
+| `<html lang>` disagrees with `Content-Language` | −5 |
+| A nonexistent path returns 403/401 | −5 |
+| Empty 404 body | −5 |
+| The 404 probe was challenged by bot management | warn only, 0 |
+
+### `ai-catalog` — the index of everything callable
+
+Discovery, in the order Lighthouse's `ard-schema` audit uses: robots.txt `Agentmap:`, `Link: rel="ai-catalog"`, `<link rel="ai-catalog">`, then `/.well-known/ai-catalog.json` and `/.well-known/ard.json`. Both specifications are drafts, so absence warns and scores nothing.
+
+| Condition | Points |
+| --- | --- |
+| Catalog is not valid JSON | **→ 10** |
+| An entry points at a document that cannot be fetched | −15 each |
+| No entries | −20 |
+| Entry missing identifier, type, or url/data | −10 |
+| No `specVersion` / no `host` | −5 each |
+| Entry served with a different media type than declared | −5 |
+
+### `agent-skills` — installable procedures
+
+Conditional: **n/a** unless the site has a developer-facing surface (documentation links, llms.txt, or an API description). Probes `/.well-known/agent-skills/index.json`, `/.well-known/skills/index.json`, then `/skill.md`.
+
+| Condition | Points |
+| --- | --- |
+| Index is not valid JSON | **→ 10** |
+| A sampled skill is unreachable, or its frontmatter name disagrees with the index | −10 per problem |
+| Index lists no skills | −30 |
+| A skill has no description | −15 |
+| No entry carries a url | −15 |
+| A skill name is outside `[a-z0-9-]{1,64}` | −10 |
+| Malformed digest, unknown type, over-long description, or no `$schema` | −5 each |
+| No skill declares a digest | −5 |
+| A single `/skill.md` with no index | −20 |
+
+### `webmcp` — forms as callable tools
+
+Conditional: **n/a** on a page with no forms and no WebMCP code. Never asks for WebMCP — it is a Community Group draft in a Chrome origin trial.
+
+| Condition | Points |
+| --- | --- |
+| `toolname` with no `tooldescription`, or the reverse | −30 |
+| Parameters with no `toolparamdescription` | −15 |
+| Tool name is not a usable identifier | −10 |
+| Deprecated `navigator.modelContext` namespace | −10 |
+| Forms present, none annotated | warn only, 0 |
+
+### `agent-operability` — can a browser agent work this page?
+
+Browser agents read the accessibility tree, not the pixels. A `<div onclick>` styled as a button has no role and no name, so it does not appear in the tree at all: the agent does not see a button it cannot press, it sees nothing.
+
+| Condition | Points |
+| --- | --- |
+| Homepage HTML unavailable | **hard fail → 0** |
+| Under 90% of buttons and links have an accessible name | −20 |
+| Under 90% of form controls are labelled | −20 |
+| Clickable elements that are not buttons or links (no `role` + `tabindex`) | −15 |
+| A CAPTCHA or `<meta http-equiv="refresh">` on the entry page | −15 |
+| Links with no `href` or a `javascript:` one | −10 |
+| Tables with no `<th>` | −10 |
+| Untitled iframes, `<time>` without `datetime`, heading-level skips, unsized media, no `<html lang>` | −5 each |
+
+Names are read from visible text, `aria-label`, `aria-labelledby`, `title`, image `alt`, and SVG `<title>`. Labels from `<label for>`, a wrapping label, or ARIA.
+
+Every run ends with a method note: this reads markup, not a rendered accessibility tree, so labels attached by script and roles computed at runtime are invisible to it. A low proportion is a prompt to check the real tree, not a count to act on blindly. Every finding is also a plain accessibility defect.
+
+### `commerce-discovery` — Universal Commerce Protocol
+
+Conditional: **n/a** unless the page shows storefront signals. A lone `Offer` is a price statement, not a catalog, so it counts only alongside a cart link.
+
+| Condition | Points |
+| --- | --- |
+| Profile requires authentication | **hard fail → 0** |
+| Profile is not valid JSON | **→ 10** |
+| No `ucp` object | **→ 20** |
+| No services declared | −25 |
+| A declared schema URL cannot be fetched | −20 |
+| No version | −20 |
+| No payment handlers | −15 |
+| Version is not a specification date | −10 |
+| No signing keys | −10 |
+| No schema URL declared | −10 |
+| Service name not in reverse-DNS form, unnamed handler | −5 each |
+
+The OpenAI and Stripe Agentic Commerce Protocol defines no manifest, and AP2 advertises through an A2A card extension, so neither is probed.
+
+### `auth-discovery` — can an agent get credentials?
+
+Conditional: **n/a** unless the site exposes an API description, API catalog, MCP server card or commerce profile. Follows the RFC 9728 chain from `WWW-Authenticate` or `/.well-known/oauth-protected-resource` to the authorization server's RFC 8414 or OpenID metadata.
+
+| Condition | Points |
+| --- | --- |
+| Metadata names no authorization server | **→ 40** |
+| Authorization server publishes no discovery metadata | −30 |
+| Invalid issuer URL | −25 |
+| Missing `issuer`, `authorization_endpoint` or `token_endpoint` | −15 each |
+| No PKCE with `S256` | −15 |
+| No dynamic registration and no Client ID Metadata Documents | −10 |
+| No `resource` identifier | −10 |
 
 ---
 
@@ -312,6 +543,8 @@ overall = round( Σ (score_i / 100 × weight_i) / Σ weight_i × 100 )
 
 When every selected check has weight 0 (e.g. `--checks rsl`), the overall falls back to a plain average of check scores.
 
+Checks reporting `applicable: false` are excluded from both the numerator and the denominator. A check whose meta exists but produced no result — because it crashed — still counts at full weight, so a broken check cannot inflate a score by shrinking the denominator.
+
 | Grade | Score | Exit code |
 | --- | --- | --- |
 | Excellent | 90–100 | 0 |

package/docs/cli.md

@@ -65,3 +65,39 @@ npx ax-audit $(cat urls.txt) --concurrency 4 --json > batch.json
 # Show me only what is broken
 npx ax-audit https://your-site.com --only-failures
 ```
+
+## Agent-surface profiles
+
+Protocol checks (`api-discovery`, `agent-card`, `mcp-discovery`, `agent-skills`, `auth-discovery`, `commerce-discovery`) apply only where the site has the corresponding surface. A blog has no API to describe, so scoring it zero would say something false; those checks report **n/a** and leave the denominator.
+
+`--profile` overrides that, for auditing against what a site intends to build rather than what it already has.
+
+```bash
+ax-audit https://example.com --profile api        # audit as though it offered an API
+ax-audit https://example.com --profile all        # every protocol check applies
+```
+
+Values: `auto` (default, detect from the site), `api`, `mcp`, `agent`, `docs`, `commerce`, `all`.
+
+## Focusing on one area
+
+```bash
+ax-audit https://example.com --category access              # only access checks
+ax-audit https://example.com --category content,discovery   # two areas
+```
+
+`--category` narrows an explicit `--checks` selection rather than replacing it, and errors if the intersection is empty.
+
+## Per-area CI gates
+
+An overall score hides an area that is entirely broken: a site can score 80 while every access check fails, because the other four carry it.
+
+```bash
+ax-audit https://example.com --fail-on-category access:70,content:80
+```
+
+Each area is reported against its threshold on stderr, so the output survives `--output json` being piped elsewhere. An area with no applicable checks is reported as not evaluated rather than failed: a build must not break because a site has no commerce surface.
+
+## Baselines across a scoring change
+
+Baselines record which scoring model produced them. Comparing a baseline written by an older version shows the deltas but suspends regression gating, because a rescore is not something the site did. Re-save with `--save-baseline` to resume gating. Checks that changed applicability are excluded from regressions and improvements for the same reason.

package/docs/concepts.md

@@ -7,7 +7,7 @@
 Agents are not browsers. Three differences drive every check:
 
 1. **They mostly don't run JavaScript.** GPTBot, ClaudeBot, CCBot and most crawlers fetch raw HTML. A client-rendered SPA that returns an empty `<div id="root">` is, to them, a blank page. (`html-rendering`, `content-negotiation`)
-2. **They look for declared structure, not visual layout.** An agent would rather read a `/llms.txt` summary or a JSON-LD graph than infer meaning from your CSS grid. (`llms-txt`, `structured-data`, `meta-tags`, `agent-json`, `mcp`, `openapi`)
+2. **They look for declared structure, not visual layout.** An agent would rather read a `/llms.txt` summary or a JSON-LD graph than infer meaning from your CSS grid. (`llms-txt`, `structured-data`, `meta-tags`, `agent-card`, `mcp-discovery`, `api-discovery`)
 3. **Their access is a policy and economic question, not just a technical one.** Who may crawl, for what use, at what price, under what license — these now have machine-readable answers. (`robots-txt`, Content Signals, `rsl`, `agent-access`)
 
 Bot traffic is projected to exceed human traffic by 2029. AX is the interface layer for that shift.
@@ -30,11 +30,15 @@ These answer: *can an agent find your content and actually read it?*
 
 | Standard | What it is | Check |
 | --- | --- | --- |
-| **[A2A — Agent2Agent](https://a2a-protocol.org)** | An "Agent Card" at `/.well-known/agent.json` advertising your agent's identity and skills, so other agents can interoperate. | `agent-json` |
-| **[MCP — Model Context Protocol](https://modelcontextprotocol.io)** | A manifest at `/.well-known/mcp.json` describing tools and resources an agent can call. The emerging standard for exposing capabilities to LLMs. | `mcp` |
-| **[OpenAPI](https://www.openapis.org)** | The long-standing machine-readable API description; agents use it to call your endpoints. | `openapi` |
-| **Emerging discovery files** | `ai.txt`, `genai.txt`, `ai-plugin.json`, `agents.json`, `nlweb.json` — competing/early conventions, scored as coverage bonus. | `well-known-ai` |
-| **AI meta tags & discovery links** | `ai:*` meta tags and `rel="alternate"` links pointing agents to your llms.txt / agent.json. | `meta-tags` |
+| **[A2A — Agent2Agent](https://a2a-protocol.org)** | An Agent Card at `/.well-known/agent-card.json` advertising your identity and skills, so other agents can interoperate. The path moved there in v0.3.0 and is IANA-registered; v1.0 changed the card's shape. | `agent-card` |
+| **[MCP — Model Context Protocol](https://modelcontextprotocol.io)** | A **server card** identifying your MCP server and its remote endpoints. Note it carries no tool list: tools come from a live `tools/list` call, and a static copy drifts. `/.well-known/mcp.json` was never part of the specification. | `mcp-discovery` |
+| **[OpenAPI](https://www.openapis.org)** and **[RFC 9727](https://www.rfc-editor.org/rfc/rfc9727.html)** | The machine-readable API description, plus the registered `/.well-known/api-catalog` that points at it. Agents use both to call your endpoints without a human reading your docs. | `api-discovery` |
+| **[Agent Skills](https://agentskills.io)** | SKILL.md documents an agent installs and follows — setup steps, argument shapes, mistakes to avoid. Answers "how do I do the thing this site is for", where llms.txt answers "what is here". | `agent-skills` |
+| **[RFC 9728 / RFC 8414](https://www.rfc-editor.org/rfc/rfc9728.html)** | OAuth metadata. A human hitting a 401 reads your docs; an agent cannot, so the answer has to be in the response. | `auth-discovery` |
+| **[UCP](https://developers.google.com/merchant/ucp/guides/ucp-profile)** | The Universal Commerce Protocol profile at `/.well-known/ucp` — the one agentic-commerce specification with published site-side discovery. | `commerce-discovery` |
+| **[WebMCP](https://webmachinelearning.github.io/webmcp/)** | Declaring a form as a callable tool, so an agent invokes it rather than driving it pixel by pixel. A Community Group draft in a Chrome origin trial. | `webmcp` |
+| **AI catalogs** | `ai-catalog.json` and `ard.json`, two competing drafts for one index listing everything above, so a client stops probing four conventions. | `ai-catalog` |
+| **AI meta tags & discovery links** | `ai:*` meta tags and the `rel` relations — `alternate`, `describedby`, `service-desc`, `api-catalog` — that point agents at your files instead of making them guess paths. | `meta-tags`, `http-headers` |
 
 These answer: *once an agent arrives, can it understand what you offer and act on it?*
 
@@ -44,14 +48,20 @@ This is the newest and fastest-moving family — the response to "AI scraped my
 
 | Standard | What it is | Check |
 | --- | --- | --- |
-| **[Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)** | The original robots.txt — *who* may crawl *what*. ax-audit grades coverage of 48 known AI crawlers. | `robots-txt` |
-| **[Content Signals](https://contentsignals.org)** | A robots.txt extension (Cloudflare, CC0) declaring *how* content may be used after access: `search`, `ai-input`, `ai-train`. Served by default on 3.8M+ Cloudflare domains. | `robots-txt` (findings) |
-| **[RSL — Really Simple Licensing](https://rslstandard.org)** | A full machine-readable licensing layer (license.xml): permits/prohibits vocabularies, payment models (free, attribution, pay-per-crawl, pay-per-inference). Endorsed by 1,500+ publishers. | `rsl` |
-| **Cloaking integrity** | Not a standard but a failure mode: your stated policy (robots.txt allows GPTBot) contradicting enforcement (WAF returns 403). | `agent-access` |
+| **[Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)** | The original robots.txt — *who* may crawl *what*. ax-audit knows 57 AI clients, grouped by what they do with a page, because blocking a search crawler costs citations while blocking a training crawler is a policy choice. | `robots-txt` |
+| **[Content Signals](https://contentsignals.org)** | A robots.txt extension (Cloudflare, CC0) declaring *how* content may be used after access: `search`, `ai-input`, `ai-train`, plus the `use` field added in 2026. Served by default on millions of Cloudflare domains. | `robots-txt`, `usage-policy` |
+| **[IETF AIPREF](https://datatracker.ietf.org/wg/aipref/documents/)** | The standards-track answer to the same question: a `Content-Usage` directive or header over `train-ai` and `search`. Still pre-last-call. Note the token inversion against Content Signals — `train-ai` here, `ai-train` there. | `robots-txt`, `usage-policy` |
+| **[RSL — Really Simple Licensing](https://rslstandard.org)** | A full machine-readable licensing layer: permits/prohibits vocabularies, payment models, and an Open Licensing Protocol for negotiating access. | `rsl`, `usage-policy` |
+| **[TDMRep](https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/)** | A text-and-data-mining reservation, named in the EU GPAI Code of Practice. Its weight is legal rather than technical. | `usage-policy` |
+| **Page-level AI directives** | `nosnippet` and `max-snippet` for Google AI Overviews, `noarchive` and `nocache` for Copilot grounding. Unlike everything above, the vendors document that they honor these. | `ai-directives` |
+| **Cloaking and blocking integrity** | Not a standard but a failure mode: robots.txt allows GPTBot and the firewall returns 403. | `agent-access` |
+| **Signal consistency** | Also not a standard: whether your Content Signals, AIPREF, RSL and TDMRep declarations agree with each other. Five documents maintained by hand do not stay in sync, and the terms that apply then depend on which file a crawler read. | `usage-policy` |
 
 These answer: *have you expressed your access and usage policy in a form agents can honor — and does your infrastructure actually match it?*
 
-The progression is one of increasing expressiveness: robots.txt says **who/where**, Content Signals adds **how it may be used**, RSL adds **under what license and price**.
+The progression is one of increasing expressiveness: robots.txt says **who and where**, Content Signals and AIPREF add **how it may be used**, RSL adds **under what licence and price**.
+
+One caveat worth stating plainly: only robots.txt access rules are documented as honored by Google, OpenAI, Anthropic and Microsoft. Content Signals, AIPREF, RSL and TDMRep are declarations. Their weight is legal rather than technical, and ax-audit says so on every run rather than implying a crawler will obey.
 
 ### 4. Transport, efficiency & hygiene
 
@@ -78,10 +88,14 @@ A fully AX-ready site tells a coherent story across all four:
 
 > "Here's my content in a form you can read **(family 1)**, here's the interface to interact with me **(family 2)**, here's exactly who may use it and how, for what license **(family 3)**, over a fast and trustworthy connection **(family 4)**."
 
-ax-audit's weighting reflects today's leverage: discovery and readability (`llms-txt`, `robots-txt`, `html-rendering`, `structured-data`, `http-headers`) carry the most weight because they're the highest-impact, most-adopted signals. The governance and efficiency standards are informational in 3.x — real and worth adopting, but still stabilizing — and gain weight in v4.0.
+ax-audit's weighting follows what actually stops an agent. **Content** carries the most weight, led by `html-rendering`, because a page whose content only exists after hydration is invisible to the crawlers that do not run JavaScript, no matter how many discovery files it publishes. **Access** is next, because a firewall rule or a `nosnippet` directive silently undoes everything else.
+
+`llms-txt` sits at 5 rather than the 11 it carried in 3.x. Adoption studies find most published files are never fetched by an AI search crawler, and Google has stated that Search ignores them; the vendors that do read them are coding agents like Claude Code and Cursor. It is worth publishing, as developer tooling, and it is not worth twice what having content is worth.
+
+Checks resting on draft specifications — `ai-catalog`, `webmcp`, `commerce-discovery` — are reported but never scored. A specification that may be renamed next quarter should not move your number.
 
 ## See also
 
 - [getting-started.md](./getting-started.md) — run your first audit
 - [checks.md](./checks.md) — exact scoring per standard
-- The [remediation guides](https://lucioduran.com/projects/ax-audit/guides) — how to implement each one
+- The [remediation guides](https://axrush.com/guides) — how to implement each one