ax-audit 3.0.0 → 3.6.0

package/docs/architecture.md

@@ -0,0 +1,88 @@
+# Architecture
+
+ax-audit is a dependency-light TypeScript codebase: two runtime dependencies (`chalk`, `commander`), Node 18+ built-in `fetch`, no HTTP libraries, no XML/HTML parser dependencies (regex-based primitives), and the built-in `node:test` runner.
+
+## Pipeline
+
+```
+cli.ts ──► orchestrator.ts ──► checks/* (Promise.allSettled, parallel)
+              │                     │
+              ▼                     ▼
+          fetcher.ts            scorer.ts ──► reporter/{terminal,json,html,markdown}
+        (cache + retries)                          │
+              ▲                                baseline.ts (save / load / diff)
+              └── shared by every check via CheckContext.fetch
+```
+
+1. **cli.ts** parses and validates flags, loads the baseline if requested, and dispatches to single or batch mode.
+2. **orchestrator.ts** (`audit`) creates one fetcher per run, fetches the homepage once, builds the `CheckContext` (`url`, `html`, `headers`, `fetch`), and runs all selected checks in parallel. A check that throws is converted into a score-0 result with the error as a finding — one bad check never kills the audit. `batchAudit` runs `audit` per URL through an order-preserving work queue with configurable `concurrency`.
+3. **fetcher.ts** wraps `fetch` with: per-run in-memory caching keyed on URL + normalized (lowercased, sorted) custom headers — mirroring HTTP `Vary` semantics so a `text/markdown` probe never collides with the HTML fetch; case-insensitive header merging over defaults; timeouts via `AbortController`; and retries with exponential backoff for transient failures (status 0, 408, 425, 429, 5xx). Errors never throw — they become `{ status: 0, ok: false, error }` results, also cached.
+4. **checks/** — one module per check (18). Each exports `default` (async check function) and `meta` (`{ id, name, description, weight }`).
+5. **scorer.ts** computes the weighted average; when all selected checks have weight 0 it falls back to a plain average.
+6. **reporter/** renders to terminal (chalk), JSON, self-contained HTML, or Markdown.
+7. **baseline.ts** persists minimal score snapshots and computes per-check diffs for regression gating.
+
+## Anatomy of a check
+
+```typescript
+import { guideUrl } from '../guide-urls.js';
+import type { CheckContext, CheckResult, CheckMeta, Finding } from '../types.js';
+import { buildResult } from './utils.js';
+
+export const meta: CheckMeta = {
+  id: 'my-check',
+  name: 'My Check',
+  description: 'One-line description shown in reports',
+  weight: 0, // new checks start informational in 3.x
+};
+
+export default async function check(ctx: CheckContext): Promise<CheckResult> {
+  const start = performance.now();
+  const findings: Finding[] = [];
+  let score = 100;
+
+  const res = await ctx.fetch(`${ctx.url}/something`, { headers: { Accept: 'application/json' } });
+  if (!res.ok) {
+    findings.push({
+      status: 'fail',
+      message: '/something not found',
+      hint: 'Actionable, copy-pasteable advice.',
+      learnMoreUrl: guideUrl(meta.id, 'not-found'),
+    });
+    return buildResult(meta, 0, findings, start);
+  }
+
+  // ... validations, each pushing a pass/warn/fail Finding and adjusting score
+
+  return buildResult(meta, score, findings, start);
+}
+```
+
+Conventions:
+
+- **Findings are actionable.** Every `warn`/`fail` carries a `hint` with concrete remediation and a `learnMoreUrl` pointing to `lucioduran.com/projects/ax-audit/guides/<check-id>#<anchor>`. Every anchor must have a section in that guide.
+- **Scores are clamped** to [0, 100] by `buildResult`.
+- **Shared HTML primitives** live in `checks/html-utils.ts` (`getMetaContent`, `findLinkTags`, `getAttribute`, `extractVisibleText`, …) — no per-check regex duplication.
+- **Content-Type validation** uses `checkContentType` from `checks/utils.ts` (−5 convention for mismatches).
+- **Network goes through `ctx.fetch`** — never raw `fetch` — so caching, retries, timeouts, and `--verbose` logging apply uniformly.
+
+## Adding a new check
+
+1. Create `src/checks/your-check.ts` exporting `default` + `meta` (weight 0 — see scoring policy below).
+2. Register it in `src/checks/index.ts`.
+3. Add its weight to `CHECK_WEIGHTS` in `src/constants.ts`.
+4. Add a test suite in `test/checks/your-check.test.js` using `mockContext` / `mockResponse` from `test/helpers.js`. Route values can be functions `(url, fetchOptions) => response` when the response must vary by request headers.
+5. Document it in `docs/checks.md` and the README table.
+6. Write the remediation guide covering every `learnMoreUrl` anchor you emit.
+
+## Scoring policy (3.x)
+
+Score deltas on the same site are treated as **breaking** (see CHANGELOG 3.0.0). Therefore in 3.x:
+
+- New checks ship with **weight 0** (informational): full findings, no effect on the overall score or baselines.
+- New findings inside weighted checks must be informational (no score deduction) — see the Content Signals findings in `robots-txt`.
+- Weight redistribution happens in major versions (v4.0).
+
+## Testing
+
+`npm test` builds (`tsc`) and runs `node --test`. The suite (301 tests) covers every check, the scorer, baseline logic, the Markdown reporter, plus integration tests that spin up real local HTTP servers for the fetcher (per-header caching, retries) and the batch orchestrator (ordering, concurrency caps). No test dependencies beyond Node.

package/docs/checks.md

@@ -0,0 +1,322 @@
+# Checks Reference
+
+ax-audit runs 18 checks. Fourteen are **weighted** (summing to 100% of the overall score); four are **informational** in 3.x — they run and report findings but carry weight 0 until v4.0, because score-affecting changes are treated as breaking (see [CHANGELOG 3.0.0](../CHANGELOG.md)).
+
+This page documents the **exact scoring** of every check: each deduction, bonus, and formula, extracted from the source. Every finding links to a step-by-step remediation guide at `lucioduran.com/projects/ax-audit/guides/<check-id>`.
+
+**Reading the tables:** each check starts at 100 unless noted. Deductions stack additively; `buildResult` clamps the final score to [0, 100]. "Hard fail" rows short-circuit the check.
+
+---
+
+## Weighted checks
+
+### `llms-txt` — 11%
+
+`/llms.txt` presence and [llmstxt.org](https://llmstxt.org) spec compliance.
+
+| Condition | Points |
+| --- | --- |
+| `/llms.txt` not found | **hard fail → 0** |
+| Wrong Content-Type (expected `text/plain` or `text/markdown`) | −5 |
+| First line is not an H1 (`# `) | −15 |
+| No blockquote description (`> `) | −10 |
+| No `##` section headings | −10 |
+| No Markdown links | −10 |
+| Content under 100 characters | −10 |
+| `/llms-full.txt` also available | **+10** (capped at 100) |
+
+### `robots-txt` — 11%
+
+AI-crawler configuration. Core crawlers: GPTBot, ClaudeBot, ChatGPT-User, Claude-SearchBot, Google-Extended, PerplexityBot, OAI-SearchBot, CCBot.
+
+| Condition | Points |
+| --- | --- |
+| `/robots.txt` not found | **hard fail → 0** |
+| No core AI crawler explicitly configured | −40 |
+| Some core crawlers missing | −`round(missing/8 × 30)` |
+| Core crawler(s) blocked only via `User-agent: *` + `Disallow: /` | −5 per crawler |
+| Known AI crawler(s) explicitly blocked (`Disallow: /`) | −3 per crawler |
+| No `Sitemap:` directive | −5 |
+| Partial path restrictions on AI crawlers | warn only, 0 |
+| [Content Signals](https://contentsignals.org) findings (declared / malformed / unknown / missing) | informational, 0 in 3.x |
+
+### `html-rendering` — 9%
+
+Whether the static HTML contains content — most AI crawlers do not execute JavaScript. Thresholds: 500 chars / 80 words of visible text, 5% text-to-markup ratio.
+
+| Condition | Points |
+| --- | --- |
+| No HTML body returned | **hard fail → 0** |
+| Zero visible text in static HTML | −50 |
+| Sparse content (< 500 chars or < 80 words) | −25 |
+| Text-to-markup ratio < 5% | −10 |
+| Empty SPA mount point (`#root`, `#__next`, `#__nuxt`, `#app`, `#svelte`, `#gatsby`) | −20 |
+| 0 semantic landmarks (`<main>`, `<article>`, `<header>`, `<footer>`, `<nav>`) | −15 |
+| 1–2 semantic landmarks | −10 |
+| No `<h1>` | −10 |
+| Multiple or empty `<h1>` | −5 |
+| > 15 executable scripts without `<noscript>` fallback | −5 |
+| `<img alt>` coverage < 90% | −5 |
+
+### `structured-data` — 9%
+
+JSON-LD on the homepage. Key entity types: Person, Organization, WebSite, WebPage, ProfilePage.
+
+| Condition | Points |
+| --- | --- |
+| No JSON-LD blocks | **hard fail → 0** |
+| Every JSON-LD block has invalid JSON | **→ 10** |
+| Invalid JSON in a block | −10 per block |
+| No schema.org `@context` | −15 |
+| No key entity types found | −15 |
+| Only one key entity type | −10 |
+| No `@graph` array | −5 |
+| No `BreadcrumbList` | −5 |
+
+### `http-headers` — 9%
+
+Security headers, AI discovery `Link` headers (RFC 5988-parsed), CORS on `.well-known`.
+
+| Condition | Points |
+| --- | --- |
+| No headers retrievable | **hard fail → 0** |
+| Missing critical security header (HSTS, X-Content-Type-Options) | −10 each |
+| Only 1–3 of the 7 tracked security headers present | −5 |
+| `Link` header missing both llms.txt and agent.json references | −15 |
+| `Link` header missing one of the two | −5 |
+| No CORS on `/.well-known/agent.json` | −10 |
+
+### `agent-json` — 7%
+
+`/.well-known/agent.json` [A2A Agent Card](https://a2a-protocol.org). Required fields: `name`, `description`, `url`, `skills`.
+
+| Condition | Points |
+| --- | --- |
+| Not found | **hard fail → 0** |
+| Invalid JSON | **→ 10** |
+| Wrong Content-Type (expected `application/json`) | −5 |
+| Missing required field | −15 per field |
+| `url` on a different origin | −5 |
+| `url` not an absolute URL | −5 |
+| `skills` empty | −10 |
+| `skills` entries missing `id` or `description` | −5 |
+| No `protocolVersion` | −5 |
+| No optional fields (`capabilities`, `authentication`, `documentationUrl`) | −5 |
+
+### `mcp` — 7%
+
+`/.well-known/mcp.json` [Model Context Protocol](https://modelcontextprotocol.io) server configuration.
+
+| Condition | Points |
+| --- | --- |
+| Not found | **hard fail → 0** |
+| Invalid JSON | **→ 10** |
+| Wrong Content-Type | −5 |
+| Missing `name` | −10 |
+| Missing `description` | −5 |
+| No `tools` array, or empty | −15 |
+| No tool has a description | −10 |
+| Some tools missing descriptions | −5 |
+| No `resources` | −5 |
+| No protocol version | −5 |
+| No CORS headers | −10 |
+
+### `seo-basics` — 7%
+
+Head-tag fundamentals. Bounds: title 20–70 chars, description 70–160.
+
+| Condition | Points |
+| --- | --- |
+| Homepage HTML unavailable | **hard fail → 0** |
+| `<title>` missing or empty | −25 |
+| Title too short / too long | −10 / −5 |
+| Meta description missing | −20 |
+| Description too short / too long | −8 / −5 |
+| Description duplicates the title | −5 |
+| No canonical link | −10 |
+| Multiple canonicals / missing href / relative href | −5 each |
+| `<html lang>` missing / invalid BCP 47 | −10 / −5 |
+| No UTF-8 charset | −5 |
+| Missing viewport | −5 |
+| hreflang present without `x-default` | −3 |
+
+### `security-txt` — 6%
+
+`/.well-known/security.txt` per [RFC 9116](https://www.rfc-editor.org/rfc/rfc9116).
+
+| Condition | Points |
+| --- | --- |
+| Not found | **hard fail → 0** |
+| Missing `Contact` or `Expires` | −25 per field |
+| `Expires` in the past | −20 |
+| No optional fields (Canonical, Preferred-Languages, Policy, Encryption, Hiring) | −5 |
+
+### `meta-tags` — 6%
+
+AI meta tags (`ai:summary`, `ai:content_type`, `ai:author`, `ai:api`, `ai:agent_card`), discovery links, Open Graph, Twitter Card.
+
+| Condition | Points |
+| --- | --- |
+| Homepage HTML unavailable | **hard fail → 0** |
+| 0 AI meta tags | −18 |
+| Only 1–2 AI meta tags | −12 |
+| No `rel="alternate"` → llms.txt | −12 |
+| No `rel="alternate"` → agent.json | −8 |
+| No `rel="me"` identity links | −8 |
+| No Open Graph tags at all | −12 |
+| OG required incomplete (`og:title`, `og:description`, `og:url`, `og:type`) | −8 |
+| OG recommended incomplete (`og:image`, `og:site_name`) | −3 |
+| No Twitter Card tags at all | −6 |
+| Twitter required incomplete (`twitter:card`, `twitter:title`, `twitter:description`) | −5 |
+| Twitter recommended incomplete (`twitter:image`) | −2 |
+
+### `openapi` — 6%
+
+`/.well-known/openapi.json`.
+
+| Condition | Points |
+| --- | --- |
+| Not found | **hard fail → 0** |
+| Invalid JSON | **→ 10** |
+| Wrong Content-Type | −5 |
+| No `openapi`/`swagger` version field | −20 |
+| Swagger 2.x instead of OpenAPI 3.x | −10 |
+| Missing `info.title` | −10 |
+| Missing `info.description` | −5 |
+| No `paths` documented | −15 |
+| No `servers` | −5 |
+
+### `tls-https` — 5%
+
+HTTPS, redirect, HSTS. Thresholds: max-age ≥ 15,768,000s (~6 months), preload ≥ 31,536,000s (1 year).
+
+| Condition | Points |
+| --- | --- |
+| Invalid URL | **hard fail → 0** |
+| Served over plain HTTP | −50 |
+| HTTP does not redirect to HTTPS | −15 |
+| Redirect unverifiable | −5 |
+| No HSTS header | −15 |
+| HSTS without `max-age` | −10 |
+| `max-age` < 6 months | −5 |
+| No `includeSubDomains` | −5 |
+| `preload` present but ineligible | −5 |
+| No `preload` directive | −3 |
+
+### `sitemap` — 4%
+
+Located via robots.txt `Sitemap:` or `/sitemap.xml`. Limits: 50,000 URLs / 50 MB / 365-day freshness.
+
+| Condition | Points |
+| --- | --- |
+| No sitemap found | **hard fail → 0** |
+| Response is not XML | **→ 20** |
+| Over 50 MB | −10 |
+| Unexpected Content-Type | −5 |
+| Sitemap index with no `<sitemap>` entries | −20, stop |
+| Some sampled child sitemaps unreachable | −10 |
+| `<urlset>` with no `<url>` entries | −30 |
+| Over 50,000 URLs declared | −10 |
+| `<lastmod>` coverage < 50% | −5 |
+| Newest `<lastmod>` older than 365 days | −5 |
+
+### `well-known-ai` — 3%
+
+Emerging AI discovery files. **Purely proportional** — no deductions:
+
+```
+score = round(present / 5 × 100)
+```
+
+over `/.well-known/ai.txt` (Spawning), `/.well-known/genai.txt`, `/ai-plugin.json`, `/agents.json`, `/.well-known/nlweb.json`. Files with invalid content produce warnings without counting as present.
+
+---
+
+## Informational checks (weight 0 in 3.x)
+
+These run on every audit and report full findings, but do not affect the overall score or baselines. They gain weight in v4.0.
+
+### `content-negotiation` — Markdown for Agents
+
+Probes the homepage with `Accept: text/markdown` — the pattern served by Cloudflare and Vercel and requested by Claude Code, Cursor, and OpenCode (~80% token reduction vs HTML).
+
+| Condition | Points |
+| --- | --- |
+| Probe request fails (network) | **hard fail → 0** |
+| No Markdown served, no fallback | **→ 0** |
+| No Markdown served, but `<link rel="alternate" type="text/markdown">` present | **→ 40** |
+| Markdown served (correct Content-Type, 2xx) | base 100 |
+| Body is empty | −30 |
+| Body is a relabeled HTML document | −25 |
+| `Vary` does not include `Accept` | −15 |
+| Markdown not smaller than HTML | warn only, 0 |
+
+### `rsl` — Really Simple Licensing
+
+[RSL 1.0](https://rslstandard.org/rsl) discovery (robots.txt `License:`, `Link: rel="license"` header, `<link rel="license" type="application/rsl+xml">`) and document validation. Plain CC-style license links without the RSL media type are ignored.
+
+| Condition | Points |
+| --- | --- |
+| No discovery mechanism found | **hard fail → 0** |
+| License document unreachable | **→ 25** (cap) |
+| Root `<rsl>` element missing | −40, stop |
+| No `<content>` elements | −20, stop |
+| Wrong or missing `https://rslstandard.org/rsl` namespace | −15 |
+| `<license>` elements missing | −15 |
+| robots.txt `License:` not an absolute URI | −10 |
+| `<content>` missing required `url` attribute | −10 |
+| Wrong Content-Type (expected `application/rsl+xml`) | −5 |
+| `permits`/`prohibits` with invalid `type` | −5 |
+| Tokens outside the RSL 1.0 vocabulary (incl. pre-1.0 draft tokens) | −5 |
+| Invalid `payment` type | −5 |
+
+### `agent-access` — Cloaking detection
+
+Probes the homepage with realistic UAs for the 8 core AI crawlers and compares status + visible text against the default-UA baseline. **Credit-ratio formula:**
+
+```
+score = round(credit / 8 × 100)
+```
+
+| Outcome per crawler | Credit |
+| --- | --- |
+| Equivalent response | 1 |
+| Blocked, consistent with robots.txt `Disallow` (explicit or wildcard) | 1 |
+| 200 but < 50% of baseline visible text (baseline ≥ 200 chars) | 0.5 |
+| Blocked while robots.txt allows (or doesn't restrict) it | 0 |
+| Baseline request itself fails | **hard fail → 0** |
+
+Caveat: WAFs using Web Bot Auth / IP verification may pass the real crawler while rejecting this unverified probe — confirm against WAF logs before changing rules.
+
+### `crawl-efficiency`
+
+| Condition | Points |
+| --- | --- |
+| Homepage request fails | **hard fail → 0** |
+| Uncompressed response | −30 |
+| gzip/deflate/zstd instead of Brotli | pass with suggestion, 0 |
+| No `ETag` / `Last-Modified` validator | −30 |
+| Validator present but conditional request not answered with `304` | −15 |
+| Page > 2 MB decompressed | −10 |
+| Page > 500 KB decompressed | −5 |
+
+---
+
+## Overall scoring model
+
+Each check returns 0–100. The overall score is the weighted average across the checks that ran:
+
+```
+overall = round( Σ (score_i / 100 × weight_i) / Σ weight_i × 100 )
+```
+
+When every selected check has weight 0 (e.g. `--checks rsl`), the overall falls back to a plain average of check scores.
+
+| Grade | Score | Exit code |
+| --- | --- | --- |
+| Excellent | 90–100 | 0 |
+| Good | 70–89 | 0 |
+| Fair | 50–69 | 1 |
+| Poor | 0–49 | 1 |
+
+Weights live in `src/constants.ts` (`CHECK_WEIGHTS`); a check's own `meta.weight` takes precedence. The scoring policy for 3.x — why new checks ship at weight 0 — is documented in [architecture.md](./architecture.md).

package/docs/ci.md

@@ -0,0 +1,89 @@
+# CI Integration
+
+ax-audit's exit codes (see [cli.md](./cli.md)) make it a drop-in quality gate: `0` for Good/Excellent, `1` for Fair/Poor or regressions.
+
+## GitHub Actions
+
+### Basic gate
+
+```yaml
+- name: AX Audit
+  run: npx ax-audit https://your-site.com
+  # Fails the step if the score < 70
+```
+
+### Regression gate with a committed baseline
+
+Commit `.ax-baseline.json` to the repo and fail the build only when a check drops:
+
+```yaml
+- name: AX Audit (regression gate)
+  run: npx ax-audit https://your-site.com --baseline .ax-baseline.json --fail-on-regression 5
+```
+
+Refresh the baseline deliberately (e.g., after intentional changes):
+
+```bash
+npx ax-audit https://your-site.com --save-baseline .ax-baseline.json
+git add .ax-baseline.json && git commit -m "chore: refresh AX baseline"
+```
+
+### Markdown report as a PR comment
+
+```yaml
+- name: AX Audit (markdown)
+  run: npx ax-audit ${{ env.PREVIEW_URL }} --output markdown > ax-report.md
+  continue-on-error: true
+
+- name: Comment PR
+  uses: marocchino/sticky-pull-request-comment@v2
+  with:
+    path: ax-report.md
+```
+
+This pairs naturally with Vercel/Netlify preview deployments: audit the preview URL on every PR and the reviewer sees the AX impact inline.
+
+### Artifacts
+
+```yaml
+- name: AX Audit (JSON)
+  run: npx ax-audit https://your-site.com --json > ax-report.json
+
+- uses: actions/upload-artifact@v4
+  with:
+    name: ax-audit-report
+    path: ax-report.json
+```
+
+## Auditing multiple environments
+
+```yaml
+- name: AX Audit (all properties)
+  run: npx ax-audit https://www.your-site.com https://docs.your-site.com https://api.your-site.com --concurrency 3
+  # Exit 1 if any property scores < 70
+```
+
+## Tuning for CI stability
+
+- `--retries 3` absorbs transient 5xx/timeouts from cold preview deployments (default is 2).
+- `--timeout 15000` for slow staging environments.
+- `--checks ...` to gate only on the surface you are iterating on — but remember the overall score then averages only the selected checks.
+
+## Scheduled audits
+
+A weekly audit catches drift from infrastructure changes (CDN settings, WAF rules, header changes deployed by other teams):
+
+```yaml
+on:
+  schedule:
+    - cron: '0 6 * * 1'
+
+jobs:
+  ax-audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: npx ax-audit https://your-site.com --baseline .ax-baseline.json --fail-on-regression 0
+```
+
+`--fail-on-regression 0` makes any per-check drop fail the workflow — appropriate for scheduled runs where every change is unexpected.

package/docs/cli.md

@@ -0,0 +1,67 @@
+# CLI Reference
+
+```bash
+ax-audit <urls...> [options]
+```
+
+One or more fully qualified URLs (scheme required). A single URL produces a full report; multiple URLs run in batch mode with a summary table.
+
+## Options
+
+| Flag | Default | Description |
+| --- | --- | --- |
+| `--output <format>` | `terminal` | Output format: `terminal`, `json`, `html`, `markdown`. Invalid values error out. |
+| `--json` | — | Shorthand for `--output json`. |
+| `--checks <list>` | all | Comma-separated check IDs to run (see [checks.md](./checks.md)). Unknown IDs error with the list of valid ones. |
+| `--timeout <ms>` | `10000` | Per-request timeout in milliseconds. |
+| `--retries <n>` | `2` | Retry attempts for transient fetch failures (network errors, timeouts, 408/425/429/5xx) with exponential backoff from 250ms. `0` disables retries. |
+| `--concurrency <n>` | `1` | Batch mode only: maximum URLs audited in parallel. Output order always matches input order. |
+| `--verbose` | — | Log every HTTP request, cache hit, retry, and per-check score to stderr. |
+| `--only-failures` | — | Hide passing findings; checks with only passes are omitted entirely. |
+| `--save-baseline <path>` | — | Save this audit as a baseline JSON file. |
+| `--baseline <path>` | — | Compare against a saved baseline; shows per-check deltas (▲/▼). Single-URL mode only. |
+| `--fail-on-regression <points>` | — | Exit 1 if any check regresses more than N points vs the baseline. Requires `--baseline`. |
+| `-v, --version` | — | Print version. |
+
+## Output formats
+
+- **terminal** — colored report with score bar, per-check sections, and PASS/WARN/FAIL findings.
+- **json** — the full `AuditReport` (plus `baselineDiff` when `--baseline` is used). Stable shape for CI pipelines.
+- **html** — self-contained page (score gauge, dark/light mode, collapsible sections). Pipe to a file: `ax-audit <url> --output html > report.html`.
+- **markdown** — summary table + per-check findings with status emoji. Built for CI logs and PR comments: `ax-audit <url> --output markdown > report.md`.
+
+## Exit codes
+
+| Code | Meaning |
+| --- | --- |
+| `0` | Score ≥ 70 (single), or all URLs ≥ 70 (batch), and no regression beyond the `--fail-on-regression` threshold. |
+| `1` | Score < 70, any batch URL < 70, invalid arguments, or regression beyond threshold. |
+| `2` | Fatal error (network failure on the audit itself, unreadable baseline file). |
+
+## Baseline workflow
+
+```bash
+# First run — record the baseline
+ax-audit https://your-site.com --save-baseline .ax-baseline.json
+
+# Subsequent runs — compare and gate
+ax-audit https://your-site.com --baseline .ax-baseline.json --fail-on-regression 5
+```
+
+The baseline stores the overall score and per-check scores. Checks added after the baseline was saved appear as new (no delta); removed checks are ignored.
+
+## Examples
+
+```bash
+# Quick audit
+npx ax-audit https://your-site.com
+
+# Only the AI-licensing surface
+npx ax-audit https://your-site.com --checks robots-txt,rsl,content-negotiation
+
+# Batch, 4 at a time, machine-readable
+npx ax-audit $(cat urls.txt) --concurrency 4 --json > batch.json
+
+# Show me only what is broken
+npx ax-audit https://your-site.com --only-failures
+```

package/docs/concepts.md

@@ -0,0 +1,87 @@
+# Concepts: the AX standards landscape
+
+"AI Agent Experience" (AX) is the sum of the conventions a site uses to be discovered, read, governed, and transacted with by autonomous AI agents and crawlers — the way "web accessibility" is the sum of conventions for assistive technology. This page maps the standards ax-audit checks against, why each exists, and how they relate. It's the conceptual companion to the mechanical detail in [checks.md](./checks.md).
+
+## Why AX is its own discipline
+
+Agents are not browsers. Three differences drive every check:
+
+1. **They mostly don't run JavaScript.** GPTBot, ClaudeBot, CCBot and most crawlers fetch raw HTML. A client-rendered SPA that returns an empty `<div id="root">` is, to them, a blank page. (`html-rendering`, `content-negotiation`)
+2. **They look for declared structure, not visual layout.** An agent would rather read a `/llms.txt` summary or a JSON-LD graph than infer meaning from your CSS grid. (`llms-txt`, `structured-data`, `meta-tags`, `agent-json`, `mcp`, `openapi`)
+3. **Their access is a policy and economic question, not just a technical one.** Who may crawl, for what use, at what price, under what license — these now have machine-readable answers. (`robots-txt`, Content Signals, `rsl`, `agent-access`)
+
+Bot traffic is projected to exceed human traffic by 2029. AX is the interface layer for that shift.
+
+## The four families of standards
+
+### 1. Content discovery & readability
+
+| Standard | What it is | Check |
+| --- | --- | --- |
+| **[llms.txt](https://llmstxt.org)** | A Markdown file at your root summarizing your site for LLMs, with curated links. The "sitemap for AI." | `llms-txt` |
+| **Server-side rendering** | Delivering real content in the HTML response, not assembling it client-side. | `html-rendering` |
+| **[Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)** | Content negotiation: serve clean Markdown when a client sends `Accept: text/markdown`. ~80% fewer tokens than HTML. | `content-negotiation` |
+| **schema.org / JSON-LD** | Structured data describing entities (Person, Organization, Product) in a graph agents can parse. | `structured-data` |
+| **Sitemaps** | The classic XML index, still how crawlers enumerate your URLs. | `sitemap` |
+
+These answer: *can an agent find your content and actually read it?*
+
+### 2. Agent interaction surface
+
+| Standard | What it is | Check |
+| --- | --- | --- |
+| **[A2A — Agent2Agent](https://a2a-protocol.org)** | An "Agent Card" at `/.well-known/agent.json` advertising your agent's identity and skills, so other agents can interoperate. | `agent-json` |
+| **[MCP — Model Context Protocol](https://modelcontextprotocol.io)** | A manifest at `/.well-known/mcp.json` describing tools and resources an agent can call. The emerging standard for exposing capabilities to LLMs. | `mcp` |
+| **[OpenAPI](https://www.openapis.org)** | The long-standing machine-readable API description; agents use it to call your endpoints. | `openapi` |
+| **Emerging discovery files** | `ai.txt`, `genai.txt`, `ai-plugin.json`, `agents.json`, `nlweb.json` — competing/early conventions, scored as coverage bonus. | `well-known-ai` |
+| **AI meta tags & discovery links** | `ai:*` meta tags and `rel="alternate"` links pointing agents to your llms.txt / agent.json. | `meta-tags` |
+
+These answer: *once an agent arrives, can it understand what you offer and act on it?*
+
+### 3. Access governance & licensing
+
+This is the newest and fastest-moving family — the response to "AI scraped my content and now competes with me."
+
+| Standard | What it is | Check |
+| --- | --- | --- |
+| **[Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)** | The original robots.txt — *who* may crawl *what*. ax-audit grades coverage of 48 known AI crawlers. | `robots-txt` |
+| **[Content Signals](https://contentsignals.org)** | A robots.txt extension (Cloudflare, CC0) declaring *how* content may be used after access: `search`, `ai-input`, `ai-train`. Served by default on 3.8M+ Cloudflare domains. | `robots-txt` (findings) |
+| **[RSL — Really Simple Licensing](https://rslstandard.org)** | A full machine-readable licensing layer (license.xml): permits/prohibits vocabularies, payment models (free, attribution, pay-per-crawl, pay-per-inference). Endorsed by 1,500+ publishers. | `rsl` |
+| **Cloaking integrity** | Not a standard but a failure mode: your stated policy (robots.txt allows GPTBot) contradicting enforcement (WAF returns 403). | `agent-access` |
+
+These answer: *have you expressed your access and usage policy in a form agents can honor — and does your infrastructure actually match it?*
+
+The progression is one of increasing expressiveness: robots.txt says **who/where**, Content Signals adds **how it may be used**, RSL adds **under what license and price**.
+
+### 4. Transport, efficiency & hygiene
+
+| Standard | What it is | Check |
+| --- | --- | --- |
+| **TLS / HSTS** | HTTPS everywhere; many agents refuse plaintext origins. | `tls-https` |
+| **HTTP security & discovery headers** | Security headers plus `Link` headers advertising your AI files. | `http-headers` |
+| **Compression & conditional GET** | Brotli/gzip and `ETag`/`304` — crawl cost matters when bots dominate traffic. | `crawl-efficiency` |
+| **[RFC 9116 security.txt](https://www.rfc-editor.org/rfc/rfc9116)** | A machine-readable security contact. | `security-txt` |
+| **SEO basics** | Title, description, canonical, lang, hreflang — agents use the same head-tag fundamentals search engines do. | `seo-basics` |
+
+These answer: *is the connection trustworthy, cheap, and well-formed?*
+
+## On the horizon (not yet scored)
+
+Two standards are maturing and worth watching:
+
+- **[Web Bot Auth](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/)** — cryptographic crawler verification via HTTP Message Signatures (RFC 9421). Bots sign requests with a key published at `/.well-known/http-message-signatures-directory`; sites verify identity instead of guessing from user-agent strings. Already implemented by Cloudflare and Google (`agent.bot.goog`). It directly affects the `agent-access` check: a WAF using Web Bot Auth may pass a real, signed crawler while rejecting ax-audit's unsigned probe — which is why that check's findings carry an explicit verified-bots caveat.
+- **Pay-per-crawl / HTTP 402** — Cloudflare and the RSL payment vocabulary point toward metered, paid agent access. RSL already encodes the terms; enforcement protocols (Open License Protocol, x402) are emerging.
+
+## How the families compose
+
+A fully AX-ready site tells a coherent story across all four:
+
+> "Here's my content in a form you can read **(family 1)**, here's the interface to interact with me **(family 2)**, here's exactly who may use it and how, for what license **(family 3)**, over a fast and trustworthy connection **(family 4)**."
+
+ax-audit's weighting reflects today's leverage: discovery and readability (`llms-txt`, `robots-txt`, `html-rendering`, `structured-data`, `http-headers`) carry the most weight because they're the highest-impact, most-adopted signals. The governance and efficiency standards are informational in 3.x — real and worth adopting, but still stabilizing — and gain weight in v4.0.
+
+## See also
+
+- [getting-started.md](./getting-started.md) — run your first audit
+- [checks.md](./checks.md) — exact scoring per standard
+- The [remediation guides](https://lucioduran.com/projects/ax-audit/guides) — how to implement each one