@shoalrun/cli 0.0.0-stage → 0.1.0-alpha.12

package/agent-docs/publish/content-negotiation.sections/validators-and-manifest-links.md

@@ -0,0 +1,49 @@
+---
+id: guide.content-negotiation
+title: Content negotiation
+summary: Understand content negotiation, why it helps agents, and what a hosting
+  provider needs to do.
+type: reference
+area: discovery
+subarea: discovery-agents
+canonical: /publish/content-negotiation
+---
+
+# Content negotiation
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Reference](https://www.shoalrun.com/.well-known/shoalrun/navigation/reference-tab-50f26fe22670.txt) (ID: reference-tab). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Reference](https://www.shoalrun.com/cli.md) > Output and presentation > Content negotiation
+
+**Updated: 2026-10-26.**
+
+Parent page: [Full Markdown page](/publish/content-negotiation.md)
+
+**Page reading path** — these links apply to the full page.
+
+**Next steps**
+
+- [Publish on Vercel](https://www.shoalrun.com/publish/vercel.md) — Generate a Vercel deployment with negotiation already configured.
+
+- [Add Shoalrun to another build or host](https://www.shoalrun.com/publish/host-adapters.md) — Supply the same HTTP rules on another host.
+
+Shoalrun builds two versions of every published page from one authored source. The HTML version is for people using a browser. The generated Markdown version contains shared material in a form agents can read directly. Authors can also add or suppress [format-specific content](https://www.shoalrun.com/author/components/only.md) in either version. The versions share a core but need not contain identical text.
+
+**Content negotiation** lets a visitor ask for a format at the page's usual URL. Some agents send the request header `Accept: text/markdown`. A host that supports negotiation responds with Shoalrun's Markdown version. A browser asking for HTML gets the HTML version. The URL stays the same:
+
+| Request to `/publish/output/`       | Response                                     |
+| ----------------------------------- | -------------------------------------------- |
+| Browser sends `Accept: text/html`   | HTML page, `Content-Type: text/html`         |
+| Agent sends `Accept: text/markdown` | Markdown page, `Content-Type: text/markdown` |
+
+`Accept` says what the visitor can use. `Content-Type` says what the host actually returned. The `.md` URL, such as `/publish/output.md`, also works directly.
+
+## Validators and manifest links
+
+A Markdown response includes its canonical HTML link and a `rel="describedby"` link to its current manifest record. The latter uses a JSON Pointer fragment, such as `/documents.json#/documents/3`. Fetch the manifest and select that array entry to read the page's ID, publication variant, and digests. The index can move between builds; use the response's current link and verify the Markdown digest.
+
+Generated hosting metadata specifies a strong `ETag` equal to the quoted SHA-256 Markdown digest and `Cache-Control: public, max-age=0, must-revalidate`. The Node server handles matching `If-None-Match` requests with `304` and no body, including Markdown negotiated at an HTML URL. Negotiated responses retain `Vary: Accept`. Vercel receives the generated headers but can replace the ETag with its own validator, including a weak validator for compressed responses. Send the ETag returned by the host in `If-None-Match`; treat its value as opaque. To verify downloaded Markdown bytes, compare their SHA-256 with `digest` in the document manifest, independently of the HTTP validator.
+
+Focused contracts have content-addressed URLs and a one-year immutable cache policy. Mutable JSON manifests and indexes require revalidation. See [focused contracts](https://www.shoalrun.com/api-docs/focused-contracts.md) for URL identity and retention limits. `resource-headers.json` describes selected hosting headers, including Markdown, JSON metadata, and contracts; it is not an exhaustive inventory of every static asset.

package/agent-docs/publish/content-negotiation.sections/verify-the-deployed-url.md

@@ -0,0 +1,62 @@
+---
+id: guide.content-negotiation
+title: Content negotiation
+summary: Understand content negotiation, why it helps agents, and what a hosting
+  provider needs to do.
+type: reference
+area: discovery
+subarea: discovery-agents
+canonical: /publish/content-negotiation
+---
+
+# Content negotiation
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Reference](https://www.shoalrun.com/.well-known/shoalrun/navigation/reference-tab-50f26fe22670.txt) (ID: reference-tab). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Reference](https://www.shoalrun.com/cli.md) > Output and presentation > Content negotiation
+
+**Updated: 2026-10-26.**
+
+Parent page: [Full Markdown page](/publish/content-negotiation.md)
+
+**Page reading path** — these links apply to the full page.
+
+**Next steps**
+
+- [Publish on Vercel](https://www.shoalrun.com/publish/vercel.md) — Generate a Vercel deployment with negotiation already configured.
+
+- [Add Shoalrun to another build or host](https://www.shoalrun.com/publish/host-adapters.md) — Supply the same HTTP rules on another host.
+
+Shoalrun builds two versions of every published page from one authored source. The HTML version is for people using a browser. The generated Markdown version contains shared material in a form agents can read directly. Authors can also add or suppress [format-specific content](https://www.shoalrun.com/author/components/only.md) in either version. The versions share a core but need not contain identical text.
+
+**Content negotiation** lets a visitor ask for a format at the page's usual URL. Some agents send the request header `Accept: text/markdown`. A host that supports negotiation responds with Shoalrun's Markdown version. A browser asking for HTML gets the HTML version. The URL stays the same:
+
+| Request to `/publish/output/`       | Response                                     |
+| ----------------------------------- | -------------------------------------------- |
+| Browser sends `Accept: text/html`   | HTML page, `Content-Type: text/html`         |
+| Agent sends `Accept: text/markdown` | Markdown page, `Content-Type: text/markdown` |
+
+`Accept` says what the visitor can use. `Content-Type` says what the host actually returned. The `.md` URL, such as `/publish/output.md`, also works directly.
+
+## Verify the deployed URL
+
+Run the [post-deploy checker](https://www.shoalrun.com/publish/deployment-check.md) against the real site origin from `framework/`. It tests every page in the deployed manifest, direct `.md` URLs, required headers, discovery resources, and 404 behavior:
+
+```sh
+npm run deploy:check -- https://your-docs.example/
+```
+
+For a manual check of one page, replace the domain and page path below with one from your site:
+
+```sh
+curl -i -H 'Accept: text/html' https://your-docs.example/publish/output/
+curl -i -H 'Accept: text/markdown' https://your-docs.example/publish/output/
+curl -i https://your-docs.example/publish/output.md
+curl -i -H 'Accept: text/markdown;q=0' https://your-docs.example/publish/output/
+```
+
+Check the status, headers, and body. The first and fourth requests should return HTML. The second and third should return readable Markdown that gives its reader what they need for the intended task. Shared facts and API constraints should stay consistent, while human- and agent-oriented content may differ; see [Shoalrun's authoring position](https://www.shoalrun.com/publish/agent-friendliness.md). The first two responses should include `Vary: Accept` so caches keep the formats separate, and a Markdown `Link` so clients can find the twin. The negotiated Markdown response should include `Content-Location` with its `.md` URL. Also check that a missing page returns 404 and the published discovery files have the types declared in `resource-headers.json`.
+
+Run these checks against the production domain after deployment. Local preview results establish what Shoalrun's Node server does; they do not prove that a CDN applies the same rules. For a whole-site audit, run the [AFDocs scorecard](https://afdocs.dev/) against the deployed site and review its findings against the site's content requirements.

package/agent-docs/publish/content-negotiation.sections/what-if-the-host-cannot-negotiate.md

@@ -0,0 +1,47 @@
+---
+id: guide.content-negotiation
+title: Content negotiation
+summary: Understand content negotiation, why it helps agents, and what a hosting
+  provider needs to do.
+type: reference
+area: discovery
+subarea: discovery-agents
+canonical: /publish/content-negotiation
+---
+
+# Content negotiation
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Reference](https://www.shoalrun.com/.well-known/shoalrun/navigation/reference-tab-50f26fe22670.txt) (ID: reference-tab). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Reference](https://www.shoalrun.com/cli.md) > Output and presentation > Content negotiation
+
+**Updated: 2026-10-26.**
+
+Parent page: [Full Markdown page](/publish/content-negotiation.md)
+
+**Page reading path** — these links apply to the full page.
+
+**Next steps**
+
+- [Publish on Vercel](https://www.shoalrun.com/publish/vercel.md) — Generate a Vercel deployment with negotiation already configured.
+
+- [Add Shoalrun to another build or host](https://www.shoalrun.com/publish/host-adapters.md) — Supply the same HTTP rules on another host.
+
+Shoalrun builds two versions of every published page from one authored source. The HTML version is for people using a browser. The generated Markdown version contains shared material in a form agents can read directly. Authors can also add or suppress [format-specific content](https://www.shoalrun.com/author/components/only.md) in either version. The versions share a core but need not contain identical text.
+
+**Content negotiation** lets a visitor ask for a format at the page's usual URL. Some agents send the request header `Accept: text/markdown`. A host that supports negotiation responds with Shoalrun's Markdown version. A browser asking for HTML gets the HTML version. The URL stays the same:
+
+| Request to `/publish/output/`       | Response                                     |
+| ----------------------------------- | -------------------------------------------- |
+| Browser sends `Accept: text/html`   | HTML page, `Content-Type: text/html`         |
+| Agent sends `Accept: text/markdown` | Markdown page, `Content-Type: text/markdown` |
+
+`Accept` says what the visitor can use. `Content-Type` says what the host actually returned. The `.md` URL, such as `/publish/output.md`, also works directly.
+
+## What if the host cannot negotiate?
+
+GitHub Pages can publish Shoalrun's HTML, `.md` files, and `/llms.txt`. It serves static files and has no request handler for choosing a file from `Accept` or setting the page response headers described above. Agents can still read the `.md` URLs, but a request to the HTML URL cannot negotiate there. A programmable proxy or CDN can add that behavior. You can also publish the same built files on a host with request rules. Bare GitHub Pages cannot pass AFDocs' content-negotiation check.
+
+`site.basePath` also prefixes assets, API contracts, `/llms.txt`, and discovery files. For a GitHub Pages project site at `/<repository>/`, set `site.basePath: /<repository>` and publish the matching subtree from the build output as the project site's root. GitHub Pages still cannot provide Markdown content negotiation at an HTML URL.

package/agent-docs/publish/content-negotiation.sections/what-shoalrun-builds-and-what-the-host-does.md

@@ -0,0 +1,49 @@
+---
+id: guide.content-negotiation
+title: Content negotiation
+summary: Understand content negotiation, why it helps agents, and what a hosting
+  provider needs to do.
+type: reference
+area: discovery
+subarea: discovery-agents
+canonical: /publish/content-negotiation
+---
+
+# Content negotiation
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Reference](https://www.shoalrun.com/.well-known/shoalrun/navigation/reference-tab-50f26fe22670.txt) (ID: reference-tab). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Reference](https://www.shoalrun.com/cli.md) > Output and presentation > Content negotiation
+
+**Updated: 2026-10-26.**
+
+Parent page: [Full Markdown page](/publish/content-negotiation.md)
+
+**Page reading path** — these links apply to the full page.
+
+**Next steps**
+
+- [Publish on Vercel](https://www.shoalrun.com/publish/vercel.md) — Generate a Vercel deployment with negotiation already configured.
+
+- [Add Shoalrun to another build or host](https://www.shoalrun.com/publish/host-adapters.md) — Supply the same HTTP rules on another host.
+
+Shoalrun builds two versions of every published page from one authored source. The HTML version is for people using a browser. The generated Markdown version contains shared material in a form agents can read directly. Authors can also add or suppress [format-specific content](https://www.shoalrun.com/author/components/only.md) in either version. The versions share a core but need not contain identical text.
+
+**Content negotiation** lets a visitor ask for a format at the page's usual URL. Some agents send the request header `Accept: text/markdown`. A host that supports negotiation responds with Shoalrun's Markdown version. A browser asking for HTML gets the HTML version. The URL stays the same:
+
+| Request to `/publish/output/`       | Response                                     |
+| ----------------------------------- | -------------------------------------------- |
+| Browser sends `Accept: text/html`   | HTML page, `Content-Type: text/html`         |
+| Agent sends `Accept: text/markdown` | Markdown page, `Content-Type: text/markdown` |
+
+`Accept` says what the visitor can use. `Content-Type` says what the host actually returned. The `.md` URL, such as `/publish/output.md`, also works directly.
+
+## What Shoalrun builds and what the host does
+
+Shoalrun creates the HTML page, its `.md` twin, and a [`/documents.json`](https://www.shoalrun.com/documents.json) manifest listing both URLs. It also writes [`/resource-headers.json`](https://www.shoalrun.com/resource-headers.json) with content types for Markdown and discovery files.
+
+The host receives each HTTP request. It must read `Accept` and select the right built file. Providers use different names for this step: a server handler, a routing rule, or an edge function. The HTTP behavior is the same. A static file alone cannot read a request header.
+
+Shoalrun's Node server handles this. The standalone [Vercel build](https://www.shoalrun.com/publish/vercel.md) generates Vercel rules. On another host, use the contract below to configure its routing feature. The [host adapter guide](https://www.shoalrun.com/publish/host-adapters.md) covers adding Shoalrun's files to another build.

package/agent-docs/publish/content-negotiation.sections/why-this-matters.md

@@ -0,0 +1,47 @@
+---
+id: guide.content-negotiation
+title: Content negotiation
+summary: Understand content negotiation, why it helps agents, and what a hosting
+  provider needs to do.
+type: reference
+area: discovery
+subarea: discovery-agents
+canonical: /publish/content-negotiation
+---
+
+# Content negotiation
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Reference](https://www.shoalrun.com/.well-known/shoalrun/navigation/reference-tab-50f26fe22670.txt) (ID: reference-tab). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Reference](https://www.shoalrun.com/cli.md) > Output and presentation > Content negotiation
+
+**Updated: 2026-10-26.**
+
+Parent page: [Full Markdown page](/publish/content-negotiation.md)
+
+**Page reading path** — these links apply to the full page.
+
+**Next steps**
+
+- [Publish on Vercel](https://www.shoalrun.com/publish/vercel.md) — Generate a Vercel deployment with negotiation already configured.
+
+- [Add Shoalrun to another build or host](https://www.shoalrun.com/publish/host-adapters.md) — Supply the same HTTP rules on another host.
+
+Shoalrun builds two versions of every published page from one authored source. The HTML version is for people using a browser. The generated Markdown version contains shared material in a form agents can read directly. Authors can also add or suppress [format-specific content](https://www.shoalrun.com/author/components/only.md) in either version. The versions share a core but need not contain identical text.
+
+**Content negotiation** lets a visitor ask for a format at the page's usual URL. Some agents send the request header `Accept: text/markdown`. A host that supports negotiation responds with Shoalrun's Markdown version. A browser asking for HTML gets the HTML version. The URL stays the same:
+
+| Request to `/publish/output/`       | Response                                     |
+| ----------------------------------- | -------------------------------------------- |
+| Browser sends `Accept: text/html`   | HTML page, `Content-Type: text/html`         |
+| Agent sends `Accept: text/markdown` | Markdown page, `Content-Type: text/markdown` |
+
+`Accept` says what the visitor can use. `Content-Type` says what the host actually returned. The `.md` URL, such as `/publish/output.md`, also works directly.
+
+## Why this matters
+
+An agent reading HTML may receive navigation, styles, scripts, and interactive controls along with the article. It may then have to extract the useful content or convert the HTML to Markdown. Shoalrun's generated Markdown gives it the article directly, including material inside tabs and collapsibles and resolved OpenAPI examples. This can make the page easier to retrieve and less likely to lose content during conversion. [AFDocs checks](https://afdocs.dev/checks/markdown-availability) both `.md` URLs and content negotiation for this reason.
+
+The direct `.md` links in each HTML page and [`/llms.txt`](https://www.shoalrun.com/llms.txt) help agents that know to follow them. Negotiation helps an agent that arrives at the ordinary page URL and requests Markdown without first discovering a second URL. It does not require writers to maintain a separate article.

package/agent-docs/publish/deployment-check.md

@@ -0,0 +1,72 @@
+---
+id: guide.deployment-check
+title: Check a deployed site
+summary: Run the provider-neutral HTTP checker after publishing and before release.
+type: how-to
+area: quality
+subarea: quality-audits
+canonical: /publish/deployment-check
+---
+
+# Check a deployed site
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Examples](https://www.shoalrun.com/.well-known/shoalrun/navigation/examples-tab-50f26fe22670.txt) (ID: examples-tab). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Examples](https://www.shoalrun.com/config/examples.md) > Maintain this documentation site > Check a deployed site
+
+**Updated: 2026-10-26.**
+
+A local build verifies the generated files. After deployment, check what the host actually serves. This guide uses `scripts/check-deployment.mjs` from Shoalrun's docs repository; the script is not included in the `@shoalrun/cli` package. From the docs repository root, pass the deployed **site origin** to the checker:
+
+```sh
+npm run deploy:check -- https://docs.example.com/
+```
+
+Use the real HTTPS site URL and a trailing slash, including a configured base path such as `https://docs.example.com/docs/`. The command does not change the deployed site.
+
+For a branch deployment, pass the full hash that the deployment job checked out. The checker reads `/build-info.json` and fails if that domain serves a different revision:
+
+```sh
+npm run deploy:check -- https://docs.example.com/ --source-commit "$SOURCE_COMMIT"
+```
+
+`SOURCE_COMMIT` must be a 40- or 64-character Git commit hash. Supply it from the job that deploys this origin, for example `git rev-parse HEAD` after checkout. A missing or mismatched record is a failure.
+
+For a site with hosted chat and search, add `--chat`. It checks the public chat settings and both endpoints: GET returns 405, a foreign origin returns 403, and an invalid same-origin request returns 400. Invalid requests stop before model or embedding calls. If chat is already paused, use `--chat --chat-paused` instead; the checker expects its 503 paused response. It does not change the pause flag.
+
+```sh
+npm run deploy:check -- https://docs.example.com/ --chat
+```
+
+## What it checks
+
+The checker reads the deployed `/documents.json` and tests every listed page:
+
+- HTML at the page URL with `Accept: text/html`, including its content type, `Vary: Accept`, and alternate Markdown `Link` header.
+- Markdown at the same URL with `Accept: text/markdown`, including its content type, `Vary`, `Content-Location`, and alternate link.
+- The direct `.md` URL and exact text equality between its body and the negotiated Markdown body.
+- Each page's `og:url` and `og:image` point to the checked public origin. When pages have share images, it fetches one image and checks that the host serves it as an image.
+- Canonical links point to the page being checked, and noindex matches the manifest's indexing choice.
+- `/robots.txt` advertises the sitemap when one exists. Every sitemap URL matches an indexable manifest page, every indexable page is listed once, and `lastmod` matches its metadata. The checker fetches child sitemaps and catches redirects and HTTP errors.
+
+It also checks `/llms.txt`, `/resource-headers.json`, and the API catalog when the resource headers declare one. It verifies discovery content types and that a random missing route returns 404 for both HTML and Markdown requests. This establishes the host's response behavior; it does not judge whether the authored HTML and Markdown prose mean the same thing.
+
+## Read the result
+
+The command prints a `PASS` or `FAIL` summary with request, document, and failure counts. On failure, each `FAIL` line names the route and the missing or incorrect response. For example, a Markdown response served as HTML is reported as a `Content-Type` failure. Fix the host route or headers, deploy again, and rerun the checker before release.
+
+Exit status `0` means every check passed, `1` means the deployed site failed one or more checks, and `2` means the command input is invalid. A passing result does not replace manual review, accessibility checks, or an external scorecard.
+
+The [content negotiation guide](https://www.shoalrun.com/publish/content-negotiation.md) defines the host-independent HTTP contract. The [Vercel guide](https://www.shoalrun.com/publish/vercel.md) shows one hosting recipe. The checker works against any provider that serves the same contract.
+
+## Related reading
+
+### Related topics
+
+- [Host and limit documentation chat](https://www.shoalrun.com/chat/host.md) — Check a deployed site's headers, redirects, and Markdown negotiation. (Status: experimental.)
+
+### Often confused with
+
+- [Audit a docs project](https://www.shoalrun.com/publish/audit.md) — The audit checks project sources in CI; the deployment check tests the published site over HTTP.

package/agent-docs/publish/evals.md

@@ -0,0 +1,175 @@
+---
+id: guide.evals
+title: Evaluate agent tasks against your docs
+summary: Define realistic tasks, record reviewed answers from your own agent,
+  and check the results against a documentation build.
+type: how-to
+area: quality
+subarea: quality-evaluations
+canonical: /publish/evals
+---
+
+# Evaluate agent tasks against your docs
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Docs](https://www.shoalrun.com/.well-known/shoalrun/navigation/docs-50f26fe22670.txt) (ID: docs). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Docs](https://www.shoalrun.com/what-is-shoalrun.md) > Publish and maintain > Evaluate agent tasks against your docs
+
+**Updated: 2026-10-26.**
+
+Use evaluation tasks to check whether an agent can accomplish important work using your documentation. Shoalrun defines task and result formats, prepares report templates, and checks those reports during an audit. You run your own agent and arrange human or external review of its answers. Shoalrun does not launch agents, choose a model, or grade prose.
+
+You need a built docs site, a few realistic tasks, and an agent or manual workflow for running them. The ordinary build does not execute evaluations or call a model. Evaluations are opt-in audit checks.
+
+## Define a task file
+
+Save YAML or JSON with `schemaVersion: 1` and a nonempty `tasks` list. Each task needs a unique lowercase ID, a prompt, an `allowedTools` list, and criteria with unique IDs and descriptions. Tasks are required by default; set `required: false` for an exploratory task that should not block an audit.
+
+```yaml title="evals/tasks.yaml"
+schemaVersion: 1
+tasks:
+  - id: reader-views
+    prompt: Explain what reader views hide and whether they protect private content.
+    allowedTools: [docs.search, docs.read]
+    criteria:
+      - id: visibility
+        description: Explain visibility versus access control and cite the reader-views page.
+      - id: markdown
+        description: Explain that published Markdown retains reader-marked content.
+```
+
+Use the same tool names in your task file and runner report, such as `search`, `fetch`, `docs.search`, or `mcp.get_page`. Shoalrun validates the recorded runner tools against each executed task's allowed list. It cannot observe or enforce what tools an external agent actually used. Use a shared allowed tool set within a report; prepare separate suites for runs with different tool permissions.
+
+Write criteria that an expert can assess. Valid citations and complete report fields are mechanical checks. Whether the answer correctly explains your product remains a review judgment.
+
+## Prepare a report for the built site
+
+```sh
+shoalrun build docs.config.yaml site-dist
+shoalrun eval prepare site-dist --tasks evals/tasks.yaml --out evals/results.json
+```
+
+Create the `evals` directory first. Preparation refuses to overwrite an existing report; use a new output filename for another run.
+
+The template contains a task-file digest, a documentation-build digest, a per-file `buildFiles` inventory, a timestamp, empty runner details, and each task marked `skipped`. It records no passing results and does not run the tasks. A template with skipped required tasks fails the check until you complete the run and review.
+
+The task digest uses normalized definitions, so whitespace-only changes do not invalidate a run. Prompt, criteria, tools, or required-status changes do. The build digest covers published output files, including HTML, Markdown, indexes, discovery, and assets. It excludes `build-info.json`, private `ai/` artifacts, and `.shoalrun/` state. File touches do not invalidate otherwise identical output; changed published bytes do. Other published metadata, including build IDs embedded in manifests, remains covered even when it reflects deployment provenance.
+
+## Diagnose a different build
+
+Save a file inventory alongside an evaluation build:
+
+```sh
+shoalrun eval snapshot site-dist > evaluated-build.json
+shoalrun eval snapshot rebuilt-site > current-build.json
+shoalrun eval compare current-build.json --since evaluated-build.json
+```
+
+The comparison lists added, changed, and removed file paths. It exits 0 for identical snapshots and 1 for differences or invalid input. Each inventory binds relative file paths and SHA-256 byte digests to the same build digest used by evaluation reports. It contains no answers or grades.
+
+New report templates include this inventory in `buildFiles`. When a report is stale, `eval check` includes the file comparison in `details.buildChanges`. Older reports still receive the exact-build check, but cannot identify which earlier files differ. Editing the stored digest or deleting changed files is not a new evaluation.
+
+For local review of a hosted build, download and retain that exact output, task file, and report template. Run the tasks against the downloaded site and check the completed report against it. Rebuilding the source on another operating system can change rendered assets; a report about the downloaded build does not automatically cover that rebuild. Archive the source revision and build inventory with the run. A successful build capture records no evaluation pass.
+
+## Run and review the tasks
+
+Use your own agent with the declared tool access. Record the actual runner name, optional model, available tools, and any limitations in `runner`. Replace `recordedAt` with the run's UTC ISO timestamp.
+
+For each task, record its status (`pass`, `fail`, or `skipped`), answer, citations, and criterion judgments. A passing result needs a nonempty answer, at least one citation, and a passing judgment with a reason for every defined criterion. Identify citations by stable page ID and publication, so editions and languages cannot be confused:
+
+```json
+{
+  "taskId": "reader-views",
+  "status": "pass",
+  "answer": "Reader views change visibility. Public Markdown still includes reader-marked sections, so the views do not protect private content.",
+  "citations": [
+    {
+      "id": "guide.reader-filters",
+      "publication": { "channel": "production", "version": "current", "language": "en" }
+    }
+  ],
+  "criteria": [
+    { "id": "visibility", "status": "pass", "reason": "The answer distinguishes visibility from authorization." },
+    { "id": "markdown", "status": "pass", "reason": "The answer states that reader-marked sections remain in Markdown." }
+  ]
+}
+```
+
+This is an example result shape. Record your agent's actual answer and the reviewer’s assessment rather than copying the example as evidence of a run.
+
+Complete `review` with `reviewer`, `method` (`human` or `external`), `reviewedAt`, and `notes`. The review timestamp must be at or after `recordedAt`; both must be valid UTC timestamps and not in the future. Identify an external grader honestly, and record its limitations. A named reviewer is evidence about who assessed the answer, not independent proof that the assessment is correct.
+
+## Check the report
+
+```sh
+shoalrun eval check site-dist --tasks evals/tasks.yaml --report evals/results.json
+```
+
+The command prints JSON and exits 1 on failure. It checks schema versions and fields, unique task and criterion IDs, the two digests, timestamps, runner and review details, declared tool access, criterion coverage, and whether each citation identifies a published page in the selected publication. Required tasks must pass. Missing or failed optional tasks appear in the outcomes without blocking, but malformed results, bad citations, or unsupported passing claims still fail validation.
+
+Shoalrun verifies that cited pages exist; it does not verify that their text supports an answer. Reported tool use and grades are assertions supplied by the runner and reviewer. A passing report does not establish broad agent reliability or factual correctness.
+
+## Add the audit hook
+
+To check an existing report whenever you run `shoalrun audit` or `shoalrun check`, add:
+
+```yaml title="docs.config.yaml"
+audit:
+  evals:
+    tasks: ./evals/tasks.yaml
+    report: ./evals/results.json
+```
+
+Paths resolve beside the config file declaring each value, including included config files. The audit builds a temporary site, then checks the report against that output. Changes to published output or task definitions require a new evaluation. Reports remain project files; Shoalrun does not publish them into the site automatically.
+
+If your team already has an evaluator command, configure it as an optional `run` list:
+
+```yaml title="docs.config.yaml"
+audit:
+  evals:
+    tasks: ./evals/tasks.yaml
+    report: ./evals/results.json
+    run: [node, scripts/evaluate-docs.mjs]
+```
+
+That script belongs to your project. It runs after a successful build, from the root config directory, with no shell interpolation and a two-minute timeout. It receives these environment variables:
+
+| Variable                     | Value                               |
+| ---------------------------- | ----------------------------------- |
+| `SHOALRUN_EVAL_SITE`         | Absolute temporary build directory. |
+| `SHOALRUN_EVAL_TASKS`        | Absolute task-file path.            |
+| `SHOALRUN_EVAL_REPORT`       | Absolute report-file path to write. |
+| `SHOALRUN_EVAL_BUILD_DIGEST` | Digest of the built documentation.  |
+| `SHOALRUN_EVAL_TASKS_DIGEST` | Digest of the normalized tasks.     |
+
+The script must run and assess tasks, write a report, and exit nonzero on execution failure. A successful exit does not bypass report checks. A nonzero exit fails the audit even if an older report still validates. Hook execution and report validation appear separately as `eval-run` and `evals` in the audit results. A failed build prevents the eval hook from running.
+
+Configured commands can use your provider keys, make network requests, and write files. Shoalrun supplies the interface; your team owns the execution, costs, review, and credentials. The temporary build is removed after the audit. For longer runs, execute your evaluator separately and import the resulting report.
+
+## Shoalrun's own example
+
+This repository retains `docs/evals/tasks.yaml` and `docs/evals/results.json` as an example of a small agent self-assessment. Its five questions cover custom search providers, search off, bounded lexical coverage, reader choices, and freshness. This narrow, familiar task set is not a general quality gate. The site's routine audit does not import the report, and ordinary documentation changes do not require rerunning it.
+
+The original October 1 report is preserved unchanged in `docs/evals/history/results-2026-10-01.json`. The October 15 local run retains its actual tool responses, new answers, self-review judgments, and exact-build inventory under `docs/evals/runs/2026-10-15-local/`. The evaluator knew the project and had seen the earlier report; tasks shared retrieved pages. These are historical observations, not blind or independent evidence of general agent reliability. Earlier exploratory results remain in `plans/documentation/agent-docs-eval.json`.
+
+Run a new evaluation when it answers a concrete product question, with relevant tasks and a clearly described runner and reviewer. Preserve older reports rather than changing their dates, digests, or grades. An exact-build report can legitimately fail freshness against a later build; that does not make its original evidence invalid or require routine refreshes.
+
+Shoalrun's evaluation commands and optional audit hook remain available for projects that choose them. This site's normal validation still covers the build, links, schemas, output consistency, relevant behavior tests, and browser checks. To inspect the archived October 15 evidence without requiring the current site to match its old build, run from the repository root after compiling the framework:
+
+```sh
+node docs/evals/runs/2026-10-15-local/verify.mjs
+```
+
+That command checks the recorded responses and report provenance; it does not rerun tasks, grade answers, or certify the current site.
+
+## Related reading
+
+### Reference while working
+
+- [CLI reference](https://www.shoalrun.com/cli.md) — Look up report preparation and validation commands.
+
+### Related topics
+
+- [Non-goals](https://www.shoalrun.com/non-goals.md) — Understand why agent execution remains outside Shoalrun.

package/agent-docs/publish/evals.sections/add-the-audit-hook.md

@@ -0,0 +1,63 @@
+---
+id: guide.evals
+title: Evaluate agent tasks against your docs
+summary: Define realistic tasks, record reviewed answers from your own agent,
+  and check the results against a documentation build.
+type: how-to
+area: quality
+subarea: quality-evaluations
+canonical: /publish/evals
+---
+
+# Evaluate agent tasks against your docs
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Docs](https://www.shoalrun.com/.well-known/shoalrun/navigation/docs-50f26fe22670.txt) (ID: docs). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Docs](https://www.shoalrun.com/what-is-shoalrun.md) > Publish and maintain > Evaluate agent tasks against your docs
+
+**Updated: 2026-10-26.**
+
+Parent page: [Full Markdown page](/publish/evals.md)
+
+Use evaluation tasks to check whether an agent can accomplish important work using your documentation. Shoalrun defines task and result formats, prepares report templates, and checks those reports during an audit. You run your own agent and arrange human or external review of its answers. Shoalrun does not launch agents, choose a model, or grade prose.
+
+You need a built docs site, a few realistic tasks, and an agent or manual workflow for running them. The ordinary build does not execute evaluations or call a model. Evaluations are opt-in audit checks.
+
+## Add the audit hook
+
+To check an existing report whenever you run `shoalrun audit` or `shoalrun check`, add:
+
+```yaml title="docs.config.yaml"
+audit:
+  evals:
+    tasks: ./evals/tasks.yaml
+    report: ./evals/results.json
+```
+
+Paths resolve beside the config file declaring each value, including included config files. The audit builds a temporary site, then checks the report against that output. Changes to published output or task definitions require a new evaluation. Reports remain project files; Shoalrun does not publish them into the site automatically.
+
+If your team already has an evaluator command, configure it as an optional `run` list:
+
+```yaml title="docs.config.yaml"
+audit:
+  evals:
+    tasks: ./evals/tasks.yaml
+    report: ./evals/results.json
+    run: [node, scripts/evaluate-docs.mjs]
+```
+
+That script belongs to your project. It runs after a successful build, from the root config directory, with no shell interpolation and a two-minute timeout. It receives these environment variables:
+
+| Variable                     | Value                               |
+| ---------------------------- | ----------------------------------- |
+| `SHOALRUN_EVAL_SITE`         | Absolute temporary build directory. |
+| `SHOALRUN_EVAL_TASKS`        | Absolute task-file path.            |
+| `SHOALRUN_EVAL_REPORT`       | Absolute report-file path to write. |
+| `SHOALRUN_EVAL_BUILD_DIGEST` | Digest of the built documentation.  |
+| `SHOALRUN_EVAL_TASKS_DIGEST` | Digest of the normalized tasks.     |
+
+The script must run and assess tasks, write a report, and exit nonzero on execution failure. A successful exit does not bypass report checks. A nonzero exit fails the audit even if an older report still validates. Hook execution and report validation appear separately as `eval-run` and `evals` in the audit results. A failed build prevents the eval hook from running.
+
+Configured commands can use your provider keys, make network requests, and write files. Shoalrun supplies the interface; your team owns the execution, costs, review, and credentials. The temporary build is removed after the audit. For longer runs, execute your evaluator separately and import the resulting report.

package/agent-docs/publish/evals.sections/check-the-report.md

@@ -0,0 +1,36 @@
+---
+id: guide.evals
+title: Evaluate agent tasks against your docs
+summary: Define realistic tasks, record reviewed answers from your own agent,
+  and check the results against a documentation build.
+type: how-to
+area: quality
+subarea: quality-evaluations
+canonical: /publish/evals
+---
+
+# Evaluate agent tasks against your docs
+
+> Documentation index: [llms.min.txt](https://www.shoalrun.com/llms.min.txt) · Document manifest: [documents.json](https://www.shoalrun.com/documents.json)
+
+Navigation section: [Docs](https://www.shoalrun.com/.well-known/shoalrun/navigation/docs-50f26fe22670.txt) (ID: docs). [All sections and pages](https://www.shoalrun.com/llms.min.txt)
+
+Breadcrumbs: [Docs](https://www.shoalrun.com/what-is-shoalrun.md) > Publish and maintain > Evaluate agent tasks against your docs
+
+**Updated: 2026-10-26.**
+
+Parent page: [Full Markdown page](/publish/evals.md)
+
+Use evaluation tasks to check whether an agent can accomplish important work using your documentation. Shoalrun defines task and result formats, prepares report templates, and checks those reports during an audit. You run your own agent and arrange human or external review of its answers. Shoalrun does not launch agents, choose a model, or grade prose.
+
+You need a built docs site, a few realistic tasks, and an agent or manual workflow for running them. The ordinary build does not execute evaluations or call a model. Evaluations are opt-in audit checks.
+
+## Check the report
+
+```sh
+shoalrun eval check site-dist --tasks evals/tasks.yaml --report evals/results.json
+```
+
+The command prints JSON and exits 1 on failure. It checks schema versions and fields, unique task and criterion IDs, the two digests, timestamps, runner and review details, declared tool access, criterion coverage, and whether each citation identifies a published page in the selected publication. Required tasks must pass. Missing or failed optional tasks appear in the outcomes without blocking, but malformed results, bad citations, or unsupported passing claims still fail validation.
+
+Shoalrun verifies that cited pages exist; it does not verify that their text supports an answer. Reported tool use and grades are assertions supplied by the runner and reviewer. A passing report does not establish broad agent reliability or factual correctness.