@iowarp/clio-coder 0.4.1 → 0.4.3

package/docs/{eval-runner.md → process/eval-runner.md}

@@ -1,11 +1,11 @@
 # Clio Coder Local Evaluation Runner
 
-> [!TIP]
-> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.4.0).
+> **Visual blueprint:** The source checkout includes the complete
+> [Clio Coder Local Evaluation Runner visual reference](https://github.com/iowarp/clio-coder/blob/main/docs/html/eval_blueprint.html).
 
 The local evaluation runner executes repository-local YAML task suites as deterministic subprocess checks. It is useful for comparing harness changes, prompts, tools, or local workflows.
 
-Source of truth: [src/domains/eval/](../src/domains/eval/) and [src/cli/eval.ts](../src/cli/eval.ts).
+Source of truth: [src/domains/eval/](../../src/domains/eval/) and [src/cli/eval.ts](../../src/cli/eval.ts).
 
 ---
 
@@ -20,6 +20,7 @@ clio-coder eval run --task-file <tasks.yaml> [--repeat <n>] [--out <path>] [--cl
 clio-coder eval report <evalId> --format text|json|md|swe-jsonl|junit
 clio-coder eval compare <baselineEvalId> <candidateEvalId> [--metric <name>] [--format text|json|md|junit] [--allow-config-drift]
 clio-coder eval gate <candidateEvalId> --baseline <baselineEvalId> [--thresholds <file>]
+clio-coder eval inventory --json
 ```
 
 ### Command Roles
@@ -33,6 +34,7 @@ clio-coder eval gate <candidateEvalId> --baseline <baselineEvalId> [--thresholds
   * `junit`: XML report for CI/CD integration.
 * **`compare`**: Compares two evaluation artifacts (baseline and candidate) by matching tasks.
 * **`gate`**: Compares candidate metrics against baseline and absolute thresholds. Correctness and safety regressions fail independently of informational budgets.
+* **`inventory`**: Prints the fixed machine-readable inventory used by GUI hosts. It includes stored report identity, provenance, serving facts, accounting, and per-scenario outcomes without report attachments.
 
 Exit codes:
 
@@ -42,7 +44,7 @@ Exit codes:
 | `eval run` | `0` when all task repetitions pass | `1` when any task fails, `2` for invalid configs |
 | `eval report` | `0` when artifact loads | `1` if artifact cannot be read, `2` for invalid ID |
 | `eval compare` | `0` when both artifacts compare and the behavioral hard gate passes | `1` for a hard regression or unreadable artifact, `2` for invalid ID |
-| `eval gate` | `0` when correctness, safety, and hard threshold assertions pass | `1` for any hard failure, `2` for config/invalid ID errors |
+| `eval gate` | `0` when correctness, safety, and hard threshold assertions pass | `1` for hard failures, unreadable inputs, and malformed threshold files; `2` for an invalid eval ID or usage error |
 
 ---
 
@@ -75,7 +77,7 @@ tasks:
       excludes:
         - "**/node_modules/**"
     runner:
-      kind: "clio-run" # clio-run | context-index | context-init | external-command
+      kind: "clio-coder-run" # clio-coder-run | context-index | context-init | external-command
       prompt: "Optimize the FFT tolerance bounds in solver.ts"
       timeoutMs: 60000
     verify:
@@ -103,12 +105,12 @@ tasks:
 | --- | --- | --- |
 | `version` | - | Must equal `2`. |
 | `suite` | `id`, `title`, `visibility`, `description` | Metadata identifying the evaluation suite. |
-| `matrix` | `targets[]`, `repeats`, `dimensions[]` | Matrix of execution targets, repetition count, and the execution-envelope fields intentionally varied by the suite. |
-| `workspace` | `kind`, `path`, `url`, `commit`, `checkout`, `excludes` | Workspace strategy: `local` (run in-place), `git` (clone from URL), or `temp-copy` (isolated copy of a directory). |
-| `runner` | `kind`, `prompt`, `command`, `commands`, `args`, `timeoutMs` | Runner type: `clio-run` (starts Clio agent loop), `context-index` (runs indexer), `context-init` (initializes context), `external-command` (spawns subprocess). |
-| `behavioral` | `schema`, `corpus`, `execution`, `expectedBehavior`, `forbiddenBehavior`, `judge` | Optional `clio.eval.scenario.v1` behavioral contract. Rules name a closed category and a typed predicate over transcript, tool, receipt, or grader facts. |
+| `matrix` | `targets[]`, `repeats`, `dimensions[]`, `maxCostUsd` | Matrix of execution targets, repetition count, execution-envelope fields intentionally varied by the suite, and an optional cumulative known-cost ceiling. |
+| `workspace` | `kind`, `path`, `url`, `commit`, `checkout`, `excludes`, `setup` | Workspace strategy: `local` (run in-place), `git` (clone from URL), or `temp-copy` (isolated copy of a directory). Optional `setup` commands prepare the workspace before the runner starts. |
+| `runner` | `kind`, `prompt`, `autonomy`, `agent`, `command`, `commands`, `args`, `timeoutMs` | Runner type: `clio-coder-run` (starts Clio's agent loop), `context-index` (runs the indexer), `context-init` (initializes context), or `external-command` (spawns a subprocess). `agent` selects a worker recipe and `autonomy` sets one-run headless authority. |
+| `behavioral` | `schema`, `corpus`, `execution`, `expectedBehavior`, `forbiddenBehavior`, `judge` | Optional `clio-coder.eval.scenario.v1` behavioral contract. Rules name a closed category and a typed predicate over transcript, tool, receipt, or grader facts. |
 | `verify` | `commands`, `measure`, `assertions`, `forbidPaths` | Validation steps: shell commands, a task-outcome grader, metric assertions (e.g. `op: lt` for max token counts), and files/directories that must not be created or modified (`forbidPaths`). |
-| `metrics` | `collect` | List of metric names to compile for the evaluation runs. |
+| `metrics` | `collect`, `readObservation` | Metric names to compile plus optional public allowlisted and decoy paths reduced to bounded read counters. Raw path strings do not enter behavioral facts. |
 
 ---
 
@@ -120,7 +122,7 @@ tasks:
 ---
 
 ## Runner Kinds
-* **`clio-run`**: Invokes the main Clio Coder agent loop with the task's prompt, tracing all tools.
+* **`clio-coder-run`**: Invokes the main Clio Coder agent loop with the task's prompt, tracing all tools. The released `clio-run` spelling is accepted only as a legacy input alias and is normalized before validation; writers and new suites use `clio-coder-run`.
 * **`context-index`**: Triggers the context engine to build index structures (`codewiki`).
 * **`context-init`**: Initializes workspace files (such as generating `CLIO-CODER.md`).
 * **`external-command`**: Spawns an external command or sequence of commands in the task workspace.
@@ -137,7 +139,37 @@ Metrics collected during runs can be validated automatically using the `verify.a
 * `eq` (equal)
 * `neq` (not equal)
 
-Metrics that can be validated include `tokens.input`, `tokens.output`, `tokens.total`, `latency.wallMs`, `tools.totalCalls`, `tools.failed`, `tools.blocked`, `verifier.exitCode`, and `result.pass`.
+Metrics that can be validated include `tokens.input`, `tokens.output`, `tokens.total`, `latency.wallMs`, `tools.totalCalls`, `tools.failed`, `tools.blocked`, `verifier.exitCode`, `result.pass`, and the `provider.*` metrics below. Each `verify.assertions` condition must hold; an unavailable metric fails closed.
+
+### Optional Provider-Health Gates
+
+A task that recovers from a provider error can still pass its task checks by default. Provider health is a separate, opt-in requirement. To require observed provider events and no observed terminal errors, add these assertions to the task:
+
+```yaml
+verify:
+  assertions:
+    - metric: "provider.measured"
+      op: "eq"
+      value: true
+    - metric: "provider.stopReason.error"
+      op: "eq"
+      value: 0
+```
+
+Suite-level `thresholds.fail` uses the opposite condition: a matching condition is a failure. The corresponding hard gate checks each run as follows:
+
+```yaml
+thresholds:
+  fail:
+    - metric: "provider.measured"
+      op: "eq"
+      value: false
+    - metric: "provider.stopReason.error"
+      op: "gt"
+      value: 0
+```
+
+Unavailable metrics also fail closed in a hard threshold. `thresholds.informational` records findings without changing exit status. These examples reject an observed error followed by a successful recovery while leaving the default ungated task-pass policy unchanged. To reject any observed retry start or assistant abort as well, add conditions on `provider.retryStarted` or `provider.stopReason.aborted` with the same assertion-versus-failure polarity.
 
 ---
 
@@ -173,14 +205,44 @@ Under the hood, these are parsed and wrapped into a Suite v2 adapter with:
 
 Clio maintains two distinct token accounting streams with different provenances. These accounts are never merged, reconciled, or treated as interchangeable:
 
-1. **`tokens.*` (Wire Streaming)**: Folded live off stdout from assistant `message_end` events watched by `token-stream.ts` / `createStreamInvariantFold`. This represents usage reported by the provider for assistant messages watched over the wire. On surfaces without stdout streaming (such as `clio-coder fleet run --json`), `tokens.measured` is `false`.
+1. **`tokens.*` (Wire Streaming)**: Folded live off stdout from assistant `message_end` events. These totals include all known usage on errored calls as well as successful calls; recovery never subtracts earlier spend. Only finite, nonnegative usage facts are admitted. On surfaces without the relevant stdout events (such as `clio-coder fleet run --json`), `tokens.measured` is `false`.
 2. **`receiptUsage.*` (Journal Receipts)**: Summed from an evaluation item's run journal. Every attempt writes a receipt carrying token counts and USD cost authenticated against its own ledger envelope.
 
 ### Fail-Closed Reporting
 Both accounting streams report unmeasured state with no counts at all rather than a numeric zero. Reporting zero for an unmeasured run would falsely claim the run cost nothing. On an unmeasured run, `tokens.total` resolves to `null` and fails closed on metric threshold comparisons.
 
+An errored call must carry at least one positive reported token, reasoning, or cost fact before its usage is considered observed. Reasoning-only or cost-only observations do not establish ordinary token totals. A stream containing only errored calls with missing or synthetic all-zero usage remains `tokens.measured: false`. The current event shape cannot distinguish synthetic all-zero failures from genuinely reported zero usage, so it cannot establish measured zero spending in either case. Partial positive usage remains included as known spend. On failed calls, adapters can also initialize individual absent fields to zero; those zeros remain unattributed and make coverage incomplete. The existing inclusive numeric fields are known subtotals, so their zeros do not prove complete zero spending when failed usage is incomplete.
+
+### Provider Observations and Failed-Call Share
+
+The `provider.*` metrics describe events observed on live stdout, folded before diagnostic output is truncated. Native runs and multi-command external runners retain these observations from their executed commands. They do not enumerate SDK-internal retries or network attempts that were never emitted. Filtered or opaque output can leave provider health unobserved even when the process exits successfully or a receipt reports task success.
+
+| Metric | Meaning |
+| --- | --- |
+| `provider.measured` | Whether an assistant terminal reason or a counted retry phase was observed. With no such observations this is `false`, and provider counters are absent. It does not certify complete provider coverage. |
+| `provider.stopReason.stop`, `provider.stopReason.toolUse`, `provider.stopReason.length` | Counts of these terminal reasons on assistant `message_end` events. |
+| `provider.stopReason.error`, `provider.stopReason.aborted`, `provider.stopReason.other` | Separate counts of errored, aborted, and other observed terminal reasons. An unrecognized terminal reason goes into `other`. Partial updates and repeated messages in `turn_end` or `agent_end` do not add counts. |
+| `provider.retryScheduled` | Observed `scheduled` phases: planned retries, including ones cancelled before execution. |
+| `provider.retryStarted` | Observed `retrying` phases: retry execution starts. Repeated `waiting` countdown frames do not count as attempts. |
+| `provider.retryCancelled`, `provider.retryExhausted`, `provider.retryRecovered` | Counts of the corresponding observed phases. They describe retry-chain outcomes and do not fabricate additional assistant calls. Attempt numbers can restart for each chain. |
+| `provider.errorUsageObservedCalls` | Errored calls with at least one positive reported token, reasoning, or cost fact. |
+| `provider.errorUsageUnobservedCalls` | Errored calls with no positive reported token, reasoning, or cost fact, including absent or all-zero usage. |
+| `provider.errorUsageIncompleteCalls` | Errored calls with unobserved usage, incomplete token fields, or ambiguous normalized zero fields. This can overlap `errorUsageObservedCalls` when only part of the usage is known. |
+| `provider.errorCostUnobservedCalls` | Errored calls without a positive cost fact. Zero or absent cost does not prove that an error was free, including when some token usage is known. |
+| `provider.errorTokens.input`, `provider.errorTokens.output`, `provider.errorTokens.total`, `provider.errorTokens.cacheRead`, `provider.errorTokens.cacheWrite` | Known positive token subtotals for errored calls. Absent or ambiguous zero fields remain unattributed; when total usage is absent, a total can still be summed from known token fields and remains incomplete. |
+| `provider.errorCostUsd` | Known positive cost subtotal from errored calls' stream usage objects. Cost can come from adapter pricing; it is not independently certified provider billing. |
+| `provider.errorReasoningTokens`, `provider.errorReasoningUnobservedCalls` | Known failed-call reasoning subtotal and calls without attributable reported reasoning. Reasoning can overlap output, so it is never added to ordinary token totals. |
+
+The failed-call share covers `stopReason: error`; aborted calls remain separately labeled. Failed-share amounts and usage-coverage counters appear only after an errored terminal message is observed. These share metrics supplement the inclusive `tokens.*` totals without changing the summary token shape, receipt schema, or verdict schema. Missing or partial failed usage makes them known subtotals, not a complete amount to subtract from total spend. The native runner's `cost.usd` can use receipt evidence, so equality with the stream-based `provider.errorCostUsd` is not guaranteed.
+
+Positive reported reasoning uses the normalized `usage.reasoning` field, `reasoning_tokens`, and supported nested provider-detail fields. The root `reasoningTokens` alias can be an adapter estimate without a provenance marker, so it is left unattributed rather than promoted to reported provider usage. Normalized zero reasoning is also unattributed because adapters can fill it when provider detail is absent. A reasoning-only failure is observed but has incomplete ordinary-token coverage; no output or total is inferred from it.
+
+These observations also do not reconcile the separate `trackedMetrics` ledger selection (#276). Tracked metrics prefer durable assistant-call facts when available, retain durable compaction and tool records, and otherwise fall back to stream calls. Artifacts expose source counts and warnings, but partial or mixed ledgers can omit stream-only calls and fork-inherited history remains unreconciled. Neither those tracked values nor the new provider counters prove complete run accounting; failed-compaction usage is retained separately in the out-of-turn usage ledger and usage report, and is not included by this eval fold.
+
 ---
 
+Full reconciliation across session, stdout, fork and out-of-turn evidence is deferred to v0.4.5 or later. Version 0.4.3 does not add an automatic rejection of cost or efficiency comparisons merely because those sources are partial or mixed. Matching source counts do not prove complete coverage or shared call identity. Existing missing-metric, serving-configuration and execution-envelope comparison gates still apply.
+
 ## Eval Artifact Format (v4)
 
 Evaluation artifacts use format version 4 (`EvalArtifactV4`). Summary token metrics report `measuredRuns` out of total `runs`:
@@ -190,7 +252,7 @@ export interface EvalArtifactV4 {
   version: 4;
   evalId: string;
   suite: { id: string; hash: string };
-  clio: EvalClioProvenance;
+  clioCoder: EvalClioProvenance;
   environment: EvalEnvironmentProvenance;
   matrix: { target: string; model: string | null; thinking: string | null };
   summary: EvalArtifactSummaryV4;
@@ -206,11 +268,11 @@ export interface EvalArtifactV4 {
 
 ## The verdict envelope
 
-Every result carries a strictly parsed `clio.eval.verdict.v1` envelope (`src/domains/eval/schema/verdict.ts`). Suite v2 results are adapted into it at one explicit boundary (`src/domains/eval/schema/adapter.ts`) rather than by widening the artifact version, because the envelope carries no information a v4 artifact cannot hold.
+Every result carries a strictly parsed `clio-coder.eval.verdict.v1` envelope (`src/domains/eval/schema/verdict.ts`). Suite v2 results are adapted into it at one explicit boundary (`src/domains/eval/schema/adapter.ts`) rather than by widening the artifact version, because the envelope carries no information a v4 artifact cannot hold.
 
 ```json
 {
-  "schema": "clio.eval.verdict.v1",
+  "schema": "clio-coder.eval.verdict.v1",
   "scenarioId": "latency-nonnegative",
   "trialIndex": 0,
   "outcome": "pass",
@@ -230,7 +292,7 @@ The envelope is fail-closed by construction. `outcome` is one of `pass`, `fail`,
 
 ### Behavioral scenario and verdict documents
 
-Behavioral evaluation is additive and does not change the persisted `clio.eval.verdict.v1` reader. A Suite v2 task may declare a `clio.eval.scenario.v1` block, and its Artifact v4 result then carries a sibling `clio.eval.behavior.v1` document whose `verdictRef` names the verdict schema, scenario id, and trial index. This preserves existing verdicts and the tracked-metrics baseline while making a cross-linked behavioral document independently parseable.
+Behavioral evaluation is additive and does not change the persisted `clio-coder.eval.verdict.v1` reader. A Suite v2 task may declare a `clio-coder.eval.scenario.v1` block, and its Artifact v4 result then carries a sibling `clio-coder.eval.behavior.v1` document whose `verdictRef` names the verdict schema, scenario id, and trial index. This preserves existing verdicts and the tracked-metrics baseline while making a cross-linked behavioral document independently parseable. Readers normalize the released `clio.eval.*` identifiers for compatibility, but current writers emit only `clio-coder.eval.*` identifiers.
 
 The closed categories are `tool_choice`, `exploration`, `delegation`, `safety_comprehension`, `claim_grounding`, `denied_tool_recovery`, `completion_behavior`, and `task_correctness`. Each category result is exactly one of `satisfied`, `violated`, `unknown`, or `unmeasured`. The document outcome is `pass`, `behavioral_failure`, `unknown`, `unmeasured`, or `infrastructure_failure`; missing facts are never invented as successes, and an infrastructure failure cannot become a behavioral pass.
 
@@ -240,8 +302,8 @@ Suite execution adapts scalar run metrics into these observable facts at the Sui
 
 ### Public built-in behavioral corpus
 
-The repository ships corpus `public-built-in-behavior` version `1.0.0` under
-`benchmarks/eval/`. It contains no private prompts, endpoints, credentials, or
+The source repository carries corpus `public-built-in-behavior` version `1.0.0`
+under `evals/`. It contains no private prompts, endpoints, credentials, or
 mutable external dataset:
 
 - `behavioral-machinery.yaml` provides one positive and one adversarial
@@ -262,12 +324,15 @@ mutable external dataset:
   that the rules can reject observed model behavior rather than merely restate
   aggregate success counters.
 
-Build once, then run either focused suite from the repository root:
+These are source-checkout workflows: the npm archive keeps the inputs for
+inspection and reproducibility, but the deterministic TypeScript driver uses
+the repository development toolchain. Build once, then run either focused
+suite from the repository root:
 
 ```sh
-node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-machinery.yaml --clio-coder-entry dist/cli/index.js
-node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-model.yaml --target mini --clio-coder-entry dist/cli/index.js
-node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-model-negative-control.yaml --target mini --clio-coder-entry dist/cli/index.js
+node dist/cli/index.js eval run --suite evals/behavioral-machinery.yaml --clio-coder-entry dist/cli/index.js
+node dist/cli/index.js eval run --suite evals/behavioral-model.yaml --target mini --clio-coder-entry dist/cli/index.js
+node dist/cli/index.js eval run --suite evals/behavioral-model-negative-control.yaml --target mini --clio-coder-entry dist/cli/index.js
 ```
 
 The machinery tasks use the repository read-only and create only private
@@ -292,7 +357,7 @@ Eleven numbers plus a reason histogram, each carrying the source it came from. `
 | `generatedTokens` | ledger |
 | `reasoningTokens` | receipt; nullable, because absent and zero are different claims |
 | `toolCalls`, `toolErrors` | ledger when present, otherwise receipt |
-| `ttftMsFirstCall` | ledger |
+| `ttftMsFirstCall` | ledger; nullable when first-call timing is absent |
 | `wallClockMs` | receipt |
 | `contextTokensAtEnd` | ledger |
 | `compactions` | ledger |
@@ -300,14 +365,18 @@ Eleven numbers plus a reason histogram, each carrying the source it came from. `
 
 A dispatched worker's receipt reports `sessionId: null` and writes no session archive, which is why the ledger source exists at all: the runner folds structured usage, backend timing, cache, and monotonic TTFT facts out of the worker's `message_end` events. It keeps no prompt text, no model prose, and no tool-result content in that fold.
 
+First-call TTFT uses the earliest recorded assistant-call timestamp across the selected ledgers; equal timestamps retain their observed order. Missing or invalid timing, or an invalid timestamp that prevents ordering the calls, yields `{ value: null, source: "estimated" }`. A measured zero remains `{ value: 0, source: "ledger" }`. Native session timing starts at each stream invocation and includes the provider's response-header wait. Stdout-only fallback timing starts at the provider's `message_start`, which can arrive after headers; it requires first output, but is not complete request latency and must not be compared as equivalent to native invocation timing. A completion alone supplies neither a zero duration nor a first-token measurement. Verdict v1 consumers must accept nullable TTFT. Historical numeric values, including estimated zeros and native spans that omitted the pre-header wait, remain readable and are not rewritten.
+
+When a stream message lacks a valid timestamp, its ledger payload marks `timestampEstimated: true` beside the legacy ISO placeholder. This leaves first-call chronology unmeasured while preserving any observed per-call monotonic timing.
+
 ### Scenario aggregates
 
 `aggregates` groups verdicts by `scenarioId`, sets `k` to the trial count, and records `passAtK` (any trial passed) and `passPowK` (every trial passed). Each tracked numeric metric reports observation, measured, and unmeasured counts, mean, min, max, nearest-rank p90, population variance, standard deviation, and the set of sources observed. A metric with no observation keeps every numeric statistic `null`; it never becomes zero. At `k: 1`, variance and standard deviation are zero only when the value was actually measured.
 
 ### Behavioral multi-metric results
 
-A result with a `clio.eval.behavior.v1` verdict also carries the additive
-`clio.eval.behavior.metrics.v1` projection. The projection binds the scenario
+A result with a `clio-coder.eval.behavior.v1` verdict also carries the additive
+`clio-coder.eval.behavior.metrics.v1` projection. The projection binds the scenario
 to its role and target/model envelope and records one `number | null`
 observation for each closed metric. The source travels beside every value:
 
@@ -356,8 +425,8 @@ is emitted as testcase output rather than a failed testcase.
 ### Execution-envelope provenance and comparability
 
 Every newly written behavioral result carries an additive
-`clio.eval.execution-envelope.v1` sibling. Artifact v4,
-`clio.eval.verdict.v1`, and `clio.eval.behavior.metrics.v1` retain their
+`clio-coder.eval.execution-envelope.v1` sibling. Artifact v4,
+`clio-coder.eval.verdict.v1`, and `clio-coder.eval.behavior.metrics.v1` retain their
 existing identities. The envelope records the selected prompt fragment ids,
 authored versions or `unversioned` marker, fragment content hashes, prompt
 composition hash, recipe id/version/fingerprint when a worker recipe applies,
@@ -382,31 +451,15 @@ metric means and variances. When the prompt or recipe identity changes, the
 generated evidence names each affected corpus scenario and role instead of
 hiding it behind an aggregate score.
 
-### Checked behavioral release baseline
+### Reference behavioral baseline
 
-The checked deterministic baseline is
-`benchmarks/eval/behavioral-machinery-baseline.json`. The release gate runs all
-26 machinery-only scenarios through the built CLI and compares a stable
-projection of their labels, metrics, and execution envelopes with that file.
-It requires no model, private endpoint, credential, or mutable dataset.
-
-When an intentional prompt, recipe, policy, or expected-behavior change moves
-the evidence, run the same machinery suite first, inspect the failing diff and
-the named affected corpus results, then update explicitly:
-
-```sh
-npm run build
-node benchmarks/eval/check-behavioral-release.mjs --update
-git diff -- benchmarks/eval/behavioral-machinery-baseline.json
-```
-
-The baseline update belongs in the reviewed change that caused it. Do not use
-the update command merely to make a red gate green. The model-required and
-negative-control suites remain manual release evidence because their outputs
-depend on a live target; they are never folded into the deterministic baseline.
-The projection excludes `latency.wallMs` because scheduler timing is not stable
-evidence. Behavioral labels, deterministic metrics, and the execution envelope
-remain checked byte for byte.
+`evals/behavioral-machinery-baseline.json` is retained as reviewable reference
+evidence for the machinery corpus. It is not a CI or release gate. Run the
+current `evals/behavioral-machinery.yaml` through the built CLI when a prompt,
+recipe, policy, or expected-behavior change needs a fresh measurement, inspect
+the named scenario evidence, and update any retained baseline deliberately in
+the reviewed change. Model-required and negative-control suites remain manual
+measurements tied to their exact target and serving configuration.
 
 ### Hard thresholds and informational budgets
 
@@ -443,10 +496,12 @@ cannot offset a task or safety regression.
 
 ```text
 serving configuration drift; pass --allow-config-drift to compare these runs
-baseline serving: target=mini runtime=llamacpp model=... server_build=b226-2115b73d8 total_slots=1 thinking=off compiled_prompt_hash=...
+baseline serving: target=mini runtime=llamacpp model=ornith1.5-35b-moe server_build=... total_slots=4 thinking=off compiled_prompt_hash=...
 candidate serving: ...
 ```
 
+The current reference `mini` endpoint is the llama.cpp router at `192.168.86.141:8080`. It serves `ornith1.5-35b-moe` with four parallel slots and 262144 context tokens per slot. These deployment facts are reference topology, not defaults imposed on another target; retain the artifact's observed serving configuration with every comparison.
+
 `--allow-config-drift` proceeds and labels the comparison `config drift: allowed`. There is a second refusal that has no override: a metric whose baseline distribution contains an `estimated` observation and whose candidate does not, or the reverse, raises `EvalTrackedMetricSourceMismatchError` rather than printing a delta, because subtracting a measurement from an estimate produces a number that looks like evidence and is not. `--metric <name>` filters tracked or behavioral rows, accepts `expectedColdReasons`, a specific `expectedColdReasons.<reason>`, a behavioral family, or a behavioral metric, and errors when the name matches nothing.
 
 ---

package/docs/{evals-internal.md → process/evals-internal.md}

@@ -1,7 +1,7 @@
 # Internal Eval Suites
 
-> [!TIP]
-> **Interactive Spec Available:** An interactive blueprint is available at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) (Version: 0.4.0).
+> **Visual blueprint:** The source checkout includes the complete
+> [Internal Eval Suites visual reference](https://github.com/iowarp/clio-coder/blob/main/docs/html/evals_internal_blueprint.html).
 
 Private suites should live outside this repository. Keep datasets, prompts,
 live fleet coordinates, calibration outputs, and raw run artifacts in a private
@@ -15,13 +15,13 @@ clio-coder eval run --suite <external-path> --clio-coder-entry dist/cli/index.js
 ```
 
 Use `--out <dir>` when the artifact should be written outside the default Clio
-data directory. Product eval artifacts and external benchmark campaigns are
-separate: public benchmark adapters live under `benchmarks/community/` and do
-not use the eval runner.
+data directory. External benchmark campaigns should adapt their cases and
+grader observations into the same eval engine while keeping private datasets,
+credentials, endpoints, and raw artifacts outside this repository.
 
 The public behavioral corpus is the deliberate exception to the otherwise
 private Suite v2 data policy. Its reviewable, synthetic suites live under
-`benchmarks/eval/`: a model-free positive/adversarial authority pair for every
+`evals/`: a model-free positive/adversarial authority pair for every
 built-in worker recipe, four tiny main-agent model scenarios covering all eight
 behavioral categories with event- and grader-derived facts, and an intentional
 decoy negative control. The model-free driver uses the shipped recipe catalog,
@@ -83,7 +83,7 @@ thinking level is measuring the server, not the change under test.
 
 The verdict envelope keeps its original `behavioral: null` field for compatibility.
 A suite that declares a versioned behavioral scenario records the result as a
-separate `clio.eval.behavior.v1` document on the Artifact v4 result, cross-linked
+separate `clio-coder.eval.behavior.v1` document on the Artifact v4 result, cross-linked
 to the unchanged verdict identity. Its labels come only from bounded transcript,
 tool, receipt, or grader facts, never from an ungrounded judge paragraph. A run whose harness broke records
 `machinery: "infrastructure_failure"`, which the parser refuses to pair with a
@@ -210,7 +210,7 @@ tasks:
         - node_modules
         - dist
     runner:
-      kind: clio-run
+      kind: clio-coder-run
       prompt: Fix the intentionally broken function so the local verifier passes.
     verify:
       commands:
@@ -271,7 +271,7 @@ tasks:
         - dist
         - .clio-coder
     runner:
-      kind: clio-run
+      kind: clio-coder-run
       prompt: Summarize the repository purpose and make no file changes.
     verify:
       forbidPaths:
@@ -305,7 +305,7 @@ tasks:
         - dist
         - .clio-coder
     runner:
-      kind: clio-run
+      kind: clio-coder-run
       prompt: Fix the failing unit test with the smallest source change.
     verify:
       commands:

package/docs/{evolution.md → process/evolution.md}

@@ -1,7 +1,7 @@
 # Evolution and Change Manifests
 
-> [!TIP]
-> **Interactive Spec Available:** An interactive change manifest editor, authority risk assessor, and checklist workspace is located at [docs/html/evolution_blueprint.html](html/evolution_blueprint.html) (Version: 0.4.0).
+> **Visual blueprint:** The source checkout includes the complete
+> [Evolution and Change Manifests visual reference](https://github.com/iowarp/clio-coder/blob/main/docs/html/evolution_blueprint.html).
 
 Clio Coder uses change manifests to make harness changes reviewable, falsifiable, and rollback-friendly. CLIO stands for Context Layer for Input/Output, named for the Greek muse of history. A manifest is JSON, generated or checked with `clio-coder evolve manifest`, and should describe what changed, why, what evidence supports it, what could regress, how to validate it, and how to roll it back.
 

package/docs/{fleet-demo-runbook.md → process/fleet-demo-runbook.md}

@@ -1,10 +1,13 @@
 # Fleet Demo Runbook
 
+> **Visual blueprint:** The source checkout includes the complete
+> [Fleet Demo Runbook visual reference](https://github.com/iowarp/clio-coder/blob/main/docs/html/fleet_demo_blueprint.html).
+
 A repeatable multi-node demonstration: one orchestrator drives a real
 CMake/C++ fix through a reviewer-gated dispatch across SSH nodes, and every
 worker's receipt (including the remote ones) verifies afterward. The steps
 are executable in order; this document doubles as the recording script.
-Background and reference: [fleet-dispatch.md](fleet-dispatch.md).
+Background and reference: [fleet-dispatch.md](../guide/fleet-dispatch.md).
 
 ## Reference fabric
 
@@ -139,9 +142,9 @@ clio-coder evidence inspect <evidenceId>
 run ledger; a tampered or mismatched receipt fails the build with the field
 that diverged. The receipts of the remote runs verify on the orchestrator host because the
 ledger and receipts live on the shared filesystem. Current receipts use strict
-v16 and authenticate every current receipt and reconstructed-ledger field.
-Every other receipt version is rejected rather than reported as partial; the
-current binary has no historical receipt reader.
+v20 and authenticate every current receipt and reconstructed-ledger field.
+Lower versions are reported as retired and are never read as evidence or
+migrated; malformed and future shapes fail verification.
 
 ## Provenance walkthrough: what a PI can verify from receipts alone
 
@@ -171,9 +174,10 @@ reconstruct:
    complete receipt schema and its stable ledger row. `clio-coder evidence build
    --run <id>` recomputes and cross-checks it; `verifyReceiptIntegrity` in
    `src/domains/dispatch/receipt-integrity.ts` is the reference
-	implementation. Current receipts use v16 and every other version fails
-	verification. Incompatible state must be archived or removed; it is never
-	read as evidence through a compatibility verifier.
+	implementation. Current receipts use v20. Lower versions are reported as
+	retired, while malformed and future shapes fail verification. Incompatible
+	state may be archived for inspection, but it is never read as evidence through
+	a compatibility verifier.
 
 The walkthrough for an audience is three commands: `clio-coder evidence build
 --run <id>` (it verifies), open the receipt JSON (read `node`, `gate`,

package/docs/{git-commit-provenance.md → process/git-commit-provenance.md}

@@ -1,13 +1,20 @@
 # Git Commit Provenance
 
+> **Visual blueprint:** The source checkout includes the complete
+> [Git Commit Provenance visual reference](https://github.com/iowarp/clio-coder/blob/main/docs/html/git_commit_provenance_blueprint.html).
+
 Clio Coder adds evidence-aware role trailers to commits created through Clio.
 The feature is enabled by default:
 
 ```yaml
-attribution:
-  gitCommits: true
+integrations:
+  git:
+    commitAttribution: true
 ```
 
+The released `attribution.gitCommits` path is a migration alias. Current
+settings files and writers use `integrations.git.commitAttribution`.
+
 Settings -> Advanced exposes the same switch as **Clio commit provenance**, with
 `enabled` and `disabled` values. A change applies immediately to subsequent
 commits in the session. When disabled, Clio leaves commit messages entirely
@@ -49,11 +56,11 @@ Co-authored-by: Clio Coder <clio-coder@iowarp.ai>
 Existing human trailers stay in place. A Clio trailer already present in any
 letter case is respected rather than repeated, line endings are normalized only
 while attribution is enabled, and repeated processing is idempotent. When a directly relevant
-receipt-v19 digest passes integrity verification, Clio may additionally add the
+receipt-v20 digest passes integrity verification, Clio may additionally add the
 full digest:
 
 ```text
-Clio-Evidence: receipt-v19/sha256:<64-character digest>
+Clio-Evidence: receipt-v20/sha256:<64-character digest>
 ```
 
 Clio does not invent, shorten, or add an unrelated digest. The role trailers do