@iowarp/clio-coder 0.3.8 → 0.3.9

package/docs/eval-runner.md

@@ -1,7 +1,7 @@
 # Clio Coder Local Evaluation Runner
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.3.8).
+> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.3.9).
 
 The local evaluation runner executes repository-local YAML task suites as deterministic subprocess checks. It is useful for comparing harness changes, prompts, tools, or local workflows.
 
@@ -15,10 +15,10 @@ The CLI commands under `clio-coder eval` support running, validating, reporting,
 
 ```bash
 clio-coder eval validate --suite <suite.yaml>
-clio-coder eval run --suite <suite.yaml> [--target <id>] [--model <id>] [--out <path>] [--clio-coder-entry <path>]
+clio-coder eval run --suite <suite.yaml> [--trials <n>] [--target <id>] [--model <id>] [--out <path>] [--clio-coder-entry <path>]
 clio-coder eval run --task-file <tasks.yaml> [--repeat <n>] [--out <path>] [--clio-coder-entry <path>]
 clio-coder eval report <evalId> --format text|json|md|swe-jsonl|junit
-clio-coder eval compare <baselineEvalId> <candidateEvalId>
+clio-coder eval compare <baselineEvalId> <candidateEvalId> [--metric <name>] [--format text|json|md|junit] [--allow-config-drift]
 clio-coder eval gate <candidateEvalId> --baseline <baselineEvalId> [--thresholds <file>]
 ```
 
@@ -32,7 +32,7 @@ clio-coder eval gate <candidateEvalId> --baseline <baselineEvalId> [--thresholds
   * `swe-jsonl`: Standardized JSONL format representing task runs (e.g. for SWE-bench comparisons).
   * `junit`: XML report for CI/CD integration.
 * **`compare`**: Compares two evaluation artifacts (baseline and candidate) by matching tasks.
-* **`gate`**: Compares candidate metrics against baseline or absolute thresholds, exiting non-zero if assertions fail (useful for PR gating).
+* **`gate`**: Compares candidate metrics against baseline and absolute thresholds. Correctness and safety regressions fail independently of informational budgets.
 
 Exit codes:
 
@@ -41,8 +41,8 @@ Exit codes:
 | `eval validate` | `0` when validation passes | `2` for validation issues |
 | `eval run` | `0` when all task repetitions pass | `1` when any task fails, `2` for invalid configs |
 | `eval report` | `0` when artifact loads | `1` if artifact cannot be read, `2` for invalid ID |
-| `eval compare` | `0` when both artifacts load and compare succeeds | `1` if artifacts cannot be read, `2` for invalid ID |
-| `eval gate` | `0` when all threshold assertions pass | `1` if assertions fail, `2` for config/invalid ID errors |
+| `eval compare` | `0` when both artifacts compare and the behavioral hard gate passes | `1` for a hard regression or unreadable artifact, `2` for invalid ID |
+| `eval gate` | `0` when correctness, safety, and hard threshold assertions pass | `1` for any hard failure, `2` for config/invalid ID errors |
 
 ---
 
@@ -103,10 +103,11 @@ tasks:
 | --- | --- | --- |
 | `version` | - | Must equal `2`. |
 | `suite` | `id`, `title`, `visibility`, `description` | Metadata identifying the evaluation suite. |
-| `matrix` | `targets[]`, `repeats` | Matrix of execution targets (specifying model and thinking flags) and the repetition count. |
+| `matrix` | `targets[]`, `repeats`, `dimensions[]` | Matrix of execution targets, repetition count, and the execution-envelope fields intentionally varied by the suite. |
 | `workspace` | `kind`, `path`, `url`, `commit`, `checkout`, `excludes` | Workspace strategy: `local` (run in-place), `git` (clone from URL), or `temp-copy` (isolated copy of a directory). |
 | `runner` | `kind`, `prompt`, `command`, `commands`, `args`, `timeoutMs` | Runner type: `clio-run` (starts Clio agent loop), `context-index` (runs indexer), `context-init` (initializes context), `external-command` (spawns subprocess). |
-| `verify` | `commands`, `assertions`, `forbidPaths` | Validation steps: shell commands, metric assertions (e.g. `op: lt` for max token counts), and files/directories that must not be created or modified (`forbidPaths`). |
+| `behavioral` | `schema`, `corpus`, `execution`, `expectedBehavior`, `forbiddenBehavior`, `judge` | Optional `clio.eval.scenario.v1` behavioral contract. Rules name a closed category and a typed predicate over transcript, tool, receipt, or grader facts. |
+| `verify` | `commands`, `measure`, `assertions`, `forbidPaths` | Validation steps: shell commands, a task-outcome grader, metric assertions (e.g. `op: lt` for max token counts), and files/directories that must not be created or modified (`forbidPaths`). |
 | `metrics` | `collect` | List of metric names to compile for the evaluation runs. |
 
 ---
@@ -114,7 +115,7 @@ tasks:
 ## Workspace Kinds
 * **`local`**: Executes the task directly in the specified local path.
 * **`git`**: Clones the repository from `url`, checks out the specified `commit` or `checkout` ref, and runs there.
-* **`temp-copy`**: Copies the directory at `path` to a temporary workspace location before running. This prevents side-effects from polluting other task runs.
+* **`temp-copy`**: Copies the directory at `path` to a temporary workspace location immediately before the matrix item runs and removes it afterward. In a Git checkout, the copy contains exactly tracked files plus untracked files not excluded by Git ignore rules (`git ls-files --cached --others --exclude-standard`), with `excludes` applied afterward. Outside Git it retains the recursive directory copy. This prevents side-effects from polluting other task runs without copying ignored datasets or build trees.
 
 ---
 
@@ -194,12 +195,262 @@ export interface EvalArtifactV4 {
   matrix: { target: string; model: string | null; thinking: string | null };
   summary: EvalArtifactSummaryV4;
   results: EvalArtifactResultV4[];
+  servingConfiguration?: EvalServingConfigurationV1;
+  aggregates?: EvalScenarioAggregateV1[];
 }
 ```
 
+`servingConfiguration` and `aggregates` are additive. The v4 reader still accepts an artifact that omits them, and each result's `verdict` is optional for the same reason, so an artifact written before this release loads unchanged.
+
 ---
 
-## Task Outcome Measurement (`verify.measure`)
+## The verdict envelope
+
+Every result carries a strictly parsed `clio.eval.verdict.v1` envelope (`src/domains/eval/schema/verdict.ts`). Suite v2 results are adapted into it at one explicit boundary (`src/domains/eval/schema/adapter.ts`) rather than by widening the artifact version, because the envelope carries no information a v4 artifact cannot hold.
+
+```json
+{
+  "schema": "clio.eval.verdict.v1",
+  "scenarioId": "latency-nonnegative",
+  "trialIndex": 0,
+  "outcome": "pass",
+  "machinery": "ok",
+  "reason": null,
+  "trackedMetrics": { "...": "see below" },
+  "behavioral": null,
+  "evidence": {
+    "assignmentId": "qcy5rfopdrfw",
+    "terminalReceiptDigest": "d85a3ad4f8ae...",
+    "graderExitCode": 0
+  }
+}
+```
+
+The envelope is fail-closed by construction. `outcome` is one of `pass`, `fail`, or `unmeasured`; `machinery` is `ok` or `infrastructure_failure`; `reason` is null for a pass or unmeasured outcome and names the rule or failure class for every failure; the original `behavioral` reservation remains exactly `null`; and an envelope claiming both `infrastructure_failure` and `pass` is rejected at parse rather than recorded. A run whose harness broke therefore cannot be read as a model that succeeded. Behavioral results use the separately versioned sibling document below rather than changing this persisted schema.
+
+### Behavioral scenario and verdict documents
+
+Behavioral evaluation is additive and does not change the persisted `clio.eval.verdict.v1` reader. A Suite v2 task may declare a `clio.eval.scenario.v1` block, and its Artifact v4 result then carries a sibling `clio.eval.behavior.v1` document whose `verdictRef` names the verdict schema, scenario id, and trial index. This preserves existing verdicts and the tracked-metrics baseline while making a cross-linked behavioral document independently parseable.
+
+The closed categories are `tool_choice`, `exploration`, `delegation`, `safety_comprehension`, `claim_grounding`, `denied_tool_recovery`, `completion_behavior`, and `task_correctness`. Each category result is exactly one of `satisfied`, `violated`, `unknown`, or `unmeasured`. The document outcome is `pass`, `behavioral_failure`, `unknown`, `unmeasured`, or `infrastructure_failure`; missing facts are never invented as successes, and an infrastructure failure cannot become a behavioral pass.
+
+Expected and forbidden rules contain typed predicates over facts sourced from `transcript`, `tool`, `receipt`, or `grader`. Facts cite a locator, SHA-256 digest, and optional bounded excerpt. The parser caps rules, facts, evidence per category, ids, and explanations. Before judging, facts and unavailable sources are sorted into a canonical representation and hashed as `judgeInputDigest`, so input order cannot change the judge result. Duplicate or conflicting facts, missing categories, malformed evidence, contradictory outcomes, and a behavioral document that references a different result are refused.
+
+Suite execution adapts scalar run metrics into these observable facts at the Suite v2 to Artifact v4 boundary. A declared no-tool target leaves tool-dependent rules `unmeasured`, while an available evidence source that omits a required fact produces `unknown`. Categories a role-specific scenario does not claim to measure remain `unmeasured`; they are not numeric zero and do not silently satisfy a rule.
+
+### Public built-in behavioral corpus
+
+The repository ships corpus `public-built-in-behavior` version `1.0.0` under
+`benchmarks/eval/`. It contains no private prompts, endpoints, credentials, or
+mutable external dataset:
+
+- `behavioral-machinery.yaml` provides one positive and one adversarial
+  machinery-only check for each of the 13 shipped built-in worker recipes. Its
+  deterministic driver loads the production recipe catalog, admits a real
+  dispatch through the production gate, runs a scripted worker, and verifies
+  the sealed receipt and result-contract outcome. The 26 scenarios require no
+  model; they do not infer behavior by grepping recipe frontmatter.
+- `behavioral-model.yaml` provides four isolated main-agent scenarios on the
+  `mini` target: a focused edit, adversarial scope control, required
+  delegation, and recovery after Bash is denied. Together they cover all eight
+  behavioral categories with per-tool call and blocked-call counts, distinct
+  and allowlisted read-path counts, declared decoy hits, and grader-emitted
+  claim-support and completion facts.
+- `behavioral-model-negative-control.yaml` intentionally reads a declared
+  decoy. A healthy run solves its literal task while recording
+  `behavioral_failure` with violated exploration and safety labels, proving
+  that the rules can reject observed model behavior rather than merely restate
+  aggregate success counters.
+
+Build once, then run either focused suite from the repository root:
+
+```sh
+node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-machinery.yaml --clio-coder-entry dist/cli/index.js
+node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-model.yaml --target mini --clio-coder-entry dist/cli/index.js
+node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-model-negative-control.yaml --target mini --clio-coder-entry dist/cli/index.js
+```
+
+The machinery tasks use the repository read-only and create only private
+scratch state under `TMPDIR`; model tasks use a fresh `temp-copy` workspace and
+remove it after the matrix item settles. The machinery suite is the fast
+admission, worker, and receipt contract. The model suite is the live behavioral
+measurement: keep its Artifact v4 output as evidence for the exact target and
+serving configuration that ran, rather than treating one observed model result
+as a universal guarantee. Behavioral facts and their evidence store only
+bounded read counters, not path strings. As with other eval runs, the artifact's
+bounded diagnostic stdout may retain the underlying tool event stream.
+
+### `trackedMetrics`
+
+Eleven numbers plus a reason histogram, each carrying the source it came from. `source` is `ledger` (the per-call ledger folded from the worker's own JSON stream), `receipt` (the sealed run receipt), or `estimated`, and `estimated` is what a missing observation is marked as rather than being silently counted as measured.
+
+| Metric | Usual source |
+| --- | --- |
+| `modelCalls` | ledger |
+| `uncachedPrefillTokens` | ledger, from `promptCache.backend` |
+| `cacheReadTokens` | ledger, from `promptCache.backend`, falling back to pi-ai cache reads |
+| `generatedTokens` | ledger |
+| `reasoningTokens` | receipt; nullable, because absent and zero are different claims |
+| `toolCalls`, `toolErrors` | ledger when present, otherwise receipt |
+| `ttftMsFirstCall` | ledger |
+| `wallClockMs` | receipt |
+| `contextTokensAtEnd` | ledger |
+| `compactions` | ledger |
+| `expectedColdReasons` | ledger, one sourced count per reason |
+
+A dispatched worker's receipt reports `sessionId: null` and writes no session archive, which is why the ledger source exists at all: the runner folds structured usage, backend timing, cache, and monotonic TTFT facts out of the worker's `message_end` events. It keeps no prompt text, no model prose, and no tool-result content in that fold.
+
+### Scenario aggregates
+
+`aggregates` groups verdicts by `scenarioId`, sets `k` to the trial count, and records `passAtK` (any trial passed) and `passPowK` (every trial passed). Each tracked numeric metric reports observation, measured, and unmeasured counts, mean, min, max, nearest-rank p90, population variance, standard deviation, and the set of sources observed. A metric with no observation keeps every numeric statistic `null`; it never becomes zero. At `k: 1`, variance and standard deviation are zero only when the value was actually measured.
+
+### Behavioral multi-metric results
+
+A result with a `clio.eval.behavior.v1` verdict also carries the additive
+`clio.eval.behavior.metrics.v1` projection. The projection binds the scenario
+to its role and target/model envelope and records one `number | null`
+observation for each closed metric. The source travels beside every value:
+
+| Family | Metric | Direction | Gate | Source |
+|---|---|---|---|---|
+| correctness | `correctness.taskSolved` | higher | hard | grader |
+| safety | `safety.violations` | lower | hard | behavioral label |
+| behavior | `behavior.labelViolations` | lower | informational | behavioral labels |
+| efficiency | `efficiency.toolCalls` | lower | informational | terminal tool events |
+| exploration | `exploration.unnecessaryReads` | lower | informational | read observation counters |
+| delegation | `delegation.quality` | higher | informational | behavioral label |
+| claims | `claims.unsupported` | lower | informational | grader |
+| tokens | `tokens.total` | lower | informational | runner usage stream |
+| latency | `latency.wallMs` | lower | informational | monotonic runner clock |
+| cost | `cost.usd` | lower | informational | sealed receipt |
+
+Label metrics are numeric projections only when the category is `satisfied` or
+`violated`; `unknown` and `unmeasured` remain null. A missing grader, token
+stream, receipt, read observation, or category label likewise remains null.
+The projection therefore records observation coverage without claiming that
+silence was success, safety, or zero cost.
+
+### Behavioral comparisons and variance
+
+`eval compare` reduces the behavioral projection independently for every
+scenario, role, target id, and model id. Each row contains the baseline and
+candidate distributions, coverage, mean delta, variance delta, and two closed
+classifications: `improved`, `regressed`, `unchanged`, or `incomparable` for the
+mean and for variability. Lower variance is the improvement direction for the
+variability classification.
+
+Correctness and safety rows are hard. A measured regression fails the hard
+gate even when pass rate, tokens, latency, or cost improved. Losing a
+correctness or safety measurement that existed in the baseline is also a hard
+failure; a category explicitly unmeasured on both sides stays incomparable but
+does not invent a regression. Other families remain visible informational
+tradeoffs. `--metric` accepts either a behavioral metric or family as well as a
+tracked metric, but filtering displayed rows never filters the hard-gate
+decision.
+
+Comparison output supports `text`, `json`, `md`, and `junit`. All four carry
+the same hard-gate result and closed classifications. JUnit failures represent
+only hard behavioral failures; an informational efficiency or cost regression
+is emitted as testcase output rather than a failed testcase.
+
+### Execution-envelope provenance and comparability
+
+Every newly written behavioral result carries an additive
+`clio.eval.execution-envelope.v1` sibling. Artifact v4,
+`clio.eval.verdict.v1`, and `clio.eval.behavior.metrics.v1` retain their
+existing identities. The envelope records the selected prompt fragment ids,
+authored versions or `unversioned` marker, fragment content hashes, prompt
+composition hash, recipe id/version/fingerprint when a worker recipe applies,
+target, wire model, runtime, thinking level, tool signature, effective
+autonomy, rule-pack and project-policy hashes, bounded project-context
+provenance, and corpus id/version. A machinery-only scenario uses explicit
+nulls for model concepts that did not apply; null is not substituted for a
+fact that was observed.
+
+Suite v2 may declare `matrix.dimensions` from `prompt`, `recipe`, `target`,
+`wireModel`, `runtime`, `thinkingLevel`, `toolSignature`, `autonomy`, `policy`,
+`projectContext`, and `corpus`. Comparison ignores only dimensions declared by
+both artifacts. Any other envelope difference marks every metric row for that
+scenario/role/target incomparable and fails the behavioral gate. A missing
+envelope on only one side is also incomparable. Two older artifacts that both
+predate the sibling remain readable and compare under their existing data.
+
+Text, JSON, Markdown, and JUnit comparison reports carry the same envelope
+mismatch. Text and Markdown also include independent per-scenario and per-role
+baseline/candidate counts for improved, regressed, unchanged, and incomparable
+metric means and variances. When the prompt or recipe identity changes, the
+generated evidence names each affected corpus scenario and role instead of
+hiding it behind an aggregate score.
+
+### Checked behavioral release baseline
+
+The checked deterministic baseline is
+`benchmarks/eval/behavioral-machinery-baseline.json`. The release gate runs all
+26 machinery-only scenarios through the built CLI and compares a stable
+projection of their labels, metrics, and execution envelopes with that file.
+It requires no model, private endpoint, credential, or mutable dataset.
+
+When an intentional prompt, recipe, policy, or expected-behavior change moves
+the evidence, run the same machinery suite first, inspect the failing diff and
+the named affected corpus results, then update explicitly:
+
+```sh
+npm run build
+node benchmarks/eval/check-behavioral-release.mjs --update
+git diff -- benchmarks/eval/behavioral-machinery-baseline.json
+```
+
+The baseline update belongs in the reviewed change that caused it. Do not use
+the update command merely to make a red gate green. The model-required and
+negative-control suites remain manual release evidence because their outputs
+depend on a live target; they are never folded into the deterministic baseline.
+The projection excludes `latency.wallMs` because scheduler timing is not stable
+evidence. Behavioral labels, deterministic metrics, and the execution envelope
+remain checked byte for byte.
+
+### Hard thresholds and informational budgets
+
+Suite and external threshold files keep two separate assertion lists:
+
+```yaml
+thresholds:
+  fail:
+    - metric: task.solved
+      op: eq
+      value: false
+  informational:
+    - metric: cost.usd
+      op: gt
+      value: 0.25
+```
+
+`fail` is the backwards-compatible hard list. A firing or unresolved hard
+assertion makes `eval run` or `eval gate` exit nonzero. `informational` uses the
+same typed predicates and reports every firing budget or missing measurement,
+but never changes the exit status. `eval gate` additionally evaluates the
+baseline-to-candidate correctness and safety hard gate, so a cheaper candidate
+cannot offset a task or safety regression.
+
+### `--trials N`
 
-Task outcome commands declared under `verify.measure` evaluate whether the model solved the workload and record metrics (`task.solved`, `task.exitCode`). A non-zero exit from `verify.measure` is recorded as data and **never fails the evaluation item**. Task solution outcome is a measurement, while only machinery invariant behavior operates as a gate.
+`--trials N` overrides the suite's `matrix.repeats` and asks for an isolated workspace per matrix item. A `local` workspace is converted to a temporary copy immediately before that item runs, so an explicit trial run never mutates the directory it was pointed at; `git` and `temp-copy` workspaces already produce a distinct preparation directory per item. Workspace and state directories are removed on the item's `finally` path, including runner, setup, and copy failures. The trial index rides through to each verdict's `trialIndex`.
+
+### Serving-configuration provenance and drift refusal
+
+`servingConfiguration` records what the numbers were measured against: `targetId`, `runtimeId`, `modelId`, `serverBuild`, `total_slots`, `thinkingLevel`, and `compiledPromptHash`. The build string and slot count are read from the server after the matrix has run while it is still awake, by fetching `/props` and falling back to the model-qualified slots query when `/props` exposes no `total_slots`. The prompt hash is the receipt's static composition hash, so a prompt change is visible as a configuration change rather than as a mysterious metric shift.
+
+`eval compare` prints both configurations and refuses outright when they differ:
+
+```text
+serving configuration drift; pass --allow-config-drift to compare these runs
+baseline serving: target=mini runtime=llamacpp model=... server_build=b226-2115b73d8 total_slots=1 thinking=off compiled_prompt_hash=...
+candidate serving: ...
+```
+
+`--allow-config-drift` proceeds and labels the comparison `config drift: allowed`. There is a second refusal that has no override: a metric whose baseline distribution contains an `estimated` observation and whose candidate does not, or the reverse, raises `EvalTrackedMetricSourceMismatchError` rather than printing a delta, because subtracting a measurement from an estimate produces a number that looks like evidence and is not. `--metric <name>` filters tracked or behavioral rows, accepts `expectedColdReasons`, a specific `expectedColdReasons.<reason>`, a behavioral family, or a behavioral metric, and errors when the name matches nothing.
+
+---
+
+## Task Outcome Measurement (`verify.measure`)
 
+Task outcome commands declared under `verify.measure` are the code grader for whether the model solved the workload and record metrics (`task.solved`, `task.exitCode`). A non-zero exit fails the final result and is named on its verdict as `reason: grader_failed`, while `machinery` remains `ok` when the runner and machinery verifiers succeeded. This keeps the artifact's `pass`, verdict outcome, scenario aggregates, and summary on one pass decision without misreporting a grader failure as broken machinery.

package/docs/evals-internal.md

@@ -1,7 +1,7 @@
 # Internal Eval Suites
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive blueprint is available at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) (Version: 0.3.8).
+> **Interactive Spec Available:** An interactive blueprint is available at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) (Version: 0.3.9).
 
 Private suites should live outside this repository. Keep datasets, prompts,
 live fleet coordinates, calibration outputs, and raw run artifacts in a private
@@ -19,6 +19,77 @@ data directory. Product eval artifacts and external benchmark campaigns are
 separate: public benchmark adapters live under `benchmarks/community/` and do
 not use the eval runner.
 
+The public behavioral corpus is the deliberate exception to the otherwise
+private Suite v2 data policy. Its reviewable, synthetic suites live under
+`benchmarks/eval/`: a model-free positive/adversarial authority pair for every
+built-in worker recipe, four tiny main-agent model scenarios covering all eight
+behavioral categories with event- and grader-derived facts, and an intentional
+decoy negative control. The model-free driver uses the shipped recipe catalog,
+real dispatch admission, scripted workers, and sealed receipts rather than
+frontmatter inspection. See
+[eval-runner.md](eval-runner.md#public-built-in-behavioral-corpus) for the
+focused commands. Private prompts, calibration cases, fleet coordinates, and
+campaign artifacts still belong outside this repository and must not be copied
+into the public corpus.
+
+## Running a private suite as a measurement
+
+A private suite is usually run to answer whether a harness change moved
+something, which makes it a measurement rather than a pass or fail. Three
+mechanics matter for that, all documented in full in
+[eval-runner.md](eval-runner.md#the-verdict-envelope).
+
+Run repeated trials with `--trials N` rather than by editing `matrix.repeats`.
+The flag overrides the suite's repeat count and asks for an isolated workspace
+per matrix item, so a `local` workspace is copied for the run instead of being
+mutated across trials. Each result's verdict carries its `trialIndex`, and the
+artifact's `aggregates` reduce them per scenario: `k`, `passAtK` (any trial
+passed), `passPowK` (every trial passed), and a mean and nearest-rank p90 for
+every tracked metric. A single trial produces a `k: 1` aggregate whose mean and
+p90 are the same observed value, which is a fact to state in a report rather
+than a distribution to reason about.
+
+Read the tracked metrics with their sources attached. A private suite on a
+local target is measuring prefill economics as much as correctness, so
+`uncachedPrefillTokens`, `cacheReadTokens`, `ttftMsFirstCall`, and the
+`expectedColdReasons` histogram are the interesting columns, and each one says
+whether it came from the ledger, from the receipt, or was `estimated`. A metric
+marked `estimated` on one side of a comparison and measured on the other is
+refused rather than differenced.
+
+Behavioral suites add a second projection beside those tracked performance
+metrics. Compare it per scenario, role, and target/model envelope rather than
+reducing unlike roles into one pass rate. Correctness and safety are hard
+regression gates; tool efficiency, unnecessary exploration, delegation
+quality, unsupported claims, tokens, latency, and receipt cost remain separate
+families with their own measured coverage and repeat variance. A missing value
+is null and makes that row incomparable, never zero.
+
+Put release-blocking assertions under `thresholds.fail` and non-blocking spend
+or latency budgets under `thresholds.informational`. Informational findings are
+printed in every gate run but do not change its exit status. Do not put a cost
+budget in the hard list to compensate for weak correctness, and do not turn a
+correctness rule into an informational budget; the comparison gate evaluates
+correctness and safety before either kind of operator-authored threshold.
+
+Record the serving configuration or the comparison is not one. The artifact
+captures `targetId`, `runtimeId`, `modelId`, `serverBuild`, `total_slots`,
+`thinkingLevel`, and `compiledPromptHash`, read from the server after the matrix
+has run while it is still awake. `eval compare` refuses two artifacts whose
+configurations differ unless `--allow-config-drift` is passed, and prints both
+either way. Treat that refusal as the useful behavior it is: a private suite
+compared across a server restart that changed a flag, a quantization, or the
+thinking level is measuring the server, not the change under test.
+
+The verdict envelope keeps its original `behavioral: null` field for compatibility.
+A suite that declares a versioned behavioral scenario records the result as a
+separate `clio.eval.behavior.v1` document on the Artifact v4 result, cross-linked
+to the unchanged verdict identity. Its labels come only from bounded transcript,
+tool, receipt, or grader facts, never from an ungrounded judge paragraph. A run whose harness broke records
+`machinery: "infrastructure_failure"`, which the parser refuses to pair with a
+`pass`, so a private suite cannot report a passing rate that includes runs
+nothing measured.
+
 ## Context Regression Seed
 
 ```yaml
@@ -267,4 +338,3 @@ thresholds:
       op: gt
       value: 0
 ```
-

package/docs/evidence-and-memory.md

@@ -1,7 +1,7 @@
 # Evidence Corpus and Long-Term Memory
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive memory lifecycle dashboard and simulator is located at [docs/html/memory_blueprint.html](html/memory_blueprint.html) (Version: 0.3.8). Use it to design, validate, and simulate memory proposals, approval loops, pruning rules, and token budgets.
+> **Interactive Spec Available:** An interactive memory lifecycle dashboard and simulator is located at [docs/html/memory_blueprint.html](html/memory_blueprint.html) (Version: 0.3.9). Use it to design, validate, and simulate memory proposals, approval loops, pruning rules, and token budgets.
 
 Clio Coder treats run claims and agent lessons as structured artifacts to support reproducibility and scientific provenance. In evaluations such as [SWE-bench](https://www.swebench.com), capturing granular execution evidence is essential for validating agent claims. Evidence corpora are deterministic directories built from run ledgers, receipts, sessions, audits, and eval artifacts. In v0.3.7, forensic evidence auto-builds on dispatch run completion: when a run finalizes, the observability domain automatically compiles the evidence bundle under `<dataDir>/evidence/run-<id>/` and updates a compact sidecar index row in `<stateDir>/evidence-index.json`. Long-term memory records are local, evidence-linked, and only injected after explicit approval. Use the TUI [`/view`](observability.md) command for interactive inspection of receipts, dispatch output, durable tool output, compaction summaries, and session accountability before building or citing evidence.
 
@@ -68,9 +68,9 @@ Eval evidence adds `eval-result.json` and uses empty receipt/protected-artifact
 | `audit-linked.jsonl` | Audit rows linked to run/session context when available. |
 | `receipt.json` | Receipt bundle (`{ version: 1, receipts: [...] }`); only receipts that pass integrity verification contribute verified fields. |
 | `gate-decisions.json` | Integrity-verified review verdicts, compete winner selections, and winner confirmations discovered from linked receipt ids. |
-| `trust-status.json` | Canonical per-run six-axis trust projections derived from authenticated receipts, gate decisions, grounded validation artifacts, and exact finish-contract audit rows. |
+| `trust-status.json` | Canonical per-run six-axis trust projections derived from authenticated receipts, gate decisions, and grounded validation artifacts. |
 | `protected-artifacts.json` | Protected artifact state/events. |
-| `findings.json` / `findings.md` | Structured and readable findings. |
+| `findings.json` / `findings.md` | Structured findings plus a readable report that begins with each linked run's canonical tier, fixed-order summary, and six axes. |
 
 ### Run attribution under concurrency
 
@@ -219,19 +219,20 @@ They do not mutate receipt, gate-decision, evidence-bundle, or session formats.
 | Valid bounded project context, valid none-tier workspace-root record, or valid briefing hash | Context provenance is `recorded`. A `none`-tier run still receives the workspace-root message, so a none-tier block naming exactly `workspace-root` with a well-formed count and hash is `recorded`. Explicit project-context tier `none` with no content and no briefing is `not_applicable`; a missing historical field is `unknown`; a contradictory block (a handbook section under a none policy, a hash with no section, a malformed count) is `invalid`. |
 | Gate decision | An authenticated independent pass or fail maps to `passed` or `failed`. Correlated review maps to `not_independent`. Unauthenticated artifacts map to `unknown`; operator or full-auto confirmation alone is `not_applicable` to independent review. |
 | Receipt autonomy grade | `mediated`, `approximated`, and `bypassed` map to `enforced`, `approximated`, and `bypassed`. A dangerous-bypass flag always normalizes to `bypassed`; a missing historical block is `unknown`. |
-| Finish-contract assessment | `validation_evidence`, `unvalidated_mutation`, `explicit_limitation`, and `no_mutation` map to `evidenced`, `incomplete`, `limited`, and `not_applicable`. A run whose receipt was presented and rejected downgrades `evidenced` to `unknown`: the row still points at its own record, but a rejected receipt authenticates nothing about the run it names. |
-| Malformed audit row identifier | A blank or whitespace-only optional identifier is treated as absent. The row falls back to its derived correlation id or drops out of the trust projection; it never aborts the bundle. |
+| Finish-contract assessment | The assessment remains linked in `audit-linked.jsonl` and contributes its domain findings and tags. It does not override the receipt-derived `completionEvidence` axis on the evidence surface alone. |
+| Malformed audit row identifier | A blank or whitespace-only optional identifier remains linked as audit input and never aborts the bundle. It cannot affect the receipt-derived trust projection. |
 | Bundle without `trust-status.json` | Inspection reports `projection: historical_format` with no canonical run projections. It never reconstructs positive states from older summary tags. |
 
 Receipt inspection, worker output, monitor details, and evidence rebuilding all
 use the same authenticated receipt projection boundary. Evidence rebuilding
-then composes independently authenticated gate decisions and exact
-finish-contract records without changing receipt-owned axes. Findings such as
+then composes independently authenticated gate decisions without changing
+receipt-owned axes. Findings such as
 `no-validation`, `proxy-validation`, `external-approximation`,
 `external-bypass`, `independent-review`, `context-provenance`, and
-`completion-evidence` are selected from the canonical states, so every axis
-reaches `findings.md`, while their detailed domain artifacts remain in the
-receipt, gate, audit, and trace files.
+`completion-evidence` are selected from the canonical states. `findings.md`
+prints the tier, summary, and every axis before those diagnostic records, while
+their detailed domain artifacts remain in the receipt, gate, audit, and trace
+files.
 
 The canonical aggregate is an additive projection for downstream work. Receipt
 integrity remains version 18, evidence bundles remain version 1, gate decisions

package/docs/evolution.md

@@ -1,7 +1,7 @@
 # Evolution and Change Manifests
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive change manifest editor, authority risk assessor, and checklist workspace is located at [docs/html/evolution_blueprint.html](html/evolution_blueprint.html) (Version: 0.3.8).
+> **Interactive Spec Available:** An interactive change manifest editor, authority risk assessor, and checklist workspace is located at [docs/html/evolution_blueprint.html](html/evolution_blueprint.html) (Version: 0.3.9).
 
 Clio Coder uses change manifests to make harness changes reviewable, falsifiable, and rollback-friendly. CLIO stands for Context Layer for Input/Output, named for the Greek muse of history. A manifest is JSON, generated or checked with `clio-coder evolve manifest`, and should describe what changed, why, what evidence supports it, what could regress, how to validate it, and how to roll it back.
 

package/docs/extensions-and-sharing.md

@@ -1,7 +1,7 @@
 # Extensions, Prompt Templates, Skills, and Share Archives
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive dashboard is located at [docs/html/extensions_blueprint.html](html/extensions_blueprint.html) (Version: 0.3.8).
+> **Interactive Spec Available:** An interactive dashboard is located at [docs/html/extensions_blueprint.html](html/extensions_blueprint.html) (Version: 0.3.9).
 
 Clio Coder has lightweight community-oriented resource packaging. Extensions are filesystem bundles that contribute prompts and skills. Share archives are portable JSON files for moving project/user Clio resources between machines or collaborators. Themes are built into the engine and are no longer loaded from extensions.
 
@@ -197,6 +197,8 @@ Required fields are `manifestVersion: 1`, `id`, `version`, and `description`. `n
 
 IDs must be lowercase and may include numbers, dots, underscores, and hyphens; they must start/end alphanumeric.
 
+`compatibility.clio` is optional. When present, it must be a valid SemVer range such as `>=0.3.8`, `^0.3.8`, or `0.3.x`. Installation refuses a package whose range excludes the running Clio version and names the extension, its declared range, and that running version. Clio repeats the check whenever it loads installed extensions, so a package that becomes incompatible after a Clio version change stays visible in `extensions list` with its diagnostic but contributes no resources. An incompatible project package does not hide a compatible user package with the same ID. A manifest without `compatibility.clio` keeps the existing unrestricted behavior.
+
 ---
 
 ## Extension CLI

package/docs/fleet-dispatch.md

@@ -1,6 +1,6 @@
 # Fleet Dispatch
 
-> **Interactive Spec Available:** An interactive fleet node topology planner, scout router, receipt verifier, and failure taxonomy simulator is located at [docs/html/fleet_dispatch_blueprint.html](html/fleet_dispatch_blueprint.html) (Version: 0.3.8).
+> **Interactive Spec Available:** An interactive fleet node topology planner, scout router, receipt verifier, and failure taxonomy simulator is located at [docs/html/fleet_dispatch_blueprint.html](html/fleet_dispatch_blueprint.html) (Version: 0.3.9).
 
 Clio Coder dispatches bounded worker agents. With a fleet configured, those
 workers run on remote machines over SSH while the orchestrator keeps every
@@ -531,14 +531,14 @@ The grammar for declared write boundary entries requires repository-relative POS
 Write boundary enforcement is detect-and-rollback, never OS or filesystem sandboxing. A step runs with whatever filesystem permissions its underlying execution environment possesses. Upon step completion, the orchestrator inspects the working tree to verify compliance:
 1. Snapshot baseline: Before a step executes, the orchestrator captures a snapshot (`captureWorkspaceSnapshot`) recording the baseline git HEAD commit and existing dirty path content tokens.
 2. Workspace diffing: After step completion, the orchestrator runs git status inspection (`diffWorkspace`) to identify changed paths relative to the snapshot baseline commit.
-3. Authorship attribution: A changed path outside the allowlist is blamed on the window only when it intersects what the window's own runs recorded writing. That record is the run's tool-call stream, folded by the same recorder that grounds a sealed mutation report and read back through `DispatchContract.observedRunWrites`. A change that no run in the window recorded is an unattributed concurrent change: it is listed under `unattributed` in the verdict and reported to the operator, and it is never rolled back. This is what keeps a file an operator edited while a fleet ran from being overwritten with its committed version.
-4. Open records: A run's recorded write set is read as a closed list only when the run could not have written outside it. A window whose steps do not all offer a closed record falls back to blaming every change outside the allowlist, which is the behavior that predates attribution. Three things open a record: a `kind: code` step, whose registered command publishes no tool events; a run whose tool telemetry coverage is not `complete`, such as one on a subprocess runtime; and a run that made a successful call to a tool able to mutate a path its own arguments do not name. That last set is derived from the tool surface rather than authored, as every registered tool outside the `read` and `write` action classes, which today is `bash`, `verify`, `dispatch`, and `steer`, plus any dynamic or MCP tool whose schema this process cannot read. The `git` tool is a closed status, diff, and log surface and stays enumerable. The verdict records `attributionComplete: false` and the operator-facing detail says the blame was inferred from the checkout rather than from the step's own record.
+3. Authorship attribution: A changed path outside the allowlist is blamed on the window only when it intersects what the window's own runs recorded writing. That record is the run's tool-call stream, folded by the same recorder that grounds a sealed mutation report and read back through `DispatchContract.observedRunWriteAttribution`. A change that no run in the window recorded is an unattributed concurrent change: it is listed under `unattributed` in the verdict and reported to the operator, and it is never rolled back. This is what keeps a file an operator edited while a fleet ran from being overwritten with its committed version.
+4. Open records: A run's recorded write set is read as a closed list only when the run could not have written outside it. A window whose steps do not all offer a closed record falls back to blaming every change outside the allowlist, which is the behavior that predates attribution. Three things open a record: a `kind: code` step, whose registered command publishes no tool events; a run whose tool telemetry coverage is not `complete`, such as one on a subprocess runtime; and a run that made a successful call to a tool able to mutate a path its own arguments do not name. That last set is derived from the tool surface rather than authored, as every registered tool outside the `read` and `write` action classes, which today is `bash`, `verify`, `dispatch`, and `steer`, plus any dynamic or MCP tool whose schema this process cannot read. The `git` tool is a closed status, diff, and log surface and stays enumerable. When a successful opaque call opens the record, the live Alt+W card immediately names the tool and says that its arguments cannot enumerate every path it may write. The durable verdict records `attributionComplete: false` and an `attributionDowngrades` entry with the reason, tool name, tool call id, run id, and step id. The after-the-fact write-boundary message repeats that cause before explaining why the whole outside diff was blamed on the window.
 5. Rollback execution: Attributed unauthorized changes are automatically rolled back (`rollbackPath`).
 6. Content source: Rollback restores content strictly from what git already has in the pinned baseline commit (`snapshot.head`). If a path was already dirty when the step snapshot was captured, its prior content is not stored in git, so in-place restoration cannot be guaranteed. The working tree is left as the step made it, and the status settles as `rollback-incomplete`.
 7. Violation handling: Any attributed unauthorized change fails the step with the typed reason `writes_boundary_violation`.
 8. Window attribution: Enforcement evaluates scheduling windows (`wave-<n>` or `revalidate-<stepId>-<n>`). A wave window cannot combine steps with overlapping declared boundaries or multiple concurrent step writers, ensuring single-step attribution.
 9. Ignored paths and state subtraction: Enforcement evaluates paths reported by git status, which never lists a git-ignored path. A declared `writes` entry the repository ignores is therefore refused before anything runs, by `fleet validate`, by `fleet run` preflight, and by the `/fleet run` preview, with a diagnostic naming the entry and the ignoring rule (for example `'work/' is ignored by .gitignore:1:work/`). Silently certifying such a window as clean is not an option, because nothing about it was observed. The Clio state directory (`.clio-coder/` or `clioStateDir()`) is subtracted from status checks so orchestrator receipts, code step log artifacts, and boundary verdicts do not trigger false violations.
-10. Durable records: Verdicts are serialized as JSON records at `write-boundaries/<rootId>/<window>.json` under the Clio state directory, carrying the baseline HEAD commit, checked paths, violations, unattributed concurrent changes, the attribution completeness flag, rollback actions, status, and SHA-256 digest.
+10. Durable records: Verdicts are serialized as JSON records at `write-boundaries/<rootId>/<window>.json` under the Clio state directory, carrying the baseline HEAD commit, checked paths, violations, unattributed concurrent changes, the attribution completeness flag and downgrade causes, rollback actions, status, and SHA-256 digest.
 
 ### Bounded check/repair loops
 

package/docs/installation-and-lifecycle.md

@@ -3,7 +3,7 @@
 Clio Coder is designed to be self-contained and platform-compliant. This document outlines the default directory paths, file purposes, permission levels, and lifecycle commands (`install`, `reset`, `upgrade`, and `uninstall`). Clio Coder installs from npm as `@iowarp/clio-coder` (`npm install -g @iowarp/clio-coder`, published since v0.3.0) or from a source checkout with a deterministic local symlink; the CLI classifies both install kinds and `clio-coder upgrade` handles each.
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive dashboard with a path simulator and visual flowcharts is located at [docs/html/lifecycle_blueprint.html](html/lifecycle_blueprint.html) (Version: 0.3.8). You can open it directly in any web browser to view details dynamically.
+> **Interactive Spec Available:** An interactive dashboard with a path simulator and visual flowcharts is located at [docs/html/lifecycle_blueprint.html](html/lifecycle_blueprint.html) (Version: 0.3.9). You can open it directly in any web browser to view details dynamically.
 
 ---
 

package/docs/middleware-and-components.md

@@ -1,7 +1,7 @@
 # Middleware and Component Registry
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive dashboard with an interactive component scanner and a dynamic hook-and-effect pipeline is located at [docs/html/middleware_blueprint.html](html/middleware_blueprint.html) (Version: 0.3.8).
+> **Interactive Spec Available:** An interactive dashboard with an interactive component scanner and a dynamic hook-and-effect pipeline is located at [docs/html/middleware_blueprint.html](html/middleware_blueprint.html) (Version: 0.3.9).
 
 Clio Coder has two related but separate surfaces:
 

package/docs/model-catalog.md

@@ -1,7 +1,7 @@
 # Model Catalog, Runtime Refresh, and Field Notes
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive dashboard mapping capabilities, probe discovery, and target resolution is located at [docs/html/models_blueprint.html](html/models_blueprint.html) (Version: 0.3.8).
+> **Interactive Spec Available:** An interactive dashboard mapping capabilities, probe discovery, and target resolution is located at [docs/html/models_blueprint.html](html/models_blueprint.html) (Version: 0.3.9).
 
 Clio Coder treats a selectable model as the intersection of three sources: