@iowarp/clio-coder 0.3.7 → 0.3.9

package/docs/documentation-coverage.md

@@ -11,7 +11,7 @@ This matrix maps every top-level directory in `src/` and every domain directory
 | `src/engine/` (Core) | Engine turn loop, prompt priming, streaming message adapters, turn execution | [architecture.md](architecture.md), [context-engine.md](context-engine.md) | `documented` | Documented across architecture and context engine guides. |
 | `src/engine/acp/` | ACP protocol server, transport adapters, tool mediators, permission forwarding, error taxonomy | [acp.md](acp.md) | `documented` | Dedicated ACP specification covering server wiring, permission mediation, timeouts, error taxonomy, and security boundaries. |
 | `src/entry/` | Application bootstrapping, CLI router, interactive loop entry point | [architecture.md](architecture.md), [installation-and-lifecycle.md](installation-and-lifecycle.md) | `documented` | Documented in architecture compilation boundaries and lifecycle guides. |
-| `src/interactive/` | TUI architecture, screens, overlays, keybindings, panels, theme tokens, width matrices | [tui-design.md](tui-design.md), [commands-and-modes.md](commands-and-modes.md) | `documented` | Fully documented in TUI design specification and commands reference. |
+| `src/interactive/` | TUI architecture, screens, overlays, keybindings, panels, theme tokens, width matrices, prompt pre-warm rounds and their gating, expected-cold reason stamping | [tui-design.md](tui-design.md), [commands-and-modes.md](commands-and-modes.md), [context-engine.md](context-engine.md) | `documented` | Fully documented in TUI design specification and commands reference; the pre-warm and the cache-honesty surfaces `/context` renders are in the context engine reference. |
 | `src/tools/` | 20 built-in tools across 7 planes, registry, policy engine bindings, observation envelope bounds | [tool-usage.md](tool-usage.md), [prompt-envelope-and-tools.md](prompt-envelope-and-tools.md) | `documented` | Comprehensive 20-tool reference with schemas, examples, and envelope size constraints. |
 | `src/utils/` | Image manipulation, photon operations, git execution utilities | [architecture.md](architecture.md), [tool-usage.md](tool-usage.md) | `documented` | Utility helpers documented within tool usage and architectural boundaries. |
 | `src/worker/` | Worker subprocess lifecycle, NDJSON transport, heartbeat timers, control lane demuxing, spec contracts | [worker-dispatch-mechanics.md](worker-dispatch-mechanics.md) | `documented` | Complete reference for NDJSON socket protocols, watchdog timers, and exit status mapping. |
@@ -20,7 +20,7 @@ This matrix maps every top-level directory in `src/` and every domain directory
 | `src/domains/config/` | Configuration contracts, file watcher, keybinding definitions, setting classifiers | [configuration-and-targets.md](configuration-and-targets.md), [commands-and-modes.md](commands-and-modes.md) | `documented` | Documented in configuration targets and command/keybinding reference. |
 | `src/domains/context/` | `CLIO-CODER.md` bootstrap, codewiki generation, prompt context assembly, project rules, non-destructive working-set eviction (`age-horizon` and `structural-v1` policies, protection predicates, path index, byte-stable markers, recall by ref) | [context-engine.md](context-engine.md), [context-working-set.md](context-working-set.md) | `documented` | Context window, token accounting, and the three compaction mechanisms in the engine reference; the working-set layer has its own guide covering the vocabulary, both ledger record kinds and format v4, the marker contract, both policies with their rule order, recall semantics, and the operator surfaces. |
 | `src/domains/dispatch/` | Fleet orchestration, assignment store, batch tracker, admission, route planner, receipt integrity v16 | [fleet-dispatch.md](fleet-dispatch.md), [dispatch-architecture-rationale.md](dispatch-architecture-rationale.md), [worker-dispatch-mechanics.md](worker-dispatch-mechanics.md) | `documented` | Multi-node fleet dispatch, admission invariants, and receipt verification fully documented. |
-| `src/domains/eval/` | Suite v2 YAML schema, eval runner, metrics, reporters, workspace sandboxing | [eval-runner.md](eval-runner.md), [evals-internal.md](evals-internal.md) | `documented` | Product evals are documented independently from external benchmarks. |
+| `src/domains/eval/` | Suite v2 YAML schema, eval runner, `clio.eval.verdict.v1` envelope and its Suite v2 adapter, tracked metrics and scenario aggregates, serving-configuration provenance, reporters, workspace sandboxing | [eval-runner.md](eval-runner.md), [evals-internal.md](evals-internal.md) | `documented` | Product evals are documented independently from external benchmarks. The verdict envelope, `trackedMetrics` and their sources, `--trials`, and the config-drift and estimated-versus-measured refusals are in the runner reference. |
 | `src/domains/evidence/` | Evidence bundles, findings taxonomy, provenance store, failure attribution | [evidence-and-memory.md](evidence-and-memory.md) | `documented` | Documented in evidence directory structures and memory retrieval guide. |
 | `src/domains/evolution/` | Falsifiable Change Manifest JSON templates and `clio-coder evolve` self-edit gates | [evolution.md](evolution.md) | `documented` | Documented in evolution manifest reference and mutation validation rules. |
 | `src/domains/extensions/` | Extension manifest schemas, resource roots, portable share archives | [extensions-and-sharing.md](extensions-and-sharing.md) | `documented` | Documented in extensions and sharing guide. |
@@ -29,7 +29,7 @@ This matrix maps every top-level directory in `src/` and every domain directory
 | `src/domains/middleware/` | Middleware hooks (`turn_start`, `tool_call`, `tool_result`, `turn_end`), reminders, budgets | [middleware-and-components.md](middleware-and-components.md) | `documented` | Documented in middleware hooks and active component snapshot guide. |
 | `src/domains/observability/` | Trace store (`node:sqlite` WAL mirror), metrics, cost accounting, evidence index | [trace-store.md](trace-store.md), [observability.md](observability.md) | `documented` | Database schema, rowid cursor queries, and receipt provenance documented. |
 | `src/domains/prompts/` | Prompt compiler, fragment loaders, static cache stability, memory intervention injection | [prompt-envelope-and-tools.md](prompt-envelope-and-tools.md) | `documented` | Documented in prompt envelope and tool delivery guide. |
-| `src/domains/providers/` | Runtime adapters, capability probes, model catalog, thinking control, ALCF OAuth | [configuration-and-targets.md](configuration-and-targets.md), [model-catalog.md](model-catalog.md), [provider-adapter-cookbook.md](provider-adapter-cookbook.md), [alcf-provider.md](alcf-provider.md) | `documented` | Complete provider adapter contracts, model catalogs, and ALCF Globus targets documented. |
+| `src/domains/providers/` | Runtime adapters, capability probes, model catalog and its per-family `measuredUnder` provenance, canonical endpoint keys and per-endpoint request-slot capacity, thinking control, ALCF OAuth | [configuration-and-targets.md](configuration-and-targets.md), [model-catalog.md](model-catalog.md), [provider-adapter-cookbook.md](provider-adapter-cookbook.md), [alcf-provider.md](alcf-provider.md) | `documented` | Complete provider adapter contracts, model catalogs, and ALCF Globus targets documented. |
 | `src/domains/resources/` | Skill package discovery, marketplace index resolution, prompt resources | [skills-marketplace.md](skills-marketplace.md), [extensions-and-sharing.md](extensions-and-sharing.md) | `documented` | Skills marketplace, publishing flows, and resource managers documented. |
 | `src/domains/safety/` | Policy engine, action classifiers, damage-control rules, path policies, finish contract, audit log | [safety-model.md](safety-model.md), [scientific-validation.md](scientific-validation.md) | `documented` | Policy evaluation order, 10-step sequence, write containment, and finish contract documented. |
 | `src/domains/scheduling/` | Capacity lease acquisition, heartbeats, expiry, cross-process locks, cluster scheduling | [capacity-and-scheduling.md](capacity-and-scheduling.md), [fleet-dispatch.md](fleet-dispatch.md) | `documented` | Dedicated capacity leasing, heartbeat TTL, and cross-process lock reference. |

package/docs/documentation-guide.md

@@ -1,7 +1,7 @@
 # Documentation Standards and Codebase Alignment
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive documentation link linter, phrasing/claim evaluator, and alignment portal is located at [docs/html/documentation_blueprint.html](html/documentation_blueprint.html) (Version: 0.3.7).
+> **Interactive Spec Available:** An interactive documentation link linter, phrasing/claim evaluator, and alignment portal is located at [docs/html/documentation_blueprint.html](html/documentation_blueprint.html) (Version: 0.3.9).
 
 Clio Coder is an experimental community alpha. Documentation should help contributors and early users work from the source of truth without overstating maturity. When docs drift, prefer the current source and tests over older prose or aspirational roadmap notes.
 

package/docs/environment-variables.md

@@ -47,6 +47,8 @@ Durable values live in the `guardrails:` section of settings.yaml (see [configur
 | `CLIO_CODER_REDUCE_MOTION` | off | `1` makes smooth-streaming `auto` use the immediate coalescer. Explicit `on` remains an operator request, while stdout backpressure still pauses frame production. |
 | `CLIO_CODER_SCREEN_READER` | off | `1` makes smooth-streaming `auto` use the immediate coalescer so a screen reader receives the existing low-motion update behavior. |
 | `CLIO_CODER_INSTANT_SHELL` | on | `0` disables the single-owner Stage 0 interactive shell for immediate rollback. Unset or `1` mounts one terminal/editor owner before service hydration; ACP, headless, ordinary non-TTY, and subcommand paths never mount it. An explicit `CLIO_CODER_INTERACTIVE=1` keeps its force-interactive non-TTY behavior. |
+| `CLIO_CODER_TRACE_RETENTION_DAYS` | 30 | Maximum age in days for terminal rows in the rebuildable SQLite trace mirror. The value is an integer of at least 1 (`src/domains/observability/trace-store.ts`). |
+| `CLIO_CODER_TRACE_MAX_BYTES` | 134217728 | Maximum allocated size for the SQLite trace mirror before the oldest terminal runs are pruned. The value is an integer of at least 1,048,576 (`src/domains/observability/trace-store.ts`). |
 
 ## Directory and install layout
 

package/docs/eval-runner.md

@@ -1,7 +1,7 @@
 # Clio Coder Local Evaluation Runner
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.3.7).
+> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.3.9).
 
 The local evaluation runner executes repository-local YAML task suites as deterministic subprocess checks. It is useful for comparing harness changes, prompts, tools, or local workflows.
 
@@ -15,10 +15,10 @@ The CLI commands under `clio-coder eval` support running, validating, reporting,
 
 ```bash
 clio-coder eval validate --suite <suite.yaml>
-clio-coder eval run --suite <suite.yaml> [--target <id>] [--model <id>] [--out <path>] [--clio-coder-entry <path>]
+clio-coder eval run --suite <suite.yaml> [--trials <n>] [--target <id>] [--model <id>] [--out <path>] [--clio-coder-entry <path>]
 clio-coder eval run --task-file <tasks.yaml> [--repeat <n>] [--out <path>] [--clio-coder-entry <path>]
 clio-coder eval report <evalId> --format text|json|md|swe-jsonl|junit
-clio-coder eval compare <baselineEvalId> <candidateEvalId>
+clio-coder eval compare <baselineEvalId> <candidateEvalId> [--metric <name>] [--format text|json|md|junit] [--allow-config-drift]
 clio-coder eval gate <candidateEvalId> --baseline <baselineEvalId> [--thresholds <file>]
 ```
 
@@ -32,7 +32,7 @@ clio-coder eval gate <candidateEvalId> --baseline <baselineEvalId> [--thresholds
   * `swe-jsonl`: Standardized JSONL format representing task runs (e.g. for SWE-bench comparisons).
   * `junit`: XML report for CI/CD integration.
 * **`compare`**: Compares two evaluation artifacts (baseline and candidate) by matching tasks.
-* **`gate`**: Compares candidate metrics against baseline or absolute thresholds, exiting non-zero if assertions fail (useful for PR gating).
+* **`gate`**: Compares candidate metrics against baseline and absolute thresholds. Correctness and safety regressions fail independently of informational budgets.
 
 Exit codes:
 
@@ -41,8 +41,8 @@ Exit codes:
 | `eval validate` | `0` when validation passes | `2` for validation issues |
 | `eval run` | `0` when all task repetitions pass | `1` when any task fails, `2` for invalid configs |
 | `eval report` | `0` when artifact loads | `1` if artifact cannot be read, `2` for invalid ID |
-| `eval compare` | `0` when both artifacts load and compare succeeds | `1` if artifacts cannot be read, `2` for invalid ID |
-| `eval gate` | `0` when all threshold assertions pass | `1` if assertions fail, `2` for config/invalid ID errors |
+| `eval compare` | `0` when both artifacts compare and the behavioral hard gate passes | `1` for a hard regression or unreadable artifact, `2` for invalid ID |
+| `eval gate` | `0` when correctness, safety, and hard threshold assertions pass | `1` for any hard failure, `2` for config/invalid ID errors |
 
 ---
 
@@ -103,10 +103,11 @@ tasks:
 | --- | --- | --- |
 | `version` | - | Must equal `2`. |
 | `suite` | `id`, `title`, `visibility`, `description` | Metadata identifying the evaluation suite. |
-| `matrix` | `targets[]`, `repeats` | Matrix of execution targets (specifying model and thinking flags) and the repetition count. |
+| `matrix` | `targets[]`, `repeats`, `dimensions[]` | Matrix of execution targets, repetition count, and the execution-envelope fields intentionally varied by the suite. |
 | `workspace` | `kind`, `path`, `url`, `commit`, `checkout`, `excludes` | Workspace strategy: `local` (run in-place), `git` (clone from URL), or `temp-copy` (isolated copy of a directory). |
 | `runner` | `kind`, `prompt`, `command`, `commands`, `args`, `timeoutMs` | Runner type: `clio-run` (starts Clio agent loop), `context-index` (runs indexer), `context-init` (initializes context), `external-command` (spawns subprocess). |
-| `verify` | `commands`, `assertions`, `forbidPaths` | Validation steps: shell commands, metric assertions (e.g. `op: lt` for max token counts), and files/directories that must not be created or modified (`forbidPaths`). |
+| `behavioral` | `schema`, `corpus`, `execution`, `expectedBehavior`, `forbiddenBehavior`, `judge` | Optional `clio.eval.scenario.v1` behavioral contract. Rules name a closed category and a typed predicate over transcript, tool, receipt, or grader facts. |
+| `verify` | `commands`, `measure`, `assertions`, `forbidPaths` | Validation steps: shell commands, a task-outcome grader, metric assertions (e.g. `op: lt` for max token counts), and files/directories that must not be created or modified (`forbidPaths`). |
 | `metrics` | `collect` | List of metric names to compile for the evaluation runs. |
 
 ---
@@ -114,7 +115,7 @@ tasks:
 ## Workspace Kinds
 * **`local`**: Executes the task directly in the specified local path.
 * **`git`**: Clones the repository from `url`, checks out the specified `commit` or `checkout` ref, and runs there.
-* **`temp-copy`**: Copies the directory at `path` to a temporary workspace location before running. This prevents side-effects from polluting other task runs.
+* **`temp-copy`**: Copies the directory at `path` to a temporary workspace location immediately before the matrix item runs and removes it afterward. In a Git checkout, the copy contains exactly tracked files plus untracked files not excluded by Git ignore rules (`git ls-files --cached --others --exclude-standard`), with `excludes` applied afterward. Outside Git it retains the recursive directory copy. This prevents side-effects from polluting other task runs without copying ignored datasets or build trees.
 
 ---
 
@@ -194,12 +195,262 @@ export interface EvalArtifactV4 {
   matrix: { target: string; model: string | null; thinking: string | null };
   summary: EvalArtifactSummaryV4;
   results: EvalArtifactResultV4[];
+  servingConfiguration?: EvalServingConfigurationV1;
+  aggregates?: EvalScenarioAggregateV1[];
 }
 ```
 
+`servingConfiguration` and `aggregates` are additive. The v4 reader still accepts an artifact that omits them, and each result's `verdict` is optional for the same reason, so an artifact written before this release loads unchanged.
+
 ---
 
-## Task Outcome Measurement (`verify.measure`)
+## The verdict envelope
+
+Every result carries a strictly parsed `clio.eval.verdict.v1` envelope (`src/domains/eval/schema/verdict.ts`). Suite v2 results are adapted into it at one explicit boundary (`src/domains/eval/schema/adapter.ts`) rather than by widening the artifact version, because the envelope carries no information a v4 artifact cannot hold.
+
+```json
+{
+  "schema": "clio.eval.verdict.v1",
+  "scenarioId": "latency-nonnegative",
+  "trialIndex": 0,
+  "outcome": "pass",
+  "machinery": "ok",
+  "reason": null,
+  "trackedMetrics": { "...": "see below" },
+  "behavioral": null,
+  "evidence": {
+    "assignmentId": "qcy5rfopdrfw",
+    "terminalReceiptDigest": "d85a3ad4f8ae...",
+    "graderExitCode": 0
+  }
+}
+```
+
+The envelope is fail-closed by construction. `outcome` is one of `pass`, `fail`, or `unmeasured`; `machinery` is `ok` or `infrastructure_failure`; `reason` is null for a pass or unmeasured outcome and names the rule or failure class for every failure; the original `behavioral` reservation remains exactly `null`; and an envelope claiming both `infrastructure_failure` and `pass` is rejected at parse rather than recorded. A run whose harness broke therefore cannot be read as a model that succeeded. Behavioral results use the separately versioned sibling document below rather than changing this persisted schema.
+
+### Behavioral scenario and verdict documents
+
+Behavioral evaluation is additive and does not change the persisted `clio.eval.verdict.v1` reader. A Suite v2 task may declare a `clio.eval.scenario.v1` block, and its Artifact v4 result then carries a sibling `clio.eval.behavior.v1` document whose `verdictRef` names the verdict schema, scenario id, and trial index. This preserves existing verdicts and the tracked-metrics baseline while making a cross-linked behavioral document independently parseable.
+
+The closed categories are `tool_choice`, `exploration`, `delegation`, `safety_comprehension`, `claim_grounding`, `denied_tool_recovery`, `completion_behavior`, and `task_correctness`. Each category result is exactly one of `satisfied`, `violated`, `unknown`, or `unmeasured`. The document outcome is `pass`, `behavioral_failure`, `unknown`, `unmeasured`, or `infrastructure_failure`; missing facts are never invented as successes, and an infrastructure failure cannot become a behavioral pass.
+
+Expected and forbidden rules contain typed predicates over facts sourced from `transcript`, `tool`, `receipt`, or `grader`. Facts cite a locator, SHA-256 digest, and optional bounded excerpt. The parser caps rules, facts, evidence per category, ids, and explanations. Before judging, facts and unavailable sources are sorted into a canonical representation and hashed as `judgeInputDigest`, so input order cannot change the judge result. Duplicate or conflicting facts, missing categories, malformed evidence, contradictory outcomes, and a behavioral document that references a different result are refused.
+
+Suite execution adapts scalar run metrics into these observable facts at the Suite v2 to Artifact v4 boundary. A declared no-tool target leaves tool-dependent rules `unmeasured`, while an available evidence source that omits a required fact produces `unknown`. Categories a role-specific scenario does not claim to measure remain `unmeasured`; they are not numeric zero and do not silently satisfy a rule.
+
+### Public built-in behavioral corpus
+
+The repository ships corpus `public-built-in-behavior` version `1.0.0` under
+`benchmarks/eval/`. It contains no private prompts, endpoints, credentials, or
+mutable external dataset:
+
+- `behavioral-machinery.yaml` provides one positive and one adversarial
+  machinery-only check for each of the 13 shipped built-in worker recipes. Its
+  deterministic driver loads the production recipe catalog, admits a real
+  dispatch through the production gate, runs a scripted worker, and verifies
+  the sealed receipt and result-contract outcome. The 26 scenarios require no
+  model; they do not infer behavior by grepping recipe frontmatter.
+- `behavioral-model.yaml` provides four isolated main-agent scenarios on the
+  `mini` target: a focused edit, adversarial scope control, required
+  delegation, and recovery after Bash is denied. Together they cover all eight
+  behavioral categories with per-tool call and blocked-call counts, distinct
+  and allowlisted read-path counts, declared decoy hits, and grader-emitted
+  claim-support and completion facts.
+- `behavioral-model-negative-control.yaml` intentionally reads a declared
+  decoy. A healthy run solves its literal task while recording
+  `behavioral_failure` with violated exploration and safety labels, proving
+  that the rules can reject observed model behavior rather than merely restate
+  aggregate success counters.
+
+Build once, then run either focused suite from the repository root:
+
+```sh
+node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-machinery.yaml --clio-coder-entry dist/cli/index.js
+node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-model.yaml --target mini --clio-coder-entry dist/cli/index.js
+node dist/cli/index.js eval run --suite benchmarks/eval/behavioral-model-negative-control.yaml --target mini --clio-coder-entry dist/cli/index.js
+```
+
+The machinery tasks use the repository read-only and create only private
+scratch state under `TMPDIR`; model tasks use a fresh `temp-copy` workspace and
+remove it after the matrix item settles. The machinery suite is the fast
+admission, worker, and receipt contract. The model suite is the live behavioral
+measurement: keep its Artifact v4 output as evidence for the exact target and
+serving configuration that ran, rather than treating one observed model result
+as a universal guarantee. Behavioral facts and their evidence store only
+bounded read counters, not path strings. As with other eval runs, the artifact's
+bounded diagnostic stdout may retain the underlying tool event stream.
+
+### `trackedMetrics`
+
+Eleven numbers plus a reason histogram, each carrying the source it came from. `source` is `ledger` (the per-call ledger folded from the worker's own JSON stream), `receipt` (the sealed run receipt), or `estimated`, and `estimated` is what a missing observation is marked as rather than being silently counted as measured.
+
+| Metric | Usual source |
+| --- | --- |
+| `modelCalls` | ledger |
+| `uncachedPrefillTokens` | ledger, from `promptCache.backend` |
+| `cacheReadTokens` | ledger, from `promptCache.backend`, falling back to pi-ai cache reads |
+| `generatedTokens` | ledger |
+| `reasoningTokens` | receipt; nullable, because absent and zero are different claims |
+| `toolCalls`, `toolErrors` | ledger when present, otherwise receipt |
+| `ttftMsFirstCall` | ledger |
+| `wallClockMs` | receipt |
+| `contextTokensAtEnd` | ledger |
+| `compactions` | ledger |
+| `expectedColdReasons` | ledger, one sourced count per reason |
+
+A dispatched worker's receipt reports `sessionId: null` and writes no session archive, which is why the ledger source exists at all: the runner folds structured usage, backend timing, cache, and monotonic TTFT facts out of the worker's `message_end` events. It keeps no prompt text, no model prose, and no tool-result content in that fold.
+
+### Scenario aggregates
+
+`aggregates` groups verdicts by `scenarioId`, sets `k` to the trial count, and records `passAtK` (any trial passed) and `passPowK` (every trial passed). Each tracked numeric metric reports observation, measured, and unmeasured counts, mean, min, max, nearest-rank p90, population variance, standard deviation, and the set of sources observed. A metric with no observation keeps every numeric statistic `null`; it never becomes zero. At `k: 1`, variance and standard deviation are zero only when the value was actually measured.
+
+### Behavioral multi-metric results
+
+A result with a `clio.eval.behavior.v1` verdict also carries the additive
+`clio.eval.behavior.metrics.v1` projection. The projection binds the scenario
+to its role and target/model envelope and records one `number | null`
+observation for each closed metric. The source travels beside every value:
+
+| Family | Metric | Direction | Gate | Source |
+|---|---|---|---|---|
+| correctness | `correctness.taskSolved` | higher | hard | grader |
+| safety | `safety.violations` | lower | hard | behavioral label |
+| behavior | `behavior.labelViolations` | lower | informational | behavioral labels |
+| efficiency | `efficiency.toolCalls` | lower | informational | terminal tool events |
+| exploration | `exploration.unnecessaryReads` | lower | informational | read observation counters |
+| delegation | `delegation.quality` | higher | informational | behavioral label |
+| claims | `claims.unsupported` | lower | informational | grader |
+| tokens | `tokens.total` | lower | informational | runner usage stream |
+| latency | `latency.wallMs` | lower | informational | monotonic runner clock |
+| cost | `cost.usd` | lower | informational | sealed receipt |
+
+Label metrics are numeric projections only when the category is `satisfied` or
+`violated`; `unknown` and `unmeasured` remain null. A missing grader, token
+stream, receipt, read observation, or category label likewise remains null.
+The projection therefore records observation coverage without claiming that
+silence was success, safety, or zero cost.
+
+### Behavioral comparisons and variance
+
+`eval compare` reduces the behavioral projection independently for every
+scenario, role, target id, and model id. Each row contains the baseline and
+candidate distributions, coverage, mean delta, variance delta, and two closed
+classifications: `improved`, `regressed`, `unchanged`, or `incomparable` for the
+mean and for variability. Lower variance is the improvement direction for the
+variability classification.
+
+Correctness and safety rows are hard. A measured regression fails the hard
+gate even when pass rate, tokens, latency, or cost improved. Losing a
+correctness or safety measurement that existed in the baseline is also a hard
+failure; a category explicitly unmeasured on both sides stays incomparable but
+does not invent a regression. Other families remain visible informational
+tradeoffs. `--metric` accepts either a behavioral metric or family as well as a
+tracked metric, but filtering displayed rows never filters the hard-gate
+decision.
+
+Comparison output supports `text`, `json`, `md`, and `junit`. All four carry
+the same hard-gate result and closed classifications. JUnit failures represent
+only hard behavioral failures; an informational efficiency or cost regression
+is emitted as testcase output rather than a failed testcase.
+
+### Execution-envelope provenance and comparability
+
+Every newly written behavioral result carries an additive
+`clio.eval.execution-envelope.v1` sibling. Artifact v4,
+`clio.eval.verdict.v1`, and `clio.eval.behavior.metrics.v1` retain their
+existing identities. The envelope records the selected prompt fragment ids,
+authored versions or `unversioned` marker, fragment content hashes, prompt
+composition hash, recipe id/version/fingerprint when a worker recipe applies,
+target, wire model, runtime, thinking level, tool signature, effective
+autonomy, rule-pack and project-policy hashes, bounded project-context
+provenance, and corpus id/version. A machinery-only scenario uses explicit
+nulls for model concepts that did not apply; null is not substituted for a
+fact that was observed.
+
+Suite v2 may declare `matrix.dimensions` from `prompt`, `recipe`, `target`,
+`wireModel`, `runtime`, `thinkingLevel`, `toolSignature`, `autonomy`, `policy`,
+`projectContext`, and `corpus`. Comparison ignores only dimensions declared by
+both artifacts. Any other envelope difference marks every metric row for that
+scenario/role/target incomparable and fails the behavioral gate. A missing
+envelope on only one side is also incomparable. Two older artifacts that both
+predate the sibling remain readable and compare under their existing data.
+
+Text, JSON, Markdown, and JUnit comparison reports carry the same envelope
+mismatch. Text and Markdown also include independent per-scenario and per-role
+baseline/candidate counts for improved, regressed, unchanged, and incomparable
+metric means and variances. When the prompt or recipe identity changes, the
+generated evidence names each affected corpus scenario and role instead of
+hiding it behind an aggregate score.
+
+### Checked behavioral release baseline
+
+The checked deterministic baseline is
+`benchmarks/eval/behavioral-machinery-baseline.json`. The release gate runs all
+26 machinery-only scenarios through the built CLI and compares a stable
+projection of their labels, metrics, and execution envelopes with that file.
+It requires no model, private endpoint, credential, or mutable dataset.
+
+When an intentional prompt, recipe, policy, or expected-behavior change moves
+the evidence, run the same machinery suite first, inspect the failing diff and
+the named affected corpus results, then update explicitly:
+
+```sh
+npm run build
+node benchmarks/eval/check-behavioral-release.mjs --update
+git diff -- benchmarks/eval/behavioral-machinery-baseline.json
+```
+
+The baseline update belongs in the reviewed change that caused it. Do not use
+the update command merely to make a red gate green. The model-required and
+negative-control suites remain manual release evidence because their outputs
+depend on a live target; they are never folded into the deterministic baseline.
+The projection excludes `latency.wallMs` because scheduler timing is not stable
+evidence. Behavioral labels, deterministic metrics, and the execution envelope
+remain checked byte for byte.
+
+### Hard thresholds and informational budgets
+
+Suite and external threshold files keep two separate assertion lists:
+
+```yaml
+thresholds:
+  fail:
+    - metric: task.solved
+      op: eq
+      value: false
+  informational:
+    - metric: cost.usd
+      op: gt
+      value: 0.25
+```
+
+`fail` is the backwards-compatible hard list. A firing or unresolved hard
+assertion makes `eval run` or `eval gate` exit nonzero. `informational` uses the
+same typed predicates and reports every firing budget or missing measurement,
+but never changes the exit status. `eval gate` additionally evaluates the
+baseline-to-candidate correctness and safety hard gate, so a cheaper candidate
+cannot offset a task or safety regression.
+
+### `--trials N`
 
-Task outcome commands declared under `verify.measure` evaluate whether the model solved the workload and record metrics (`task.solved`, `task.exitCode`). A non-zero exit from `verify.measure` is recorded as data and **never fails the evaluation item**. Task solution outcome is a measurement, while only machinery invariant behavior operates as a gate.
+`--trials N` overrides the suite's `matrix.repeats` and asks for an isolated workspace per matrix item. A `local` workspace is converted to a temporary copy immediately before that item runs, so an explicit trial run never mutates the directory it was pointed at; `git` and `temp-copy` workspaces already produce a distinct preparation directory per item. Workspace and state directories are removed on the item's `finally` path, including runner, setup, and copy failures. The trial index rides through to each verdict's `trialIndex`.
+
+### Serving-configuration provenance and drift refusal
+
+`servingConfiguration` records what the numbers were measured against: `targetId`, `runtimeId`, `modelId`, `serverBuild`, `total_slots`, `thinkingLevel`, and `compiledPromptHash`. The build string and slot count are read from the server after the matrix has run while it is still awake, by fetching `/props` and falling back to the model-qualified slots query when `/props` exposes no `total_slots`. The prompt hash is the receipt's static composition hash, so a prompt change is visible as a configuration change rather than as a mysterious metric shift.
+
+`eval compare` prints both configurations and refuses outright when they differ:
+
+```text
+serving configuration drift; pass --allow-config-drift to compare these runs
+baseline serving: target=mini runtime=llamacpp model=... server_build=b226-2115b73d8 total_slots=1 thinking=off compiled_prompt_hash=...
+candidate serving: ...
+```
+
+`--allow-config-drift` proceeds and labels the comparison `config drift: allowed`. There is a second refusal that has no override: a metric whose baseline distribution contains an `estimated` observation and whose candidate does not, or the reverse, raises `EvalTrackedMetricSourceMismatchError` rather than printing a delta, because subtracting a measurement from an estimate produces a number that looks like evidence and is not. `--metric <name>` filters tracked or behavioral rows, accepts `expectedColdReasons`, a specific `expectedColdReasons.<reason>`, a behavioral family, or a behavioral metric, and errors when the name matches nothing.
+
+---
+
+## Task Outcome Measurement (`verify.measure`)
 
+Task outcome commands declared under `verify.measure` are the code grader for whether the model solved the workload and record metrics (`task.solved`, `task.exitCode`). A non-zero exit fails the final result and is named on its verdict as `reason: grader_failed`, while `machinery` remains `ok` when the runner and machinery verifiers succeeded. This keeps the artifact's `pass`, verdict outcome, scenario aggregates, and summary on one pass decision without misreporting a grader failure as broken machinery.

package/docs/evals-internal.md

@@ -1,7 +1,7 @@
 # Internal Eval Suites
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive blueprint is available at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) (Version: 0.3.7).
+> **Interactive Spec Available:** An interactive blueprint is available at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) (Version: 0.3.9).
 
 Private suites should live outside this repository. Keep datasets, prompts,
 live fleet coordinates, calibration outputs, and raw run artifacts in a private
@@ -19,6 +19,77 @@ data directory. Product eval artifacts and external benchmark campaigns are
 separate: public benchmark adapters live under `benchmarks/community/` and do
 not use the eval runner.
 
+The public behavioral corpus is the deliberate exception to the otherwise
+private Suite v2 data policy. Its reviewable, synthetic suites live under
+`benchmarks/eval/`: a model-free positive/adversarial authority pair for every
+built-in worker recipe, four tiny main-agent model scenarios covering all eight
+behavioral categories with event- and grader-derived facts, and an intentional
+decoy negative control. The model-free driver uses the shipped recipe catalog,
+real dispatch admission, scripted workers, and sealed receipts rather than
+frontmatter inspection. See
+[eval-runner.md](eval-runner.md#public-built-in-behavioral-corpus) for the
+focused commands. Private prompts, calibration cases, fleet coordinates, and
+campaign artifacts still belong outside this repository and must not be copied
+into the public corpus.
+
+## Running a private suite as a measurement
+
+A private suite is usually run to answer whether a harness change moved
+something, which makes it a measurement rather than a pass or fail. Three
+mechanics matter for that, all documented in full in
+[eval-runner.md](eval-runner.md#the-verdict-envelope).
+
+Run repeated trials with `--trials N` rather than by editing `matrix.repeats`.
+The flag overrides the suite's repeat count and asks for an isolated workspace
+per matrix item, so a `local` workspace is copied for the run instead of being
+mutated across trials. Each result's verdict carries its `trialIndex`, and the
+artifact's `aggregates` reduce them per scenario: `k`, `passAtK` (any trial
+passed), `passPowK` (every trial passed), and a mean and nearest-rank p90 for
+every tracked metric. A single trial produces a `k: 1` aggregate whose mean and
+p90 are the same observed value, which is a fact to state in a report rather
+than a distribution to reason about.
+
+Read the tracked metrics with their sources attached. A private suite on a
+local target is measuring prefill economics as much as correctness, so
+`uncachedPrefillTokens`, `cacheReadTokens`, `ttftMsFirstCall`, and the
+`expectedColdReasons` histogram are the interesting columns, and each one says
+whether it came from the ledger, from the receipt, or was `estimated`. A metric
+marked `estimated` on one side of a comparison and measured on the other is
+refused rather than differenced.
+
+Behavioral suites add a second projection beside those tracked performance
+metrics. Compare it per scenario, role, and target/model envelope rather than
+reducing unlike roles into one pass rate. Correctness and safety are hard
+regression gates; tool efficiency, unnecessary exploration, delegation
+quality, unsupported claims, tokens, latency, and receipt cost remain separate
+families with their own measured coverage and repeat variance. A missing value
+is null and makes that row incomparable, never zero.
+
+Put release-blocking assertions under `thresholds.fail` and non-blocking spend
+or latency budgets under `thresholds.informational`. Informational findings are
+printed in every gate run but do not change its exit status. Do not put a cost
+budget in the hard list to compensate for weak correctness, and do not turn a
+correctness rule into an informational budget; the comparison gate evaluates
+correctness and safety before either kind of operator-authored threshold.
+
+Record the serving configuration or the comparison is not one. The artifact
+captures `targetId`, `runtimeId`, `modelId`, `serverBuild`, `total_slots`,
+`thinkingLevel`, and `compiledPromptHash`, read from the server after the matrix
+has run while it is still awake. `eval compare` refuses two artifacts whose
+configurations differ unless `--allow-config-drift` is passed, and prints both
+either way. Treat that refusal as the useful behavior it is: a private suite
+compared across a server restart that changed a flag, a quantization, or the
+thinking level is measuring the server, not the change under test.
+
+The verdict envelope keeps its original `behavioral: null` field for compatibility.
+A suite that declares a versioned behavioral scenario records the result as a
+separate `clio.eval.behavior.v1` document on the Artifact v4 result, cross-linked
+to the unchanged verdict identity. Its labels come only from bounded transcript,
+tool, receipt, or grader facts, never from an ungrounded judge paragraph. A run whose harness broke records
+`machinery: "infrastructure_failure"`, which the parser refuses to pair with a
+`pass`, so a private suite cannot report a passing rate that includes runs
+nothing measured.
+
 ## Context Regression Seed
 
 ```yaml
@@ -267,4 +338,3 @@ thresholds:
       op: gt
       value: 0
 ```
-