@iowarp/clio-coder 0.3.0

package/docs/environment-variables.md

@@ -0,0 +1,105 @@
+# Environment Variables
+
+Every environment variable the shipped `src/` tree reads, grouped by role. Settings.yaml is the durable home for operator policy; env vars exist for per-process overrides (CI, one-off experiments), directory layout, debugging, and internal plumbing. When prose and source disagree, prefer the source; the table cites the read site.
+
+This page is the complete inventory, and `tests/contracts/environment-variable-inventory.test.ts` fails if `src/` reads a variable that has no row here.
+
+> [!TIP]
+> [docs/html/environment_blueprint.html](html/environment_blueprint.html) is a browsable walkthrough of the most commonly set variables with an effective-path resolver. It covers a curated subset, so use the tables below when you need the full list.
+
+## Guardrail overrides
+
+Durable values live in the `guardrails:` section of settings.yaml (see [configuration-and-targets.md](configuration-and-targets.md)). These env vars override them for one process; resolution is env > settings > built-in default, and every value is a positive integer. Resolution lives in `src/core/guardrails.ts`.
+
+| Variable | Settings key | Default | Controls |
+| --- | --- | --- | --- |
+| `CLIO_CODER_TURN_TOOL_CALL_BUDGET` | `guardrails.turnToolCallBudget` | 60 | Orchestrator per-turn soft tool-call budget; the hard interrupt ceiling sits 15 above it (`src/engine/loop-guard.ts`). |
+| `CLIO_CODER_WORKER_TOOL_CALL_CAP` | `guardrails.workerToolCallCap` | 150 | Lifetime ceiling on tool calls one dispatched worker may execute. Calls the harness refused (reserve steering, synthesis-lockout denials) never spend it. Agent recipe budgets may narrow but never widen it (`src/engine/loop-guard.ts`). |
+| `CLIO_CODER_MAX_RUNS` | `guardrails.maxDispatchRuns` | 1000 | Dispatch run-ledger retention cap (`src/domains/dispatch/state.ts`). |
+| `CLIO_CODER_READ_MAX_BYTES` | `guardrails.readMaxBytes` | 51200 | Per-call byte cap for the read tool, floored at 1024 (`src/tools/read.ts`). |
+| `CLIO_CODER_OBSERVATION_TURN_BUDGET_BYTES` | `guardrails.observationTurnBudgetBytes` | 196608 | Shared per-turn byte pool across observation tools (`src/tools/observation.ts`). |
+| `CLIO_CODER_INTERNAL_DISPATCH_TIMEOUT_MS` | `guardrails.internalDispatchTimeoutMs` | 900000 | Wall-clock cap for one internal generator dispatch: the wiki documenter and the bootstrap scout (`src/cli/internal-dispatch.ts`). |
+
+## Behavior knobs without a settings key
+
+| Variable | Default | Controls |
+| --- | --- | --- |
+| `NO_COLOR` | unset | Set to any non-empty value to drop every foreground and background color. Bold, dim, italic, and underline stay, because they are what is left to read the interface by (`src/interactive/theme/tokens.ts`). |
+| `CLIO_CODER_RIGOR` | repo-derived | Finish-contract evidence bar, `normal` or `high`, layered over the repo-derived default (`src/domains/safety/rigor.ts`). |
+| `CLIO_CODER_RESIDENCY` | managed | `observe`/`off` stops Clio managing model residency on every local runtime path, llama.cpp routers included; per-target opt-out via `lifecycle: user-managed` (`src/engine/apis/residency.ts`). |
+| `CLIO_CODER_TRUST_PROJECT_SKILLS` | off | `1` trusts project-local skills for execution (`src/domains/resources/skills/loader.ts`). |
+| `CLIO_CODER_ALLOW_EXTERNAL_FULL_ACCESS` | off | `1` lets full-auto pass through to external CLI runtimes with their own full access (`src/engine/claude/subprocess-runtime.ts`, `src/engine/antigravity/subprocess-runtime.ts`). |
+| `CLIO_CODER_FORCE_COMPACT` | off | `1` forces compaction on the next interactive turn (`src/interactive/chat-loop.ts`). |
+| `CLIO_CODER_STATUS_STUCK_MS` | 180000 | Stuck-turn watchdog threshold (`src/interactive/status/watchdog.ts`). |
+| `CLIO_CODER_SHUTDOWN_HOOK_MS` | 500 | Wall-clock budget per shutdown hook (`src/core/termination.ts`). |
+| `CLIO_CODER_HOOK_BUDGET_MS` | per-phase built-ins | Global middleware hook wall-clock budget (`src/domains/middleware/budget.ts`). |
+| `CLIO_CODER_HOOK_BUDGET_<PHASE>_MS` | per-phase built-ins | Per-phase hook budget, e.g. `CLIO_CODER_HOOK_BUDGET_TURN_END_MS`; beats the global var. |
+| `CLIO_CODER_HOOK_BUDGET_WARMUP_CALLS` | 1 | Hook calls exempted from budget accounting at startup. |
+| `CLIO_CODER_HOOK_BUDGET_WINDOW` | 5 | Sliding-window size for steady-state hook-budget warnings. |
+| `CLIO_CODER_HOOK_BUDGET_THRESHOLD` | 3 | Overruns within the window before a steady-state warning. |
+| `CLIO_CODER_LMSTUDIO_SDK_PREDICT` | off | `1` sends LM Studio predictions over the SDK again instead of its OpenAI-compatible port. Predictions moved to HTTP because the SDK surface ignores the thinking control, so this is an escape hatch back to the older transport and not a debug toggle. Listing, loading, and unloading always use the SDK (`src/engine/apis/lmstudio-native.ts`). |
+| `CLIO_CODER_SKILL_CATALOG_DIR` | unset | Local skill-catalog directory override (`src/domains/resources/skills/marketplace.ts`). |
+| `CLIO_CODER_SKILL_MARKETPLACE_INDEX` | unset | Skill-marketplace index path override (`src/domains/resources/skills/marketplace.ts`). |
+| `CLIO_CODER_MODEL_CATALOG_DIRS` | unset | Extra model-catalog directories (`src/domains/providers/knowledge-base-path.ts`). |
+| `CLIO_CODER_NO_NETWORK_TOOLS` | off | `1` strips network tools from every registry in the process; the skills-eval harness sets it for hermetic arms; `--allow-network` clears it (`src/tools/network-policy.ts`). |
+
+## Directory and install layout
+
+| Variable | Default | Controls |
+| --- | --- | --- |
+| `CLIO_CODER_HOME` | unset | Single-tree install root; the per-role vars below beat it (`src/core/xdg.ts`). |
+| `CLIO_CODER_CONFIG_DIR`, `CLIO_CODER_DATA_DIR`, `CLIO_CODER_STATE_DIR`, `CLIO_CODER_CACHE_DIR` | XDG platform defaults | Per-role directory overrides (`src/core/xdg.ts`). |
+| `CLIO_CODER_BIN_DIR` | `~/.local/bin` | Launcher symlink location (`src/cli/uninstall.ts`). |
+| `CLIO_CODER_PACKAGE_ROOT` | auto-detected | Package root for bundled-asset resolution (`src/core/package-root.ts`). |
+
+## Debug and trace toggles
+
+All default off; enable with `1`.
+
+| Variable | Controls |
+| --- | --- |
+| `CLIO_CODER_BUS_TRACE` | Event-bus channel tracing to stderr (`src/core/bus-trace.ts`). |
+| `CLIO_CODER_TRACE_BOOT` | Boot-phase timing trace (`src/core/boot-trace.ts`). |
+| `CLIO_CODER_TIMING` | Startup timing report (`src/entry/orchestrator.ts`). |
+| `CLIO_CODER_DEBUG_SHUTDOWN` | Shutdown-path diagnostics (`src/core/termination.ts`). |
+| `CLIO_CODER_DEBUG_LMSTUDIO` | LM Studio wire logging (`src/domains/providers/runtimes/common/lmstudio-logger.ts`). |
+| `CLIO_CODER_RUNTIME_VERBOSE` | Verbose runtime logging (`src/engine/apis/lmstudio-native.ts`). |
+| `CLIO_CODER_HOOK_BUDGET_DEBUG` | Per-overrun hook-budget diagnostics (`src/domains/middleware/runtime.ts`). |
+
+### File-writing traces
+
+These two take a path, not `1`. Setting either to `1` writes a file named `1` in the working directory. Both are off when unset or empty, and both create parent directories.
+
+| Variable | Contents | Controls |
+| --- | --- | --- |
+| `CLIO_CODER_RENDER_TRACE` | timing only | Per-frame render timing for the interactive TUI, truncated on open so one file is one session. Records frame durations and counts and no conversation text, which makes it the instrument for reproducing a frame-cost claim at a given terminal size (`src/interactive/render-trace.ts`). |
+| `CLIO_CODER_MEMORY_TRACE` | conversation text | Proactive task-memory step envelopes, including up to 8000 characters of the text each step saw. This is content-bearing by construction, so the file carries whatever the session carried. Do not enable it on work you would not paste, and do not attach the file to a bug report without reading it first (`src/domains/memory/task-memory-trace.ts`). |
+
+Example:
+
+```bash
+CLIO_CODER_RENDER_TRACE=/tmp/clio-render.jsonl clio-coder
+```
+
+## Internal plumbing
+
+Set by Clio for its own processes; not operator knobs.
+
+| Variable | Purpose |
+| --- | --- |
+| `CLIO_CODER_INTERACTIVE` | Marks the interactive TUI process; scrubbed from bash-tool children so nested invocations do not inherit it (`src/cli/clio.ts`, `src/core/bash-exec.ts`). |
+| `CLIO_CODER_RUN_OVERRIDES` | JSON envelope for run-scoped CLI options (`--max-context-tokens`, `--kv-cache-mode`, sampling flags). One typed variable instead of one env var per option; worker subprocesses inherit it (`src/core/run-overrides.ts`). |
+| `CLIO_CODER_RESUME_SESSION_ID` | Session id handed across a self-restart; consumed and deleted at boot (`src/entry/orchestrator.ts`). |
+| `CLIO_CODER_BOOTSTRAP_GENERATE_CHILD` | Marks the CLIO-CODER.md-generation child so it skips recursion (`src/domains/context/extension.ts`). |
+| `CLIO_CODER_WORKER_LABELS` | Comma-separated labels a dispatched worker reports as its own (`src/domains/dispatch/transport.ts`, `src/worker/entry.ts`). |
+| `CLIO_CODER_WORKER_PGID` | Process-group id the transport assigns a worker so its whole tree can be signalled (`src/domains/dispatch/transport.ts`, `src/worker/entry.ts`). |
+
+## Test-only
+
+| Variable | Purpose |
+| --- | --- |
+| `CLIO_CODER_WORKER_FAUX` (+ `_MODEL`, `_TEXT`, `_STOP_REASON`, `_ERROR_MESSAGE`) | Fake worker model for tests (`src/engine/ai.ts`). |
+| `CLIO_CODER_TEST_UPGRADE_NO_NETWORK` | Skips npm install during upgrade tests (`src/cli/upgrade.ts`). |
+| `CLIO_CODER_REQUIRE_HOME_PREFIX` | Test guardrail: abort if resolved directories escape `CLIO_CODER_HOME` (`src/core/init.ts`). |
+
+Variables used only by `scripts/` and `benchmarks/` harnesses (the `CLIO_CODER_LIVE_*` smoke-test family, benchmark fleet configuration, install-script inputs) are not part of the shipped runtime and are documented inline where they are consumed.

package/docs/eval-runner.md

@@ -0,0 +1,205 @@
+# Clio Coder Local Evaluation Runner
+
+> [!TIP]
+> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.3.0).
+
+The local evaluation runner executes repository-local YAML task suites as deterministic subprocess checks. It is useful for comparing harness changes, prompts, tools, or local workflows.
+
+Source of truth: [src/domains/eval/](../src/domains/eval/) and [src/cli/eval.ts](../src/cli/eval.ts).
+
+---
+
+## CLI Commands
+
+The CLI commands under `clio-coder eval` support running, validating, reporting, comparing, and gating evaluation suites.
+
+```bash
+clio-coder eval validate --suite <suite.yaml>
+clio-coder eval run --suite <suite.yaml> [--target <id>] [--model <id>] [--out <path>] [--clio-entry <path>]
+clio-coder eval run --task-file <tasks.yaml> [--repeat <n>] [--out <path>] [--clio-entry <path>]
+clio-coder eval report <evalId> --format text|json|md|swe-jsonl|junit
+clio-coder eval compare <baselineEvalId> <candidateEvalId>
+clio-coder eval gate <candidateEvalId> --baseline <baselineEvalId> [--thresholds <file>]
+```
+
+### Command Roles
+* **`validate`**: Validates the structure and constraints of a Suite v2 YAML file without executing it.
+* **`run`**: Runs a Suite v2 (via `--suite`) or a compatibility v1 task file (via `--task-file`). Outputs a text summary and writes an eval artifact under `<dataDir>/evals/` and evidence under `<dataDir>/evidence/eval-<evalId>/`.
+* **`report`**: Formats and prints a report from a saved `evalId`. Supports multiple `--format` outputs:
+  * `text` (default): Human-readable stdout summary.
+  * `json`: Raw JSON structure of the artifact.
+  * `md`: Markdown document with tables and summaries.
+  * `swe-jsonl`: Standardized JSONL format representing task runs (e.g. for SWE-bench comparisons).
+  * `junit`: XML report for CI/CD integration.
+* **`compare`**: Compares two evaluation artifacts (baseline and candidate) by matching tasks.
+* **`gate`**: Compares candidate metrics against baseline or absolute thresholds, exiting non-zero if assertions fail (useful for PR gating).
+
+Exit codes:
+
+| Command | Success | Failure |
+| --- | --- | --- |
+| `eval validate` | `0` when validation passes | `2` for validation issues |
+| `eval run` | `0` when all task repetitions pass | `1` when any task fails, `2` for invalid configs |
+| `eval report` | `0` when artifact loads | `1` if artifact cannot be read, `2` for invalid ID |
+| `eval compare` | `0` when both artifacts load and compare succeeds | `1` if artifacts cannot be read, `2` for invalid ID |
+| `eval gate` | `0` when all threshold assertions pass | `1` if assertions fail, `2` for config/invalid ID errors |
+
+---
+
+## Suite v2 Schema
+
+Suite v2 files define matrix targets, workspaces, runner parameters, validation metrics, assertions, and path blocklists.
+
+```yaml
+version: 2
+suite:
+  id: "science-suite"
+  title: "Scientific Software Evaluation"
+  visibility: "local"
+  description: "Suite for verifying HPC integrations."
+matrix:
+  targets:
+    - id: "local-gemini"
+      model: "gemini-3.5-flash"
+    - id: "local-claude"
+      model: "claude-sonnet-5"
+  repeats: 3
+tasks:
+  - id: "fft-tolerance-check"
+    tags:
+      - numeric
+      - fast
+    workspace:
+      kind: "temp-copy" # local | git | temp-copy
+      path: "fixtures/fft-src"
+      excludes:
+        - "**/node_modules/**"
+    runner:
+      kind: "clio-run" # clio-run | context-index | context-init | external-command
+      prompt: "Optimize the FFT tolerance bounds in solver.ts"
+      timeoutMs: 60000
+    verify:
+      commands:
+        - "npm run test"
+      assertions:
+        - metric: "result.pass"
+          op: "eq"
+          value: true
+        - metric: "tokens.total"
+          op: "lt"
+          value: 15000
+      forbidPaths:
+        - "**/credentials.yaml"
+    metrics:
+      collect:
+        - "tokens.total"
+        - "latency.wallMs"
+    timeoutMs: 90000
+```
+
+### Schema Field Reference
+
+| Field / Section | Sub-fields | Description |
+| --- | --- | --- |
+| `version` | - | Must equal `2`. |
+| `suite` | `id`, `title`, `visibility`, `description` | Metadata identifying the evaluation suite. |
+| `matrix` | `targets[]`, `repeats` | Matrix of execution targets (specifying model and thinking flags) and the repetition count. |
+| `workspace` | `kind`, `path`, `url`, `commit`, `checkout`, `excludes` | Workspace strategy: `local` (run in-place), `git` (clone from URL), or `temp-copy` (isolated copy of a directory). |
+| `runner` | `kind`, `prompt`, `command`, `commands`, `args`, `timeoutMs` | Runner type: `clio-run` (starts Clio agent loop), `context-index` (runs indexer), `context-init` (initializes context), `external-command` (spawns subprocess). |
+| `verify` | `commands`, `assertions`, `forbidPaths` | Validation steps: shell commands, metric assertions (e.g. `op: lt` for max token counts), and files/directories that must not be created or modified (`forbidPaths`). |
+| `metrics` | `collect` | List of metric names to compile for the evaluation runs. |
+
+---
+
+## Workspace Kinds
+* **`local`**: Executes the task directly in the specified local path.
+* **`git`**: Clones the repository from `url`, checks out the specified `commit` or `checkout` ref, and runs there.
+* **`temp-copy`**: Copies the directory at `path` to a temporary workspace location before running. This prevents side-effects from polluting other task runs.
+
+---
+
+## Runner Kinds
+* **`clio-run`**: Invokes the main Clio Coder agent loop with the task's prompt, tracing all tools.
+* **`context-index`**: Triggers the context engine to build index structures (`codewiki`).
+* **`context-init`**: Initializes workspace files (such as generating `CLIO-CODER.md`).
+* **`external-command`**: Spawns an external command or sequence of commands in the task workspace.
+
+---
+
+## Metric Assertions
+
+Metrics collected during runs can be validated automatically using the `verify.assertions` list. Supported operator fields (`op`) are:
+* `lt` (less than)
+* `lte` (less than or equal)
+* `gt` (greater than)
+* `gte` (greater than or equal)
+* `eq` (equal)
+* `neq` (not equal)
+
+Metrics that can be validated include `tokens.input`, `tokens.output`, `tokens.total`, `latency.wallMs`, `tools.totalCalls`, `tools.failed`, `tools.blocked`, `verifier.exitCode`, and `result.pass`.
+
+---
+
+## Failure Classes
+
+Evaluation tasks may fail with one of the following classes:
+
+| Failure Class | Meaning |
+| --- | --- |
+| `setup_failed` | A setup command exited non-zero. |
+| `verifier_failed` | A verifier command exited non-zero. |
+| `timeout` | A setup or verifier command timed out. |
+| `cwd_missing` | Resolved task cwd does not exist. |
+| `command_error` | Reserved class for command spawn/system errors. |
+
+---
+
+## v1 Compatibility
+
+Version 1 task files can still be run directly via:
+```bash
+clio-coder eval run --task-file tasks.yaml [--repeat <n>]
+```
+Under the hood, these are parsed and wrapped into a Suite v2 adapter with:
+* Workspace kind: `local` (using task `cwd` as workspace path)
+* Runner kind: `external-command` (executing task `setup` commands)
+* Verify commands: Task `verifier` commands
+* Timeout and tags mapped directly
+
+---
+
+## Token Accounting & Provenance
+
+Clio maintains two distinct token accounting streams with different provenances. These accounts are never merged, reconciled, or treated as interchangeable:
+
+1. **`tokens.*` (Wire Streaming)**: Folded live off stdout from assistant `message_end` events watched by `token-stream.ts` / `createStreamInvariantFold`. This represents usage reported by the provider for assistant messages watched over the wire. On surfaces without stdout streaming (such as `clio-coder fleet run --json`), `tokens.measured` is `false`.
+2. **`receiptUsage.*` (Journal Receipts)**: Summed from an evaluation item's run journal. Every attempt writes a receipt carrying token counts and USD cost authenticated against its own ledger envelope.
+
+### Fail-Closed Reporting
+Both accounting streams report unmeasured state with no counts at all rather than a numeric zero. Reporting zero for an unmeasured run would falsely claim the run cost nothing. On an unmeasured run, `tokens.total` resolves to `null` and fails closed on metric threshold comparisons.
+
+---
+
+## Eval Artifact Format (v4)
+
+Evaluation artifacts use format version 4 (`EvalArtifactV4`). Summary token metrics report `measuredRuns` out of total `runs`:
+
+```typescript
+export interface EvalArtifactV4 {
+  version: 4;
+  evalId: string;
+  suite: { id: string; hash: string };
+  clio: EvalClioProvenance;
+  environment: EvalEnvironmentProvenance;
+  matrix: { target: string; model: string | null; thinking: string | null };
+  summary: EvalArtifactSummaryV4;
+  results: EvalArtifactResultV4[];
+}
+```
+
+---
+
+## Task Outcome Measurement (`verify.measure`)
+
+Task outcome commands declared under `verify.measure` evaluate whether the model solved the workload and record metrics (`task.solved`, `task.exitCode`). A non-zero exit from `verify.measure` is recorded as data and **never fails the evaluation item**. Task solution outcome is a measurement, while only machinery invariant behavior operates as a gate.
+

package/docs/evals-internal.md

@@ -0,0 +1,298 @@
+# Internal Eval Suites
+
+> [!TIP]
+> **Interactive Spec Available:** Interactive blueprints are available for internal evaluation suites at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) and soak benchmark suites at [docs/html/soak_blueprint.html](html/soak_blueprint.html) (Version: 0.3.0).
+
+Private suites should live outside this repository. Keep datasets, prompts,
+live fleet coordinates, calibration outputs, and raw run artifacts in a private
+checkout or object store.
+
+Run a private suite from this source checkout with:
+
+```sh
+npm run build
+clio-coder eval run --suite <external-path> --clio-entry dist/cli/index.js
+```
+
+Use `--out <dir>` when the artifact should be written outside the default Clio
+data directory. Public summaries can be copied into
+`benchmarks/results/<suite>/<run-id>/` only after they have been sanitized down
+to `manifest.json` and `summary.json`.
+
+## Context Regression Seed
+
+```yaml
+version: 2
+suite:
+  id: internal-context-regression
+  title: Internal context regression
+  visibility: private
+  description: Private context index determinism and coverage regression.
+  provenance:
+    owner: internal-eval
+    source: private
+matrix:
+  targets:
+    - id: local
+  repeats: 3
+tasks:
+  - id: context-index-private-checkout
+    tags:
+      - internal
+      - context
+      - offline
+    workspace:
+      kind: local
+      path: .
+      excludes:
+        - .git
+        - node_modules
+        - dist
+        - coverage
+    runner:
+      kind: context-index
+    verify:
+      assertions:
+        - metric: context.indexedFiles
+          op: gt
+          value: 0
+        - metric: context.coverage
+          op: gt
+          value: 0
+      forbidPaths:
+        - data/private-dump
+        - runs/raw
+    metrics:
+      collect:
+        - context.indexedFiles
+        - context.coverage
+        - context.structuralHash
+        - context.digestTokens
+        - latency.wallMs
+    timeoutMs: 120000
+thresholds:
+  fail:
+    - metric: result.pass
+      op: eq
+      value: false
+    - metric: context.coverage
+      op: lte
+      value: 0
+```
+
+## Model Matrix Seed
+
+```yaml
+version: 2
+suite:
+  id: internal-model-matrix
+  title: Internal model matrix
+  visibility: private
+  description: Private model target smoke matrix for headless Clio runs.
+  provenance:
+    owner: internal-eval
+    source: private
+matrix:
+  targets:
+    - id: local-fast
+      model: example-fast-model
+      thinking: low
+    - id: local-deep
+      model: example-deep-model
+      thinking: high
+  repeats: 2
+tasks:
+  - id: version-smoke
+    tags:
+      - internal
+      - matrix
+      - smoke
+    workspace:
+      kind: temp-copy
+      path: fixtures/version-smoke
+      excludes:
+        - node_modules
+        - dist
+    runner:
+      kind: external-command
+      commands:
+        - node -e "console.log('matrix smoke')"
+    verify:
+      assertions:
+        - metric: verifier.exitCode
+          op: eq
+          value: 0
+    metrics:
+      collect:
+        - latency.wallMs
+        - verifier.exitCode
+    timeoutMs: 30000
+  - id: tiny-clio-run
+    tags:
+      - internal
+      - matrix
+      - model-required
+    workspace:
+      kind: temp-copy
+      path: fixtures/tiny-repair
+      excludes:
+        - node_modules
+        - dist
+    runner:
+      kind: clio-run
+      prompt: Fix the intentionally broken function so the local verifier passes.
+    verify:
+      commands:
+        - npm test
+      assertions:
+        - metric: result.pass
+          op: eq
+          value: true
+    metrics:
+      collect:
+        - tokens.input
+        - tokens.output
+        - tokens.total
+        - latency.wallMs
+        - patch.filesChanged
+        - verifier.exitCode
+    timeoutMs: 600000
+thresholds:
+  fail:
+    - metric: result.pass
+      op: eq
+      value: false
+    - metric: tools.blocked
+      op: gt
+      value: 0
+```
+
+## Live Turns Seed
+
+```yaml
+version: 2
+suite:
+  id: internal-live-turns
+  title: Internal live turns
+  visibility: private
+  description: Private live-fleet prompts for end-to-end Clio turn checks.
+  provenance:
+    owner: internal-eval
+    source: private
+matrix:
+  targets:
+    - id: live-orchestrator
+      model: example-live-model
+      thinking: low
+  repeats: 1
+tasks:
+  - id: readonly-repo-summary
+    tags:
+      - internal
+      - live
+      - readonly
+      - model-required
+    workspace:
+      kind: temp-copy
+      path: fixtures/live-readonly
+      excludes:
+        - node_modules
+        - dist
+        - .clio-coder
+    runner:
+      kind: clio-run
+      prompt: Summarize the repository purpose and make no file changes.
+    verify:
+      forbidPaths:
+        - unexpected-output.txt
+      assertions:
+        - metric: result.pass
+          op: eq
+          value: true
+        - metric: patch.filesChanged
+          op: eq
+          value: 0
+    metrics:
+      collect:
+        - tokens.total
+        - latency.wallMs
+        - tools.totalCalls
+        - tools.failed
+        - patch.filesChanged
+    timeoutMs: 300000
+  - id: live-small-edit
+    tags:
+      - internal
+      - live
+      - edit
+      - model-required
+    workspace:
+      kind: temp-copy
+      path: fixtures/live-small-edit
+      excludes:
+        - node_modules
+        - dist
+        - .clio-coder
+    runner:
+      kind: clio-run
+      prompt: Fix the failing unit test with the smallest source change.
+    verify:
+      commands:
+        - npm test
+      assertions:
+        - metric: result.pass
+          op: eq
+          value: true
+        - metric: patch.testFilesModified
+          op: eq
+          value: 0
+    metrics:
+      collect:
+        - tokens.input
+        - tokens.output
+        - tokens.total
+        - latency.wallMs
+        - tools.totalCalls
+        - tools.failed
+        - patch.filesChanged
+        - patch.testFilesModified
+        - verifier.exitCode
+    timeoutMs: 600000
+thresholds:
+  fail:
+    - metric: result.pass
+      op: eq
+      value: false
+    - metric: tools.failed
+      op: gt
+      value: 0
+```
+
+---
+
+## Soak Benchmark Suite
+
+The soak benchmark suite located under [`benchmarks/soak/`](../benchmarks/soak/) measures Clio's own machinery performance, integrity, and structural invariant promises under load. Unlike standard evaluation suites, the soak suite evaluates the reliability of Clio rather than model capability. A weak model that fails to solve the workload still passes the suite if Clio's machinery behaves correctly; a strong model fails the suite if Clio fails to seal a receipt, cannot authenticate a receipt, or violates a system invariant.
+
+The soak suite comprises four specialized suite files:
+
+### 1. Machinery Under Load (`clio-soak.yaml`)
+Evaluates the same task workload across two execution surfaces: the headless main-agent surface (`clio-run`) and a dispatched worker surface (`agent: coder`). It tests single-file bugs, multi-file bugs, and compaction continuity across restarts.
+- **Surface Differences**: Main-agent tasks verify session ledger continuity (`ledger.formatVersion`, `ledger.toolPairsUnmatched`, `ledger.assistantBetweenCallAndResult`), while dispatch worker tasks verify process group cleanup (`process.orphanedChildren == 0`).
+- **Compaction Continuity**: Verifies that compaction summaries are present (`continuity.compactionSummaryPresent`) and that pre-compaction facts are preserved (`continuity.answeredFromPreCompaction`).
+- **Suite-Wide Gates**: Gates on `receipt.sealed`, `receipt.integrityValid`, `receipt.outcomeMatchesExit`, `tokens.measured`, `stream.cumulativeSnapshots == 0`, `stream.usageDoubleCounted == false`, and `stream.segmentUsageMatchesMessages == true`.
+
+### 2. Per-Step Write Boundaries (`clio-soak-boundary.yaml`)
+Validates write boundary enforcement across steps without model participation. Enforcement is strictly detect-and-rollback and is never sandboxing.
+- `write-boundary.rolled-back`: Verifies clean detection of allowlist violations (`writes_boundary_violation`), git-level file restoration, and sealed verdict generation (`boundary.violationsRolledBack == 1`, `boundary.rollbackIncomplete == 0`).
+- `write-boundary.rollback-incomplete`: Tests honest failure reporting when a path was dirty prior to snapshot taking. prior bytes exist only in the overwritten tree, so rollback leaves the tree unchanged and records incomplete rollback (`boundary.rollbackIncomplete == 1`, `boundary.violationsRolledBack == 0`).
+
+### 3. Fault Injection Chaos (`clio-soak-chaos.yaml`)
+Evaluates system resilience against process signals.
+- `chaos.sigint-mid-tool`: Prompts Clio for a long-running bash tool call and injects `SIGINT` once the subprocess initializes. Asserts exit code `130`, confirms no orphaned children remain (`process.orphanedChildren == 0`), and verifies receipt sealing, receipt integrity, and provider token reporting.
+
+### 4. Bounded Loops (`clio-soak-loop.yaml`)
+Validates iteration bounds and receipt accounting for fleet loops (`bounded-loop.fleet`).
+- **Loop Bounds**: Asserts that verification attempts do not exceed declared limits (`loop.attemptsSpent <= 3`), recovery attempts seal individual receipts (`loop.receiptsMatchRepairs == true`), and unneeded nodes report as `unneeded` rather than skipped or failed (`loop.skippedNodes == 0`).
+- **Two Token Accountings**: Distinguishes `tokens.*` (folded live off wire stdout by `createStreamInvariantFold`) from `receiptUsage.*` (journal receipts sealed and authenticated against ledger envelopes). On fleet runs, wire streaming is absent (`tokens.measured == false`), while journal receipts provide authenticated usage (`receiptUsage.measured == true`).
+