@bjoernrochel/sevro 0.1.0-rc.2 → 0.1.0-rc.21

package/docs/releases.md

@@ -7,12 +7,19 @@ exact semantic version; development versions containing `-dev` are not release
 candidates. Darrow pins the published version in development dependencies and
 its lockfile. Marketplace plugins do not depend on Sevro.
 
+The [current release](installing.md#current-release) uses the current [licensing terms](licensing.md).
+Published `rc.1` and `rc.2` retain their original BSL grants. Always select a new
+version for publication; a local development tarball does not replace an
+already published artifact.
+
 ## Candidate contract
 
 Prepare one candidate from the intended release tag and package metadata. The
 tag must equal `v<package version>`. Package metadata must identify the public
 repository, registry, distribution tag, and an owner-selected license with its
-nonempty `LICENSE` file. Prereleases cannot use npm's `latest` distribution tag.
+nonempty `LICENSE` file. `latest` identifies the recommended default release,
+including a release candidate. Use `next` for a preview that should require an
+explicit opt-in. Preparation accepts either tag for prerelease and stable versions.
 
 Preparation creates a new absolute output directory and never replaces an
 existing directory. It packs with lifecycle scripts disabled, retains the npm
@@ -21,8 +28,9 @@ The record identifies the package, version, release tag, distribution tag,
 tarball SHA-256 and npm SHA-512 integrity, and the preparation runtimes.
 Preparation does not publish a package or create a Git tag.
 
-The package installation check accepts `--tarball <absolute file>` to verify
-that exact candidate without repacking. The supplied tarball remains intact.
+For development or explicitly requested diagnostics, the package installation
+check accepts `--tarball <absolute file>` to verify an exact candidate without
+repacking. It is not a release step. The supplied tarball remains intact.
 Its installed name and version must match the source metadata. Installation,
 CLI execution, generic grading, prompt-only execution, release provenance,
 reporting, and supported deterministic native fixtures run without source Git
@@ -30,59 +38,92 @@ metadata in the installed package.
 
 ## Maintainer workflow
 
-1. Select the license, update `package.json` to the intended release version,
-   and commit those changes. Preserve the existing public schemas and document
-   behavior changes. For the first candidate, use `0.1.0-rc.1` with npm tag
-   `next`; a prerelease does not establish full Darrow migration completion.
-2. Run frozen installation, typecheck, formatting, and the full test suite.
-3. Prepare and test the exact candidate:
+Release already verified code. Do not rerun tests locally or in CI as part of a
+release: reuse the successful source CI run. Code, dependencies, schemas, tests,
+examples, license changes, and other quality inputs belong in ordinary changes
+with their own CI before starting the release.
+
+1. Start a release branch from verified `main`. Update `package.json.version`
+   to a new release version, keep the selected license, and run `bun run docs:sync`.
+   Set `publishConfig.tag` to `latest` for the recommended default or `next`
+   for an opt-in preview. Commit and open a pull request; every release commit
+   must reach `main` through a PR. Never attempt a direct release push to `main`.
+2. Merge the release PR after its static checks and reused CI evidence pass.
+   Release CI compares the candidate against a successful ancestor run on
+   `main`, permitting only version/distribution-tag changes and the exact
+   release documentation and workflow files inventoried in
+   [the CI selector](../.github/scripts/release-ci.py). All other tracked inputs
+   must match. The original run must have completed all four quality phases
+   and their gate steps, or the historical complete quality gate;
+   another release's reused status is insufficient. CI retains the source SHA
+   and run URL in `release-ci.json`. Missing evidence fails the release check;
+   it does not trigger another test run. Ordinary code changes and explicit
+   workflow dispatches continue to run the full gate.
+3. From the clean, merged release commit, use macOS, Bun 1.3.13, Node 24,
+   and npm to prepare one exact candidate:
+
+   ```sh
+   bun install --frozen-lockfile
+   bun run release:prepare --tag "v${SEVRO_RELEASE_VERSION:?set the selected new release version}" --output /absolute/new/release-directory
+   ```
+
+4. Inspect the version, repository, license, distribution tag, file inventory,
+   and retained checksums. Publish that tarball with lifecycle scripts disabled
+   using the commands below. Do not run `check:typescript`, `test:package-install`,
+   Darrow's archive/installed gates, or native trials during release. Diagnostic
+   tests require a separate explicit request.
+5. Once npm serves the version, compare `dist.integrity` with `release.json`
+   and verify the intended distribution tag:
 
    ```sh
-   bun run release:prepare --tag v0.1.0-rc.1 --output /absolute/new/release-directory
-   bun run test:package-install --tarball /absolute/new/release-directory/bjoernrochel-sevro-0.1.0-rc.1.tgz
+   npm view "@bjoernrochel/sevro@${SEVRO_RELEASE_VERSION}" version dist.integrity --json
+   npm view @bjoernrochel/sevro dist-tags --json
    ```
 
-4. Run Darrow's public integration against that same tarball:
+   npm may take a few minutes to process the accepted upload. Wait for the
+   registry record rather than republishing. Retain the reviewed tarball,
+   checksums, and original CI evidence.
+
+6. Create the matching Git tag at the merged release commit and push only the tag:
 
    ```sh
-   SEVRO_PACKAGE_TARBALL=/absolute/new/release-directory/bjoernrochel-sevro-0.1.0-rc.1.tgz bun run test:eval-runner-sevro-package
+   git tag -a "v${SEVRO_RELEASE_VERSION}" -m "Release ${SEVRO_RELEASE_VERSION}"
+   git push origin "refs/tags/v${SEVRO_RELEASE_VERSION}"
    ```
 
-5. Review the changes, file inventory, retained checks, and compatibility
-   limitations before authorizing publication. Keep the reviewed tarball and
-   its checksums. Publish that artifact rather than repacking after review.
-6. After publication, verify registry integrity, install the exact version in
-   Darrow, and run its frozen installed compatibility gate. Retain the prior
-   version's evidence for updates. Roll back through a new dependency/lockfile
-   commit and rerun the installed gate; never relabel failed-version evidence.
+Darrow dependency/lockfile updates and compatibility gates are a separate
+Darrow PR, not prerequisites for npm publication. A prerelease does not establish
+full Darrow migration completion. Preserve prior-version evidence for updates
+and rollback; never relabel failed-version evidence.
+
+## Local macOS publication
+
+There is no GitHub release job. The maintainer prepares, inspects, and publishes
+the reviewed tarball from macOS with an authenticated npm account. Check its
+retained checksum immediately before publication:
 
-## Manual release workflow
+```sh
+cd /absolute/reviewed
+shasum -a 256 -c SHA256SUMS
+npm publish ./bjoernrochel-sevro-0.1.0-rc.1.tgz --ignore-scripts --access public --tag latest
+```
 
-`.github/workflows/release.yml` runs only through explicit dispatch against a
-release tag. Its default prepares and checks the artifact. `publish: true`
-additionally publishes that verified artifact using npm trusted publishing.
-No push or pull-request event publishes a package.
+Replace the example version and distribution tag with the reviewed candidate's
+values. The package repository URL must match this public repository. Preparing
+a tarball or opening a pull request does not authorize publication.
 
-Before the first publication, an npm owner must establish the package and its
-publishing authorization. The initial reviewed tarball can be published
-manually with an authenticated npm account:
+An existing published candidate can become the default without republishing:
 
 ```sh
-npm publish /absolute/reviewed/bjoernrochel-sevro-0.1.0-rc.1.tgz --ignore-scripts --access public --tag next
+npm dist-tag add @bjoernrochel/sevro@0.1.0-rc.2 latest
+npm view @bjoernrochel/sevro dist-tags --json
 ```
 
-After the package exists, configure its trusted publisher for GitHub owner
-`BjRo`, repository `sevro`, and workflow filename `release.yml`. The workflow
-uses a GitHub-hosted runner, Node 24, npm 11.20.0, and job-scoped `id-token: write`.
-The package repository URL must match this public repository. See npm's
-[trusted publishing documentation](https://docs.npmjs.com/trusted-publishers/).
-The workflow checks the downloaded artifact's SHA-256 before publishing and
-retains its release record. Configuring an owner or workflow does not itself
-authorize this coding session to publish or push.
+Promotion changes the registry tag; it does not update the published tarball.
 
-The first local candidate is `0.1.0-rc.1`, using the owner-selected
+The first published candidate was `0.1.0-rc.1`, using the owner-selected
 `BUSL-1.1` license. Its `LICENSE` preserves Darrow's terms and parameters,
-changing only the Licensed Work name and description to Sevro. Neither local
-checks nor a workflow file claim a remote CI run or publication. Darrow's exact
+changing only the Licensed Work name and description to Sevro. Local checks do
+not claim a remote CI run or publication. Darrow's exact
 registry pin, default caller switch, remaining workflow migrations, focused live validation, and
 generic runner removal remain separate extraction gates.

package/docs/report-v1.md

@@ -3,10 +3,14 @@
 `sevro report` reads one or more retained `sevro.cli-result.v1` files and their
 referenced `sevro.run-evidence.v1` files. It validates both public schemas and
 checks run identity, exit category, case count, and trial references before
-reporting. Paths must be absolute and distinct.
+reporting. Programmatic `createReport` paths must be absolute and distinct.
+CLI `--result-file` paths may be relative to the invoking directory; they are
+canonicalized before duplicate detection and loading.
 
 ```sh
 sevro report --result-file /absolute/path/to/result.json
+sevro report --latest
+sevro report --latest --results-root ./other-results
 sevro report --json --result-file /absolute/path/to/first.json \
   --result-file /absolute/path/to/second.json
 ```
@@ -17,6 +21,30 @@ inputs return exit 64 with a diagnostic on stderr. Reporting does not change
 the exit category of any input run; each row retains its original code.
 Pre-run failures with no case rows remain visible in the input-run section.
 
+`--latest` selects
+`<cwd>/.sevro/results/latest.json`; `--results-root` changes that root and requires
+`--latest`. It is exclusive with `--result-file`. Each retained CLI evaluation
+saves a `result.json` beside its run evidence and atomically updates latest,
+including dry and failed runs. Pre-run failures do not replace latest. Missing
+latest or referenced evidence is an error. See
+[the evaluation workflow](evaluation-workflow.md).
+
+[`sevro prune`](reading-results.md#prune-retained-results) maintains this index
+when removing retained evidence. After pruning, latest selects the newest
+remaining completed result; if none remain, the index is removed and `--latest`
+reports a missing-input diagnostic. Explicit reports still require their selected
+results and evidence to exist.
+
+After [`sevro prune --native-transcripts`](reading-results.md#remove-native-transcript-bundles),
+reports accept absent native bundles only with the matching
+[pruning receipt](results-v1.md#native-transcript-bundles). Present bundles must
+still match their recorded hashes; unexplained missing bytes remain an error.
+New report rows include optional `nativeTranscripts` with nonnegative integer
+`retained` and `pruned` bundle counts across the case's trials and host roles.
+Markdown shows the same counts beside retained evidence. Older report documents
+without this field remain valid. Pruning does not change verdicts, routes,
+identity, usage, or duration.
+
 Each case row keeps task verdict, execution, and grading separate. Task pass
 rate is measured only when every trial completed execution and grading with an
 assessed task verdict. Candidate duration sums the measured host call time

package/docs/results-v1.md

@@ -9,6 +9,17 @@ Each domain outcome also gets one human-readable line with its case, trial,
 outcome ID, and status. Domain outcomes remain independent of the task verdict
 and exit code.
 
+The CLI also shows trial states, bounded failed/unavailable
+check details, measured candidate duration and known/unknown usage in human
+completion. Actual preparation/candidate/grading/evidence stage messages use
+stderr in both modes; they do not assert task success. Dry execution/grading are
+explicitly skipped. JSON stdout stays unchanged. The CLI saves `result.json`
+beside retained run evidence and atomically publishes `<results-root>/latest.json`
+for [report access](report-v1.md). These output files stay inside excluded storage
+and do not change evaluation identity. A failure to retain CLI/latest output is
+reported as an internal error after an otherwise successful run; an earlier
+nonzero run outcome keeps its precedence and its evidence reference.
+
 ## Separate states
 
 Each trial and aggregate result carries:
@@ -98,6 +109,17 @@ persistence error.
 [`identity-v1.md`](identity-v1.md) defines the canonical digest and each
 comparison dimension.
 
+Current CLI runs additionally retain the redacted effective-settings preview as
+optional `configuration.resolved` (`sevro.resolved-run-settings.v1`), including
+saved-setting selections and resolved operational paths. This snapshot excludes
+credentials, credential-option paths, private source-closure/mapping paths,
+private extension configuration and runtime
+environment values; it is outside `configuration.digest`. Assessment settings
+remain bound by the existing identity inputs and `configuration.redacted`.
+Historical evidence without this optional field remains valid; missing settings
+stay unknown. See [saved run settings](run-settings-v1.md) for precedence and
+privacy limits.
+
 The versioned machine-readable contracts are
 [`cli-result-v1.schema.json`](../schemas/cli-result-v1.schema.json) and
 [`run-evidence-v1.schema.json`](../schemas/run-evidence-v1.schema.json). A
@@ -110,9 +132,167 @@ An interrupted or failed run keeps completed trials plus an explicit partial
 attempt or diagnostic record. Failed persistence is an internal error and
 cannot be presented as a successful assessment.
 
+Successful temporary workspaces are removed after their evidence is saved.
+Failed and cancelled workspaces, including trials with unavailable required
+grading evidence, are kept for 24 hours after becoming inactive. Evaluation
+startup removes expired workspaces with verified inactive ownership. Active or
+unverifiable workspaces and unregistered legacy folders are preserved; saved
+results and evidence remain intact. Failed cleanup is retried on later evaluations.
+
+`sevro.observation.workspaces` records candidate and retained auxiliary workspace
+paths for inspection. These paths are temporary and do not affect evaluation identity.
+
+Retained results have no automatic expiration. The explicit
+[`sevro prune` command](reading-results.md#prune-retained-results) applies an
+operator-selected age or count policy, defaulting to the newest ten results.
+It deletes complete retained result directories and maintains `latest.json`
+for the remaining results. Active or unrecognized attempts remain protected;
+fixture cleanup and run-state coordination metadata have separate lifecycles.
+
+Opted-in [candidate artifact runtimes](runtime-v1.md#candidate-artifact-access)
+remain available through grading and are removed afterward, including failed
+and cancelled trials. Failed-fixture retention does not preserve them.
+Read access does not automatically retain artifact bytes in run evidence.
+
+Checker private home, temporary storage, and copied dependency slots are removed
+after grading on success, failure, and cancellation. Retained failed fixtures and
+saved evidence do not retain checker runtime copies. Explicitly declared shared
+caches survive this cleanup and remain under operator control.
+
 A host adapter can return `executionFailed: true` with bounded artifacts from
 a failed process. The engine retains those artifacts, skips all grading, and
 reports `failed` / `not_requested` / `not_assessed` with exit code `2`. A final
 message that happens to match a check cannot turn that failure into a pass.
 Claude retains its bounded event stream and, when present, up to 64 KiB of
 stderr as private host artifacts. Failure bodies do not appear in CLI diagnostics.
+Codex preflight failures retain `codex.process` (stage, reason, exit code, signal,
+duration, and stderr truncation) and `codex.stderr` (the first 64 KiB), with role
+prefixes for semantic/advisory hosts. Diagnostics distinguish failure, timeout,
+and cancellation without exposing raw stderr.
+
+## Native host diagnostics
+
+The CLI's opt-in
+[`--retain-host-diagnostics`](development-cli.md#retain-native-host-diagnostics)
+retains candidate artifact `sevro.host.diagnostics` with format
+`sevro.host-diagnostics.v1`. Semantic and advisory requests use the existing
+`sevro.semantic[.<source digest>].` and `sevro.advisory.` artifact ID prefixes.
+The flag is independent of native transcript retention.
+
+The private JSON contains `host`, `role`, `stage`, `error`, `processes`,
+and `processesDropped`. Error evidence preserves the original name, message,
+stack and available error code, with up to eight error entries in the cause chain. Repeated
+or deeper causes end in an explicit `truncated` entry. Messages and stacks
+are bounded to 8,192 characters per entry. A host-returned failure without a
+thrown exception receives a runner failure summary; subprocess receipts and
+bounded output supply its native evidence.
+
+Each process receipt records stage, actual `code`, `signal`, `durationMs`,
+reason, stdout/stderr prefixes, and separate truncation markers. Unknown native
+exit information remains `null`; app-server's synthesized protocol result is
+not substituted for a process exit code. Each stream retains at most 64 KiB,
+and hosts continue consuming output after filling the diagnostic prefix.
+Stream parsing and decoding failures also retain available prefixes.
+Text uses replacement decoding for diagnostic inspection; it is not an exact
+native-byte archive.
+
+The journal keeps at most 32 recent process receipts and at most 2 MiB of
+encoded receipt data. It drops oldest receipts first and reports their count.
+The failing stage and bounded error remain independent of those dropped
+receipts. Known credentials from selected authentication files, forwarded
+Claude authentication variables, and common credential patterns are redacted
+from the diagnostic JSON. Sevro does not explicitly dump environment values,
+full argv, or settings into diagnostics. Treat retained diagnostics as private evidence and keep
+them out of Git; redaction is not authorization to publish subprocess output.
+
+The engine supplies an optional `retainHostDiagnostics(bytes)` callback to
+host adapters. Bundled opted-in hosts invoke it before removing native state.
+The callback saves the artifact with private permissions and registers its
+file URL and SHA-256 in trial evidence. Direct adapter calls without that
+callback return diagnostic bytes in ordinary host artifacts.
+
+Failure diagnostics name the retained file URL when it fits the CLI message
+limit; `evidencePath` and trial `artifactPath` resolve artifact references
+even when a message cannot contain the URL. Native failures retain the existing
+execution, grading, task and exit semantics. Failed diagnostic persistence is
+an internal runner error, not a successful assessment or an ordinary host
+failure. Cleanup errors do not replace an earlier host or diagnostic-persistence
+failure; cleanup warnings contain no raw error/output bodies.
+
+## Native transcript bundles
+
+With [runtime transcript opt-in](runtime-v1.md#native-transcript-access), each
+bundled host captures full native files before removing its private state.
+Candidate artifact `sevro.native-transcripts.bundle` has format
+`sevro.native-transcripts.v2`. It contains `host` (`codex` or `claude`), `role`,
+native `rootSessionId` (or `null`), `completeness`, `issues`, and `files`.
+Each file records its native relative `path`, SHA-256 `sha256` of the original
+bytes, and `gzipBase64`: the exact file bytes compressed with gzip, then encoded
+as base64. Decode base64, decompress gzip, and verify the original-byte hash
+before using a file. Bound decompression to 5 MiB across all files in the bundle.
+Decode bytes without rewriting native records. Native
+session paths, root/child records and sidecar metadata preserve associations;
+the bundle does not normalize model, effort, timing, token, or cost statistics.
+Existing unknown metrics remain unknown.
+
+Sevro reads both v2 bundles and existing `sevro.native-transcripts.v1` bundles.
+V1 files use uncompressed `bytesBase64`; decode that field directly without
+gzip decompression and verify the same original-byte hash. Readers must select
+the payload field and decoding by `format` and reject unsupported formats,
+corrupt compressed payloads, hash mismatches, or a decoded total above 5 MiB.
+New captures emit v2; reading retained v1 evidence does not rewrite it.
+The retained view exposes the same exact native files for either format, with
+the original bundle in `index.json`.
+
+`sevro.host.native-transcripts` records the same capture state and file hashes
+without raw bytes. Semantic and advisory bundles retain their role in the
+bundle and use the existing `sevro.semantic[.<source digest>].` and
+`sevro.advisory.` artifact ID prefixes. Run/trial artifact references supply
+stable private file URLs and whole-bundle hashes after native home cleanup.
+Extensions receive these references through the existing `evaluate` protocol;
+benchmarks can use the same references and decode the native files.
+
+Capture includes native transcript-tree sidecars and refuses descendant links,
+special files and escaping paths. It is bounded to 4,096 directory entries,
+5 MiB of native file bytes before compression and 7 MiB of encoded file
+metadata/gzip content. The
+bundle remains inside the existing 8 MiB item and 32 MiB host artifact budgets;
+it groups sessions instead of spending an artifact per child. Limits produce
+`byte_limit`, `entry_limit`, or `bundle_limit` issues. Unreadable, unsafe or
+changing entries, malformed/empty native records, malformed JSON sidecars,
+an unknown/missing original session, or incomplete host
+execution produce explicit partial/unavailable capture. `complete` means the
+identified root's tree was captured within those bounds after completed host
+execution and generic syntax/basic record validation. JSONL must contain
+nonempty JSON object records with a nonempty string `type`; JSON sidecars must
+parse as JSON. UTF-8 decoding is strict. One trailing newline and CRLF are
+accepted; blank interior records, invalid encoding and truncated JSON are
+malformed. Unknown native fields and record types remain valid. Issues
+`empty_transcript`, `malformed_transcript` and `malformed_sidecar` downgrade
+capture without rewriting or discarding retained bytes. This validates basic
+structure, not native semantic correctness.
+An empty/missing native JSONL tree is `unavailable`. Failure, cancellation and
+timeout retain available native bytes with incomplete capture state. Raw
+bundles and views are private evidence and must stay out of Git.
+
+The explicit [`sevro prune --native-transcripts` mode](reading-results.md#remove-native-transcript-bundles)
+can remove these bundles without rewriting CLI results, run/trial evidence,
+capture observations, native file hashes, or whole-bundle hashes. It retains
+other artifacts and does not change evaluation identity or assessment states.
+Reports validate present native bundles against their recorded SHA-256 and
+require an exact pruning receipt for absent bundles.
+
+Before deletion, Sevro atomically writes private
+`native-transcripts-pruned.json` beside `run.json`, with format
+`sevro.pruned-native-transcripts.v1`, the original `runId`, and `artifacts`.
+Each entry contains the original artifact reference (`id`, `path`, `sha256`,
+and any optional reference flags), plus its `caseId` and `trial`. Entries follow
+run trial/reference order and cover every candidate, semantic (including
+source-specific), and advisory native bundle in that run. A receipt must match
+these original references exactly; malformed, inconsistent, or linked receipts
+are errors. Bundle references must name regular files directly inside their
+result directory. Missing bytes without a receipt, changed hashes, and escaping
+references remain errors. An interrupted deletion may leave some marked bundles
+present; reports count actual availability, and a repeated prune completes removal.
+Transcript references that alias result/trial metadata, raw responses, or other
+retained artifacts are refused before pruning.

package/docs/run-settings-v1.md

@@ -0,0 +1,147 @@
+# Saved run settings v1
+
+Save recurring evaluation selections in a JSON run manifest, then use
+`sevro run --run-manifest-file ./evaluation.json`. The examples use the
+[installed command](installing.md#make-the-command-available).
+
+## Repeat an evaluation
+
+Start with an existing [resolved case and adapter](getting-started.md), and save
+this `evaluation.json` beside `case.json` and `host-adapter.ts`:
+
+```json
+{
+  "format": "sevro.run.v1",
+  "settings": {
+    "case-file": "./case.json",
+    "adapter-module": "./host-adapter.ts",
+    "project-root": ".",
+    "condition": "passive",
+    "trials": 3,
+    "jobs": 1,
+    "threshold": 1
+  }
+}
+```
+
+Run it twice with the same saved selections, or explicitly override one setting:
+
+```sh
+sevro run --run-manifest-file ./evaluation.json
+sevro run --run-manifest-file ./evaluation.json
+sevro run --run-manifest-file ./evaluation.json --trials 1 --json
+```
+
+Every run prints effective settings on stderr before candidate execution, then
+prints the result and evidence location. The example requires all assessed
+trials to pass. A missing assessment remains `not_assessed`. Use `--dry` to
+check preparation without calling the candidate or graders; dry preparation
+does not prove native authentication or task success.
+
+## Select candidate and grading routes once
+
+For native execution, replace the adapter selection with an explicit host,
+model and effort. Each grading role has its own selection. For example, a case
+with semantic checks can use this manifest:
+
+```json
+{
+  "format": "sevro.run.v1",
+  "settings": {
+    "case-file": "./semantic-case.json",
+    "project-root": ".",
+    "host": "codex",
+    "model": "<candidate-model>",
+    "effort": "medium",
+    "codex-auth-file": "../private/auth.json",
+    "semantic-host": "codex",
+    "semantic-model": "<grading-model>",
+    "semantic-effort": "high",
+    "trials": 3,
+    "threshold": 1
+  }
+}
+```
+
+Replace model placeholders with installed-host-supported model names and supply
+the [native prerequisites and authentication](native-hosts.md). Models and
+efforts are never inferred or copied between roles. Candidate Claude uses
+`host: "claude"` and its own explicit `model`/`effort`. An optional advisory
+route uses `advisory-host`, `advisory-model` and `advisory-effort`; advisory
+review requires a Git fixture and remains separate from the task verdict.
+Injected adapters continue to declare their own model and effort.
+
+## Format, precedence and paths
+
+The root has exactly `format: "sevro.run.v1"` and an object `settings`. Settings
+use the [CLI option names](development-cli.md) without the leading `--`.
+All run options can be saved except `json` and `run-manifest-file`. Keep output
+presentation explicit on the command line, including `--json` when collecting
+a versioned success or failure document. Manifests do not include other
+manifests and are read only when explicitly selected; there is no discovery.
+
+| Option kind                                                                             | JSON value                                            |
+| --------------------------------------------------------------------------------------- | ----------------------------------------------------- |
+| `trials`, `jobs`, `threshold`, `codex-timeout-ms`                                       | Number, with the CLI's existing ranges                |
+| `dry`, `shell-isolation`, `retain-host-diagnostics`, `claude-project-settings`          | Boolean                                               |
+| `extension-source-file`, `protected-root`, `replace-builtin-grader`, `advisory-exclude` | Array of nonempty strings; `[]` selects an empty list |
+| Other run options                                                                       | Nonempty string                                       |
+
+Precedence is **built-in defaults → saved settings → explicit CLI options**.
+Omitted settings retain the [documented defaults](development-cli.md#run-defaults).
+An explicit repeatable flag replaces its saved array in full; repeated CLI
+occurrences build the replacement array. `--no-dry`, `--no-shell-isolation`,
+`--no-retain-host-diagnostics` and `--no-claude-project-settings` override saved
+`true` values with `false`.
+
+Selecting an alternative route on the CLI removes its saved alternative and
+route-only settings. For example, `--adapter-module ./host-adapter.ts` replaces
+a saved native candidate and its model/effort. Selecting `--host claude` over
+a saved Codex candidate requires fresh explicit `--model` and `--effort`;
+grading roles keep their separate selections. Shared Codex settings remain
+while any Codex route is selected. CLI `--case-file` replaces a saved extension
+source and its extension-only options. CLI `--extension-command-file` replaces
+a saved case file; supply the extension's required ID and source closure.
+Supplying two exclusive alternatives explicitly remains an error.
+
+Paths saved in a manifest resolve from the **canonical directory containing
+that manifest**, including `project-root`, output roots, adapters, protected
+roots and extension source files. CLI paths always resolve from the invoking
+current directory. `advisory-exclude` contains fixture-relative names, not host
+paths. Paths inside case files, extension command argv, source maps and runtime
+files retain their existing contracts. A manifest must be a regular,
+non-symlink JSON file of at most 64 KiB. Unsupported versions, unknown fields,
+invalid types and unreadable files fail before execution with an actionable
+file/field diagnostic and exit `64`.
+
+## Runtime, evidence and identity
+
+The existing [`sevro.json` runtime contract](runtime-v1.md) is unchanged.
+A run manifest selects recurring evaluation settings; `runtime-config-file`
+can select a separate runtime file. Otherwise Sevro still discovers exactly
+`<project-root>/sevro.json`. Runtime declarations do not become run options.
+
+Current CLI runs retain the effective preview as optional
+`configuration.resolved` in `sevro.run-evidence.v1`, with format
+`sevro.resolved-run-settings.v1`. It records selected case and extension command
+paths, source-closure count, mapping selection, adapters, routes, model/effort,
+trial settings, runtime file selection and operational output locations. It
+omits credentials and credential-option paths, private source-closure and mapping
+paths, private extension configuration, runtime environment values and native
+diagnostic contents. Explicit protected roots, paths and model labels are still operator metadata;
+inspect retained evidence before sharing it. The runtime policy's existing
+redacted evidence still retains explicitly declared non-secret environment
+values. Older evidence without `configuration.resolved` remains valid and
+does not acquire inferred settings.
+
+Assessment settings enter the [evaluation identity](identity-v1.md) through
+resolved cases, fixtures, checks, routes and `configuration.redacted`. The
+additional `configuration.redacted.runSettings` binds shell isolation, explicit
+protected roots, deprecated runtime selections, selected native timeouts and
+Claude project-settings authority. Equivalent saved and
+explicit selections have the same identity. Semantic changes such as model,
+effort, trial count, job limit or threshold change identity. The manifest's
+location, CLI output mode, and output/run-state paths are excluded from the
+semantic configuration digest; `configuration.resolved` is an operational
+snapshot outside that digest. Project content, including a tracked manifest,
+still follows the existing project identity rules.

package/docs/running-evaluations.md

@@ -0,0 +1,31 @@
+# Run an evaluation
+
+Begin with the [deterministic example](getting-started.md). For your own case,
+make [the installed command available](installing.md#make-the-command-available),
+then select a trusted adapter and absolute paths:
+
+```sh
+sevro run --json \
+  --case-file /absolute/path/case.json \
+  --adapter-module /absolute/path/host.ts \
+  --project-root /absolute/path/project \
+  --results-root /absolute/path/results \
+  --condition passive --trials 1 --threshold 1
+```
+
+For a contributor checkout, use `bun src/cli.ts` in place of `sevro`.
+An adapter is trusted executable code; injection does not automatically provide
+native isolation. `--project-root` identifies the evaluated project, not
+necessarily the runner checkout. `--results-root` receives evidence.
+Trials control repetition, and the threshold controls the required pass rate.
+
+A passive condition cannot request execution-changing instrumentation.
+`--dry` prepares without execution/grading: successful preparation is still
+`not_assessed`. For model routes follow [native hosts](native-hosts.md).
+An [extension command](creating-cases.md) and a direct case file are exclusive.
+Read [results](reading-results.md) to interpret failures and missing data.
+
+The [detailed CLI reference](development-cli.md) owns configuration roots,
+concurrency, cancellation, semantic/advisory routes, fixtures, and extra options.
+Preserve its limitations when composing commands. Cancellation stops admission
+and retains completed evidence before cleanup.