@bjoernrochel/sevro 0.1.0-rc.7 → 1.0.0

package/docs/results-v1.md

@@ -9,6 +9,17 @@ Each domain outcome also gets one human-readable line with its case, trial,
 outcome ID, and status. Domain outcomes remain independent of the task verdict
 and exit code.
 
+The CLI also shows trial states, bounded failed/unavailable
+check details, measured candidate duration and known/unknown usage in human
+completion. Actual preparation/candidate/grading/evidence stage messages use
+stderr in both modes; they do not assert task success. Dry execution/grading are
+explicitly skipped. JSON stdout stays unchanged. The CLI saves `result.json`
+beside retained run evidence and atomically publishes `<results-root>/latest.json`
+for [report access](report-v1.md). These output files stay inside excluded storage
+and do not change evaluation identity. A failure to retain CLI/latest output is
+reported as an internal error after an otherwise successful run; an earlier
+nonzero run outcome keeps its precedence and its evidence reference.
+
 ## Separate states
 
 Each trial and aggregate result carries:
@@ -98,6 +109,17 @@ persistence error.
 [`identity-v1.md`](identity-v1.md) defines the canonical digest and each
 comparison dimension.
 
+Current CLI runs additionally retain the redacted effective-settings preview as
+optional `configuration.resolved` (`sevro.resolved-run-settings.v1`), including
+saved-setting selections and resolved operational paths. This snapshot excludes
+credentials, credential-option paths, private source-closure/mapping paths,
+private extension configuration and runtime
+environment values; it is outside `configuration.digest`. Assessment settings
+remain bound by the existing identity inputs and `configuration.redacted`.
+Historical evidence without this optional field remains valid; missing settings
+stay unknown. See [saved run settings](run-settings-v1.md) for precedence and
+privacy limits.
+
 The versioned machine-readable contracts are
 [`cli-result-v1.schema.json`](../schemas/cli-result-v1.schema.json) and
 [`run-evidence-v1.schema.json`](../schemas/run-evidence-v1.schema.json). A
@@ -110,26 +132,118 @@ An interrupted or failed run keeps completed trials plus an explicit partial
 attempt or diagnostic record. Failed persistence is an internal error and
 cannot be presented as a successful assessment.
 
+Successful temporary workspaces are removed after their evidence is saved.
+Failed and cancelled workspaces, including trials with unavailable required
+grading evidence, are kept for 24 hours after becoming inactive. Evaluation
+startup removes expired workspaces with verified inactive ownership. Active or
+unverifiable workspaces and unregistered legacy folders are preserved; saved
+results and evidence remain intact. Failed cleanup is retried on later evaluations.
+
+`sevro.observation.workspaces` records candidate and retained auxiliary workspace
+paths for inspection. These paths are temporary and do not affect evaluation identity.
+
+Retained results have no automatic expiration. The explicit
+[`sevro prune` command](reading-results.md#prune-retained-results) applies an
+operator-selected age or count policy, defaulting to the newest ten results.
+It deletes complete retained result directories and maintains `latest.json`
+for the remaining results. Active or unrecognized attempts remain protected;
+fixture cleanup and run-state coordination metadata have separate lifecycles.
+
+Opted-in [candidate artifact runtimes](runtime-v1.md#candidate-artifact-access)
+remain available through grading and are removed afterward, including failed
+and cancelled trials. Failed-fixture retention does not preserve them.
+Read access does not automatically retain artifact bytes in run evidence.
+
+Checker private home, temporary storage, and copied dependency slots are removed
+after grading on success, failure, and cancellation. Retained failed fixtures and
+saved evidence do not retain checker runtime copies. Explicitly declared shared
+caches survive this cleanup and remain under operator control.
+
 A host adapter can return `executionFailed: true` with bounded artifacts from
 a failed process. The engine retains those artifacts, skips all grading, and
 reports `failed` / `not_requested` / `not_assessed` with exit code `2`. A final
 message that happens to match a check cannot turn that failure into a pass.
 Claude retains its bounded event stream and, when present, up to 64 KiB of
 stderr as private host artifacts. Failure bodies do not appear in CLI diagnostics.
+Codex preflight failures retain `codex.process` (stage, reason, exit code, signal,
+duration, and stderr truncation) and `codex.stderr` (the first 64 KiB), with role
+prefixes for semantic/advisory hosts. Diagnostics distinguish failure, timeout,
+and cancellation without exposing raw stderr.
+
+## Native host diagnostics
+
+The CLI's opt-in
+[`--retain-host-diagnostics`](development-cli.md#retain-native-host-diagnostics)
+retains candidate artifact `sevro.host.diagnostics` with format
+`sevro.host-diagnostics.v1`. Semantic and advisory requests use the existing
+`sevro.semantic[.<source digest>].` and `sevro.advisory.` artifact ID prefixes.
+The flag is independent of native transcript retention.
+
+The private JSON contains `host`, `role`, `stage`, `error`, `processes`,
+and `processesDropped`. Error evidence preserves the original name, message,
+stack and available error code, with up to eight error entries in the cause chain. Repeated
+or deeper causes end in an explicit `truncated` entry. Messages and stacks
+are bounded to 8,192 characters per entry. A host-returned failure without a
+thrown exception receives a runner failure summary; subprocess receipts and
+bounded output supply its native evidence.
+
+Each process receipt records stage, actual `code`, `signal`, `durationMs`,
+reason, stdout/stderr prefixes, and separate truncation markers. Unknown native
+exit information remains `null`; app-server's synthesized protocol result is
+not substituted for a process exit code. Each stream retains at most 64 KiB,
+and hosts continue consuming output after filling the diagnostic prefix.
+Stream parsing and decoding failures also retain available prefixes.
+Text uses replacement decoding for diagnostic inspection; it is not an exact
+native-byte archive.
+
+The journal keeps at most 32 recent process receipts and at most 2 MiB of
+encoded receipt data. It drops oldest receipts first and reports their count.
+The failing stage and bounded error remain independent of those dropped
+receipts. Known credentials from selected authentication files, forwarded
+Claude authentication variables, and common credential patterns are redacted
+from the diagnostic JSON. Sevro does not explicitly dump environment values,
+full argv, or settings into diagnostics. Treat retained diagnostics as private evidence and keep
+them out of Git; redaction is not authorization to publish subprocess output.
+
+The engine supplies an optional `retainHostDiagnostics(bytes)` callback to
+host adapters. Bundled opted-in hosts invoke it before removing native state.
+The callback saves the artifact with private permissions and registers its
+file URL and SHA-256 in trial evidence. Direct adapter calls without that
+callback return diagnostic bytes in ordinary host artifacts.
+
+Failure diagnostics name the retained file URL when it fits the CLI message
+limit; `evidencePath` and trial `artifactPath` resolve artifact references
+even when a message cannot contain the URL. Native failures retain the existing
+execution, grading, task and exit semantics. Failed diagnostic persistence is
+an internal runner error, not a successful assessment or an ordinary host
+failure. Cleanup errors do not replace an earlier host or diagnostic-persistence
+failure; cleanup warnings contain no raw error/output bodies.
 
 ## Native transcript bundles
 
 With [runtime transcript opt-in](runtime-v1.md#native-transcript-access), each
 bundled host captures full native files before removing its private state.
 Candidate artifact `sevro.native-transcripts.bundle` has format
-`sevro.native-transcripts.v1`. It contains `host` (`codex` or `claude`), `role`,
+`sevro.native-transcripts.v2`. It contains `host` (`codex` or `claude`), `role`,
 native `rootSessionId` (or `null`), `completeness`, `issues`, and `files`.
-Each file records its native relative `path`, SHA-256 `sha256`, and exact bytes
-as `bytesBase64`. Decode the bytes without rewriting native records. Native
+Each file records its native relative `path`, SHA-256 `sha256` of the original
+bytes, and `gzipBase64`: the exact file bytes compressed with gzip, then encoded
+as base64. Decode base64, decompress gzip, and verify the original-byte hash
+before using a file. Bound decompression to 5 MiB across all files in the bundle.
+Decode bytes without rewriting native records. Native
 session paths, root/child records and sidecar metadata preserve associations;
 the bundle does not normalize model, effort, timing, token, or cost statistics.
 Existing unknown metrics remain unknown.
 
+Sevro reads both v2 bundles and existing `sevro.native-transcripts.v1` bundles.
+V1 files use uncompressed `bytesBase64`; decode that field directly without
+gzip decompression and verify the same original-byte hash. Readers must select
+the payload field and decoding by `format` and reject unsupported formats,
+corrupt compressed payloads, hash mismatches, or a decoded total above 5 MiB.
+New captures emit v2; reading retained v1 evidence does not rewrite it.
+The retained view exposes the same exact native files for either format, with
+the original bundle in `index.json`.
+
 `sevro.host.native-transcripts` records the same capture state and file hashes
 without raw bytes. Semantic and advisory bundles retain their role in the
 bundle and use the existing `sevro.semantic[.<source digest>].` and
@@ -140,7 +254,8 @@ benchmarks can use the same references and decode the native files.
 
 Capture includes native transcript-tree sidecars and refuses descendant links,
 special files and escaping paths. It is bounded to 4,096 directory entries,
-5 MiB of native file bytes and 7 MiB of encoded file metadata/content. The
+5 MiB of native file bytes before compression and 7 MiB of encoded file
+metadata/gzip content. The
 bundle remains inside the existing 8 MiB item and 32 MiB host artifact budgets;
 it groups sessions instead of spending an artifact per child. Limits produce
 `byte_limit`, `entry_limit`, or `bundle_limit` issues. Unreadable, unsafe or
@@ -159,3 +274,25 @@ structure, not native semantic correctness.
 An empty/missing native JSONL tree is `unavailable`. Failure, cancellation and
 timeout retain available native bytes with incomplete capture state. Raw
 bundles and views are private evidence and must stay out of Git.
+
+The explicit [`sevro prune --native-transcripts` mode](reading-results.md#remove-native-transcript-bundles)
+can remove these bundles without rewriting CLI results, run/trial evidence,
+capture observations, native file hashes, or whole-bundle hashes. It retains
+other artifacts and does not change evaluation identity or assessment states.
+Reports validate present native bundles against their recorded SHA-256 and
+require an exact pruning receipt for absent bundles.
+
+Before deletion, Sevro atomically writes private
+`native-transcripts-pruned.json` beside `run.json`, with format
+`sevro.pruned-native-transcripts.v1`, the original `runId`, and `artifacts`.
+Each entry contains the original artifact reference (`id`, `path`, `sha256`,
+and any optional reference flags), plus its `caseId` and `trial`. Entries follow
+run trial/reference order and cover every candidate, semantic (including
+source-specific), and advisory native bundle in that run. A receipt must match
+these original references exactly; malformed, inconsistent, or linked receipts
+are errors. Bundle references must name regular files directly inside their
+result directory. Missing bytes without a receipt, changed hashes, and escaping
+references remain errors. An interrupted deletion may leave some marked bundles
+present; reports count actual availability, and a repeated prune completes removal.
+Transcript references that alias result/trial metadata, raw responses, or other
+retained artifacts are refused before pruning.

package/docs/run-settings-v1.md

@@ -0,0 +1,147 @@
+# Saved run settings v1
+
+Save recurring evaluation selections in a JSON run manifest, then use
+`sevro run --run-manifest-file ./evaluation.json`. The examples use the
+[installed command](installing.md#make-the-command-available).
+
+## Repeat an evaluation
+
+Start with an existing [resolved case and adapter](getting-started.md), and save
+this `evaluation.json` beside `case.json` and `host-adapter.ts`:
+
+```json
+{
+  "format": "sevro.run.v1",
+  "settings": {
+    "case-file": "./case.json",
+    "adapter-module": "./host-adapter.ts",
+    "project-root": ".",
+    "condition": "passive",
+    "trials": 3,
+    "jobs": 1,
+    "threshold": 1
+  }
+}
+```
+
+Run it twice with the same saved selections, or explicitly override one setting:
+
+```sh
+sevro run --run-manifest-file ./evaluation.json
+sevro run --run-manifest-file ./evaluation.json
+sevro run --run-manifest-file ./evaluation.json --trials 1 --json
+```
+
+Every run prints effective settings on stderr before candidate execution, then
+prints the result and evidence location. The example requires all assessed
+trials to pass. A missing assessment remains `not_assessed`. Use `--dry` to
+check preparation without calling the candidate or graders; dry preparation
+does not prove native authentication or task success.
+
+## Select candidate and grading routes once
+
+For native execution, replace the adapter selection with an explicit host,
+model and effort. Each grading role has its own selection. For example, a case
+with semantic checks can use this manifest:
+
+```json
+{
+  "format": "sevro.run.v1",
+  "settings": {
+    "case-file": "./semantic-case.json",
+    "project-root": ".",
+    "host": "codex",
+    "model": "<candidate-model>",
+    "effort": "medium",
+    "codex-auth-file": "../private/auth.json",
+    "semantic-host": "codex",
+    "semantic-model": "<grading-model>",
+    "semantic-effort": "high",
+    "trials": 3,
+    "threshold": 1
+  }
+}
+```
+
+Replace model placeholders with installed-host-supported model names and supply
+the [native prerequisites and authentication](native-hosts.md). Models and
+efforts are never inferred or copied between roles. Candidate Claude uses
+`host: "claude"` and its own explicit `model`/`effort`. An optional advisory
+route uses `advisory-host`, `advisory-model` and `advisory-effort`; advisory
+review requires a Git fixture and remains separate from the task verdict.
+Injected adapters continue to declare their own model and effort.
+
+## Format, precedence and paths
+
+The root has exactly `format: "sevro.run.v1"` and an object `settings`. Settings
+use the [CLI option names](development-cli.md) without the leading `--`.
+All run options can be saved except `json` and `run-manifest-file`. Keep output
+presentation explicit on the command line, including `--json` when collecting
+a versioned success or failure document. Manifests do not include other
+manifests and are read only when explicitly selected; there is no discovery.
+
+| Option kind                                                                             | JSON value                                            |
+| --------------------------------------------------------------------------------------- | ----------------------------------------------------- |
+| `trials`, `jobs`, `threshold`, `codex-timeout-ms`                                       | Number, with the CLI's existing ranges                |
+| `dry`, `shell-isolation`, `retain-host-diagnostics`, `claude-project-settings`          | Boolean                                               |
+| `extension-source-file`, `protected-root`, `replace-builtin-grader`, `advisory-exclude` | Array of nonempty strings; `[]` selects an empty list |
+| Other run options                                                                       | Nonempty string                                       |
+
+Precedence is **built-in defaults → saved settings → explicit CLI options**.
+Omitted settings retain the [documented defaults](development-cli.md#run-defaults).
+An explicit repeatable flag replaces its saved array in full; repeated CLI
+occurrences build the replacement array. `--no-dry`, `--no-shell-isolation`,
+`--no-retain-host-diagnostics` and `--no-claude-project-settings` override saved
+`true` values with `false`.
+
+Selecting an alternative route on the CLI removes its saved alternative and
+route-only settings. For example, `--adapter-module ./host-adapter.ts` replaces
+a saved native candidate and its model/effort. Selecting `--host claude` over
+a saved Codex candidate requires fresh explicit `--model` and `--effort`;
+grading roles keep their separate selections. Shared Codex settings remain
+while any Codex route is selected. CLI `--case-file` replaces a saved extension
+source and its extension-only options. CLI `--extension-command-file` replaces
+a saved case file; supply the extension's required ID and source closure.
+Supplying two exclusive alternatives explicitly remains an error.
+
+Paths saved in a manifest resolve from the **canonical directory containing
+that manifest**, including `project-root`, output roots, adapters, protected
+roots and extension source files. CLI paths always resolve from the invoking
+current directory. `advisory-exclude` contains fixture-relative names, not host
+paths. Paths inside case files, extension command argv, source maps and runtime
+files retain their existing contracts. A manifest must be a regular,
+non-symlink JSON file of at most 64 KiB. Unsupported versions, unknown fields,
+invalid types and unreadable files fail before execution with an actionable
+file/field diagnostic and exit `64`.
+
+## Runtime, evidence and identity
+
+The existing [`sevro.json` runtime contract](runtime-v1.md) is unchanged.
+A run manifest selects recurring evaluation settings; `runtime-config-file`
+can select a separate runtime file. Otherwise Sevro still discovers exactly
+`<project-root>/sevro.json`. Runtime declarations do not become run options.
+
+Current CLI runs retain the effective preview as optional
+`configuration.resolved` in `sevro.run-evidence.v1`, with format
+`sevro.resolved-run-settings.v1`. It records selected case and extension command
+paths, source-closure count, mapping selection, adapters, routes, model/effort,
+trial settings, runtime file selection and operational output locations. It
+omits credentials and credential-option paths, private source-closure and mapping
+paths, private extension configuration, runtime environment values and native
+diagnostic contents. Explicit protected roots, paths and model labels are still operator metadata;
+inspect retained evidence before sharing it. The runtime policy's existing
+redacted evidence still retains explicitly declared non-secret environment
+values. Older evidence without `configuration.resolved` remains valid and
+does not acquire inferred settings.
+
+Assessment settings enter the [evaluation identity](identity-v1.md) through
+resolved cases, fixtures, checks, routes and `configuration.redacted`. The
+additional `configuration.redacted.runSettings` binds shell isolation, explicit
+protected roots, deprecated runtime selections, selected native timeouts and
+Claude project-settings authority. Equivalent saved and
+explicit selections have the same identity. Semantic changes such as model,
+effort, trial count, job limit or threshold change identity. The manifest's
+location, CLI output mode, and output/run-state paths are excluded from the
+semantic configuration digest; `configuration.resolved` is an operational
+snapshot outside that digest. Project content, including a tracked manifest,
+still follows the existing project identity rules.