@bjoernrochel/sevro 0.1.0-rc.11 → 0.1.0-rc.13

package/CONTRIBUTING.md

@@ -38,6 +38,13 @@ bun run test:docs-examples
 ```
 
 The second command checks the tutorial in fresh consumer/contributor directories.
+Its complete recipe checks also require UV, Python 3, mise, and an existing
+`mise install node@24.13.0` installation; see [recipe prerequisites](docs/recipes.md).
+CI explicitly provisions these tools. Missing prerequisites fail with a named
+setup instruction rather than skipping a recipe.
+To run recipe commands from this source checkout, set
+`export SEVRO_RECIPES="$PWD/examples/recipes"` from the repository root, then
+follow the [recipe commands](docs/recipes.md#prepare-a-recipe-project).
 Native model trials need their own [credentials and prerequisites](docs/native-hosts.md).
 
 ## Quality contract and checks
@@ -68,6 +75,10 @@ Before committing, use `bun run check:typescript --fast` after installing the
 locked dependencies. This runs inventory, schema, formatting, lint, typing and
 documentation checks; it does not replace full acceptance. CI retains machine-readable coverage
 reports under `.quality/` and exposes the stable `TypeScript quality` status.
+The complete command's `static`, `docs`, `tests` and `package` phases run as
+concurrent CI jobs. After frozen installation, use
+`bun run check:typescript --phase <name>` to run one phase locally; it does not
+replace acceptance across all four. See the [phase inventory](docs/typescript-quality.md).
 
 Run `bun run check:docs:external` separately and [triage network failures](docs/documentation-quality.md#external-links).
 Guide changes also need [native evals](docs/guide-evaluation.md).
@@ -84,8 +95,8 @@ repairing in-scope failures; it does not authorize merging or unrelated changes.
   non-root user, and isolation prerequisites. Reproducing that environment locally
   with Docker is optional for diagnosis and is not a publication prerequisite.
 - After the local pre-commit checks, publish the authorized candidate to its PR
-  and require `bun run check:typescript` to succeed in GitHub CI before reporting
-  delivery as complete. A local run, fast check, targeted suite, or historical
+  and require all four canonical `check:typescript` phases to succeed in GitHub CI
+  before reporting delivery as complete. A local run, fast check, targeted suite, or historical
   result does not establish that the current Ubuntu gate passes.
 - Keep the quality bar intact. Repair behavior or add meaningful coverage for
   accepted behavior; do not lower thresholds, exempt authored sources, skip tests,

package/README.md

@@ -40,6 +40,11 @@ Continue with [your first evaluation](docs/getting-started.md): the exact comman
 expected `passed` result, and retained evidence. Read [installation](docs/installing.md)
 for updates, removal, and source checkouts.
 
+The development checkout and freshly built packages provide a
+[demo-to-iteration workflow](docs/evaluation-workflow.md): initialize real
+assertions, prepare, execute, read progress and inspect the latest result.
+The published `rc.11` package uses the installed tutorial above.
+
 ## Find your next task
 
 The [documentation hub](docs/README.md) routes to running evaluations, authoring

package/docs/README.md

@@ -4,19 +4,22 @@
 
 Start with the task you want to complete. These pages work without an agent.
 
-| I want to…                                    | Read                                                     |
-| --------------------------------------------- | -------------------------------------------------------- |
-| Get a result without a model account          | [Your first evaluation](getting-started.md)              |
-| Install, update, remove, or use a checkout    | [Installation](installing.md)                            |
-| Run evaluations and select a host             | [Running evaluations](running-evaluations.md)            |
-| Create cases and reusable policy              | [Cases and extensions](creating-cases.md)                |
-| Interpret verdicts, evidence, and comparisons | [Reading results](reading-results.md)                    |
-| Run Codex or Claude with isolation            | [Native hosts](native-hosts.md)                          |
-| Reuse host tools or enable goals and hooks    | [Runtime configuration](runtime-v1.md)                   |
-| Resolve a symptom                             | [Troubleshooting](troubleshooting.md)                    |
-| Understand runner responsibilities            | [Architecture](architecture.md)                          |
-| Change Sevro or its documentation             | [Contributing](../CONTRIBUTING.md)                       |
-| Understand licensing and branding             | [Licensing](licensing.md) and [assets](assets/README.md) |
+| I want to…                                     | Read                                                     |
+| ---------------------------------------------- | -------------------------------------------------------- |
+| Get a result without a model account           | [Your first evaluation](getting-started.md)              |
+| Initialize, prepare, run and iterate           | [Evaluation workflow](evaluation-workflow.md)            |
+| Install, update, remove, or use a checkout     | [Installation](installing.md)                            |
+| Run evaluations and select a host              | [Running evaluations](running-evaluations.md)            |
+| Create cases and reusable policy               | [Cases and extensions](creating-cases.md)                |
+| Copy a complete coding, skill or runtime setup | [Runnable recipes](recipes.md)                           |
+| Interpret verdicts, evidence, and comparisons  | [Reading results](reading-results.md)                    |
+| Run Codex or Claude with isolation             | [Native hosts](native-hosts.md)                          |
+| Reuse host tools or enable goals and hooks     | [Runtime configuration](runtime-v1.md)                   |
+| Save recurring evaluation selections           | [Run settings](run-settings-v1.md)                       |
+| Resolve a symptom                              | [Troubleshooting](troubleshooting.md)                    |
+| Trace runner lifecycle, ownership, and changes | [Architecture](architecture.md)                          |
+| Change Sevro or its documentation              | [Contributing](../CONTRIBUTING.md)                       |
+| Understand licensing and branding              | [Licensing](licensing.md) and [assets](assets/README.md) |
 
 ## Ask the guide
 
@@ -35,6 +38,7 @@ There is no accepted Rust migration decision here;
 [issue #2](https://github.com/BjRo/sevro/issues/2) tracks research.
 
 Read [documentation quality](documentation-quality.md),
+[the usability study protocol](usability-study.md),
 [guide evaluation](guide-evaluation.md), [releases](releases.md), and
 [acknowledgements](acknowledgements.md) for maintenance.
 Return to the [Sevro introduction](../README.md).

package/docs/architecture.md

@@ -4,22 +4,143 @@ Sevro owns generic execution/evidence contracts. Projects own cases and policy.
 This lets Darrow and other integrations use the same runner without importing
 their benchmark policy into it.
 
-| Responsibility                                    | Sources                                                                                                            |
-| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
-| Parse commands/select explicit routes             | [CLI](../src/cli.ts), [CLI reference](development-cli.md)                                                          |
-| Prepare, execute, grade, retain, clean up         | [Engine](../src/engine.ts), [engine slice](engine-slice.md)                                                        |
-| Negotiate trusted extension calls                 | [Client](../src/extension-client.ts), [session](../src/extension-session.ts), [protocol](extension-protocol-v1.md) |
-| Run native hosts and retain bounded observations  | [Codex](codex-host.md), [native controls](native-controls.md), [host sources](../src/hosts/)                       |
-| Reduce execution/grading/task states              | [Result source](../src/results.ts), [contract](results-v1.md)                                                      |
-| Bind comparison identities and release provenance | [Identity](identity-v1.md), [releases](releases.md)                                                                |
-| Summarize retained results                        | [Report](report-v1.md)                                                                                             |
-
-Completed evidence is retained before cleanup. Missing observations remain
-unavailable; unknown measurements do not become zero. Extensions cannot
-silently replace isolation/persistence or built-in policy. The
+This page explains this checkout's implementation and points to acceptance
+tests for changes. [Current contracts](contracts.md) define public behavior;
+[historical validation](evidence.md) records bounded past observations. The
+[engine slice](engine-slice.md) retains its earlier scope and detailed lifecycle
+explanation. Implementation paths and test scenarios here are navigation, not
+evidence that a particular candidate passed validation.
+
+[`engine.ts`](../src/engine.ts) remains the programmatic facade, and
+[`cli.ts`](../src/cli.ts) remains the installed command entrypoint. The CLI
+loads and assembles evaluation options; evaluation modules coordinate the run
+and its trials.
+
+## Evaluation lifecycle
+
+[`runEvaluation`](../src/evaluation/run.ts) prepares inputs before allocating
+run resources, retains preparation artifacts, and reserves candidate and
+semantic workspace roots as needed before calling the trial scheduler. The map
+follows the path of a retained trial; individual stages can be skipped or fail.
+
+```mermaid
+flowchart TD
+    P["Validate and prepare evaluation inputs"] --> R["Allocate run storage and owner; reserve workspace roots"]
+    R --> W["Admit trial and materialize its reserved fixture"]
+    subgraph Trial["Each admitted trial"]
+        W --> C["Initialize grading bases; execute candidate"]
+        C --> E["Retain candidate artifacts; observe output; prepare grading views"]
+        E --> G["Output, shell, Git, semantic, and extension grading"]
+        G --> A["Assess execution, grading, and task separately"]
+        A --> V["Optional independent advisory review"]
+        V --> T["Retain raw candidate response; atomically write trial evidence"]
+        T --> K["Collect completed trial and checkpoint its reference"]
+        K --> D["Release temporary views; clear or retain fixture contents"]
+    end
+    D --> J["All admitted trials drain"]
+    J --> F["Write final run evidence; finalize owner checkpoint"]
+    F --> X["Finalize retained workspaces; release other reservations"]
+```
+
+[`scheduleTrials`](../src/evaluation/scheduler.ts) bounds active work by the
+prepared job limit. It admits a trial before
+[`prepareTrialWorkspace`](../src/evaluation/resources.ts) materializes that
+trial's fixture. Each [`EvaluationTrial`](../src/evaluation/trial.ts) owns its
+mutable state and writer. Its `run()` method keeps candidate execution,
+artifact retention, grading, assessment, advisory review, persistence, and
+cleanup in one visible order. Grading helpers return checks, observations,
+and failures; the coordinator incorporates those results and stops new
+admission when an execution or required grading error is detected, before
+advisory work, persistence, or cleanup. The scheduler also stops admission on
+cancellation and waits for active workers. A failed task check alone does not
+close admission. Already admitted peers drain and retain their evidence, including
+when they finish out of order.
+
+[`runAdvisoryReview`](../src/evaluation/advisory.ts) handles its own host call,
+assessment, retained response, and auxiliary workspace cleanup. It depends on
+candidate completion, independently of the task assessment. The
 [advisory fixture](advisory-fixture.md) gives review a separate Git view with
-evaluator inputs withheld.
+evaluator inputs withheld. Its recommendation or host failure does not change
+task grading; a failure to retain its evidence remains a persistence error.
+
+Successful [semantic response parsing](../src/evaluation/semantic.ts) clears
+auxiliary fixture contents; successful advisory response parsing removes its
+auxiliary fixture. Both retain response evidence first, including valid failed
+check verdicts or advisory recommendations.
+
+The [trial writer](../src/evidence/trial-writer.ts) writes the trial artifact
+before the coordinator marks it persisted and hands it to the
+[result collector](../src/evaluation/collection.ts). The collector registers
+the completed record, orders records by trial number, and checkpoints retained
+artifact paths. After active work drains, the run coordinator validates and
+writes `run.json` before finalizing the owner as complete or interrupted.
+Persistence failures cannot produce a successful result; the run's failure
+path attempts a diagnostic checkpoint while preserving the original failure.
+
+Trial cleanup releases candidate artifact runtimes and transcript views even
+when the trial fails. Persisted successful fixture contents are cleared while
+their empty reserved roots remain until active work drains. Failed, cancelled,
+grading-error, unavailable-evidence, and failed-task fixtures are retained;
+[workspace ownership](../src/fixtures/workspace-state.ts) tracks their inactive
+retention and startup expiry after 24 hours. Saved evidence stays outside that
+cleanup. See the [results retention contract](results-v1.md#identity-and-provenance)
+for the public boundary, including ownership that cannot be verified.
+
+Dry preparation still validates and materializes fixtures, including declared
+fixture setup, and retains trial records. It skips candidate execution and
+grading, recording `not_run` / `not_requested` / `not_assessed`, as exercised by
+the [engine acceptance suite](../tests/engine.test.ts). Missing observations
+remain unavailable; missing required evidence cannot become a pass. Optional
+unknown measurements remain unknown rather than becoming zero. Extensions
+cannot silently replace isolation, persistence,
+or built-in policy. The [results](results-v1.md) and
+[extension protocol](extension-protocol-v1.md) contracts define those semantics.
+
+## State ownership
+
+[`EvaluationContext`](../src/evaluation/context.ts) carries four explicit
+collaborators: prepared inputs, run resources, the result collector, and retained
+preparation-artifact references. It does not combine mutable trial fields with
+the preparation plan.
+
+| Data or state                  | Owner                                                                                                                                                           | Boundary                                                                                                                                                                                                                                                                          |
+| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Prepared inputs                | [`PreparedEvaluation` in `plan.ts`](../src/evaluation/plan.ts)                                                                                                  | Named fixture, check, and invocation groups plus routes, provenance, redacted configuration, and comparison identity. Preparation completes before run allocation.                                                                                                                |
+| Run resources and reservations | [`RunResources` and `RunWorkspaceReservations` in `resources.ts`](../src/evaluation/resources.ts)                                                               | Run ID, evidence/state locations, process owner, and candidate/semantic reservations. Reservations outlive completed fixture contents; retained workspaces are tracked separately from completed roots.                                                                           |
+| Mutable trial state            | [`TrialRuntimeState` and `EvaluationTrial` in `trial.ts`](../src/evaluation/trial.ts)                                                                           | Candidate result, observations, checks, assessment, temporary views, advisory result, and persisted milestone belong to one admitted trial. Grading receives narrow inputs and explicit effects rather than this state object.                                                    |
+| Collected run results          | [`ResultCollector` in `collection.ts`](../src/evaluation/collection.ts)                                                                                         | Private completed summaries, evidence, and diagnostics; trial-number ordering, snapshots for aggregation, and checkpoint requests. [`run-owner.ts`](../src/run-owner.ts) verifies process ownership and writes owner/checkpoint state.                                            |
+| Evidence writes                | [`TrialEvidenceWriter`](../src/evidence/trial-writer.ts), [preparation retention](../src/evaluation/resources.ts), [run finalization](../src/evaluation/run.ts) | The trial writer owns filenames, private writes, hashes, artifact references, and typed trial-persistence failures. Resource helpers retain preparation bytes; run finalization writes aggregate evidence. [`storage.ts`](../src/storage.ts) supplies atomic private JSON writes. |
+
+Fixture modules own materialization and workspace preparation. Runtime modules
+own declared host runtime policy and private state. Evidence modules validate
+host observations and artifacts, verify retained bytes, and normalize optional
+usage without constructing fixtures. Native host implementations and graders
+remain in their own families.
+
+## Where to change behavior
+
+Start with the responsibility, then inspect its observable acceptance scenarios.
+These are useful starting suites, not an exhaustive validation list. The
+[contribution checks](../CONTRIBUTING.md#quality-contract-and-checks) still govern
+candidate acceptance.
+
+| Responsibility                                                    | Implementation and reference                                                                                                                                                                                                                      | Relevant acceptance tests                                                                                                                                                                                         |
+| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Parse commands/select explicit routes                             | [CLI entrypoint](../src/cli.ts), [invocation parsing](../src/cli/invocation.ts), [CLI reference](development-cli.md)                                                                                                                              | [Invalid invocations](../tests/cli/invocation-validation.test.ts)                                                                                                                                                 |
+| Load and prepare a CLI evaluation                                 | [Input loading](../src/cli/loading.ts), [preparation](../src/cli/preparation.ts), [host construction](../src/cli/hosts.ts), [extension resolution](../src/cli/extension.ts), [evaluation assembly](../src/cli/evaluation.ts)                      | [Case/configuration validation](../tests/cli/case-validation.test.ts)                                                                                                                                             |
+| Present CLI results and own cancellation                          | [Presentation](../src/cli/presentation.ts), [cancellation](../src/cli/cancellation.ts), [command lifecycle](../src/cli/command.ts)                                                                                                                | [CLI states and exits](../tests/cli/results.test.ts), [signal retention](../tests/cli/trial-retention.test.ts)                                                                                                    |
+| Prepare evaluation inputs and grade trials                        | [Engine facade](../src/engine.ts), [plan](../src/evaluation/plan.ts), [grading](../src/evaluation/grading.ts), [semantic grading](../src/evaluation/semantic.ts), [evaluation modules](../src/evaluation/), [engine slice](engine-slice.md)       | [Preparation refusals](../tests/evaluation/preparation-validation.test.ts), [semantic grading failures](../tests/evaluation/semantic-grading-failures.test.ts), [engine lifecycle](../tests/engine.test.ts)       |
+| Schedule trials, retain evidence, and finalize ownership          | [Scheduler](../src/evaluation/scheduler.ts), [trial coordinator](../src/evaluation/trial.ts), [reservations](../src/evaluation/resources.ts), [collection](../src/evaluation/collection.ts), [run owner](../src/run-owner.ts)                     | [Concurrent admission and ordering](../tests/cli-concurrency.test.ts), [trial failures and retention](../tests/evaluation/trial-retention.test.ts), [checkpoint refusals](../tests/run-owner-checkpoints.test.ts) |
+| Prepare, construct, and clean up fixtures                         | [Fixture modules](../src/fixtures/), [construction](../src/fixtures/create.ts), [preparation artifacts](../src/fixtures/preparation.ts), [content cleanup](../src/fixtures/cleanup.ts), [workspace ownership](../src/fixtures/workspace-state.ts) | [Artifact preparation](../tests/fixtures/artifact-preparation.test.ts), [workspace lifecycle](../tests/fixtures/workspace-lifecycle.test.ts)                                                                      |
+| Resolve runtime policy, discovery, and private state              | [Runtime modules](../src/runtime/), [configuration](../src/runtime/config.ts)                                                                                                                                                                     | [Runtime environment](../tests/runtime/environment.test.ts), [candidate artifact lifetime](../tests/runtime/candidate-artifacts.test.ts)                                                                          |
+| Validate host results and retain bounded evidence                 | [Host results](../src/evidence/host-results.ts), [trial writer](../src/evidence/trial-writer.ts), [retained artifacts](../src/evidence/retained-artifacts.ts), [usage](../src/evidence/usage.ts)                                                  | [Host evidence refusals](../tests/evaluation/host-evidence.test.ts), [failed-host evidence retention](../tests/evaluation/trial-retention.test.ts)                                                                |
+| Negotiate trusted extension calls                                 | [Client](../src/extension-client.ts), [session](../src/extension-session.ts), [extension grading](../src/evaluation/grading.ts), [protocol](extension-protocol-v1.md)                                                                             | [Negotiated engine behavior](../tests/extension-engine.test.ts), [extension grading outcomes](../tests/evaluation/extension-grading.test.ts)                                                                      |
+| Run independent advisory review                                   | [Advisory execution](../src/evaluation/advisory.ts), [blind fixture](../src/fixtures/advisory.ts), [advisory fixture explanation](advisory-fixture.md)                                                                                            | [Advisory failures](../tests/evaluation/advisory-failures.test.ts), [blind change view](../tests/advisory-fixture.test.ts)                                                                                        |
+| Run native hosts and retain bounded observations                  | [Codex](codex-host.md), [native controls](native-controls.md), [host sources](../src/hosts/), [shared process mechanics](../src/process-lifecycle.ts), [native waiting](../src/hosts/process-lifecycle.ts)                                        | [Native process admission and byte bounds](../tests/hosts/native-process-boundaries.test.ts)                                                                                                                      |
+| Reduce execution/grading/task states and maintain contract shapes | [Result reduction](../src/results.ts), [shared semantic types](../src/contract-types.ts), [wire types](../src/schema-types.ts), [runtime schema boundary](../src/schema.ts), [results contract](results-v1.md)                                    | [State and exit precedence](../tests/results.test.ts), [evidence schema boundaries](../tests/schema/evidence-boundaries.test.ts)                                                                                  |
+| Bind comparison identities and release provenance                 | [Preparation identity inputs](../src/evaluation/plan.ts), [identity hashing](../src/identity.ts), [provenance](../src/provenance.ts), [identity contract](identity-v1.md), [releases](releases.md)                                                | [CLI identity](../tests/cli/identity.test.ts), [package release provenance](../tests/release.test.ts)                                                                                                             |
+| Summarize retained results                                        | [Report implementation](../src/report.ts), [report contract](report-v1.md)                                                                                                                                                                        | [Report states and CLI](../tests/report.test.ts)                                                                                                                                                                  |
 
-These paths describe this checkout. [Historical validation](evidence.md) records
-bounded past observations. [Rust research](https://github.com/BjRo/sevro/issues/2)
-does not establish a migration; public interfaces remain useful independently.
+[Rust research](https://github.com/BjRo/sevro/issues/2) does not establish a
+migration; public interfaces remain useful independently. Return to the
+[documentation hub](README.md).

package/docs/builtin-graders-v1.md

@@ -38,8 +38,10 @@ The shell-check process runner accepts a bounded evaluator-owned `run` string,
 optional `expectedExitCode` and `timeoutMs`, plus `expectExact`, `expectRegex`,
 `notRegex`, and regex `flags` for stdout. Exact matching removes one final
 newline; regex matching always uses multiline mode. The runner executes with
-`sh -e`. On macOS and Linux it runs through the outer sandbox with network access denied,
-a credential-free environment, and caller-declared protected roots. Without a
+`sh -e`. On macOS and Linux it runs through the outer sandbox with network access denied
+by default, a credential-free environment, and caller-declared protected roots.
+The runtime [local HTTP option](runtime-v1.md#local-http-test-servers) permits
+socket-using upstream tests while retaining outbound isolation. Without a
 stdout assertion, process output is discarded. With one, the runner captures
 at most 1 MiB in memory and retains only its digest and byte length. Oversized
 or invalid UTF-8 output is a grading error.

package/docs/codex-host.md

@@ -31,8 +31,8 @@ because its parent directory belongs to a mounted skill; an existing mounted
 target that fails or produces incomplete bytes remains partial. A completed direct
 `cat` of one mounted `SKILL.md`, alone or alongside literal documentation
 paths, counts only when the command output contains that file's exact body.
-Batched reads naming multiple `SKILL.md` files or more than 64 paths remain
-partial, as do failed, malformed, or incomplete recognized reads. Direct
+Batched reads naming more than 64 paths remain partial, as do failed,
+malformed, or incomplete recognized reads. Direct
 `sed -n` pages count after their verified line ranges cover the whole body. An
 exact `cat` wrapped by `lean-ctx -c` is accepted only when its reported output
 still contains the full body; a compressed summary cannot prove the read.
@@ -47,18 +47,36 @@ cat .agents/skills/darrow-guide/SKILL.md; cat docs/specs/repository-guide.md; rg
 ```
 
 This sequence uses the shared stream and native read check. Quoted separators
-remain part of their literal operand. The initial skill body must match the
-beginning of the retained output, so later documentation or search output cannot
-replace an altered or incomplete initial read. Subsequent reads must not name
-another `SKILL.md` or resolve to a mounted or receipt-bound skill through an alias.
+remain part of their literal operand. The ordered file bodies must match the
+retained output, so documentation or search output cannot replace an altered
+or incomplete skill read. Literal `cat` commands may read multiple mounted
+skills and documentation, before or after skills, with or without a terminal
+search. Multiple literal terminal `rg` commands are accepted within the command
+bound. Each verified skill receives its own ordered receipt. Documentation
+must resolve within the workspace or a receipt-bound installed plugin root;
+each file is bounded to 1 MiB. Aliases of mounted skills without a literal
+`SKILL.md` operand remain unsupported.
 The 64-operand bound applies across all reads in the sequence. Shell expansions,
 including unquoted glob and tilde operands, are unsupported in this form.
 
+The bounded composed-read grammar also accepts `unset` with literal variable
+names, `pwd` or `pwd -P`, `git rev-parse --show-toplevel` (optionally with
+`-C "$PWD"`), `printenv` with literal variable names, and `rg --files` with
+literal arguments before the reads. Prefix `printenv` requires the reads to
+end the command, so environment output cannot replace missing read bytes.
+Literal `printenv` commands may follow reads without prefix `printenv`.
+The fixed pair `repo=$(git -C "$PWD" rev-parse --show-toplevel)` and
+`cd "$repo" && pwd -P` is supported only with absolute read operands. Other
+assignments, substitutions, directory changes, or control flow remain
+unsupported. Prefix and environment output cannot supply skill frontmatter;
+ambiguous repeated output cannot establish an attributable read.
+
 The check also accepts a chain of literal
 `cat` fallbacks with `2>/dev/null` and `||`, ending in `true` followed by a newline
 and one literal `rg` discovery command. A fallback chain must resolve to one
 canonical file across all its read operands; absent alternate targets do not
-invalidate the read. Multiple existing targets, including unmounted copies,
+invalidate the read. Each fallback `cat` names one literal file. Multiple
+existing targets, including unmounted copies,
 cannot establish an attributable read. The total read operands remain bounded
 at 64. The complete command must succeed and expose the mounted file's entire
 exact body. `cat` accepts literal file operands and the optional `--` delimiter;
@@ -99,6 +117,10 @@ The host also retains separate `sevro.codex.initial-skill-reads` and
 `sevro.codex.follow-up-skill-reads` observations. Each applies the same exact
 mounted-skill read check to one completed turn, so an extension can grade read
 order or a missing repeat read without inferring it from the combined list.
+For `app-server`, the verified same-thread feedback boundary separates the
+initial phase from the follow-up phase, including subsequent native goal turns.
+Absent, duplicated, foreign-thread, or failed boundaries do not yield phase
+receipts.
 
 It also checks native parent command records for exact mounted skill bodies,
 including sessions with no spawn request. The bounded diagnostic distinguishes
@@ -137,12 +159,20 @@ discarded. Extensions interpret task names and decide whether a nested reader
 satisfies their policy.
 The record proves an invocation attempt, not its success. A missing, ambiguous,
 unreadable, oversized, or malformed session is unavailable or partial, so its
-absence cannot prove that a control was unused. Calls made through submitted
-code are not classified by this first observation.
+absence cannot prove that a control was unused. Code Mode submissions now have
+separate bounded `execCalls` inspection receipts, described in the
+[native control contract](native-controls.md). Direct native call ordinals and
+acceptance receipts keep their original meaning; submitted requests do not
+acquire direct native acceptance receipts.
 For an explicitly declared `$plugin:skill`, the host verifies that the installed
 skill exists and that the exact token occurs once in the delivered prompt. A
-completed turn yields `sevro.codex.explicit-invocation` with the owner first and
-verified supporting reads after it. Incomplete reads make that receipt partial.
+completed turn yields a complete `sevro.codex.explicit-invocation` with the owner
+first and verified supporting reads after it. This receipt proves verified
+owner dispatch independently of file reads; it does not prove that all
+supporting skills were read. Consumers requiring complete supporting-read
+evidence must also require `sevro.codex.skill-reads` (or the applicable phase
+receipt) to be complete. Failed execution cannot yield a complete dispatch
+receipt.
 The `sevro.codex.repository-invocation` capability supports an explicitly
 declared `$skill` from `.agents/skills/<skill>/SKILL.md`. The host checks that
 the repository mount is a bounded regular file with no symlink components and
@@ -158,7 +188,11 @@ symlinks, and installs it with `codex plugin marketplace add` followed by
 `codex plugin add`. It checks each installation receipt against the private
 plugin cache. Candidate commands can read that cache, while the rest of the
 private Codex home, including authentication and configuration, remains denied.
-It refuses fixture-local `.codex` configuration, unsupported enforced
+With the [Codex state opt-in](runtime-v1.md#candidate-codex-configuration-and-user-skills),
+commands may read generated configuration and write actual non-system user skills;
+graders retain read-only access to that request's original paths. Fixture-local
+configuration is accepted as inert represented input on this route. Otherwise,
+the host refuses fixture-local `.codex` configuration. It also refuses unsupported enforced
 conditions, unreadable or oversized auth files, malformed event streams, and
 incomplete turns. The host bounds output and runtime, kills its process group
 on timeout or cancellation, and removes private state after the turn. It is
@@ -180,14 +214,18 @@ explicit workspace rule, denies all other filesystem roots except minimal
 runtime files and the selected executable's install directories, and
 specifically denies protected roots and the private Codex home. An executable
 inside a protected root is rejected. The profile also disables command
-network access and clears inherited command environment variables. The Codex
+network access by default and clears inherited command environment variables.
+The runtime [local HTTP option](runtime-v1.md#local-http-test-servers) enables
+Codex's managed network proxy and local binding for socket-using tests.
+The Codex
 parent can read authentication before it starts sandboxed commands. Tests
 invoke the installed `codex sandbox` CLI to prove fixture access, denied
 source/auth reads, and executable access. The process adapter verifies these
 properties in one sandbox invocation before each fresh host request, with a
 60-second bounded startup deadline. The combined probe still requires readable
 private command storage, denied project listing, denied authentication and
-configuration reads, and a successful native executable `--version`. Continuation
+configuration reads (unless configuration access was explicitly enabled), and a
+successful native executable `--version`. Continuation
 turns reuse the same isolated state. The longer deadline accommodates observed
 native policy startup; it is not counted as a performance saving.
 
@@ -207,3 +245,8 @@ yielded chunk count, and separate `completedCall` and `literalCommandCall`
 flags. These facts distinguish recovery sources without retaining commands,
 submitted code, or skill bodies. The diagnostic's completeness and truncation
 fields still determine whether its read evidence can support grading.
+
+[Issue #70 representative retests](evidence/skill-reads-70/results.json) retain
+separate task, activation, and phase-read outcomes from the private candidate,
+plus its source fingerprints and test-first regression results. Private native
+transcripts remain outside the repository.

package/docs/contracts.md

@@ -7,6 +7,7 @@ Read these human contracts and their schemas together:
 - [Report v1](report-v1.md): summaries and comparison limitations.
 - [Identity v1](identity-v1.md): versioned comparison dimensions and hashing.
 - [Runtime configuration v1](runtime-v1.md): environment, tools, private state, goals, and hooks.
+- [Run settings v1](run-settings-v1.md): recurring evaluation selections and CLI overrides.
 - [Built-in graders v1](builtin-graders-v1.md): output, shell, Git HEAD, and semantic checks.
 - [Schemas](../schemas/): machine-readable public forms.
 

package/docs/creating-cases.md

@@ -4,18 +4,25 @@ Start with a direct case for a prompt, fixture, and built-in success criteria.
 Copy [the basic graded case](../examples/basic/graded.json), give it a stable
 ID, and replace the prompt/check. Read [built-in graders](builtin-graders-v1.md)
 for regex, JSON, and bounded schema assertions. No criteria means `not_assessed`.
+For complete coding, repository-skill, runtime, semantic and extension projects,
+follow the [runnable recipes](recipes.md), including their deliberate failures.
 
 Run it with the [CLI](running-evaluations.md). Direct cases follow `ResolvedCase`
 in [the engine](../src/engine.ts). Fixtures can provide inline files, generated
 Git history, or a declared clean repository. The [CLI reference](development-cli.md)
 owns required source roots/maps; an arbitrary returned path does not grant access.
 
+Selecting `--project-root` identifies the project; it does not copy its contents
+into a participant fixture. Declare the files or repository source the participant
+should receive. The [run defaults reference](development-cli.md#run-defaults)
+explains how project identity, fixtures and CLI path resolution interact.
+
 ## Evaluate your repository skills
 
 Start with one representative task, your actual skill, and observable success
 criteria. This example evaluates a documentation guide's answer with Codex.
 Use [Sevro from `PATH`](installing.md#make-the-command-available).
-Live runs require the [native host prerequisites](native-hosts.md), macOS
+Live runs require the [native host prerequisites](native-hosts.md), supported
 isolation, an authenticated Codex installation, and your chosen model.
 They consume that account's model quota.
 
@@ -109,7 +116,7 @@ prepares skill mounts and interprets complete native host evidence. The
 [extension protocol](extension-protocol-v1.md) defines repository and plugin
 invocations; the checkout's [guide evals](guide-evaluation.md) demonstrate the
 approach. Check those capabilities against your exact installed release:
-this checkout contains [changes beyond published `rc.2`](installing.md#published-package-and-checkout).
+this checkout can contain [changes beyond its published version](installing.md#published-package-and-checkout).
 
 ## Reusable project policy
 

package/docs/development-cli.md

@@ -11,11 +11,99 @@ grading, evidence, and extension lifecycle.
 Examples assume [Sevro and Bun are on `PATH`](installing.md#make-the-command-available).
 For a contributor checkout, use `bun src/cli.ts` in place of `sevro`.
 
+Use `sevro --help`, `sevro run --help`, or `sevro report --help` for examples,
+required options, and defaults without preparing a run. `sevro --version` prints
+the installed package version. Invocation and configuration diagnostics go to
+stderr in both output modes; `--json` keeps one `sevro.cli-result.v1` document
+on stdout. Invalid case diagnostics name the input file and offending field.
+
+<a id="run-defaults-in-this-checkout"></a>
+
+## Run defaults
+
+`sevro demo` owns a deterministic case, fixture and adapter. It needs no model
+credentials and immediately prints an assessed verdict plus an evidence path in
+`.sevro/results` beneath the invoking directory. Its temporary project and
+configuration are independent of the caller's files and runtime policy.
+
+An ordinary run needs a case source and candidate route. For example:
+
+```sh
+sevro run --case-file ./case.json --adapter-module ./host-adapter.ts
+```
+
+| Setting                                                     | Default                                                                                                                 |
+| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
+| Condition                                                   | `passive`                                                                                                               |
+| Trial count                                                 | `1`                                                                                                                     |
+| Pass threshold                                              | `1`: all assessed trials must pass; any unassessed trial prevents a pass                                                |
+| Parallel trial limit (`--jobs`)                             | `3`                                                                                                                     |
+| Project root                                                | Invoking current directory                                                                                              |
+| Results root                                                | `<project-root>/.sevro/results`                                                                                         |
+| Configuration root                                          | Project root                                                                                                            |
+| Run-state root                                              | Results root                                                                                                            |
+| Runtime policy                                              | Discover `sevro.json` at project root; none when absent                                                                 |
+| Native binary                                               | Resolve only the explicitly selected `codex` or `claude` from `PATH`; `--codex-bin` / `--claude-bin` override it        |
+| Native execution timeout                                    | `600000` ms (10 minutes); Codex accepts `--codex-timeout-ms`                                                            |
+| Codex entrypoint                                            | `exec`; a native-goal runtime policy selects `app-server`                                                               |
+| Codex agent concurrency                                     | `agents.max_concurrent_threads_per_session` from `.codex/config.toml` at configuration root, otherwise the host default |
+| Shell-check expected exit                                   | `0` unless the check declares `expectedExitCode`                                                                        |
+| Shell-check timeout                                         | `30000` ms unless the check declares `timeoutMs` (maximum `120000`)                                                     |
+| Shell isolation, dry preparation, host diagnostic retention | Off                                                                                                                     |
+
+Every run file/directory option accepts a relative path based on the invoking
+current directory, including adapter modules, extension source files, native
+binary overrides and protected roots. Sevro resolves them to canonical absolute
+paths before constructing protection lists; missing output-directory suffixes
+resolve through their nearest existing parent. A different `--project-root`
+does not change that base. The default results directory is relative to the
+selected project root. Paths inside runtime policy files retain their
+[runtime-specific resolution rules](runtime-v1.md).
+Extension command JSON still requires an absolute executable argv; arbitrary
+argv entries are not treated as CLI path options.
+
+Sevro prints a whitelist of effective run settings and routes on stderr before
+candidate execution, including resolved storage paths, native executables,
+timeouts and concurrency. JSON stdout remains one result document. The preview
+does not print credential bytes, runtime environment values or native diagnostics.
+The resolved executable is retained as `sevro.host.executable` in each used native
+role's redacted host configuration; selecting a different path changes that
+configuration's identity.
+
+Use `sevro init` with `--template response`, `coding` or
+`repository-skill` and `--directory <new-path>`. It refuses existing destinations
+and writes runnable assertions, a manifest and template instructions. Follow the
+[demo-to-iteration workflow](evaluation-workflow.md) for preparation, execution
+and negative controls. Repository-skill requires explicit native model, effort
+and authentication; dry preparation does not establish dispatch.
+
+Actual preparation, candidate, grading and evidence-retention stage events use
+stderr. Human completion includes each trial's separate states, failed-check
+reasons, measured candidate time, known/unknown usage and retained paths. JSON
+stdout remains one result. Each retained CLI run writes `result.json` beside run
+evidence and `<results-root>/latest.json`. `sevro report --latest` reads the current
+directory's default results root; add `--results-root <path>` for another root.
+Dry and failed runs can be latest.
+
+Project root identifies the evaluated project and extension discovery. It does
+not populate the participant workspace: declare inline files, generated history
+or a repository source in the case's fixture. See [case authoring](creating-cases.md).
+Keep `.sevro/` out of Git. Results and run-state paths are excluded from project
+snapshots and untracked Git identity; tracked source changes still count.
+
+## Explicit run settings
+
+To repeat these selections, use the versioned [saved run settings](run-settings-v1.md)
+with `--run-manifest-file ./evaluation.json`. Built-in defaults are replaced by
+saved settings, then by explicit CLI options. `sevro.json` keeps its separate
+runtime scope. Saved file paths use the manifest's directory; explicit CLI
+paths use the invoking directory.
+
 An installed package records its name, version, and a digest of its packaged
 runtime and public contract files without reading runner Git metadata. The CLI
 derives the project digest from its revision and dirty patch, or from a bounded
 content snapshot when the project has no Git revision. Result and run-state
-directories are excluded from that snapshot. When running this source checkout
+directories are excluded from that snapshot and untracked Git identity. When running this source checkout
 for coordinated development, pass `--runner-checkout-root` with the absolute
 path to this checkout. Sevro verifies that it is the code actually running and
 records its revision and dirty-patch digest instead of package provenance.
@@ -67,8 +155,7 @@ Add `--retain-host-diagnostics` to a failing invocation to retain private
 diagnostics from bundled Codex and Claude hosts. The switch defaults to off,
 applies to every selected native role, and is independent of runtime
 `nativeTranscripts`. It does not enable transcript access or change the
-candidate's permissions. This switch is available in the development checkout;
-the published `0.1.0-rc.8` does not accept it.
+candidate's permissions.
 
 For example, add the switch to the Codex command above:
 
@@ -239,7 +326,8 @@ For cases with `sevro.semantic` checks, also pass
 explicit second `HostAdapter` for grading. Alternatively, select the bundled
 Codex route with `--semantic-host codex --semantic-model <model>
 --semantic-effort <effort>`. That route also requires `--codex-bin` and
-`--codex-auth-file`, shared with a Codex candidate route when present. The two
+`--codex-auth-file`, shared with a Codex candidate route when present. The binary
+defaults to the explicitly selected `codex` on `PATH`. The two
 semantic route options are exclusive. The candidate and semantic routes are
 recorded separately in evidence and evaluation identity. The semantic adapter
 runs after a complete candidate response in its own empty workspace; missing
@@ -293,7 +381,7 @@ verdict and rejects unsolicited recommendations.
 
 If the extension needs configuration, supply both
 `--extension-configuration-file` and
-`--extension-redacted-configuration-file` as absolute paths to JSON objects.
+`--extension-redacted-configuration-file` as paths to JSON objects.
 The latter replaces secret values for the retained configuration digest. Keep
 secrets out of the command argv and the redacted file.
 
@@ -323,7 +411,13 @@ isolated shell checks receive `.git/fixture-bin` on `PATH`. Host adapters
 receive `fixtureBinDir` and must add it to their native tool environment; the
 bundled Codex adapter also preserves it across login shells. A negotiated
 fixture setup command may then prepare the clone before the host runs.
-Repository submodules and preparation artifacts targeting `.git/` are refused.
+Repository fixtures preserve committed `.gitmodules` files and submodule gitlink
+revisions. Trial clones leave submodules uninitialized and do not fetch their
+contents, even when the source has initialized submodules. A fixture that needs
+those contents must initialize them during its negotiated setup. Modified or
+untracked files and revision changes in initialized source submodules are
+refused, including when Git's submodule ignore settings hide them. Preparation
+artifacts targeting `.git/` are refused.
 
 A direct case or extension can declare a generated Git fixture:
 
@@ -415,6 +509,3 @@ Claude also advertises `sevro.host.native-goal`. Its observer binds one persiste
 transcript to the original successful native session and retains goal status
 and objective length. Missing, ambiguous, malformed or oversized evidence is
 unavailable; an unavailable observation cannot prove goal absence.
-
-These additions require the `0.1.0-rc.2` candidate. The published `rc.1` lacks
-the app-server selector and native goal observations.