@bjoernrochel/sevro 0.1.0-rc.2 → 0.1.0-rc.21

package/docs/development-cli.md

@@ -1,20 +1,114 @@
 # Development CLI
 
-The local `sevro run` command connects a resolved JSON case or an explicit
-extension to a host route.
-It uses the engine's normal grading and evidence path. This remains a
-development entrypoint while the full extension lifecycle and production
-provenance collection are being built.
+The `sevro run` command connects a resolved JSON case or an explicit extension
+to a host route. The installed CLI and source entrypoint use the same engine
+grading, evidence, and extension lifecycle.
 
 `sevro report` reads one or more retained JSON results through the
 [versioned report contract](report-v1.md). It emits Markdown by default or
 `sevro.report.v1` with `--json`.
 
+`sevro prune` [removes older retained results](reading-results.md#prune-retained-results)
+and maintains the latest report index. Bare pruning keeps the newest ten results;
+`--keep`, `--older-than`, and `--latest` select explicit retention policies.
+
+Examples assume [Sevro and Bun are on `PATH`](installing.md#make-the-command-available).
+For a contributor checkout, use `bun src/cli.ts` in place of `sevro`.
+
+Use `sevro --help`, `sevro run --help`, `sevro report --help`, or
+`sevro prune --help` for examples,
+required options, and defaults without preparing a run. `sevro --version` prints
+the installed package version. Invocation and configuration diagnostics go to
+stderr in both output modes; `--json` keeps one `sevro.cli-result.v1` document
+on stdout. Invalid case diagnostics name the input file and offending field.
+
+<a id="run-defaults-in-this-checkout"></a>
+
+## Run defaults
+
+`sevro demo` owns a deterministic case, fixture and adapter. It needs no model
+credentials and immediately prints an assessed verdict plus an evidence path in
+`.sevro/results` beneath the invoking directory. Its temporary project and
+configuration are independent of the caller's files and runtime policy.
+
+An ordinary run needs a case source and candidate route. For example:
+
+```sh
+sevro run --case-file ./case.json --adapter-module ./host-adapter.ts
+```
+
+| Setting                                                     | Default                                                                                                                 |
+| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
+| Condition                                                   | `passive`                                                                                                               |
+| Trial count                                                 | `1`                                                                                                                     |
+| Pass threshold                                              | `1`: all assessed trials must pass; any unassessed trial prevents a pass                                                |
+| Parallel trial limit (`--jobs`)                             | `3`                                                                                                                     |
+| Project root                                                | Invoking current directory                                                                                              |
+| Results root                                                | `<project-root>/.sevro/results`                                                                                         |
+| Configuration root                                          | Project root                                                                                                            |
+| Run-state root                                              | Results root                                                                                                            |
+| Runtime policy                                              | Discover `sevro.json` at project root; none when absent                                                                 |
+| Native binary                                               | Resolve only the explicitly selected `codex` or `claude` from `PATH`; `--codex-bin` / `--claude-bin` override it        |
+| Native execution timeout                                    | `600000` ms (10 minutes); Codex accepts `--codex-timeout-ms`                                                            |
+| Codex entrypoint                                            | `exec`; a native-goal runtime policy selects `app-server`                                                               |
+| Codex agent concurrency                                     | `agents.max_concurrent_threads_per_session` from `.codex/config.toml` at configuration root, otherwise the host default |
+| Shell-check expected exit                                   | `0` unless the check declares `expectedExitCode`                                                                        |
+| Shell-check timeout                                         | `30000` ms unless the check declares `timeoutMs` (maximum `120000`)                                                     |
+| Shell isolation, dry preparation, host diagnostic retention | Off                                                                                                                     |
+
+Every run file/directory option accepts a relative path based on the invoking
+current directory, including adapter modules, extension source files, native
+binary overrides and protected roots. Sevro resolves them to canonical absolute
+paths before constructing protection lists; missing output-directory suffixes
+resolve through their nearest existing parent. A different `--project-root`
+does not change that base. The default results directory is relative to the
+selected project root. Paths inside runtime policy files retain their
+[runtime-specific resolution rules](runtime-v1.md).
+Extension command JSON still requires an absolute executable argv; arbitrary
+argv entries are not treated as CLI path options.
+
+Sevro prints a whitelist of effective run settings and routes on stderr before
+candidate execution, including resolved storage paths, native executables,
+timeouts and concurrency. JSON stdout remains one result document. The preview
+does not print credential bytes, runtime environment values or native diagnostics.
+The resolved executable is retained as `sevro.host.executable` in each used native
+role's redacted host configuration; selecting a different path changes that
+configuration's identity.
+
+Use `sevro init` with `--template response`, `coding` or
+`repository-skill` and `--directory <new-path>`. It refuses existing destinations
+and writes runnable assertions, a manifest and template instructions. Follow the
+[demo-to-iteration workflow](evaluation-workflow.md) for preparation, execution
+and negative controls. Repository-skill requires explicit native model, effort
+and authentication; dry preparation does not establish dispatch.
+
+Actual preparation, candidate, grading and evidence-retention stage events use
+stderr. Human completion includes each trial's separate states, failed-check
+reasons, measured candidate time, known/unknown usage and retained paths. JSON
+stdout remains one result. Each retained CLI run writes `result.json` beside run
+evidence and `<results-root>/latest.json`. `sevro report --latest` reads the current
+directory's default results root; add `--results-root <path>` for another root.
+Dry and failed runs can be latest.
+
+Project root identifies the evaluated project and extension discovery. It does
+not populate the participant workspace: declare inline files, generated history
+or a repository source in the case's fixture. See [case authoring](creating-cases.md).
+Keep `.sevro/` out of Git. Results and run-state paths are excluded from project
+snapshots and untracked Git identity; tracked source changes still count.
+
+## Explicit run settings
+
+To repeat these selections, use the versioned [saved run settings](run-settings-v1.md)
+with `--run-manifest-file ./evaluation.json`. Built-in defaults are replaced by
+saved settings, then by explicit CLI options. `sevro.json` keeps its separate
+runtime scope. Saved file paths use the manifest's directory; explicit CLI
+paths use the invoking directory.
+
 An installed package records its name, version, and a digest of its packaged
 runtime and public contract files without reading runner Git metadata. The CLI
 derives the project digest from its revision and dirty patch, or from a bounded
 content snapshot when the project has no Git revision. Result and run-state
-directories are excluded from that snapshot. When running this source checkout
+directories are excluded from that snapshot and untracked Git identity. When running this source checkout
 for coordinated development, pass `--runner-checkout-root` with the absolute
 path to this checkout. Sevro verifies that it is the code actually running and
 records its revision and dirty-patch digest instead of package provenance.
@@ -22,7 +116,7 @@ records its revision and dirty-patch digest instead of package provenance.
 temporary project without runner Git metadata.
 
 ```sh
-bun src/cli.ts run --json \
+sevro run --json \
   --case-file /absolute/path/case.json \
   --adapter-module /absolute/path/host-adapter.ts \
   --project-root /absolute/path/project \
@@ -35,7 +129,7 @@ chosen by the operator. The runner does not load it from a case file or infer
 it from an installed extension. A bundled Codex route is also available:
 
 ```sh
-bun src/cli.ts run --json \
+sevro run --json \
   --case-file /absolute/path/case.json \
   --host codex --codex-bin /absolute/path/codex \
   --codex-auth-file /absolute/path/auth.json \
@@ -52,6 +146,61 @@ only passive conditions. `--host` and `--adapter-module` are exclusive. Use
 `--protected-root` to add private roots for Codex, including when no shell
 checks are selected.
 
+Use `--codex-timeout-ms <milliseconds>` to set the native Codex execution
+timeout. It accepts an integer from `1` through `1800000` (30 minutes) and
+defaults to `600000` (10 minutes) when omitted. It applies to every selected
+Codex route, including semantic and advisory roles, and to both `exec` and
+`app-server` entrypoints. For a 20-minute app-server evaluation, add
+`--codex-entrypoint app-server --codex-timeout-ms 1200000` to the Codex command
+above. The option requires a Codex host route; invalid values fail before a run.
+For `app-server`, this is the only deadline for the whole session, including
+startup requests such as `thread/start`. If it expires while a request is
+unanswered, the failure names that request, for example
+`App-server evaluation time bound reached awaiting thread/start`.
+
+### Retain native host diagnostics
+
+Add `--retain-host-diagnostics` to a failing invocation to retain private
+diagnostics from bundled Codex and Claude hosts. The switch defaults to off,
+applies to every selected native role, and is independent of runtime
+`nativeTranscripts`. It does not enable transcript access or change the
+candidate's permissions.
+
+For example, add the switch to the Codex command above:
+
+```sh
+sevro run --json \
+  --case-file /absolute/path/case.json \
+  --host codex --codex-bin /absolute/path/codex \
+  --codex-auth-file /absolute/path/auth.json \
+  --model <model> --effort medium \
+  --project-root /absolute/path/project \
+  --results-root /absolute/path/results \
+  --condition passive --trials 1 --threshold 1 \
+  --retain-host-diagnostics
+```
+
+Failed preparation or execution retains `sevro.host.diagnostics`, including
+when no native session or final response exists. Its private JSON records the
+failing stage, original error/cause chain, and recent subprocess receipts.
+Receipts contain actual exit code or signal, monotonic duration, bounded
+stdout/stderr prefixes, and truncation markers. App-server protocol failure is
+separate from its process exit code.
+
+The CLI's failure diagnostic names the artifact's file URL when it fits the
+diagnostic length limit. The result's `evidencePath` and each trial's
+`artifactPath` always lead to retained artifact references; semantic and
+advisory artifacts carry their existing role prefixes. Files have private
+permissions and are persisted before native state cleanup. The
+[results contract](results-v1.md#native-host-diagnostics) defines the format,
+bounds, redaction, and failure semantics.
+
+Enabled native roles record `sevro.host.retain-diagnostics: true` in redacted
+host configuration. This changes configuration identity, so a diagnostic rerun
+is not an exact-repeat comparison with an invocation that omitted the switch.
+Injected adapters own their diagnostic behavior; the CLI switch configures the
+bundled native routes.
+
 ### Trial concurrency
 
 `--jobs <positive integer>` bounds simultaneous trials within one case and
@@ -106,10 +255,10 @@ The same worktree protection applies to other declared source and private roots;
 a package without Git metadata remains usable. Result and active-run storage
 stay independent through `--results-root` and `--run-state-root`.
 
-A bundled Claude Code candidate route is available on macOS:
+A bundled Claude Code candidate route is available on macOS and Linux:
 
 ```sh
-bun src/cli.ts run --json \
+sevro run --json \
   --case-file /absolute/path/case.json \
   --host claude --claude-bin /absolute/path/claude \
   --model sonnet --effort medium \
@@ -132,11 +281,16 @@ values are excluded from retained evidence and configuration digests. Pass
 `--claude-credential-file` with an absolute path to select a separate saved login.
 It accepts declared Claude
 plugin directories and explicit skill invocations from extension preparation.
-It runs candidate tools under Claude Code's native macOS sandbox with private,
-source, and result paths denied. The route disables hooks and does not expose
+It runs candidate tools under Claude Code's native sandbox with private,
+source, and result paths denied. The route disables hooks by default and does not expose
 the Write tool. It currently supports passive candidate turns only. Use
 `--protected-root` to exclude additional private paths from the candidate's
-filesystem access. For plugins that run locked UV backends, use
+filesystem access. Configure shared host tools, private or reusable caches, goals, and selected
+plugin hooks with [`sevro.json` or `--runtime-config-file`](runtime-v1.md).
+The same policy reaches isolated checks and native grading hosts.
+
+The deprecated options remain available when no runtime file is selected.
+For plugins that run locked UV backends, use
 `--claude-uv-cache-dir /absolute/path/to/curated-cache` to copy a prepared UV
 cache into the trial's Git-private runtime directory before the turn. Use
 `--toolchain-bin-dir /absolute/path/to/bin` for a compatible Python and other
@@ -160,9 +314,10 @@ Without that option, the host does not advertise
 `sevro.claude.repository-invocation` and refuses explicit repository dispatch
 before launching Claude. This avoids recording an unknown command as a usable
 repository-skill route.
-The host reads only bounded Skill metadata from completed child Agent sessions
-in its temporary private configuration, then removes those session files. A
-missing or inconsistent graph leaves `sevro.claude.nested-skills` partial.
+The host collects successful `Skill` calls through the existing structured
+stream as [skill activation evidence](native-hosts.md#skill-activation-evidence).
+It uses the same collector for parent and nested actors. Missing or unmatched
+results remain partial; separate transcript-based skill detectors are retired.
 
 A case's `followUpPrompt` starts a UUID-bound session and resumes that same
 session after a complete successful initial result. Both results must name
@@ -181,7 +336,8 @@ For cases with `sevro.semantic` checks, also pass
 explicit second `HostAdapter` for grading. Alternatively, select the bundled
 Codex route with `--semantic-host codex --semantic-model <model>
 --semantic-effort <effort>`. That route also requires `--codex-bin` and
-`--codex-auth-file`, shared with a Codex candidate route when present. The two
+`--codex-auth-file`, shared with a Codex candidate route when present. The binary
+defaults to the explicitly selected `codex` on `PATH`. The two
 semantic route options are exclusive. The candidate and semantic routes are
 recorded separately in evidence and evaluation identity. The semantic adapter
 runs after a complete candidate response in its own empty workspace; missing
@@ -235,7 +391,7 @@ verdict and rejects unsolicited recommendations.
 
 If the extension needs configuration, supply both
 `--extension-configuration-file` and
-`--extension-redacted-configuration-file` as absolute paths to JSON objects.
+`--extension-redacted-configuration-file` as paths to JSON objects.
 The latter replaces secret values for the retained configuration digest. Keep
 secrets out of the command argv and the redacted file.
 
@@ -265,7 +421,13 @@ isolated shell checks receive `.git/fixture-bin` on `PATH`. Host adapters
 receive `fixtureBinDir` and must add it to their native tool environment; the
 bundled Codex adapter also preserves it across login shells. A negotiated
 fixture setup command may then prepare the clone before the host runs.
-Repository submodules and preparation artifacts targeting `.git/` are refused.
+Repository fixtures preserve committed `.gitmodules` files and submodule gitlink
+revisions. Trial clones leave submodules uninitialized and do not fetch their
+contents, even when the source has initialized submodules. A fixture that needs
+those contents must initialize them during its negotiated setup. Modified or
+untracked files and revision changes in initialized source submodules are
+refused, including when Git's submodule ignore settings hide them. Preparation
+artifacts targeting `.git/` are refused.
 
 A direct case or extension can declare a generated Git fixture:
 
@@ -289,9 +451,9 @@ fixture digest.
 
 The case file follows the `ResolvedCase`
 interface in `src/engine.ts`; this slice accepts inline files, generated Git
-history, or a declared repository, built-in output and semantic checks, and
-isolated shell checks. Add `{{sevro.workspace}}` to a case prompt when the host
-needs the trial's absolute fixture path. Sevro resolves it after fixture
+history, or a declared repository, built-in output, semantic, and Git HEAD
+checks, and isolated shell checks. Add `{{sevro.workspace}}` to a case prompt
+when the host needs the trial's absolute fixture path. Sevro resolves it after fixture
 creation for each trial and retains the template in evaluation identity. For
 cases with two participant turns, set `followUpPrompt` to a nonempty second
 prompt. Sevro renders the workspace token in both prompts and requires a host
@@ -311,8 +473,8 @@ For shell checks, add
 `--shell-isolation` and repeat `--protected-root /absolute/path` for every
 additional source worktree or private root. The engine always protects the
 selected project, results, runner source, user home, configured host homes,
-and active peer fixtures. Shell checks require macOS `sandbox-exec` in this
-development slice.
+and active peer fixtures. Shell checks require macOS `sandbox-exec` or Linux
+`bubblewrap`.
 Injected host adapters may return namespaced observations and bounded evidence
 artifacts. A case can list their IDs in `requiredEvidence`; missing or partial
 host evidence produces unavailable grading rather than a passing task. The
@@ -325,9 +487,10 @@ Its retained trials record `not_run` / `not_requested` / `not_assessed` and exit
 successfully when preparation succeeds. Dry and executed runs have different
 evaluation identities.
 Run evidence and trial files live under `--results-root`. A packaged runner
-needs no Git checkout. This private development checkout records its Git
-revision and dirty content. `--runner-build-digest` and `--project-digest`
-remain optional explicit overrides for test fixtures and coordinated
+needs no Git checkout. Source runs also use package provenance by default;
+pass `--runner-checkout-root` with the absolute path to the running checkout
+to record its Git revision and dirty content. `--runner-build-digest` and
+`--project-digest` remain optional explicit overrides for test fixtures and coordinated
 development; normal CLI runs derive both values.
 Use `--run-state-root /absolute/path/state` to keep active records and trial
 checkpoints separate from results; it defaults to `--results-root`. Shell
@@ -356,6 +519,3 @@ Claude also advertises `sevro.host.native-goal`. Its observer binds one persiste
 transcript to the original successful native session and retains goal status
 and objective length. Missing, ambiguous, malformed or oversized evidence is
 unavailable; an unavailable observation cannot prove goal absence.
-
-These additions require the `0.1.0-rc.2` candidate. The published `rc.1` lacks
-the app-server selector and native goal observations.

package/docs/documentation-quality.md

@@ -0,0 +1,61 @@
+# Documentation quality
+
+Every page should answer one reader need: tutorial, how-to, reference, or
+explanation. Landing pages can route to multiple modes. Lead with the outcome,
+use plain language, and keep commands, prerequisites, limits, and recovery steps.
+
+## Local checks
+
+Run `bun run check:docs`. The Markdown parser checks local targets/anchors,
+image alternatives, asset references, fence languages, and exact guide mirrors.
+It validates the guide inventory's source/static destinations and case coverage.
+It ignores code contents as links and excludes inactive fixture snapshots.
+Local checks need no network. Run `bun run test:docs-examples` for the exact
+marked tutorial commands in fresh consumer/contributor directories, including
+expected verdicts and saved evidence. Candidate and released-package runs are
+reported separately. These commands create and clean temporary directories.
+The [recipe verifier](../scripts/documentation-recipes.ts) runs the new packaged
+projects separately from the current published version. It executes actual
+isolated tool/check positive and negative controls and both native hosts' dry
+preparation; authenticated model, plugin-hook and goal behavior remain
+unexecuted by that check. Its [prerequisites](recipes.md#prepare-a-recipe-project)
+are mandatory, including UV and mise's declared Node installation.
+
+`package.json.version` is the authoritative current version. `bun run docs:sync`
+updates only the marked [Current release](installing.md#current-release) section
+with its exact installation command. Keep its markers and heading in place;
+the command refuses missing, duplicate, or reversed markers and invalid versions.
+The rest of the page stays ordinary Markdown. `check:docs` compares this section
+without writing; release preparation and package-install validation also reject
+stale output. Run synchronization after a version bump and include the resulting
+documentation change. Link installation and update instructions to this section.
+Keep historical versions, licensing transitions, and minimum supported versions
+fixed and explain their purpose beside each reference.
+Link to the current-release section instead of repeating the current Sevro
+version elsewhere. Describe features for the latest version, using the installed
+`sevro` command in user workflows. Keep source-checkout instructions in contributor
+guidance and omit notes about feature unavailability in older releases.
+
+## External links
+
+Run `bun run check:docs:external` separately. Transient failures are retried
+twice. The report names the referring page and URL. Fix permanent missing
+destinations; triage authentication walls, rate limits, timeouts, and bot blocks
+as network limits and retry later. Do not change a valid destination just to
+make a network job green. External success does not establish a claim's truth.
+
+## Human review
+
+- Can a new user reach installation, first success, results, troubleshooting,
+  architecture, and contributing without the guide?
+- Are supported environments, credentials, side effects, and prerequisites
+  explicit beside the relevant commands?
+- Do descriptive links, meaningful headings, source order, readable tables,
+  image alternatives, and words rather than color support accessibility?
+- Are current contracts, accepted choices, research, and past evidence distinct?
+- Are unique knowledge and existing destinations preserved?
+- Do source citations support material guide claims? Are unknowns/conflicts visible?
+
+Guide changes need [native evals](guide-evaluation.md). A structural check is
+not proof of selection or a substitute for human review. #1's
+[TypeScript quality contract](https://github.com/BjRo/sevro/issues/1) owns code standards.

package/docs/evaluation-workflow.md

@@ -0,0 +1,110 @@
+# From a demo to your next evaluation
+
+Create an evaluation, prepare it, run its assertions, and inspect the latest
+result with the installed `sevro` command.
+
+## Try the demo and create a project
+
+You need Bun 1.3.13. [Install Sevro](installing.md) and
+[make its command available on `PATH`](installing.md#make-the-command-available),
+then run:
+
+```sh
+sevro demo
+sevro init --template response --directory ./response-evaluation
+cd response-evaluation
+```
+
+The demo returns an assessed `passed` task and evidence under `.sevro/results`,
+using its own temporary project without credentials. The initializer refuses
+existing destinations, including directories and symbolic links. It writes
+real assertions, a deterministic adapter, `run.json` and template instructions.
+[Saved settings](run-settings-v1.md) use their manifest's directory for paths;
+`sevro.json` retains its separate runtime scope. Keep private `.sevro/` out of Git.
+
+## Prepare before execution
+
+```sh
+sevro run --run-manifest-file run.json --dry
+```
+
+Stderr shows the whitelisted resolved settings before candidate execution:
+case/source selection, routes/model/effort, trials, threshold, concurrency,
+runtime selection and canonical storage paths. Invalid input names its option,
+file or field; missing executables name installation or explicit override paths.
+Correct the named prerequisite and repeat preparation.
+
+Dry preparation follows normal fixture, extension and check preparation without
+calling the candidate or graders. Expect
+`execution=not_run grading=not_requested task=not_assessed`. It does not establish
+authentication, provider model availability, native skill dispatch or model success.
+See [native prerequisites](native-hosts.md) and [troubleshooting](troubleshooting.md).
+
+## Run, read and change one thing
+
+```sh
+sevro run --run-manifest-file run.json
+sevro report --latest
+SEVRO_TEMPLATE_FAIL=1 sevro run --run-manifest-file run.json
+sevro report --latest
+```
+
+The first run's adapter returns exactly `ready`; its regex assertion must pass.
+The negative control returns `wrong`: expect exit `1`, completed execution and
+grading, task `failed`, and the named `ready` check's reason. Then edit `case.json`
+and `adapter.ts`, prepare again and rerun. Explicit flags override saved settings;
+`--trials 3` repeats the task without rewriting `run.json`.
+
+Progress uses stderr at actual preparation, candidate, grading and evidence
+retention boundaries. Trial numbers distinguish concurrent work, which may
+interleave. Stage `completed` means the attempt finished; the result states whether
+execution/grading succeeded. Dry runs explicitly skip candidate and grading;
+failed candidates skip grading. Failed preparation never prints candidate start.
+Evidence completion follows persistence. Optional synchronous engine observers
+cannot change the result through a thrown exception.
+
+Human completion shows per-trial states, failed/unavailable check reasons,
+measured candidate time, known/unknown usage/cost and `result=`/`evidence=` paths.
+Candidate time measures the host call, not total setup time. Absent metrics remain
+`unknown`; dry duration is unknown. Prompt-only completion stays `not_assessed`.
+Add `--json` to emit one `sevro.cli-result.v1` stdout document; progress remains on
+stderr. See [reading results](reading-results.md).
+
+Each retained CLI evaluation saves `result.json` beside its run evidence and
+atomically publishes `<results-root>/latest.json`. Latest is the last retained CLI
+result, including dry runs and failed assessments. Pre-run failures do not replace
+it. Concurrent runs publish in completion order. Output files stay under excluded
+results storage, preserving project identity. A latest-write failure cannot turn
+an earlier failed assessment into success.
+
+`report --latest` reads `<cwd>/.sevro/results/latest.json`. Use
+`report --latest --results-root ./other-results` for another output root or working
+directory. `--result-file ./path/to/result.json` selects older runs and accepts
+relative paths. Reports validate referenced evidence; missing/moved evidence
+produces a diagnostic. To reclaim native bundle space while keeping reportable
+results and hashes, use [`sevro prune --native-transcripts`](reading-results.md#remove-native-transcript-bundles).
+Reports label bundles intentionally removed by that command as pruned.
+[Report v1](report-v1.md) defines fields and limits.
+
+## Choose another starter
+
+| Template           | Real assertions                                                                                                    | Prerequisites and interpretation                                                                           |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
+| `response`         | Exact `ready` response; negative control returns `wrong`                                                           | Bun; no model credentials                                                                                  |
+| `coding`           | Edit `add.sh`; isolated checks test positive, negative and zero addition                                           | Bun plus `sandbox-exec` on macOS or `bubblewrap`/`socat` on Linux; negative control leaves subtraction bug |
+| `repository-skill` | Actual `.agents/skills/startup-guide/SKILL.md` mount; explicit invocation evidence; correct start command and port | Bun, Git, Codex, native isolation, explicit model/effort and readable auth file; dry is unassessed         |
+
+From a new parent directory, use `sevro init --template coding --directory ./coding-eval`
+or `sevro init --template repository-skill --directory ./skill-eval`. Read its generated
+README for controls and dry/live commands. Repository-skill `command.json` records
+absolute Bun/extension argv; regenerate it after moving the project.
+
+For model evaluations, replace a saved adapter route with explicit `--host`,
+`--model`, `--effort` and host authentication options. Candidate, semantic and
+advisory model choices remain independent. Deterministic adapters measure wiring;
+they do not measure model ability. Native runs may incur provider charges.
+Automated package checks run response/coding pass/fail and native skill preparation;
+authenticated native behavior and human usability remain unmeasured. The
+[study protocol](usability-study.md) records deferred human measurements.
+[Runnable recipes](recipes.md) cover Claude, plugins/hooks, goals, semantic grading
+and host tools.

package/docs/evidence/code-mode-controls-71.md

@@ -0,0 +1,64 @@
+# Code Mode control evidence for issue 71
+
+On 2026-10-14, the implementation for [issue 71](https://github.com/BjRo/sevro/issues/71)
+replayed the four retained complete Codex native transcript bundles named in
+that ticket through the changed Sevro observers. This is artifact rechecking,
+not a fresh authenticated model trial. The original task checks and activation
+outcomes were preserved separately from the new control-absence assessment.
+Private native bundles, commands and output remain outside Git.
+
+| Case                                   | Inspected exec submissions | Submitted tool calls | Rechecked control absence | Original task checks    | Original activation |
+| -------------------------------------- | -------------------------- | -------------------- | ------------------------- | ----------------------- | ------------------- |
+| guide-mutation                         | 2                          | 4                    | passed                    | 7 passed, 1 unavailable | passed              |
+| guide-orchestration                    | 3                          | 6                    | passed                    | 8 passed, 1 unavailable | passed              |
+| code-review-empty-diff                 | 6                          | 8                    | passed                    | 6 passed, 1 unavailable | passed              |
+| code-review-reviewer-route-unavailable | 11                         | 12                   | passed                    | 6 passed, 1 unavailable | unavailable         |
+
+The recheck found no submitted agent launch or prohibited goal mutation in any
+case. All submission receipts had complete inspection and matching native
+ordinals. No accepted-agent count was substituted for invocation evidence.
+The older Darrow consumer still requires zero submitted exec calls; it needs
+to adopt the [receipt validation contract](../native-controls.md) to grade
+these new observations. Its original unavailable outcomes were not rewritten.
+
+The reviewer-route case's first read used a literal Git environment reset and
+repository-root query before its final mounted skill read. Its retained output
+ended with the exact 20,627-character skill body still present in the Darrow
+validation checkout. Rehydrating that independently matching mount and rebasing
+its expired private path let the changed skill-read observer establish a
+complete `code-review` receipt. This rechecked skill-file receipt is separate
+from the original unavailable activation outcome; no historical trial was
+changed to passed.
+
+Durable Bun regressions exercise the host's public observations for completed
+and rejected submissions, opaque code, and the repository-prefixed skill read.
+Additional real session/file regressions cover malformed/truncated completion,
+missing and duplicate receipts, forged textual labels, ambiguous code, namespace
+and size limits, and contradictory receipt counts/ordinals. Existing integration
+tests remain in place.
+
+## Observed test-first slices
+
+Each command below was run with its new regression present and its corresponding
+product behavior unchanged, then rerun after the product fix. Commands invoke
+the existing Bun harness; the shell routed their output through lean-ctx.
+
+1. Red — `bun test tests/codex-native-calls.test.ts -t "Code Mode exposes completed literal"`: missing submitted-call inventory.
+   Green — `bun test tests/codex-native-calls.test.ts -t "Code Mode exposes completed literal"`: 1 passed.
+2. Red — `bun test tests/codex-native-calls.test.ts -t "Code Mode portable"`: rejected request receipt missing and opaque submission reported complete.
+   Green — `bun test tests/codex-native-calls.test.ts -t "Code Mode portable"`: 2 passed.
+3. Red — `bun test tests/codex-native-calls.test.ts -t "Code Mode inspects the retained"`: both retained parallel-read shapes reported partial.
+   Green — `bun test tests/codex-native-calls.test.ts -t "Code Mode inspects the retained"`: 2 passed.
+4. Red — `bun test tests/codex-native-calls.test.ts -t "Code Mode refuses malformed"`: malformed and truncated completion envelopes reported complete.
+   Green — `bun test tests/codex-native-calls.test.ts -t "Code Mode refuses malformed"`: 2 passed.
+5. Red — `bun test tests/codex-native-calls.test.ts -t "Code Mode accepts native array"`: the native array envelope reported partial.
+   Green — `bun test tests/codex-native-calls.test.ts -t "Code Mode accepts native array"`: 1 passed.
+6. Red — `bun test tests/codex-host.test.ts -t "reviewer-route repository-prefixed"`: complete mounted body did not establish a skill read.
+   Green — `bun test tests/codex-host.test.ts -t "reviewer-route repository-prefixed"`: 1 passed.
+7. Red — `bun test tests/native-controls.test.ts -t "contradictory Code Mode"`: mismatched submission ordinal reported complete.
+   Green — `bun test tests/native-controls.test.ts -t "contradictory Code Mode"`: 1 passed.
+8. Red — `bun test tests/codex-native-calls.test.ts -t "attributable control evidence"`: foreign namespace and oversized tool labels reported complete.
+   Green — `bun test tests/codex-native-calls.test.ts -t "attributable control evidence"`: 2 passed.
+
+These focused slices do not substitute for the full GitHub CI acceptance gate
+on the PR's exact published commit.

package/docs/evidence/code-mode-programs-85.md

@@ -0,0 +1,64 @@
+# Template and batch inspection for issue 85
+
+On 2026-10-15, the implementation for [issue #85](https://github.com/BjRo/sevro/issues/85)
+replayed the two original Codex native transcript bundles through the changed
+Sevro observers and Darrow's checkpoint `67f4fcdb` extension. Bundle and decoded
+source SHA-256 values matched the issue before replay. Raw bundles and output
+remain private and outside Git.
+
+| Original case                     | Exec submissions | Complete receipts after replay | Regraded no-agent assertion |
+| --------------------------------- | ---------------- | ------------------------------ | --------------------------- |
+| goal-placement-direct-same-route  | 11               | 11                             | passed                      |
+| goal-placement-coupled-sequential | 10               | 10                             | passed                      |
+
+The five exact [retained program fixtures](../../tests/fixtures/code-mode-85/README.md)
+now preserve these native ordinals and ordered submission-local labels:
+
+- Direct 45: three `exec_command` calls.
+- Direct 69 and 77: `apply_patch`, then `exec_command`.
+- Coupled 30: three `exec_command` calls.
+- Coupled 68: two `exec_command` calls.
+
+Regrading replaced only the native-call and portable native-control observations
+with newly inspected observations and invoked the original case's extension
+checks. Both `darrow.evals.transcript.1` results changed from unavailable to
+passed. These are artifact rechecks, not new candidate runs or rewritten
+historical task verdicts. Deterministic consumer coverage separately confirms
+that a submitted `spawn_agent` fails the no-agent assertion even when the
+accepted-agent count is zero, and partial receipts remain unavailable.
+
+## Observed test-first slices
+
+1. Red — `bun test tests/codex-native-calls.test.ts -t 'retained direct ordinal 45'`: the template-command receipt was partial.
+   Green — `bun test tests/codex-native-calls.test.ts -t 'retained direct ordinal 45'`: 1 passed, with three bound call labels.
+2. Red — `bun test tests/codex-native-calls.test.ts -t 'retained coupled ordinal'`: both destructured batch receipts were partial.
+   Green — `bun test tests/codex-native-calls.test.ts -t 'retained coupled ordinal'`: 2 passed, with three and two bound call labels respectively.
+
+Additional regressions preserve submitted spawns inside interpolated and
+concatenated output, reject ambiguous bindings and composition, and keep
+private source bytes out of observations. The focused native-call/control
+suite passed 94 tests. These checks do not replace the full GitHub CI gate
+on the PR's exact published commit.
+
+## Fresh native trials
+
+Fresh trials use Darrow checkpoint `67f4fcdb`, native Codex 0.162.1 app-server,
+candidate and semantic grader `gpt-6-luna/medium`, passive ownership, N=1,
+jobs=1, threshold=1, and a 1,200,000 ms timeout with retained diagnostics.
+They explicitly select the changed Sevro source worktree through
+`SEVRO_CHECKOUT`; they are development evidence, not published-package trials.
+
+The direct trial, run `510f99df-882f-48fe-b893-d4b6fba8864d`, completed with
+9/10 checks passing and the no-agent assertion unavailable. Its new submission
+29 maps a callback that invokes tools over a command array; submission 74 uses
+an output callback with a conditional expression. Both forms remain outside
+the [bounded inspection contract](../native-controls.md) and correctly retain
+partial receipts. The five original programs no longer cause that limitation.
+
+The coupled trial, run `7f9f5d8a-641e-4744-a986-05298100541c`, completed with
+10/11 checks passing and the no-agent assertion unavailable. Submissions 29,
+45, and 68 each use a `for ... of` output loop with a conditional expression
+and `String(...)` conversion. That output body is also outside the bounded
+subset and remains partial. Neither fresh run is a passing placement trial;
+their unavailable verdicts are preserved rather than treating zero accepted
+agents as proof of absence.