@bjoernrochel/sevro 0.1.0-rc.8 → 1.0.0

package/docs/evaluation-workflow.md

@@ -0,0 +1,110 @@
+# From a demo to your next evaluation
+
+Create an evaluation, prepare it, run its assertions, and inspect the latest
+result with the installed `sevro` command.
+
+## Try the demo and create a project
+
+You need Bun 1.3.13. [Install Sevro](installing.md) and
+[make its command available on `PATH`](installing.md#make-the-command-available),
+then run:
+
+```sh
+sevro demo
+sevro init --template response --directory ./response-evaluation
+cd response-evaluation
+```
+
+The demo returns an assessed `passed` task and evidence under `.sevro/results`,
+using its own temporary project without credentials. The initializer refuses
+existing destinations, including directories and symbolic links. It writes
+real assertions, a deterministic adapter, `run.json` and template instructions.
+[Saved settings](run-settings-v1.md) use their manifest's directory for paths;
+`sevro.json` retains its separate runtime scope. Keep private `.sevro/` out of Git.
+
+## Prepare before execution
+
+```sh
+sevro run --run-manifest-file run.json --dry
+```
+
+Stderr shows the whitelisted resolved settings before candidate execution:
+case/source selection, routes/model/effort, trials, threshold, concurrency,
+runtime selection and canonical storage paths. Invalid input names its option,
+file or field; missing executables name installation or explicit override paths.
+Correct the named prerequisite and repeat preparation.
+
+Dry preparation follows normal fixture, extension and check preparation without
+calling the candidate or graders. Expect
+`execution=not_run grading=not_requested task=not_assessed`. It does not establish
+authentication, provider model availability, native skill dispatch or model success.
+See [native prerequisites](native-hosts.md) and [troubleshooting](troubleshooting.md).
+
+## Run, read and change one thing
+
+```sh
+sevro run --run-manifest-file run.json
+sevro report --latest
+SEVRO_TEMPLATE_FAIL=1 sevro run --run-manifest-file run.json
+sevro report --latest
+```
+
+The first run's adapter returns exactly `ready`; its regex assertion must pass.
+The negative control returns `wrong`: expect exit `1`, completed execution and
+grading, task `failed`, and the named `ready` check's reason. Then edit `case.json`
+and `adapter.ts`, prepare again and rerun. Explicit flags override saved settings;
+`--trials 3` repeats the task without rewriting `run.json`.
+
+Progress uses stderr at actual preparation, candidate, grading and evidence
+retention boundaries. Trial numbers distinguish concurrent work, which may
+interleave. Stage `completed` means the attempt finished; the result states whether
+execution/grading succeeded. Dry runs explicitly skip candidate and grading;
+failed candidates skip grading. Failed preparation never prints candidate start.
+Evidence completion follows persistence. Optional synchronous engine observers
+cannot change the result through a thrown exception.
+
+Human completion shows per-trial states, failed/unavailable check reasons,
+measured candidate time, known/unknown usage/cost and `result=`/`evidence=` paths.
+Candidate time measures the host call, not total setup time. Absent metrics remain
+`unknown`; dry duration is unknown. Prompt-only completion stays `not_assessed`.
+Add `--json` to emit one `sevro.cli-result.v1` stdout document; progress remains on
+stderr. See [reading results](reading-results.md).
+
+Each retained CLI evaluation saves `result.json` beside its run evidence and
+atomically publishes `<results-root>/latest.json`. Latest is the last retained CLI
+result, including dry runs and failed assessments. Pre-run failures do not replace
+it. Concurrent runs publish in completion order. Output files stay under excluded
+results storage, preserving project identity. A latest-write failure cannot turn
+an earlier failed assessment into success.
+
+`report --latest` reads `<cwd>/.sevro/results/latest.json`. Use
+`report --latest --results-root ./other-results` for another output root or working
+directory. `--result-file ./path/to/result.json` selects older runs and accepts
+relative paths. Reports validate referenced evidence; missing/moved evidence
+produces a diagnostic. To reclaim native bundle space while keeping reportable
+results and hashes, use [`sevro prune --native-transcripts`](reading-results.md#remove-native-transcript-bundles).
+Reports label bundles intentionally removed by that command as pruned.
+[Report v1](report-v1.md) defines fields and limits.
+
+## Choose another starter
+
+| Template           | Real assertions                                                                                                    | Prerequisites and interpretation                                                                           |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
+| `response`         | Exact `ready` response; negative control returns `wrong`                                                           | Bun; no model credentials                                                                                  |
+| `coding`           | Edit `add.sh`; isolated checks test positive, negative and zero addition                                           | Bun plus `sandbox-exec` on macOS or `bubblewrap`/`socat` on Linux; negative control leaves subtraction bug |
+| `repository-skill` | Actual `.agents/skills/startup-guide/SKILL.md` mount; explicit invocation evidence; correct start command and port | Bun, Git, Codex, native isolation, explicit model/effort and readable auth file; dry is unassessed         |
+
+From a new parent directory, use `sevro init --template coding --directory ./coding-eval`
+or `sevro init --template repository-skill --directory ./skill-eval`. Read its generated
+README for controls and dry/live commands. Repository-skill `command.json` records
+absolute Bun/extension argv; regenerate it after moving the project.
+
+For model evaluations, replace a saved adapter route with explicit `--host`,
+`--model`, `--effort` and host authentication options. Candidate, semantic and
+advisory model choices remain independent. Deterministic adapters measure wiring;
+they do not measure model ability. Native runs may incur provider charges.
+Automated package checks run response/coding pass/fail and native skill preparation;
+authenticated native behavior and human usability remain unmeasured. The
+[study protocol](https://github.com/BjRo/sevro/blob/main/docs/usability-study.md) records deferred human measurements.
+[Runnable recipes](recipes.md) cover Claude, plugins/hooks, goals, semantic grading
+and host tools.

package/docs/extension-protocol-v1.md

@@ -106,8 +106,16 @@ trial in the Git fixture after source history and working-tree files are built,
 before extension artifacts are mounted or the host starts. The command is never
 interpreted through a shell by Sevro; an extension may explicitly select a
 shell executable. `{{sevro.project}}` and `{{sevro.workspace}}` in environment
-values resolve to the declared roots at execution time. The symbolic values and
-command enter fixture and comparison identity, while retained configuration
+values resolve to the declared roots at execution time. Setup's `HOME` is the
+candidate commands' private trial home, outside the Git fixture. Ordinary
+`$HOME` paths can initialize directories and seed prior state before the candidate
+starts, including without a runtime configuration file. This home is separate
+from host authentication and native configuration, other trials, and grader homes.
+Setup's `TMPDIR` remains disposable and is removed when setup finishes; the shared
+home survives through candidate execution and grading, then is removed even on
+setup failure, cancellation, or dry runs. Trusted injected hosts must use the
+supplied `candidateRuntimeRoot` and runtime policy to preserve this boundary.
+The symbolic values and command enter fixture and comparison identity, while retained configuration
 records only their digest. Setup has a two-minute limit, inherits no credential
 environment variables, and its output is discarded. Failure or interruption
 stops the run before host execution; it cannot become a passing assessment.
@@ -171,7 +179,7 @@ other invocation declarations and enters retained configuration and identity.
 The host validates the bounded mount without following symbolic links. Its
 `sevro.claude.repository-invocation` observation binds one native command, exact
 arguments, and the complete mounted body before the first assistant entry in
-the completed session. It retains only acceptance and ordered skill metadata.
+the completed session. It retains only acceptance and the dispatched skill name.
 Missing or ambiguous session evidence yields a partial observation; later Skill
 calls cannot repair an unverified explicit repository dispatch.
 
@@ -224,6 +232,21 @@ grader error, or unavailable required evidence.
 Host adapters can supply additional namespaced observations. The engine bounds
 and retains those records, then passes them to `evaluate`. Required observation
 IDs must be present with `complete` evidence before a task can pass.
+The bundled adapters expose [skill activation evidence](native-hosts.md#skill-activation-evidence)
+as `sevro.host.skill-activations`, with separate methods for Codex tool-result
+body exposure and successful Claude `Skill` calls.
+This replaces the retired Codex skill-read and Claude nested-skill observations.
+Explicit invocation receipts remain dispatch evidence and do not contribute to
+`observedSkills`; extensions must migrate activation checks to the common ID.
+Codex additionally supplies per-skill `skillExposures` and independent
+`collectionStatus` in that observation. Its optional
+`sevro.host.skill-exposure-claims` capability provides complete evidence only
+for listed positive full or partial body exposures, even when collection is
+partial. Extensions must check the requested skill and acceptable exposure
+status explicitly; this evidence cannot establish absence or primary selection.
+See the [exposure contract](native-hosts.md#skill-activation-evidence) for scope,
+unknown states and host limitations. Required aggregate activation evidence
+still needs a `complete` observation as before.
 Host adapters can also supply bounded evidence artifacts. The engine persists
 their bytes outside the candidate workspace before `evaluate`, then passes
 file URLs and digests through the existing artifact references. A required host

package/docs/getting-started.md

@@ -5,6 +5,27 @@ Darrow checkout, Codex/Claude login, or model credentials are required. The
 adapter returns `ready`; Sevro grades that response against `^ready$`.
 This teaches the runner lifecycle, not the quality of an actual model.
 
+<a id="built-in-demo-in-this-checkout"></a>
+
+## Built-in demo
+
+With [Sevro installed and on `PATH`](installing.md#make-the-command-available), run:
+
+```sh
+sevro demo
+```
+
+It prints `execution=completed grading=completed task=passed` and an absolute
+`evidence=` path under `.sevro/results` in your current directory. The demo owns
+its packaged case and adapter, uses a separate temporary project, and ignores
+your project's `sevro.json` and native configuration. No credentials are needed.
+Effective run settings appear on stderr before evaluation. Add `--json` for one
+versioned result on stdout.
+
+Continue with [the initialized evaluation workflow](evaluation-workflow.md) to
+create response, coding or repository-skill assertions, prepare them, inspect
+progress and access the latest result without copying an evidence path.
+
 ## From an installed package
 
 Prepare a new directory:
@@ -16,8 +37,8 @@ bun init -y
 ```
 
 Install the [current release](installing.md#current-release) there, then return
-here to run the example. Read [version differences](installing.md#published-package-and-checkout)
-for the current docs, source-available license, and earlier BSL releases.
+here to run the example. See [licensing](licensing.md) for the current terms and
+the licenses retained by historical releases.
 
 Make the local installation available on `PATH` for this shell, then call
 `sevro` directly. Bun must also be on `PATH`.

package/docs/guide-evaluation.md

@@ -54,6 +54,34 @@ documentation and both skill mounts; the eval cases and grader code are withheld
 The case matrix covers orientation, explicit invocation, unrelated requests,
 missing/conflicting/stale evidence, follow-ups, pressure for effects,
 extension boundaries, contributions, and licensing.
+
+The deterministic CLI integration suite runs all eleven positive cases on each
+host. Eight negative cases also run on both hosts: attempted effects, missing
+events, incomplete native turns, missing/wrong invocation receipts, stale/missing
+follow-up reads, and missing continuation evidence. These exercise distinct host
+event formats or receipt identifiers. Ten host-independent negatives run on
+Codex once: changed files or skill mounts, extra directories, host failure,
+missing/partial common selection observations, and four answer contradictions.
+This keeps every failure scenario while reducing the matrix from 58 to 48 CLI
+runs. Two additional Codex positive controls remain, for 50 guide integration
+tests in total. The docs phase still dry-prepares all eleven cases.
+
+On 2026-10-24, an instrumented local comparison on macOS arm64 with Bun 1.3.13
+measured the negative cases at 262.55 seconds for 36 passing tests on `adf04a1`,
+then 181.50 seconds for 26 passing tests with #145: 81.05 seconds less (30.9%).
+These are sums of individual case durations in `tests.log`, not a CI speedup
+forecast. Claude's negative cases alone changed from 130.23 to 57.50 seconds;
+that ratio rescales their shard weight. Reproduce the targeted measurement with:
+
+```sh
+bun run check:typescript --coverage ./tests/guide/codex-negative.test.ts \
+  ./tests/guide/claude-negative.test.ts
+```
+
+These sparse timing runs passed their tests and report validation, but failed
+the unchanged 95% full-inventory coverage thresholds as expected. Full acceptance
+still requires every CI phase and the combined coverage gate on the candidate.
+
 Missing event artifacts, incomplete turns, missing dispatch receipts, or absent
 fresh follow-up reads cannot pass. Built-in `sevro.regex` checks assess answer
 assertions and contradictions. The `sevro.shell` check verifies Git status,
@@ -73,9 +101,13 @@ Automatic checks remain signals, not a complete truth judgment. Acceptance
 requires inspecting material claims against the sources actually read.
 File listings do not count as source inspection; follow-ups require fresh
 content. Explicit invocation requires Sevro's complete host dispatch receipt.
+Automatic selection requires the complete `sevro.host.skill-activations`
+observation on both hosts. Codex uses passive tool-result hooks; Claude uses
+successful native `Skill` calls. The evaluator does not fall back to command
+patterns, direct Claude file reads, or retired skill-read observations.
 See the [guide contract](specs/repository-guide.md).
 
-The [earlier verification record](guide-verification.json) describes historical
+The [earlier verification record](https://github.com/BjRo/sevro/blob/main/docs/guide-verification.json) describes historical
 trials of the previous harness. Its outputs and recheck observations are preserved
 as history; they do not validate this conversion. The old `--recheck` command
 and custom evidence format have been removed. Rerun changed cases through Sevro

package/docs/identity-v1.md

@@ -32,7 +32,10 @@ The CLI derives `runnerBuildDigest` from sorted packaged file paths and their
 SHA-256 bytes under `src/`, `schemas/`, `docs/`, and `examples/`, plus `README.md` and
 `package.json`. For a Git project, `projectDigest` binds the revision and dirty
 patch digest without the checkout path. Without a revision it hashes a bounded
-file snapshot, excluding `.git`, results, and run-state storage. A caller may
+file snapshot, excluding `.git`, results, and run-state storage. Results and
+run-state storage are also excluded from untracked Git identity, so repeated
+project-local output does not change the project digest. Tracked source changes
+remain included, even under a selected storage directory. A caller may
 provide explicit 64-character digests for controlled development runs.
 
 The prefix separates this digest from component digests and future identity
@@ -48,6 +51,17 @@ The effective positive trial job limit is retained as `configuration.redacted.jo
 and enters `configurationDigest`. Changing parallelism changes evaluation identity,
 even when the condition, host route, trial count, and assessment threshold match.
 
+CLI [saved run settings](run-settings-v1.md) resolve before evaluation. Equivalent
+manifest and explicit selections produce the same semantic inputs. CLI evidence
+also retains `configuration.redacted.runSettings` for shell isolation, explicit
+protected roots, deprecated runtime selections, selected native timeouts and
+Claude project-settings authority. These values enter the
+configuration digest. Optional `configuration.resolved` stores the whitelisted
+operator preview outside that digest, including operational paths and the
+selected manifest location. Relocating output storage alone does not alter
+identity. A tracked manifest remains project content under the normal project
+identity rules.
+
 Extension runs retain the operator-supplied redacted extension configuration
 under `configuration.redacted.extensionConfiguration`, alongside
 `extensionConfigurationDigest`. The retained copy is the same snapshot used
@@ -64,10 +78,19 @@ for the host default. Configuration paths and unrelated TOML bytes are not
 imported settings; a relocated configuration with the same effective values
 keeps this component unchanged.
 
+Opted-in bundled native roles also record `sevro.host.retain-diagnostics: true`.
+The [diagnostic retention switch](development-cli.md#retain-native-host-diagnostics)
+therefore changes `configurationDigest`. Omission leaves that host setting
+absent and preserves the previous configuration shape. Retained diagnostic
+contents and temporary native state paths do not enter identity.
+
 Selected [runtime configuration](runtime-v1.md) is retained under
 `configuration.redacted.runtimePolicy` and enters `configurationDigest`.
 It includes declared non-secret environment values, canonical read roots,
-private seed digests and optional role selections, and hook authority. Per-role output paths are resolved
+private seed digests and optional role selections, shared-cache sources/slots/roles,
+and hook authority. Mutable shared-cache contents are excluded; equal identities
+do not establish equal cache contents or trial independence in shared mode.
+Per-role output paths are resolved
 only during execution; runtime placeholders remain symbolic in identity.
 Runtime roots also retain lexical aliases, metadata discovery sources,
 and skipped optional directories. Moving or

package/docs/installing.md

@@ -1,8 +1,9 @@
 # Install, update, or remove Sevro
 
 Use Bun 1.3.13 for this TypeScript/Bun CLI. Installation does not make it a
-Node-only executable. Full validation and native isolation target macOS;
-Linux and Windows are unverified.
+Node-only executable. Full validation and native isolation target macOS and
+Ubuntu 24.04 x64. See the [bounded platform evidence](native-hosts.md#validated-platform-scope);
+Windows remains unverified.
 
 ## Install locally
 
@@ -22,10 +23,10 @@ Local installation records the selected version in the project lockfile.
 
 ## Current release
 
-The current release is `0.1.0-rc.8`. Install this exact version:
+The current release is `1.0.0`. Install this exact version:
 
 ```sh
-bun add --exact @bjoernrochel/sevro@0.1.0-rc.8
+bun add --exact @bjoernrochel/sevro@1.0.0
 ```
 
 <!-- sevro-current-release:end -->
@@ -46,17 +47,16 @@ no adjustment is needed. Bun must also be on `PATH`: the executable uses
 
 ## Published package and checkout
 
-The [current release](#current-release) includes runtime
-configuration, native goals, selected plugin hooks, current onboarding, and
-[source-available licensing terms](licensing.md). Earlier `rc.1` and `rc.2`
-packages retain BUSL-1.1. The license transition began with `0.1.0-rc.3`.
-Source checkouts can include changes beyond their
-package version; compare the exact release tag when establishing provenance.
+See [licensing](licensing.md) for Sevro's source-available terms and historical
+licensing transitions.
 
-The candidate npm package contains sources, schemas, examples, documentation,
+The npm package contains sources, schemas, examples, documentation,
 LICENSE, and contribution guidance. It excludes tests, developer scripts, Git
 metadata, and repository guide mounts. Use a source checkout for the guide.
 
+Source checkouts can include changes beyond their package version; compare the
+exact release tag when establishing provenance.
+
 ## Work from source
 
 ```sh

package/docs/native-controls.md

@@ -5,7 +5,8 @@ retain an observation with that ID. It exposes host facts for extension policy:
 
 - `method: native_control_calls`;
 - ordered `calls` containing only `ordinal`, `namespace`, and `name`;
-- `acceptedAgentCount`, a verified Codex acceptance count or `null` on Claude;
+- `acceptedAgentCount`, a verified direct native Codex acceptance count when
+  overall coverage is complete, otherwise `null` (also `null` on Claude);
 - `submittedExecCalls`, the Codex submitted-code count or `null` on Claude; and
 - `truncated`, which is `false` for a complete record and may be unknown (`null`)
   for an incomplete Codex record.
@@ -15,9 +16,71 @@ tool arguments, task messages, command bodies, or results. Call labels prove an
 invocation attempt, not success. Extensions decide which calls are prohibited.
 An unknown acceptance count cannot establish agent acceptance.
 
-Codex projects labels and counts from its bound native-session observation and
-preserves its completeness. Incomplete counts remain `null`. Submitted code remains explicit: extensions cannot
-infer absence of controls inside code that this observer did not inspect.
+Codex projects labels and counts from its bound native-session observation.
+It additionally retains `directAcceptedAgentCount`, the number of positively
+correlated direct native spawn acceptances in that source, independently of exec
+inspection. This is an observed count, not a total across unknown launch paths.
+`coverage.nativeCalls` records native-session completeness;
+`coverage.execInspection` records whether all submitted exec sources were
+understood and matched to the direct submission inventory;
+`coverage.acceptedAgents: direct_native_calls` names the scope of both acceptance
+counts. Accepted launches through exec are not included in these direct counts.
+When native capture is incomplete, even a zero observed count cannot establish
+absence. Native labels remain usable as positive attempt evidence regardless
+of an unrelated exec inspection gap. A rejected attempt is never an acceptance.
+
+When submitted code is present, both observations also retain `execCalls`:
+one receipt per native `exec` submission, with its native `ordinal`, inspection
+`completeness`, and ordered `calls` containing `index`, `namespace`, and `name`.
+The index is local to that submission; native ordinals are not synthesized.
+The original direct-call list and total `submittedExecCalls` stay unchanged.
+
+Sevro statically inspects bounded straight-line Code Mode
+programs without executing them. It accepts literal tool calls, constant result
+bindings, template interpolation and `+` composition of inspected expressions,
+direct `text`/`store` output, and literal `Promise.all`/`allSettled` batches.
+Batch results may use flat constant array bindings or result-only
+`forEach(text)` or `for ... of` output, including blocks and conditional
+result-field selection. `JSON.stringify`, `String`, object spread/shorthand,
+and nested result fields are supported. Numeric output loops must use the
+bounded shape `for (let i = 0; i < results.length; i++)`, indexing the known
+batch with that loop variable. Output loops and conditional expressions cannot
+invoke tools. Destructuring defaults, rest elements,
+holes, nested bindings, duplicate names, and shadowed globals remain unsupported.
+Tool argument strings, comments and printed tool-looking text
+do not become call labels. A rejected tool request returned by a completed
+script still appears in its submitted-call inventory. This inventory proves
+submission of those requests, not successful effects or acceptance; the existing
+acceptance count describes independently correlated direct native spawns.
+
+Dynamic lookup, aliases, shadowed globals, callbacks that invoke tools,
+conditional tool execution, and nested executors keep source inspection partial.
+Each receipt's `inspectionReason` is `null` on success, or `unsupported_source`,
+`malformed_source`, `source_limit`, or `foreign_namespace`. Unsupported syntax
+and expression/call-budget exhaustion use `unsupported_source`.
+Source is bounded at
+64 KiB, traversed expressions at 2,048, and calls at 128 per submission; tool
+labels are bounded at 64 characters. Private source and output bytes are discarded.
+The portable observation becomes partial if any submission is uninspected or
+receipt counts/ordinals disagree with the native list. `acceptedAgentCount`
+remains `null` in that case, while native coverage, direct acceptance facts and
+the native `submittedExecCalls` count remain independently available.
+`submittedExecCalls` and `truncated` depend on native capture, not exec output.
+
+Receipt `completeness` describes source inspection, independently of execution
+or displayed output. `execution` is `completed`, `failed`, or `incomplete`;
+`output` is `complete`, `truncated`, or `unavailable`. Output evidence requires
+a unique later output bound to the same native call ID, with recognized text
+blocks. `complete` describes the displayed receipt, not the whole program's
+execution. Execution status additionally requires a successful or failed script
+header. Missing, ambiguous or
+malformed output yields `incomplete`/`unavailable`; a yielded or unrecognized
+header stays `incomplete`. Displayed truncation does not change source coverage.
+A fully inspected source with no launch call proves that absence even if its
+shell output was truncated or the exec cell failed. Source inspection cannot
+prove execution or success. Missing fields in historical receipts are unknown;
+historical artifacts retain their original results and meanings.
+
 Claude records every tool-use label from the bounded stream, with namespace
 `claude`, without filtering to Skill or Agent. It requires a completed successful
 turn, valid unique tool-use IDs and names, and a non-truncated list of at most
@@ -26,4 +89,12 @@ Tool-looking text in arguments or output does not become a native call.
 
 Consumers must verify the source, completeness, shape, ordering, and counts
 before using an empty call list to establish absence. Missing or partial
-evidence cannot establish a negative assertion.
+source coverage cannot establish a negative assertion. For Codex, consumers must
+inspect every `execCalls` receipt in addition to the direct `calls` list,
+validate receipt ordinals against the direct `exec` submissions, and require
+all receipts to be complete with counts matching `submittedExecCalls`.
+They can use positively observed native attempt labels before requiring complete
+coverage for an absence pass. They must examine submitted labels for prohibited controls even when
+`acceptedAgentCount` is zero. Older consumers that require
+`submittedExecCalls === 0` remain conservative but cannot use this new evidence
+until they adopt receipt validation.