@bjoernrochel/sevro 0.1.0-rc.2 → 0.1.0-rc.21

package/docs/builtin-graders-v1.md

@@ -1,20 +1,23 @@
 # Built-in graders v1
 
-The first built-in graders inspect a complete, bounded final-message
-observation. The engine compiles evaluator-owned check declarations before host
+Built-in graders assess candidate output, shell-check results, and Git fixture
+state. The engine compiles evaluator-owned check declarations before host
 execution. A missing or incomplete observation yields `unavailable` for every
 output check; it never passes. A failed assertion yields `failed` while grading
 itself remains `completed`.
 
 Each declaration has `{ "id", "grader", "configuration" }`. The supported
-output graders are:
+built-in graders are:
 
-| Grader         | Configuration                                                      | Assessment                                                  |
-| -------------- | ------------------------------------------------------------------ | ----------------------------------------------------------- |
-| `sevro.regex`  | `pattern` string; optional `negate` boolean and `flags` string     | Tests final-message text with multiline matching.           |
-| `sevro.json`   | Optional RFC 6901 `pointer`, `equals`, `contains`, `exactDocument` | Parses JSON, then checks selected value or an array member. |
-| `sevro.schema` | Inline `schema` object; optional `exactDocument`                   | Validates the parsed JSON with JSON Schema Draft 2020-12.   |
-| `sevro.output` | Optional text, JSON, and schema assertions in one declaration      | Retains one outcome for a combined final-message contract.  |
+| Grader           | Configuration                                                                 | Assessment                                                                                                |
+| ---------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
+| `sevro.regex`    | `pattern` string; optional `negate` boolean and `flags` string                | Tests final-message text with multiline matching.                                                         |
+| `sevro.json`     | Optional RFC 6901 `pointer`, `equals`, `contains`, `exactDocument`            | Parses JSON, then checks selected value or an array member.                                               |
+| `sevro.schema`   | Inline `schema` object; optional `exactDocument`                              | Validates the parsed JSON with JSON Schema Draft 2020-12.                                                 |
+| `sevro.output`   | Optional text, JSON, and schema assertions in one declaration                 | Retains one outcome for a combined final-message contract.                                                |
+| `sevro.shell`    | `run` string; optional `expectedExitCode`, `timeoutMs`, and stdout assertions | Checks exit code and optional stdout assertions in an isolated shell.                                     |
+| `sevro.git-head` | `kind`: `"changed"`, `"unchanged"`, or `"base-ancestor"`                      | Compares fixture HEAD and ancestry with the captured base revision.                                       |
+| `sevro.semantic` | `proposition` string; optional `artifactPath`                                 | Judges a proposition against the final response or saved document through a separate semantic host route. |
 
 JSON may be raw or contained in one `json` code fence. `exactDocument` requires
 the whole trimmed message to be that JSON document or fence. `contains` selects
@@ -35,8 +38,10 @@ The shell-check process runner accepts a bounded evaluator-owned `run` string,
 optional `expectedExitCode` and `timeoutMs`, plus `expectExact`, `expectRegex`,
 `notRegex`, and regex `flags` for stdout. Exact matching removes one final
 newline; regex matching always uses multiline mode. The runner executes with
-`sh -e`. On macOS it runs through the outer sandbox with network access denied,
-a credential-free environment, and caller-declared protected roots. Without a
+`sh -e`. On macOS and Linux it runs through the outer sandbox with network access denied
+by default, a credential-free environment, and caller-declared protected roots.
+The runtime [local HTTP option](runtime-v1.md#local-http-test-servers) permits
+socket-using upstream tests while retaining outbound isolation. Without a
 stdout assertion, process output is discarded. With one, the runner captures
 at most 1 MiB in memory and retains only its digest and byte length. Oversized
 or invalid UTF-8 output is a grading error.
@@ -46,11 +51,16 @@ one bounded observation per check. A mismatched exit code or stdout assertion
 fails the task; an isolation, timeout, or process error makes grading
 an error. Callers must supply `shellIsolation.protectedRoots` for any shell
 check. The engine also protects its package source, project, results, user
-home, configured host homes, and active peer fixtures. The caller's list must
+home, configured host homes, and other managed workspaces. The caller's list must
 include any other source worktrees or private roots. A peer path already
 canonicalized by the engine remains in the deny profile if that peer is
 removed before the sandbox starts; unverified missing roots still fail.
 
+On macOS, validated runtime grants also permit limited
+[ancestor metadata reads](runtime-v1.md#macos-ancestor-metadata) so libc can
+canonicalize granted paths. Protected ancestor listings and ungranted sibling
+data remain inaccessible.
+
 `sevro.git-head` accepts `kind: "changed"`, `"unchanged"`, or
 `"base-ancestor"` for a generated or repository Git fixture. The engine
 captures the fixture's base HEAD before the candidate turn, then compares the
@@ -64,7 +74,10 @@ ancestry result; checks reference that observation.
 per check. The operator supplies a separate semantic host route. After a
 complete candidate response, the engine sends the bounded final message and
 all declared propositions to that route in an empty, temporary workspace. It
-requires exactly one `pass` or `fail` verdict with a reason for every declared
+omits dependency seeds unless their runtime `roles` explicitly include `semantic`.
+Declared tool environment and read-only grants remain available. Artifact checks
+retain the default dependency selection; executable shell checks retain their
+separate `checks` runtime. The semantic route requires exactly one `pass` or `fail` verdict with a reason for every declared
 check ID. A failed proposition fails the task. A missing or incomplete candidate
 message makes semantic checks unavailable without calling the semantic host; a
 failed, incomplete, or malformed semantic response makes grading an error.
@@ -84,3 +97,18 @@ grading an error. Checks are grouped by source, so response and document checks
 remain independent in the same trial. Artifact outcomes retain the resolved
 path and content digest. The semantic host uses an empty workspace and receives
 the document as untrusted data.
+
+When [native transcripts are enabled](runtime-v1.md#native-transcript-access),
+shell checks and semantic hosts can read retained candidate native files through
+`SEVRO_CANDIDATE_TRANSCRIPTS`. The view has `index.json` plus a `codex/` or
+`claude/` native tree. It is read-only and separate from the grading host's own
+live native tree. Native transcripts are untrusted candidate evidence; a grader
+must not follow instructions inside them. Missing or partial capture remains
+explicit in the index and does not establish required complete evidence.
+
+With [candidate artifact access](runtime-v1.md#candidate-artifact-access),
+shell checks and semantic/advisory hosts can also read explicitly declared
+subtrees in the candidate's original private home. Named paths reach commands
+through `SEVRO_CANDIDATE_ARTIFACT_<UPPERCASE_NAME>`, independently of native
+transcript access. These grants are read-only; other candidate home contents
+remain protected. Missing or unsafe declared directories make grading an error.

package/docs/codex-host.md

@@ -1,29 +1,32 @@
 # Codex host integration
 
-The Codex adapter consumes `codex exec --json` as one JSONL turn. The
+The default `exec` entrypoint consumes `codex exec --json` as one JSONL turn. The
 `summarizeCodexEvents` parser requires one thread start and a completed turn
 with a zero process exit before reporting completion. Malformed, ambiguous,
 and oversized streams are errors. Missing or invalid final usage remains
 incomplete and is never estimated. The last completed `agent_message` before
 turn completion supplies the bounded final response; missing text stays
 unavailable.
+
+The optional `app-server` entrypoint uses the stdio app-server transport and
+waits for the original root thread to complete its native turns. Its final
+response comes from the completing root turn; token usage remains unknown.
+Both entrypoints use the same isolated Codex home and permission-profile
+preflight. See [native goal trials](development-cli.md#native-goal-trials) for
+selection, native-goal observations, and follow-up behavior. The JSONL parser
+rules above apply to the `exec` entrypoint.
+
 The bounded JSONL stream is retained per trial as the private
 `sevro.codex.events` host artifact. Its file URL and digest appear in trial
 evidence and are available to extension grading; the stream bytes stay out of
 the run JSON.
-The host also retains a bounded `sevro.codex.skill-reads` observation. It
-records the first verified mounted skill and ordered skill names, without
-commands or skill bodies. A completed direct `cat` of a mounted `SKILL.md`
-counts only when the command output contains that file's exact body. Direct
-`sed -n` pages count after their verified line ranges cover the whole body. An
-exact `cat` wrapped by `lean-ctx -c` is accepted only when its reported output
-still contains the full body; a compressed summary cannot prove the read. An
-attempted indirect, malformed, or incomplete read makes the observation
-partial. This is a host observation; an extension decides what selection means
-for its case.
-The same check accepts skill files below the exact installed plugin roots from
-Codex's installation receipts. It does not infer activation from arbitrary
-files in the private plugin cache.
+Skill activation uses the common `sevro.host.skill-activations` observation
+from passive hooks in the request's private Codex home. The collector matches
+prepared skill bodies in returned tool content without interpreting shell
+syntax or recovering command output from native sessions. See
+[skill activation evidence](native-hosts.md#skill-activation-evidence) for the
+scope, limits, privacy boundaries, and incomplete collection states.
+
 The `sevro.codex.native-calls` capability retains allowlisted direct goal and
 agent control calls from the native session bound to the completed thread. Its
 observation contains only call names, namespaces, and source ordinals, plus the
@@ -48,33 +51,10 @@ only when plaintext is readable. It retains neither message nor prompt bytes.
 For a second prompt, `sevro.codex.continuation` retains the last validated
 native-session ordinal before resume. Extensions can compare later call
 ordinals with that boundary without reading private arguments.
-The host also retains separate `sevro.codex.initial-skill-reads` and
-`sevro.codex.follow-up-skill-reads` observations. Each applies the same exact
-mounted-skill read check to one completed turn, so an extension can grade read
-order or a missing repeat read without inferring it from the combined list.
-
-It also checks native parent command records for exact mounted skill bodies,
-including sessions with no spawn request. The bounded diagnostic distinguishes
-no read from an incomplete read without claiming skill activation.
-It retains the skill name and native ordinal when verified reads complete a
-whole body. Repeated complete reads produce separate receipts; paged reads
-produce one receipt when their ranges first cover the body. An incomplete
-later reread makes the diagnostic partial. Extensions can compare these
-ordinals with an accepted owner boundary without retaining commands or output.
-Earlier yielded output can complete a read only when its process and unique
-call identity bind it to that actor's native command completion. Completed
-literal command calls use a bounded TypeScript parser; conditional or ambiguous
-code cannot establish that binding. Recovery verifies the exact mounted bytes
-and recomputes completeness for the matching CLI command. Private commands,
-submitted code and recovered skill bodies stay out of public observations.
-For accepted spawns, it also checks up to eight child thread rollouts and
+For accepted spawns, the host checks up to eight child thread rollouts and
 reports whether each is available, unavailable, ambiguous, or partial, with an
-explicit truncation flag. An available rollout includes bounded diagnostics for
-native child commands: the number of commands and skill read attempts, whether
-the exact mounted `SKILL.md` body was read, and whether the check was complete
-or truncated. A rollout with no read differs from a missing or malformed
-rollout. These diagnostics do not establish skill activation or task completion
-by themselves and retain no child prompt, command, or output.
+explicit truncation flag. Child rollout metadata describes delegation and
+completion; skill activation remains in the common hook observation.
 An available child rollout also reports whether one nonempty final assistant
 message matches a later native completion event for the same turn. This proves
 a returned turn, not the accuracy of the child's work, and retains no message
@@ -90,12 +70,18 @@ discarded. Extensions interpret task names and decide whether a nested reader
 satisfies their policy.
 The record proves an invocation attempt, not its success. A missing, ambiguous,
 unreadable, oversized, or malformed session is unavailable or partial, so its
-absence cannot prove that a control was unused. Calls made through submitted
-code are not classified by this first observation.
+absence cannot prove that a control was unused. Code Mode submissions now have
+separate bounded `execCalls` inspection receipts, described in the
+[native control contract](native-controls.md). Direct native call ordinals and
+acceptance receipts keep their original meaning; submitted requests do not
+acquire direct native acceptance receipts.
 For an explicitly declared `$plugin:skill`, the host verifies that the installed
 skill exists and that the exact token occurs once in the delivered prompt. A
-completed turn yields `sevro.codex.explicit-invocation` with the owner first and
-verified supporting reads after it. Incomplete reads make that receipt partial.
+completed turn yields a complete `sevro.codex.explicit-invocation` naming the
+dispatched skill. This receipt proves verified dispatch and contains no
+`observedSkills` list. Activation checks require the common hook observation;
+missing hook evidence cannot be repaired by a dispatch receipt or native
+transcript. Failed execution cannot yield a complete dispatch receipt.
 The `sevro.codex.repository-invocation` capability supports an explicitly
 declared `$skill` from `.agents/skills/<skill>/SKILL.md`. The host checks that
 the repository mount is a bounded regular file with no symlink components and
@@ -103,7 +89,7 @@ that the token occurs once across the delivered turns. It uses the same
 completed-turn dispatch receipt without wrapping the skill as a plugin.
 
 `createCodexHost` copies file-based authentication into a private Codex home
-for one turn. It supplies only explicit environment variables to the parent
+for each trial. It supplies only explicit environment variables to the parent
 process and uses a runner-generated permission profile for candidate commands.
 When the extension declares a local Codex marketplace, the host verifies its
 manifest contains only the named plugins with local sources, rejects package
@@ -111,7 +97,11 @@ symlinks, and installs it with `codex plugin marketplace add` followed by
 `codex plugin add`. It checks each installation receipt against the private
 plugin cache. Candidate commands can read that cache, while the rest of the
 private Codex home, including authentication and configuration, remains denied.
-It refuses fixture-local `.codex` configuration, unsupported enforced
+With the [Codex state opt-in](runtime-v1.md#candidate-codex-configuration-and-user-skills),
+commands may read generated configuration and write actual non-system user skills;
+graders retain read-only access to that request's original paths. Fixture-local
+configuration is accepted as inert represented input on this route. Otherwise,
+the host refuses fixture-local `.codex` configuration. It also refuses unsupported enforced
 conditions, unreadable or oversized auth files, malformed event streams, and
 incomplete turns. The host bounds output and runtime, kills its process group
 on timeout or cancellation, and removes private state after the turn. It is
@@ -119,29 +109,171 @@ available through the engine's injected host interface and the development
 CLI's explicit `--host codex` route. Its protocol and evidence host ID is
 `sevro.host.codex`; `codex` remains the CLI route selector.
 
-`prepareMacSandboxCommand` provides the macOS shell-grading boundary. Callers
+`prepareMacSandboxCommand` and `prepareLinuxSandboxCommand` provide the shell-grading boundary. Callers
 provide absolute protected roots and a private state root; the primitive
 canonicalizes them, refuses overlap with the fixture workspace, and creates a
-temporary `sandbox-exec` profile that denies reads and writes. It fails if
-macOS isolation is unavailable. The caller releases the profile after the
+temporary native sandbox that denies reads and writes. It fails if
+isolation is unavailable. The caller releases the sandbox resources after the
 process exits. The runner assembles the same protected-root set for Codex
 candidate commands and shell grading.
 
 Codex cannot start its native sandbox inside an outer `sandbox-exec` process
-on this host. Its runner-generated permission profile therefore sets an
+on macOS. Its runner-generated permission profile therefore sets an
 explicit workspace rule, denies all other filesystem roots except minimal
 runtime files and the selected executable's install directories, and
 specifically denies protected roots and the private Codex home. An executable
 inside a protected root is rejected. The profile also disables command
-network access and clears inherited command environment variables. The Codex
+network access by default and clears inherited command environment variables.
+The runtime [local HTTP option](runtime-v1.md#local-http-test-servers) enables
+Codex's managed network proxy and local binding for socket-using tests.
+The Codex
 parent can read authentication before it starts sandboxed commands. Tests
 invoke the installed `codex sandbox` CLI to prove fixture access, denied
 source/auth reads, and executable access. The process adapter verifies these
-properties before every turn.
+properties in one sandbox invocation before each fresh host request, with a
+60-second bounded startup deadline. The combined probe still requires readable
+private command storage, denied project listing, denied authentication and
+configuration reads (unless configuration access was explicitly enabled), and a
+successful native executable `--version`. Continuation
+turns reuse the same isolated state. The longer deadline accommodates observed
+native policy startup; it is not counted as a performance saving.
+
+Native plugin installation remains fresh for every host request. Reusing an
+installed home would carry configuration, marketplace registrations, and
+absolute installation paths into a different private namespace. Selected hook
+wrappers additionally embed that request's runtime, sandbox profile, and receipt
+paths. Replaying an old installation receipt cannot establish a new installed
+path. Sevro therefore retains native marketplace/plugin installation and receipt
+verification, fresh credentials/configuration, hook rewriting, preflight, and
+invocation evidence for every request. See the [startup measurements](evidence/runtime-startup-30/README.md)
+for measured boundaries, copy-on-write assessment, and limitations.
+
+## Retired skill-read evidence
+
+The current host no longer emits `sevro.codex.skill-reads`,
+`sevro.codex.initial-skill-reads`, `sevro.codex.follow-up-skill-reads`, or native
+parent/child read diagnostics. Migrate activation checks to
+`sevro.host.skill-activations`; old required evidence remains unavailable rather
+than silently receiving a different meaning. Existing retained run evidence
+keeps its original interpretation.
+
+[Issue #70 representative retests](evidence/skill-reads-70/results.json) retain
+separate task, activation, and phase-read outcomes from the private candidate,
+plus its source fingerprints and test-first regression results. Private native
+transcripts remain outside the repository.
+
+<details>
+<summary>Historical command-pattern and read-recovery contract</summary>
 
+The following describes the retired collector before the hook migration.
+These rules are preserved for interpreting older evidence; they do not describe
+the current host.
+
+The host also retains a bounded `sevro.codex.skill-reads` observation. It
+records the first verified mounted skill and ordered skill names, without
+commands or skill bodies. A `cat` or `sed -n` attempt counts only when its path
+resolves to a mounted skill or a skill below an installed plugin root verified
+by Codex's installation receipt. Missing targets outside a mounted skill
+directory, listings, and other harmless mentions of `SKILL.md` do not make the
+observation partial. A missing `SKILL.md` target does not count as a read just
+because its parent directory belongs to a mounted skill; an existing mounted
+target that fails or produces incomplete bytes remains partial. A completed direct
+`cat` of one mounted `SKILL.md`, alone or alongside literal documentation
+paths, counts only when the command output contains that file's exact body.
+Batched reads naming more than 64 paths remain partial, as do failed,
+malformed, or incomplete recognized reads. Direct
+`sed -n` pages count after their verified line ranges cover the whole body. An
+exact `cat` wrapped by `lean-ctx -c` is accepted only when its reported output
+still contains the full body; a compressed summary cannot prove the read.
+The same exact-body check accepts a literal read followed by one literal `rg`
+command, separated by a semicolon or newline. It also accepts a single-file
+literal `cat` of the skill followed by one or more literal `cat` reads of
+non-skill files, then one literal `rg` command, with the same separators. For
+example:
+
+```sh
+cat .agents/skills/darrow-guide/SKILL.md; cat docs/specs/repository-guide.md; rg -n -i 'offline|\bsla\b|service.level|guarantee|deployment' --glob '*.md' --glob '!docs/research/**' --glob '!**/SKILL.md' .
+```
+
+This sequence uses the shared stream and native read check. Quoted separators
+remain part of their literal operand. The ordered file bodies must match the
+retained output, so documentation or search output cannot replace an altered
+or incomplete skill read. Literal `cat` commands may read multiple mounted
+skills and documentation, before or after skills, with or without a terminal
+search. Multiple literal terminal `rg` commands are accepted within the command
+bound. Each verified skill receives its own ordered receipt. Documentation
+must resolve within the workspace or a receipt-bound installed plugin root;
+each file is bounded to 1 MiB. Aliases of mounted skills without a literal
+`SKILL.md` operand remain unsupported.
+The 64-operand bound applies across all reads in the sequence. Shell expansions,
+including unquoted glob and tilde operands, are unsupported in this form.
+
+The bounded composed-read grammar also accepts `unset` with literal variable
+names, `pwd` or `pwd -P`, `git rev-parse --show-toplevel` (optionally with
+`-C "$PWD"`), `printenv` with literal variable names, and `rg --files` with
+literal arguments before the reads. Prefix `printenv` requires the reads to
+end the command, so environment output cannot replace missing read bytes.
+Literal `printenv` commands may follow reads without prefix `printenv`.
+The fixed pair `repo=$(git -C "$PWD" rev-parse --show-toplevel)` and
+`cd "$repo" && pwd -P` is supported only with absolute read operands. Other
+assignments, substitutions, directory changes, or control flow remain
+unsupported. Prefix and environment output cannot supply skill frontmatter;
+ambiguous repeated output cannot establish an attributable read.
+
+The check also accepts a chain of literal
+`cat` fallbacks with `2>/dev/null` and `||`, ending in `true` followed by a newline
+and one literal `rg` discovery command. A fallback chain must resolve to one
+canonical file across all its read operands; absent alternate targets do not
+invalidate the read. Each fallback `cat` names one literal file. Multiple
+existing targets, including unmounted copies,
+cannot establish an attributable read. The total read operands remain bounded
+at 64. The complete command must succeed and expose the mounted file's entire
+exact body. `cat` accepts literal file operands and the optional `--` delimiter;
+other options cannot prove an unchanged read. Substitutions, pipelines, other shell control flow, and dynamic
+search arguments cannot establish a compound read through this check.
+Unrecognized commands remain partial when their output contains the identifying
+`name: <skill>` line from a mounted or receipt-bound installed-plugin skill.
+This is a host observation; an extension decides what selection means for its
+case.
+The same check accepts skill files below the exact installed plugin roots from
+Codex's installation receipts. It does not infer activation from arbitrary
+files in the private plugin cache.
+The host also retains separate `sevro.codex.initial-skill-reads` and
+`sevro.codex.follow-up-skill-reads` observations. Each applies the same exact
+mounted-skill read check to one completed turn, so an extension can grade read
+order or a missing repeat read without inferring it from the combined list.
+For `app-server`, the verified same-thread feedback boundary separates the
+initial phase from the follow-up phase, including subsequent native goal turns.
+Absent, duplicated, foreign-thread, or failed boundaries do not yield phase
+receipts.
+
+It also checks native parent command records for exact mounted skill bodies,
+including sessions with no spawn request. The bounded diagnostic distinguishes
+no read from an incomplete read without claiming skill activation.
+It retains the skill name and native ordinal when verified reads complete a
+whole body. Repeated complete reads produce separate receipts; paged reads
+produce one receipt when their ranges first cover the body. An incomplete
+later reread makes the diagnostic partial. Extensions can compare these
+ordinals with an accepted owner boundary without retaining commands or output.
+Earlier yielded output can complete a read only when its process and unique
+call identity bind it to that actor's native command completion. Completed
+literal command calls use a bounded TypeScript parser; conditional or ambiguous
+code cannot establish that binding. Recovery verifies the exact mounted bytes
+and recomputes completeness for the matching CLI command. Private commands,
+submitted code and recovered skill bodies stay out of public observations.
+For accepted spawns, it also checks up to eight child thread rollouts and
+reports whether each is available, unavailable, ambiguous, or partial, with an
+explicit truncation flag. An available rollout includes bounded diagnostics for
+native child commands: the number of commands and skill read attempts, whether
+the exact mounted `SKILL.md` body was read, and whether the check was complete
+or truncated. A rollout with no read differs from a missing or malformed
+rollout. These diagnostics do not establish skill activation or task completion
+by themselves and retain no child prompt, command, or output.
 Native read diagnostics retain at most 64 `recoverySources` entries. Each
 records its native ordinal, whether the native completion had output, the
 yielded chunk count, and separate `completedCall` and `literalCommandCall`
 flags. These facts distinguish recovery sources without retaining commands,
 submitted code, or skill bodies. The diagnostic's completeness and truncation
 fields still determine whether its read evidence can support grading.
+
+</details>

package/docs/contracts.md

@@ -0,0 +1,19 @@
+# Current contracts
+
+Read these human contracts and their schemas together:
+
+- [Extension protocol v1](extension-protocol-v1.md): transport and operations.
+- [Results/evidence v1](results-v1.md): states, exits, identity, and retention.
+- [Report v1](report-v1.md): summaries and comparison limitations.
+- [Identity v1](identity-v1.md): versioned comparison dimensions and hashing.
+- [Runtime configuration v1](runtime-v1.md): environment, tools, private state, goals, and hooks.
+- [Run settings v1](run-settings-v1.md): recurring evaluation selections and CLI overrides.
+- [Built-in graders v1](builtin-graders-v1.md): output, shell, Git HEAD, and semantic checks.
+- [Schemas](../schemas/): machine-readable public forms.
+
+[Guide specification](specs/repository-guide.md) governs the repository assistant.
+[LICENSE](../LICENSE) governs licensing. The
+[TypeScript quality contract](typescript-quality.md) defines source inventories,
+development checks, and coverage policy.
+[Architecture](architecture.md) explains implementation context;
+[historical validation](evidence.md) records past observations. Return to the [hub](README.md).

package/docs/creating-cases.md

@@ -0,0 +1,139 @@
+# Create cases and extensions
+
+Start with a direct case for a prompt, fixture, and built-in success criteria.
+Copy [the basic graded case](../examples/basic/graded.json), give it a stable
+ID, and replace the prompt/check. Read [built-in graders](builtin-graders-v1.md)
+for regex, JSON, and bounded schema assertions. No criteria means `not_assessed`.
+For complete coding, repository-skill, runtime, semantic and extension projects,
+follow the [runnable recipes](recipes.md), including their deliberate failures.
+
+Run it with the [CLI](running-evaluations.md). Direct cases follow `ResolvedCase`
+in [the engine](../src/engine.ts). Fixtures can provide inline files, generated
+Git history, or a declared clean repository. The [CLI reference](development-cli.md)
+owns required source roots/maps; an arbitrary returned path does not grant access.
+
+Selecting `--project-root` identifies the project; it does not copy its contents
+into a participant fixture. Declare the files or repository source the participant
+should receive. The [run defaults reference](development-cli.md#run-defaults)
+explains how project identity, fixtures and CLI path resolution interact.
+
+## Evaluate your repository skills
+
+Start with one representative task, your actual skill, and observable success
+criteria. This example evaluates a documentation guide's answer with Codex.
+Use [Sevro from `PATH`](installing.md#make-the-command-available).
+Live runs require the [native host prerequisites](native-hosts.md), supported
+isolation, an authenticated Codex installation, and your chosen model.
+They consume that account's model quota.
+
+Create this layout, keeping evaluator inputs outside the participant fixture:
+
+```text
+skill-evals/
+├── cases/
+│   └── explain-startup.json
+├── sources.json
+└── fixture-repo/
+    ├── README.md
+    └── .agents/skills/my-guide/
+        └── SKILL.md
+```
+
+Copy your guide skill and any supporting resources into `fixture-repo`. Use
+its real name in place of `my-guide`. For this sample task, write a README
+that explains that `bun run dev` starts the application on port `3000`.
+Initialize and commit the fixture repository, leaving it clean. Sevro clones
+that committed revision into a fresh workspace for every trial.
+`--project-root` alone does not copy your project into the participant workspace.
+
+Save this case as `cases/explain-startup.json`:
+
+```json
+{
+  "id": "my-guide-explain-startup",
+  "prompt": "How do I start this application locally, and which port does it use?",
+  "fixture": { "sourceRef": "fixture-repo" },
+  "checks": [
+    {
+      "id": "startup-command",
+      "grader": "sevro.regex",
+      "configuration": { "pattern": "bun run dev" }
+    },
+    {
+      "id": "correct-port",
+      "grader": "sevro.regex",
+      "configuration": { "pattern": "\\b3000\\b" }
+    }
+  ],
+  "requiredEvidence": []
+}
+```
+
+Save the source map as `sources.json`, replacing the example absolute path
+with the location of your fixture repository:
+
+```json
+{
+  "fixture-repo": "file:///absolute/path/skill-evals/fixture-repo"
+}
+```
+
+From `skill-evals`, run the case with your actual model identifier in place
+of `<codex-model>`:
+
+```sh
+sevro run --json \
+  --case-file "$PWD/cases/explain-startup.json" \
+  --case-source-root "$PWD" \
+  --case-source-map-file "$PWD/sources.json" \
+  --project-root "$PWD/fixture-repo" \
+  --results-root "$PWD/results" \
+  --host codex \
+  --codex-bin "$(command -v codex)" \
+  --codex-auth-file "${CODEX_HOME:-$HOME/.codex}/auth.json" \
+  --model <codex-model> --effort medium \
+  --condition passive --trials 3 --threshold 1 > result.json
+```
+
+Add `--dry` first to validate preparation without calling the model. Three
+executed trials assess consistency; threshold `1` requires all three to pass.
+Summarize the retained result:
+
+```sh
+sevro report --result-file "$PWD/result.json"
+```
+
+These regex checks assess answer content; they do not prove that the guide was
+selected or that every claim was correct. Add cases for ordinary selection,
+unrelated requests, missing inputs, follow-ups, and skill-specific boundaries.
+For coding skills, use `sevro.shell` to run evaluator-owned tests against the
+resulting workspace, with the required `--shell-isolation` configuration.
+`sevro.semantic` can assess qualitative requirements through a separate judge.
+The [grader reference](builtin-graders-v1.md) defines their limits.
+
+To grade skill selection or verify explicit invocation, use an extension that
+prepares skill mounts and interprets complete native host evidence. The
+[extension protocol](extension-protocol-v1.md) defines repository and plugin
+invocations; the checkout's [guide evals](guide-evaluation.md) demonstrate the
+approach. Check those capabilities against your exact installed release:
+this checkout can contain [changes beyond its published version](installing.md#published-package-and-checkout).
+
+## Reusable project policy
+
+Use an extension for project-specific discovery, preparation, checks, or
+task-verdict policy. [Extension protocol v1](extension-protocol-v1.md) is the
+contract, backed by [its schema](../schemas/extension-v1.schema.json).
+
+An extension is an explicitly trusted executable. Each process accepts one JSON
+request and returns one matching response. `describe` negotiates identity and
+capabilities; `resolve` supplies cases; `prepare` supplies artifacts and
+negotiated requests; `evaluate` supplies extension results. A process boundary
+alone is not a sandbox. Diagnostics use stderr. Keep secrets out of argv,
+redacted configuration, observations, and artifact references.
+
+Use an argv-array command file, declare source files, and select a case ID as
+shown in the CLI reference. Replacing built-ins or using an extension verdict
+policy needs explicit selection. These are separate controls: choosing a task
+verdict policy does not disable built-in checks. Unsupported capabilities, incomplete evidence,
+and failed exchanges cannot produce a pass. Separate participant-visible inputs
+from evaluator checks; version policy and preserve its [comparison identity](identity-v1.md).