@bjoernrochel/sevro 0.1.0-rc.4 → 0.1.0-rc.5

package/CONTRIBUTING.md

@@ -71,6 +71,41 @@ Run `bun run check:docs:external` separately and [triage network failures](docs/
 Guide changes also need [native evals](docs/guide-evaluation.md).
 Documentation checks supplement #1 rather than replacing its contract.
 
+## Delivery and CI ownership
+
+An implementation PR is complete only when required CI checks pass for its exact
+published commit. A request to open or update a PR includes monitoring CI and
+repairing in-scope failures; it does not authorize merging or unrelated changes.
+
+- Use GitHub CI in `.github/workflows/verify.yml` for full acceptance validation.
+  It provides the configured OS, architecture, runtime and native-tool versions,
+  non-root user, and isolation prerequisites. Reproducing that environment locally
+  with Docker is optional for diagnosis and is not a publication prerequisite.
+- After the local pre-commit checks, publish the authorized candidate to its PR
+  and require `bun run check:typescript` to succeed in GitHub CI before reporting
+  delivery as complete. A local run, fast check, targeted suite, or historical
+  result does not establish that the current Ubuntu gate passes.
+- Keep the quality bar intact. Repair behavior or add meaningful coverage for
+  accepted behavior; do not lower thresholds, exempt authored sources, skip tests,
+  bypass checks, or add artificial tests just to move a counter. Remove impossible
+  branches by clarifying validated invariants rather than inventing impossible inputs.
+- Bind validation to the candidate's exact source and commit. Changes to source,
+  tests, tooling, dependencies, schemas, workflows, or guidance invalidate prior
+  candidate evidence; require the applicable CI gates on the updated published head.
+- After authorized publication, observe required checks for the exact PR head
+  until they finish. Queued or running checks, a successful older commit, and a
+  passing subset are not completion.
+- On failure, inspect the failed job and retained artifacts, reproduce the cause
+  locally when useful, repair within the authorized scope, and publish a non-force
+  update to the same PR. Continue monitoring without waiting for the user to notice
+  the failure or request correction; do not open a duplicate PR or merge it.
+- If an external blocker prevents progress, report its concrete evidence and the
+  unfinished checks. Keep the PR's status and description honest; do not report
+  delivery as complete while required checks remain failing or unobserved.
+
+Release metadata follows `docs/releases.md` and its existing verified-source CI
+reuse policy; these implementation gates do not require release test reruns.
+
 ## Tests, fixtures, and schemas
 
 Preserve real Bun filesystem/process integration tests and native isolation.
@@ -105,10 +140,16 @@ native trials or remote CI results.
 
 ## Maintainer releases
 
-Follow [releases](docs/releases.md): choose a new version, prepare and inspect
-the exact tarball, verify its license/provenance, and validate that artifact.
+Follow [releases](docs/releases.md): reuse successful source CI, choose a new
+version, and submit the release commit through a PR. Release PRs reuse the
+existing full quality gate and run static metadata checks; releases do not
+rerun local or CI tests. After the PR merges, prepare and inspect the exact
+tarball and verify its license, provenance, and checksums before publication.
 After changing `package.json.version`, run `bun run docs:sync` to refresh the
 [current release](docs/installing.md#current-release). `check:docs`, release
 preparation, and package-install validation reject stale release documentation.
+Package-install, native, and Darrow compatibility tests belong to development
+or an explicitly requested diagnostic, not the release process. Darrow's
+dependency update is a separate PR.
 Publication, tags, registry changes, and alternative licensing are separate
 maintainer actions; preparing a contribution does not authorize them.

package/docs/installing.md

@@ -22,10 +22,10 @@ Local installation records the selected version in the project lockfile.
 
 ## Current release
 
-The current release is `0.1.0-rc.4`. Install this exact version:
+The current release is `0.1.0-rc.5`. Install this exact version:
 
 ```sh
-bun add --exact @bjoernrochel/sevro@0.1.0-rc.4
+bun add --exact @bjoernrochel/sevro@0.1.0-rc.5
 ```
 
 <!-- sevro-current-release:end -->

package/docs/releases.md

@@ -28,8 +28,9 @@ The record identifies the package, version, release tag, distribution tag,
 tarball SHA-256 and npm SHA-512 integrity, and the preparation runtimes.
 Preparation does not publish a package or create a Git tag.
 
-The package installation check accepts `--tarball <absolute file>` to verify
-that exact candidate without repacking. The supplied tarball remains intact.
+For development or explicitly requested diagnostics, the package installation
+check accepts `--tarball <absolute file>` to verify an exact candidate without
+repacking. It is not a release step. The supplied tarball remains intact.
 Its installed name and version must match the source metadata. Installation,
 CLI execution, generic grading, prompt-only execution, release provenance,
 reporting, and supported deterministic native fixtures run without source Git
@@ -37,43 +38,66 @@ metadata in the installed package.
 
 ## Maintainer workflow
 
-1. Select the license, update `package.json` to the intended release version,
-   run `bun run docs:sync`, and commit those changes. Preserve the existing public schemas and document
-   behavior changes. Set `publishConfig.tag` to `latest` for the recommended
-   default or `next` for an opt-in preview. A prerelease does not establish full
-   Darrow migration completion.
-2. On macOS with Bun 1.3.13, Node 24, npm, and `sandbox-exec`, run frozen
-   installation and the full quality gate:
+Release already verified code. Do not rerun tests locally or in CI as part of a
+release: reuse the successful source CI run. Code, dependencies, schemas, tests,
+examples, license changes, and other quality inputs belong in ordinary changes
+with their own CI before starting the release.
+
+1. Start a release branch from verified `main`. Update `package.json.version`
+   to a new release version, keep the selected license, and run `bun run docs:sync`.
+   Set `publishConfig.tag` to `latest` for the recommended default or `next`
+   for an opt-in preview. Commit and open a pull request; every release commit
+   must reach `main` through a PR. Never attempt a direct release push to `main`.
+2. Merge the release PR after its static checks and reused CI evidence pass.
+   Release CI compares the candidate against a successful ancestor run on
+   `main`, permitting only version/distribution-tag changes and the exact
+   release documentation and workflow files inventoried in
+   [the CI selector](../.github/scripts/release-ci.py). All other tracked inputs
+   must match. The original run must have completed the full quality gate;
+   another release's reused status is insufficient. CI retains the source SHA
+   and run URL in `release-ci.json`. Missing evidence fails the release check;
+   it does not trigger another test run. Ordinary code changes and explicit
+   workflow dispatches continue to run the full gate.
+3. From the clean, merged release commit, use macOS, Bun 1.3.13, Node 24,
+   and npm to prepare one exact candidate:
 
    ```sh
    bun install --frozen-lockfile
-   bun run check:typescript
+   bun run release:prepare --tag "v${SEVRO_RELEASE_VERSION:?set the selected new release version}" --output /absolute/new/release-directory
    ```
 
-3. Prepare and test the exact candidate:
+4. Inspect the version, repository, license, distribution tag, file inventory,
+   and retained checksums. Publish that tarball with lifecycle scripts disabled
+   using the commands below. Do not run `check:typescript`, `test:package-install`,
+   Darrow's archive/installed gates, or native trials during release. Diagnostic
+   tests require a separate explicit request.
+5. Once npm serves the version, compare `dist.integrity` with `release.json`
+   and verify the intended distribution tag:
 
    ```sh
-   bun run release:prepare --tag "v${SEVRO_RELEASE_VERSION:?set the selected new release version}" --output /absolute/new/release-directory
-   bun run test:package-install --tarball "/absolute/new/release-directory/bjoernrochel-sevro-${SEVRO_RELEASE_VERSION}.tgz"
+   npm view "@bjoernrochel/sevro@${SEVRO_RELEASE_VERSION}" version dist.integrity --json
+   npm view @bjoernrochel/sevro dist-tags --json
    ```
 
-4. Run Darrow's public integration against that same tarball:
+   npm may take a few minutes to process the accepted upload. Wait for the
+   registry record rather than republishing. Retain the reviewed tarball,
+   checksums, and original CI evidence.
+
+6. Create the matching Git tag at the merged release commit and push only the tag:
 
    ```sh
-   SEVRO_PACKAGE_TARBALL="/absolute/new/release-directory/bjoernrochel-sevro-${SEVRO_RELEASE_VERSION}.tgz" bun run test:eval-package
+   git tag -a "v${SEVRO_RELEASE_VERSION}" -m "Release ${SEVRO_RELEASE_VERSION}"
+   git push origin "refs/tags/v${SEVRO_RELEASE_VERSION}"
    ```
 
-5. Review the changes, file inventory, retained checks, and compatibility
-   limitations before authorizing publication. Keep the reviewed tarball and
-   its checksums. Publish that artifact rather than repacking after review.
-6. After publication, verify registry integrity, install the exact version in
-   Darrow, and run its frozen installed compatibility gate. Retain the prior
-   version's evidence for updates. Roll back through a new dependency/lockfile
-   commit and rerun the installed gate; never relabel failed-version evidence.
+Darrow dependency/lockfile updates and compatibility gates are a separate
+Darrow PR, not prerequisites for npm publication. A prerelease does not establish
+full Darrow migration completion. Preserve prior-version evidence for updates
+and rollback; never relabel failed-version evidence.
 
 ## Local macOS publication
 
-There is no GitHub release job. The maintainer prepares, verifies, and publishes
+There is no GitHub release job. The maintainer prepares, inspects, and publishes
 the reviewed tarball from macOS with an authenticated npm account. Check its
 retained checksum immediately before publication:
 

package/docs/runtime-v1.md

@@ -50,8 +50,19 @@ declare only non-secret values. Credential names and runner-owned variables
 such as `HOME`, temporary-directory variables, `CODEX_HOME`, `CLAUDE_CONFIG_DIR`,
 shell startup overrides, subprocess injection flags, and `UV_OFFLINE` are refused.
 
-An inherited or explicit `PATH` must contain nonempty absolute entries. Missing
-entries are inert. Sevro grants read-only access to existing directories,
+An inherited or explicit `PATH` must contain nonempty absolute entries. For an
+inherited `PATH`, Sevro removes entries whose directory, installation prefix, or
+symlinked tools overlap credential roots, including `~/.codex` and `~/.claude`.
+This lets evaluations invoked from a Codex CLI session use ordinary host tools
+without manually removing injected paths. Lexical paths and canonical targets
+are checked, including absent credential paths. The remaining entries keep their
+original order; the effective `PATH` is retained in runtime evidence. A `PATH`
+with no remaining entries fails rather than selecting fallback tools.
+
+An explicit `environment.set.PATH` remains authoritative and conflicting grants
+fail, even when `PATH` is also inherited. Source and evaluator overlaps, invalid
+entries, and unreadable unprotected directories remain errors for either form.
+Missing unprotected entries are inert. Sevro grants read-only access to existing directories,
 resolved symlinked tools, and bounded installation prefixes above `bin`.
 An inaccessible individual symlink target is inert; an unreadable PATH directory
 still fails configuration. Explicitly declared read roots must remain readable.
@@ -145,9 +156,21 @@ than credential-bearing homes or evaluator directories.
 `{{sevro.runtime}}` in an environment value resolves to that role's writable
 root. Sevro supplies private `HOME` and temporary storage and sets `UV_OFFLINE=1`
 and `PYTHONDONTWRITEBYTECODE=1`. Candidate, hook, checker, semantic, and advisory
-state are separate. Checker and hook caches live outside candidate-controlled
-workspace state. Writable caches stay outside assessed worktree contents.
-They are removed with their owning trial or host state.
+state are separate. With a runtime policy, native candidate, semantic, and
+advisory roots live in a unique runner-owned host directory outside the Git
+fixture. The root contains `home`, `tmp`, and declared seed slots; plugins can
+use paths such as `~/.darrow/reviews` without creating repository-local review
+state. Command runtime directories are separate from host authentication,
+configuration, and live transcript sources. Checker and hook caches also live
+outside candidate-controlled workspace state. Private roots survive native
+continuation turns and children, then are removed with their owning trial or
+host state after owned hooks terminate. Runs without a selected runtime policy
+retain their previous writable-state locations.
+
+Claude runtime policies refuse a private namespace overlapping the native CLI's
+implicit `/tmp/claude` or `/private/tmp/claude` write roots, including canonical
+symlink targets. A conflicting `TMPDIR` placement fails before native launch;
+choose a temporary base outside those cache trees.
 
 The same declared tool environment reaches native candidates, isolated shell
 checks, and native semantic/advisory hosts. Candidate plugin-hook authority is

package/docs/typescript-quality.md

@@ -170,9 +170,22 @@ GitHub documents the [Ubuntu runner labels](https://docs.github.com/en/actions/r
 Ubuntu installs `bubblewrap` and `socat` and runs the same native isolation cases;
 local macOS runs retain `sandbox-exec`. Windows remains unverified.
 
-CI invokes the canonical gate, uploads `.quality` even when coverage fails, and
-publishes the stable aggregate `TypeScript quality` status. A cancelled, skipped
-or failed gate cannot turn that status green.
+Ordinary changes and explicit workflow dispatches invoke the canonical gate,
+upload `.quality` even when coverage fails, and publish the stable aggregate
+`TypeScript quality` status. A cancelled or failed gate cannot turn that status
+green.
+
+Release metadata changes follow [the release workflow](releases.md) and reuse a
+successful full gate from an ancestor on `main`, with identical code,
+dependencies, schemas, tests, and other quality inputs. The selector permits
+only version/distribution-tag changes and its exact release-file inventory.
+It verifies the original full-gate step, so a reused release run cannot serve
+as fresh test evidence. Release CI runs `check:typescript --fast`, records the
+tested source and CI run URL, and skips the test/coverage job. The aggregate
+status requires both verified prior evidence and successful static checks;
+missing, cancelled, failed, or ambiguous evidence cannot establish a pass.
+Reused results remain attributed to their original run rather than being
+reported as newly measured coverage. Releases perform no local test reruns.
 
 The [coverage tooling README](../scripts/coverage/README.md) explains the maintained
 pipeline and file responsibilities. Fresh CI reports provide candidate-specific evidence.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bjoernrochel/sevro",
-  "version": "0.1.0-rc.4",
+  "version": "0.1.0-rc.5",
   "license": "SEE LICENSE IN LICENSE",
   "description": "A standalone evaluation runner for agentic coding capabilities",
   "type": "module",

package/src/case-validation.ts

@@ -0,0 +1,82 @@
+import { type ResolvedCase } from "./engine";
+import { prepareGeneratedFixture } from "./generated-fixture";
+import { prepareRepositoryFixture } from "./repository-fixture";
+import {
+  isOptionalNonblankString,
+  isRecord,
+  isStringArray,
+} from "./value-guards";
+
+function validGeneratedFixture(value: Record<string, unknown>): boolean {
+  try {
+    prepareGeneratedFixture(value);
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+function validRepositoryFixture(value: Record<string, unknown>): boolean {
+  try {
+    prepareRepositoryFixture({ kind: "repository", ...value });
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+export function validInlineFixture(value: Record<string, unknown>): boolean {
+  return (
+    !Object.hasOwn(value, "sourceRef") &&
+    Object.hasOwn(value, "files") &&
+    isRecord(value.files) &&
+    Object.values(value.files).every((content) => typeof content === "string")
+  );
+}
+
+function fixtureFields(value: Record<string, unknown>): boolean {
+  if (Object.hasOwn(value, "sourceRef")) return validRepositoryFixture(value);
+  return validInlineFixture(value);
+}
+
+function validFixture(value: unknown): boolean {
+  if (!isRecord(value)) return false;
+  if (value.kind === "generated") return validGeneratedFixture(value);
+  if (value.kind !== undefined) return false;
+  return fixtureFields(value);
+}
+
+function nonemptyString(value: unknown): value is string {
+  return typeof value === "string" && value.length > 0;
+}
+
+function validCaseDetails(value: Record<string, unknown>): boolean {
+  return (
+    nonemptyString(value.id) &&
+    nonemptyString(value.prompt) &&
+    isOptionalNonblankString(value.followUpPrompt) &&
+    validFixture(value.fixture)
+  );
+}
+
+function validCheck(value: unknown): boolean {
+  return (
+    isRecord(value) &&
+    typeof value.id === "string" &&
+    typeof value.grader === "string" &&
+    isRecord(value.configuration)
+  );
+}
+
+function validChecks(value: unknown): boolean {
+  return Array.isArray(value) && value.every(validCheck);
+}
+
+export function isResolvedCase(value: unknown): value is ResolvedCase {
+  return (
+    isRecord(value) &&
+    validCaseDetails(value) &&
+    validChecks(value.checks) &&
+    isStringArray(value.requiredEvidence)
+  );
+}

package/src/cli.ts

@@ -1,32 +1,31 @@
 #!/usr/bin/env bun
-import { resolvedFixture } from "./resolved-case";
-import { isStringArray } from "./value-guards";
 import { readFile } from "node:fs/promises";
 import { isAbsolute, join } from "node:path";
 import { fileURLToPath, pathToFileURL } from "node:url";
+import { isResolvedCase } from "./case-validation";
 import {
   EvaluationConfigurationError,
   runEvaluation,
   type HostAdapter,
   type ResolvedCase,
 } from "./engine";
+import { openExtensionSession, type ExtensionCase } from "./extension-session";
+import { createClaudeHost } from "./hosts/claude";
 import { createCodexHost } from "./hosts/codex";
 import {
   codexAgentConcurrency,
   configurationRoot,
 } from "./hosts/codex-configuration";
-import { createClaudeHost } from "./hosts/claude";
-import { prepareGeneratedFixture } from "./generated-fixture";
-import { prepareRepositoryFixture } from "./repository-fixture";
 import { prepareInstrumentation } from "./instrumentation";
-import { openExtensionSession, type ExtensionCase } from "./extension-session";
-import { assertCliResult } from "./schema";
-import { reportCommand } from "./report";
 import {
   packageBuildDigest,
   projectIdentityDigest,
   projectProvenance,
 } from "./provenance";
+import { reportCommand } from "./report";
+import { resolvedFixture } from "./resolved-case";
+import { assertCliResult } from "./schema";
+import { isRecord, isStringArray } from "./value-guards";
 
 import { InvocationError, parseInvocation } from "./cli-invocation";
 import {
@@ -35,89 +34,10 @@ import {
 } from "./runtime-config";
 import { canonicalRuntimeRoot, requireRuntimeReadRoots } from "./runtime-paths";
 
-function record(value: unknown): value is Record<string, unknown> {
-  return Boolean(value && typeof value === "object" && !Array.isArray(value));
-}
-
-function validGeneratedFixture(value: Record<string, unknown>): boolean {
-  try {
-    prepareGeneratedFixture(value);
-    return true;
-  } catch {
-    return false;
-  }
-}
-
-function validRepositoryFixture(value: Record<string, unknown>): boolean {
-  try {
-    prepareRepositoryFixture({ kind: "repository", ...value });
-    return true;
-  } catch {
-    return false;
-  }
-}
-
-function validInlineFixture(value: Record<string, unknown>): boolean {
-  return (
-    !Object.hasOwn(value, "sourceRef") &&
-    record(value.files) &&
-    Object.values(value.files).every((content) => typeof content === "string")
-  );
-}
-
-function fixtureFields(value: Record<string, unknown>): boolean {
-  if (Object.hasOwn(value, "sourceRef")) return validRepositoryFixture(value);
-  return Object.hasOwn(value, "files") && validInlineFixture(value);
-}
-
-function validFixture(value: unknown): boolean {
-  if (!record(value)) return false;
-  if (value.kind === "generated") return validGeneratedFixture(value);
-  if (value.kind !== undefined) return false;
-  return fixtureFields(value);
-}
-
-function nonemptyString(value: unknown): value is string {
-  return typeof value === "string" && value.length > 0;
-}
-
-function followUpPrompt(value: unknown): boolean {
-  return (
-    value === undefined || (typeof value === "string" && Boolean(value.trim()))
-  );
-}
-
-function validCaseDetails(value: Record<string, unknown>): boolean {
-  return (
-    nonemptyString(value.id) &&
-    nonemptyString(value.prompt) &&
-    followUpPrompt(value.followUpPrompt) &&
-    validFixture(value.fixture)
-  );
-}
-
-function validCheck(value: unknown): boolean {
-  return (
-    record(value) &&
-    typeof value.id === "string" &&
-    typeof value.grader === "string" &&
-    record(value.configuration)
-  );
-}
-
-function validChecks(value: unknown): boolean {
-  return Array.isArray(value) && value.every(validCheck);
-}
-
 function parseCase(value: unknown): ResolvedCase {
-  if (
-    !record(value) ||
-    !validCaseDetails(value) ||
-    !validChecks(value.checks) ||
-    !isStringArray(value.requiredEvidence)
-  )
+  if (!isResolvedCase(value))
     throw new InvocationError("invalid resolved case file");
-  return value as unknown as ResolvedCase;
+  return value;
 }
 
 async function loadCase(path: string): Promise<ResolvedCase> {
@@ -169,7 +89,7 @@ async function loadExtensionOptions(
         "redacted extension configuration file",
       )
     : {};
-  if (!record(configuration) || !record(redactedConfiguration))
+  if (!isRecord(configuration) || !isRecord(redactedConfiguration))
     throw new InvocationError("extension configuration must be JSON objects");
   return { command, configuration, redactedConfiguration };
 }
@@ -183,7 +103,7 @@ async function loadPreparationSources(
     "case source map file",
   );
   if (
-    !record(refs) ||
+    !isRecord(refs) ||
     !Object.values(refs).every(
       (value) => typeof value === "string" && value.startsWith("file:///"),
     )
@@ -221,7 +141,7 @@ async function adapterModule(path: string): Promise<unknown> {
 
 async function loadHost(path: string): Promise<HostAdapter> {
   const module = await adapterModule(path);
-  const host = record(module) ? module.default : null;
+  const host = isRecord(module) ? module.default : null;
   if (!validHostAdapter(host))
     throw new InvocationError(
       "host adapter module has no valid default adapter",
@@ -248,7 +168,7 @@ function validateHostInstrumentation(host: Record<string, unknown>): void {
 
 function validHostAdapter(value: unknown): value is Record<string, unknown> {
   return (
-    record(value) &&
+    isRecord(value) &&
     typeof value.id === "string" &&
     typeof value.model === "string" &&
     typeof value.effort === "string" &&
@@ -295,6 +215,7 @@ function display(value: unknown, json: boolean): void {
     grading: { status: string };
     task: { verdict: string };
     evidencePath: string | null;
+    diagnostic?: { code: string; message: string };
     cases: Array<{
       caseId: string;
       trials: Array<{
@@ -307,6 +228,8 @@ function display(value: unknown, json: boolean): void {
     `execution=${result.execution.status} grading=${result.grading.status} task=${result.task.verdict}\n`,
   );
   displayDomainOutcomes(result.cases);
+  if (result.diagnostic?.code === "sevro.grader.error")
+    process.stdout.write(`diagnostic=${result.diagnostic.message}\n`);
   if (result.evidencePath)
     process.stdout.write(`evidence=${result.evidencePath}\n`);
 }