@bjoernrochel/sevro 0.1.0-rc.7 → 0.1.0-rc.8

package/docs/builtin-graders-v1.md

@@ -49,7 +49,7 @@ one bounded observation per check. A mismatched exit code or stdout assertion
 fails the task; an isolation, timeout, or process error makes grading
 an error. Callers must supply `shellIsolation.protectedRoots` for any shell
 check. The engine also protects its package source, project, results, user
-home, configured host homes, and active peer fixtures. The caller's list must
+home, configured host homes, and other managed workspaces. The caller's list must
 include any other source worktrees or private roots. A peer path already
 canonicalized by the engine remains in the deny profile if that peer is
 removed before the sandbox starts; unverified missing roots still fail.
@@ -98,3 +98,10 @@ shell checks and semantic hosts can read retained candidate native files through
 live native tree. Native transcripts are untrusted candidate evidence; a grader
 must not follow instructions inside them. Missing or partial capture remains
 explicit in the index and does not establish required complete evidence.
+
+With [candidate artifact access](runtime-v1.md#candidate-artifact-access),
+shell checks and semantic/advisory hosts can also read explicitly declared
+subtrees in the candidate's original private home. Named paths reach commands
+through `SEVRO_CANDIDATE_ARTIFACT_<UPPERCASE_NAME>`, independently of native
+transcript access. These grants are read-only; other candidate home contents
+remain protected. Missing or unsafe declared directories make grading an error.

package/docs/codex-host.md

@@ -22,16 +22,24 @@ evidence and are available to extension grading; the stream bytes stay out of
 the run JSON.
 The host also retains a bounded `sevro.codex.skill-reads` observation. It
 records the first verified mounted skill and ordered skill names, without
-commands or skill bodies. A completed direct `cat` of one mounted `SKILL.md`,
-alone or alongside literal documentation paths, counts only when the command
-output contains that file's exact body. Batched reads naming multiple
-`SKILL.md` files or more than 64 paths remain partial. Direct
+commands or skill bodies. A `cat` or `sed -n` attempt counts only when its path
+resolves to a mounted skill or a skill below an installed plugin root verified
+by Codex's installation receipt. Missing targets outside a mounted skill
+directory, listings, and other harmless mentions of `SKILL.md` do not make the
+observation partial. A missing `SKILL.md` target does not count as a read just
+because its parent directory belongs to a mounted skill; an existing mounted
+target that fails or produces incomplete bytes remains partial. A completed direct
+`cat` of one mounted `SKILL.md`, alone or alongside literal documentation
+paths, counts only when the command output contains that file's exact body.
+Batched reads naming multiple `SKILL.md` files or more than 64 paths remain
+partial, as do failed, malformed, or incomplete recognized reads. Direct
 `sed -n` pages count after their verified line ranges cover the whole body. An
 exact `cat` wrapped by `lean-ctx -c` is accepted only when its reported output
-still contains the full body; a compressed summary cannot prove the read. An
-attempted indirect, malformed, or incomplete read makes the observation
-partial. This is a host observation; an extension decides what selection means
-for its case.
+still contains the full body; a compressed summary cannot prove the read.
+Unrecognized commands remain partial when their output contains the identifying
+`name: <skill>` line from a mounted or receipt-bound installed-plugin skill.
+This is a host observation; an extension decides what selection means for its
+case.
 The same check accepts skill files below the exact installed plugin roots from
 Codex's installation receipts. It does not infer activation from arbitrary
 files in the private plugin cache.

package/docs/fixture-cleanup-validation.md

@@ -1,5 +1,8 @@
 # Permission-locked fixture cleanup
 
+These historical observations predate the current
+[workspace retention policy](results-v1.md).
+
 The Darrow extraction comparison exposed a cleanup difference: its legacy
 fixture destroyer removes permission-locked directories, while Sevro retained
 them after a completed trial. Sevro now restores owner access inside completed

package/docs/installing.md

@@ -22,10 +22,10 @@ Local installation records the selected version in the project lockfile.
 
 ## Current release
 
-The current release is `0.1.0-rc.7`. Install this exact version:
+The current release is `0.1.0-rc.8`. Install this exact version:
 
 ```sh
-bun add --exact @bjoernrochel/sevro@0.1.0-rc.7
+bun add --exact @bjoernrochel/sevro@0.1.0-rc.8
 ```
 
 <!-- sevro-current-release:end -->

package/docs/results-v1.md

@@ -110,12 +110,31 @@ An interrupted or failed run keeps completed trials plus an explicit partial
 attempt or diagnostic record. Failed persistence is an internal error and
 cannot be presented as a successful assessment.
 
+Successful temporary workspaces are removed after their evidence is saved.
+Failed and cancelled workspaces, including trials with unavailable required
+grading evidence, are kept for 24 hours after becoming inactive. Evaluation
+startup removes expired workspaces with verified inactive ownership. Active or
+unverifiable workspaces and unregistered legacy folders are preserved; saved
+results and evidence remain intact. Failed cleanup is retried on later evaluations.
+
+`sevro.observation.workspaces` records candidate and retained auxiliary workspace
+paths for inspection. These paths are temporary and do not affect evaluation identity.
+
+Opted-in [candidate artifact runtimes](runtime-v1.md#candidate-artifact-access)
+remain available through grading and are removed afterward, including failed
+and cancelled trials. Failed-fixture retention does not preserve them.
+Read access does not automatically retain artifact bytes in run evidence.
+
 A host adapter can return `executionFailed: true` with bounded artifacts from
 a failed process. The engine retains those artifacts, skips all grading, and
 reports `failed` / `not_requested` / `not_assessed` with exit code `2`. A final
 message that happens to match a check cannot turn that failure into a pass.
 Claude retains its bounded event stream and, when present, up to 64 KiB of
 stderr as private host artifacts. Failure bodies do not appear in CLI diagnostics.
+Codex preflight failures retain `codex.process` (stage, reason, exit code, signal,
+duration, and stderr truncation) and `codex.stderr` (the first 64 KiB), with role
+prefixes for semantic/advisory hosts. Diagnostics distinguish failure, timeout,
+and cancellation without exposing raw stderr.
 
 ## Native transcript bundles
 

package/docs/runtime-v1.md

@@ -102,6 +102,10 @@ Existing directory paths and symlink targets are canonicalized. Reads under
 the real home are allowed only through bounded runtime grants. Source and
 configuration worktrees, evaluator inputs, credentials, results, run ownership
 state, and peer fixtures remain protected. A conflicting or broad grant fails.
+Managed workspaces share a private parent that protects sibling workspaces,
+including those created after launch. Independently prepared workspaces require
+explicit protection for other private inputs. Claude hosts require Claude Code
+2.1.257 or later for managed-workspace isolation; see [checked versions](typescript-quality.md#environment-and-ci).
 
 Use `filesystem.optionalReadOnlyRoots` for supporting directories that may not
 be installed on every host, with the same path rules as `readOnlyRoots`:
@@ -179,10 +183,11 @@ state. Command runtime directories are separate from host authentication,
 configuration, and live transcript sources. Checker and hook caches also live
 outside candidate-controlled workspace state. Private roots survive native
 continuation turns and children, then are removed with their owning trial or
-host state after owned hooks terminate. Runs without a selected runtime policy
+host state after owned hooks terminate. Opted-in candidate artifact runtimes
+remain alive through grading as described below. Runs without a selected runtime policy
 retain their previous writable-state locations.
 
-Claude runtime policies refuse a private namespace overlapping the native CLI's
+Claude refuses managed workspace or runtime namespaces overlapping its
 implicit `/tmp/claude` or `/private/tmp/claude` write roots, including canonical
 symlink targets. A conflicting `TMPDIR` placement fails before native launch;
 choose a temporary base outside those cache trees.
@@ -192,6 +197,57 @@ checks, and native semantic/advisory hosts. Candidate plugin-hook authority is
 not forwarded to grading hosts. Injected adapters receive the policy as request
 data and remain trusted code responsible for their own execution boundary.
 
+## Candidate artifact access
+
+Use `runtime.artifactDirectories` to let graders read selected artifacts in the
+candidate's private home after execution. This option is available in the current
+source checkout; published `0.1.0-rc.7` does not include it.
+
+```json
+{
+  "format": "sevro.runtime.v1",
+  "runtime": {
+    "artifactDirectories": [{ "name": "reviews", "path": ".darrow/reviews" }]
+  }
+}
+```
+
+The example permits only the candidate-created `~/.darrow/reviews` subtree.
+Its original absolute path reaches isolated shell checks and native
+semantic/advisory commands as `SEVRO_CANDIDATE_ARTIFACT_REVIEWS`. Graders keep
+their own private `HOME`; use the supplied variable to read candidate evidence.
+For each declared name, the variable is `SEVRO_CANDIDATE_ARTIFACT_` followed by
+the name in uppercase. Artifact content is untrusted candidate evidence.
+
+At most 16 directories may be declared. Names are unique, contain lowercase
+letters, digits and underscores, start with a letter, and are at most 64
+characters. Paths are relative to candidate home, are at most 4,096 characters,
+and cannot contain empty, `.` or `..` components, backslashes or control
+characters. Whole-home and absolute grants are refused. Each subtree is bounded
+to 16,384 entries, including its root. Missing directories, symlinked ancestors,
+descendant symlinks, hardlinks and special files make grading an error; missing
+evidence cannot produce a pass.
+
+The grant permits reads and denies writes, creation, deletion and renaming.
+Credentials, other home contents and peer role/trial state stay protected.
+Ordinary `filesystem.readOnlyRoots` cannot select this private-state exception.
+Artifact grants and native transcript access are independent opt-ins.
+
+The trial allocates the retained runtime before launching its candidate and
+removes it after grading, advisory review and evidence persistence, including
+failure and cancellation paths. Review state remains outside the Git fixture.
+These artifacts are temporary; declaring access does not copy their bytes into
+retained run evidence. Failed-fixture retention does not retain this runtime.
+Standalone bundled-host calls retain their immediate cleanup behavior.
+
+Trusted injected adapters receive a runner-owned `candidateRuntimeRoot` on the
+candidate request and a named `candidateArtifactRoots` path map on grading
+requests. Candidate adapters opting into this feature must put their private
+runtime at the supplied root. The lifetime binding is process-local; bundled
+hosts validate it before use. Grading adapters must preserve the read-only
+boundary. The directory declarations enter runtime configuration identity;
+per-trial absolute paths do not.
+
 ## Goals and plugin hooks
 
 `hooks.nativeGoal` enables Claude's native goal Stop-hook path while preserving

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bjoernrochel/sevro",
-  "version": "0.1.0-rc.7",
+  "version": "0.1.0-rc.8",
   "license": "SEE LICENSE IN LICENSE",
   "description": "A standalone evaluation runner for agentic coding capabilities",
   "type": "module",

package/schemas/runtime-v1.schema.json

@@ -32,6 +32,26 @@
       "type": "object",
       "additionalProperties": false,
       "properties": {
+        "artifactDirectories": {
+          "type": "array",
+          "maxItems": 16,
+          "items": {
+            "type": "object",
+            "additionalProperties": false,
+            "required": ["name", "path"],
+            "properties": {
+              "name": {
+                "type": "string",
+                "pattern": "^[a-z][a-z0-9_]{0,63}$"
+              },
+              "path": {
+                "type": "string",
+                "minLength": 1,
+                "maxLength": 4096
+              }
+            }
+          }
+        },
         "seedDirectories": {
           "type": "array",
           "maxItems": 16,

package/src/advisory-fixture.ts

@@ -3,7 +3,6 @@ import { isMissingFile } from "./fixture-tools";
 import {
   lstat,
   mkdir,
-  mkdtemp,
   readlink,
   readdir,
   realpath,
@@ -11,7 +10,7 @@ import {
   symlink,
   copyFile,
 } from "node:fs/promises";
-import { tmpdir } from "node:os";
+import { allocateWorkspace, removeWorkspace } from "./workspace-state";
 import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
 import { fixtureParts } from "./preparation";
 
@@ -245,7 +244,7 @@ export async function buildBlindAdvisoryFixture(
   requireAdvisorySource(candidateWorkspace, options.baseRevision);
   const source = await realpath(candidateWorkspace);
   const exclusions = advisoryExclusions(options);
-  const temporary = await mkdtemp(join(tmpdir(), "sevro-advisory-"));
+  const temporary = await allocateWorkspace();
   try {
     const destination = await realpath(temporary);
     await initializeAdvisoryBaseline(
@@ -262,7 +261,7 @@ export async function buildBlindAdvisoryFixture(
     );
     return destination;
   } catch (error) {
-    await rm(temporary, { recursive: true, force: true });
+    await removeWorkspace(temporary);
     throw error;
   }
 }

package/src/candidate-artifacts.ts

@@ -0,0 +1,175 @@
+import { lstat, readdir, realpath, rm } from "node:fs/promises";
+import { join, relative, sep } from "node:path";
+import {
+  allocateNativeState,
+  nativeStatePath,
+} from "./native-transcript-state";
+import type { RuntimePolicy, RuntimeRole } from "./runtime-config";
+import { clearFixtureContents } from "./fixture-cleanup";
+
+export interface RuntimeArtifactDirectory {
+  name: string;
+  path: string;
+}
+
+export type CandidateArtifactRoots = Record<string, string>;
+
+const leases = new Map<string, RuntimeArtifactDirectory[]>();
+const MAX_ENTRIES = 16_384;
+
+/** These are relative candidate-home paths, never operator filesystem grants. */
+export function requireArtifactDirectories(
+  directories: RuntimeArtifactDirectory[],
+): void {
+  if (!Array.isArray(directories) || directories.length > 16)
+    throw new Error("runtime artifact directories exceed the limit");
+  const names = new Set<string>();
+  for (const directory of directories) {
+    requireArtifactDirectory(directory);
+    if (names.has(directory.name))
+      throw new Error("runtime artifact directory names overlap");
+    names.add(directory.name);
+  }
+}
+
+function requireArtifactDirectory(directory: RuntimeArtifactDirectory): void {
+  if (!/^[a-z][a-z0-9_]{0,63}$/.test(directory.name))
+    throw new Error("runtime artifact directory name is invalid");
+  const path = directory.path;
+  if (!validArtifactPath(path))
+    throw new Error("runtime artifact directory path is invalid");
+  if (path.split("/").some((part) => !part || part === "." || part === ".."))
+    throw new Error(
+      "runtime artifact directory must be a bounded relative path",
+    );
+}
+
+function validArtifactPath(path: string): boolean {
+  if (typeof path !== "string" || path.length > 4096 || path.includes("\\"))
+    return false;
+  return !Array.from(path).some(
+    (character) =>
+      character.charCodeAt(0) < 0x20 || character.charCodeAt(0) === 0x7f,
+  );
+}
+
+/** Allocate before host execution so exceptions cannot lose cleanup ownership. */
+export async function allocateCandidateRuntime(
+  policy: RuntimePolicy | undefined,
+): Promise<string | undefined> {
+  const directories = policy?.artifactDirectories ?? [];
+  if (!directories.length) return undefined;
+  requireArtifactDirectories(directories);
+  const root = await allocateNativeState("candidate-runtime-");
+  leases.set(root, directories);
+  return root;
+}
+
+export async function requireCandidateRuntime(
+  root: string | undefined,
+  role?: RuntimeRole,
+): Promise<void> {
+  if (root === undefined) return;
+  requireCandidateRole(role);
+  if (!leases.has(root) || (await realpath(root)) !== root)
+    throw new Error("candidate runtime must be owned by the current trial");
+}
+
+function requireCandidateRole(role: RuntimeRole | undefined): void {
+  if (role !== undefined && role !== "candidate")
+    throw new Error(
+      "retained candidate runtime cannot be used by another role",
+    );
+}
+
+export async function candidateArtifactRoots(
+  root: string | undefined,
+): Promise<CandidateArtifactRoots | undefined> {
+  if (root === undefined) return undefined;
+  const directories = leases.get(root);
+  if (!directories) throw new Error("candidate runtime lease is unavailable");
+  const roots = Object.fromEntries(
+    directories.map(({ name, path }) => [name, join(root, "home", path)]),
+  );
+  await requireCandidateArtifactRoots(roots);
+  return roots;
+}
+
+/** Validate the engine binding and refuse candidate-created filesystem aliases. */
+export async function requireCandidateArtifactRoots(
+  roots: CandidateArtifactRoots | undefined,
+): Promise<void> {
+  for (const [name, path] of Object.entries(roots ?? {})) {
+    const { root, directory } = artifactRootBinding(name, path);
+    await requireArtifactAncestors(root, directory.path);
+    await requireArtifactTree(path);
+  }
+}
+
+function artifactRootBinding(name: string, path: string) {
+  const parts = relative(nativeStatePath(), path).split(sep);
+  const root = join(nativeStatePath(), parts[0] ?? "");
+  const directory = leases.get(root)?.find((item) => item.name === name);
+  if (!directory || path !== join(root, "home", directory.path))
+    throw new Error("candidate artifact grant is not bound to an active trial");
+  return { root, directory };
+}
+
+async function requireArtifactAncestors(
+  root: string,
+  path: string,
+): Promise<void> {
+  let current = root;
+  for (const part of ["", "home", ...path.split("/")]) {
+    current = join(current, part);
+    const info = await lstat(current);
+    if (!info.isDirectory() || (await realpath(current)) !== current)
+      throw new Error(
+        "candidate artifact ancestors must be canonical directories",
+      );
+  }
+}
+
+async function requireArtifactTree(root: string): Promise<void> {
+  const pending = [root];
+  for (const path of pending) {
+    if (pending.length > MAX_ENTRIES)
+      throw new Error("candidate artifact tree exceeds the entry limit");
+    pending.push(...(await artifactChildren(path)));
+  }
+}
+
+async function artifactChildren(path: string): Promise<string[]> {
+  const info = await lstat(path);
+  if (info.isDirectory())
+    return (await readdir(path)).map((name) => join(path, name));
+  if (!info.isFile() || info.nlink !== 1)
+    throw new Error("candidate artifact tree contains a link or special file");
+  return [];
+}
+
+export function candidateArtifactEnvironment(
+  roots: CandidateArtifactRoots | undefined,
+): Record<string, string> {
+  return Object.fromEntries(
+    Object.entries(roots ?? {}).map(([name, path]) => [
+      `SEVRO_CANDIDATE_ARTIFACT_${name.toUpperCase()}`,
+      path,
+    ]),
+  );
+}
+
+export function candidateArtifactReadRoots(
+  roots: CandidateArtifactRoots | undefined,
+): string[] {
+  return Object.values(roots ?? {});
+}
+
+export async function releaseCandidateRuntime(
+  root: string | undefined,
+): Promise<void> {
+  if (root === undefined) return;
+  await clearFixtureContents(root);
+  await rm(root, { recursive: true, force: true });
+  leases.delete(root);
+}

package/src/engine.ts

@@ -9,3 +9,7 @@ export type {
 export { runEvaluation } from "./evaluation-run";
 export { loadRuntimeConfiguration } from "./runtime-config";
 export type { RuntimeConfiguration, RuntimePolicy } from "./runtime-config";
+export type {
+  RuntimeArtifactDirectory,
+  CandidateArtifactRoots,
+} from "./candidate-artifacts";

package/src/evaluation-context.ts

@@ -1,5 +1,9 @@
-import { mkdir, mkdtemp, realpath, rm, writeFile } from "node:fs/promises";
-import { tmpdir } from "node:os";
+import { mkdir, realpath, rm, writeFile } from "node:fs/promises";
+import {
+  allocateWorkspace,
+  retainWorkspace,
+  removeWorkspace,
+} from "./workspace-state";
 import { dirname, join, resolve } from "node:path";
 import { pathToFileURL } from "node:url";
 import { createFixture } from "./evaluation-fixture";
@@ -116,13 +120,9 @@ export async function reserveEvaluationWorkspaces(
     trial <= options.trialCount && !options.signal?.aborted;
     trial++
   ) {
-    reservedWorkspaces.push(
-      await realpath(await mkdtemp(join(tmpdir(), "sevro-case-"))),
-    );
+    reservedWorkspaces.push(await allocateWorkspace());
     if (preparedSemantic.length)
-      reservedSemanticWorkspaces.push(
-        await realpath(await mkdtemp(join(tmpdir(), "sevro-case-"))),
-      );
+      reservedSemanticWorkspaces.push(await allocateWorkspace());
   }
   const diagnostics: {
     trial: number;
@@ -243,12 +243,14 @@ export function diagnoseEvaluationFailure(context: EvaluationRuntime): void {
 export async function cleanupEvaluationWorkspaces(
   context: EvaluationRuntime,
 ): Promise<void> {
+  for (const workspace of context.retainedWorkspaces)
+    await retainWorkspace(workspace);
   await Promise.all(
     [...context.reservedWorkspaces, ...context.reservedSemanticWorkspaces]
       .filter((workspace) => !context.retainedWorkspaces.has(workspace))
       .map(async (workspace) => {
         try {
-          await rm(workspace, { recursive: true, force: true });
+          await removeWorkspace(workspace);
         } catch {
           console.warn(`fixture cleanup failed; retained at ${workspace}`);
         }

package/src/evaluation-fixture.ts

@@ -1,15 +1,7 @@
 import { isRecord } from "./value-guards";
 import { sha256 } from "./identity";
-import {
-  appendFile,
-  mkdir,
-  mkdtemp,
-  readFile,
-  realpath,
-  rm,
-  writeFile,
-} from "node:fs/promises";
-import { tmpdir } from "node:os";
+import { appendFile, mkdir, readFile, writeFile } from "node:fs/promises";
+import { allocateWorkspace, removeWorkspace } from "./workspace-state";
 import { dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
 import { canonicalJson } from "./identity";
@@ -53,9 +45,7 @@ export async function createFixture(
   reservedWorkspace?: string,
 ): Promise<string> {
   const paths = inlineFixturePaths(fixture, generated, repository);
-  const workspace =
-    reservedWorkspace ??
-    (await realpath(await mkdtemp(join(tmpdir(), "sevro-case-"))));
+  const workspace = reservedWorkspace ?? (await allocateWorkspace());
   try {
     await initializeFixtureHistory(
       fixture,
@@ -76,8 +66,7 @@ export async function createFixture(
     await fixtureBinDirectory(workspace);
     return workspace;
   } catch (error) {
-    if (!reservedWorkspace)
-      await rm(workspace, { recursive: true, force: true });
+    if (!reservedWorkspace) await removeWorkspace(workspace);
     throw error;
   }
 }

package/src/evaluation-run.ts

@@ -12,6 +12,7 @@ import {
   type Assessment,
 } from "./results";
 import { runtimePolicySnapshot } from "./runtime-config";
+import { cleanupExpiredWorkspaces } from "./workspace-state";
 import { assertCliResult, assertRunEvidence } from "./schema";
 import { atomicWriteJson } from "./storage";
 import {
@@ -178,6 +179,7 @@ export async function runEvaluation(
     ...options,
     runtimePolicy: runtimePolicySnapshot(options.runtimePolicy),
   };
+  await cleanupExpiredWorkspaces();
   const plan = await prepareEvaluation(options);
   const runtime = await allocateEvaluationRun(options, plan);
   try {