@bjoernrochel/sevro 0.1.0-rc.5 → 0.1.0-rc.7

package/docs/builtin-graders-v1.md

@@ -67,7 +67,10 @@ ancestry result; checks reference that observation.
 per check. The operator supplies a separate semantic host route. After a
 complete candidate response, the engine sends the bounded final message and
 all declared propositions to that route in an empty, temporary workspace. It
-requires exactly one `pass` or `fail` verdict with a reason for every declared
+omits dependency seeds unless their runtime `roles` explicitly include `semantic`.
+Declared tool environment and read-only grants remain available. Artifact checks
+retain the default dependency selection; executable shell checks retain their
+separate `checks` runtime. The semantic route requires exactly one `pass` or `fail` verdict with a reason for every declared
 check ID. A failed proposition fails the task. A missing or incomplete candidate
 message makes semantic checks unavailable without calling the semantic host; a
 failed, incomplete, or malformed semantic response makes grading an error.

package/docs/codex-host.md

@@ -148,7 +148,22 @@ network access and clears inherited command environment variables. The Codex
 parent can read authentication before it starts sandboxed commands. Tests
 invoke the installed `codex sandbox` CLI to prove fixture access, denied
 source/auth reads, and executable access. The process adapter verifies these
-properties before every turn.
+properties in one sandbox invocation before each fresh host request, with a
+60-second bounded startup deadline. The combined probe still requires readable
+private command storage, denied project listing, denied authentication and
+configuration reads, and a successful native executable `--version`. Continuation
+turns reuse the same isolated state. The longer deadline accommodates observed
+native policy startup; it is not counted as a performance saving.
+
+Native plugin installation remains fresh for every host request. Reusing an
+installed home would carry configuration, marketplace registrations, and
+absolute installation paths into a different private namespace. Selected hook
+wrappers additionally embed that request's runtime, sandbox profile, and receipt
+paths. Replaying an old installation receipt cannot establish a new installed
+path. Sevro therefore retains native marketplace/plugin installation and receipt
+verification, fresh credentials/configuration, hook rewriting, preflight, and
+invocation evidence for every request. See the [startup measurements](evidence/runtime-startup-30/README.md)
+for measured boundaries, copy-on-write assessment, and limitations.
 
 Native read diagnostics retain at most 64 `recoverySources` entries. Each
 records its native ordinal, whether the native completion had output, the

package/docs/development-cli.md

@@ -53,6 +53,14 @@ only passive conditions. `--host` and `--adapter-module` are exclusive. Use
 `--protected-root` to add private roots for Codex, including when no shell
 checks are selected.
 
+Use `--codex-timeout-ms <milliseconds>` to set the native Codex execution
+timeout. It accepts an integer from `1` through `1800000` (30 minutes) and
+defaults to `600000` (10 minutes) when omitted. It applies to every selected
+Codex route, including semantic and advisory roles, and to both `exec` and
+`app-server` entrypoints. For a 20-minute app-server evaluation, add
+`--codex-entrypoint app-server --codex-timeout-ms 1200000` to the Codex command
+above. The option requires a Codex host route; invalid values fail before a run.
+
 ### Trial concurrency
 
 `--jobs <positive integer>` bounds simultaneous trials within one case and

package/docs/evidence/runtime-startup-30/README.md

@@ -0,0 +1,185 @@
+# Runtime startup measurements for issue 30
+
+The matched synthetic native-host benchmark reduced candidate setup plus the
+controlled participant from 14.37–14.92 seconds to 6.20–6.66 seconds. The response-only
+grading host call, including its separate native startup, fell from 13.49–15.03
+seconds to 5.45–5.65 seconds. These are runner-boundary measurements with synthetic
+model responses, not measurements of API latency or live model quality.
+
+## Reproduction and conditions
+
+Run the same maintained [benchmark harness](../../../scripts/benchmark-runtime-startup.ts)
+against an unchanged checkout of `c205482b25d4d52a395c70434f2b392f6acfb279` and this
+candidate, sequentially on one machine:
+
+```sh
+bun scripts/benchmark-runtime-startup.ts /absolute/baseline /tmp/before.json
+bun scripts/benchmark-runtime-startup.ts /absolute/candidate /tmp/after.json
+```
+
+The retained [before](before.json) and [after](after.json) observations used
+macOS arm64, Bun 1.3.13, Codex CLI 0.162.1, the same synthetic plugin/response case,
+`synthetic-candidate` and `synthetic-semantic` model labels, and `low` effort.
+There were three candidate/semantic repetitions in each process, with a 32 MiB,
+2,049-entry public dependency tree. Every role received a fresh private home and
+native plugin installation. The first candidate used a cold verified-snapshot
+cache; repetitions two and three reused the policy. OS filesystem caches were
+uncontrolled and are not described as cold. Baseline ran before candidate;
+order effects and three repetitions limit generalization.
+
+The harness wraps the same production functions and process boundaries without
+changing their arguments or timeout. Only external model output is replaced by a
+fixed event stream. The native plugin installer and `codex sandbox` remain real.
+This fixture does not claim live skill invocation; separate native integration
+coverage and consumer smoke own that evidence. Timings overlap: runtime and
+sandbox phases are inside candidate/grading-host totals, and plugin commands are
+inside plugin installation. Do not sum both levels. The `grading` measurement surrounds the semantic host
+call with its response-only policy; it excludes engine fixture preparation,
+verdict parsing, and extension bookkeeping.
+
+## Observed timings
+
+All values below are milliseconds, in repetition order.
+
+| Boundary                                     | Before                   | After                 |
+| -------------------------------------------- | ------------------------ | --------------------- |
+| Configuration fingerprint (once)             | 262                      | 299                   |
+| Candidate runtime preparation                | 1,969 / 1,357 / 1,273    | 721 / 431 / 443       |
+| Candidate hook dependency preparation        | 1,226 / 1,465 / 1,235    | No invocation         |
+| Fresh native plugin installation             | 385 / 141 / 143          | 188 / 141 / 143       |
+| Candidate sandbox probes, summed per role    | 10,938 / 11,032 / 11,622 | 5,490 / 5,404 / 5,448 |
+| Controlled participant process               | 27 / 29 / 39             | 27 / 27 / 24          |
+| Candidate total                              | 14,925 / 14,369 / 14,666 | 6,659 / 6,199 / 6,257 |
+| Semantic runtime preparation                 | 1,174 / 1,236 / 2,262    | Under 1 each          |
+| Semantic hook dependency preparation         | 1,105 / 1,087 / 1,363    | No invocation         |
+| Semantic sandbox probes, summed per role     | 10,855 / 10,803 / 11,032 | 5,548 / 5,360 / 5,365 |
+| Controlled semantic response process         | 28 / 44 / 27             | 29 / 28 / 28          |
+| Grading host total, including native startup | 13,495 / 13,534 / 15,030 | 5,647 / 5,454 / 5,460 |
+
+The probe count fell from two to one per fresh role. Candidate dependency
+materialization reuses immutable verified bytes after its first capture, and
+response-only grading no longer materializes implicit candidate dependencies.
+Configuration hashing remains necessary for identity. Fresh-install warm timings
+were essentially unchanged; no plugin-install speedup is claimed. Participant
+and semantic outputs are synthetic controls, so no API/model saving is claimed.
+
+## Startup deadline diagnosis
+
+A separate authenticated Darrow baseline using Codex 0.162.1 and `gpt-6-luna`
+with medium effort failed before participant execution: the first sandbox probe
+hit its 10-second deadline at 10,004.8 ms. A seedless control retained the same
+tool declarations and also timed out at 10,004.2 ms. With only the diagnostic
+probe deadline increased to 60 seconds, both original full-policy probes passed
+at 10,649.1 and 10,690.4 ms. Participant execution was deliberately withheld in
+that diagnostic. A minimal-policy control without tool grants or seeds passed
+at 5,372.8 and 5,681.1 ms.
+
+Raw diagnostic records are retained for the [original timeout](baseline-timeout.jsonl),
+[seedless timeout](seedless-timeout.jsonl), [full policy with diagnostic deadline](full-policy-diagnostic.jsonl),
+and [minimal policy](minimal-policy-diagnostic.jsonl). The maintained
+[diagnostic preload](../../../scripts/diagnose-codex-startup.ts) records bounded
+phase timing/errors and withholds app-server participant execution. To reproduce
+against an explicit checkout, set `SEVRO_DIAGNOSTIC_SOURCE_ROOT` and
+`SEVRO_DIAGNOSTIC_PHASE_FILE`, then use a CLI wrapper running:
+
+```sh
+bun --preload /absolute/candidate/scripts/diagnose-codex-startup.ts /absolute/baseline/src/cli.ts "$@" --runner-checkout-root /absolute/baseline
+```
+
+Point Darrow's `SEVRO_PACKAGE_BIN` at that wrapper and run:
+
+```sh
+bun evals/runner/run.ts --harness codex --owner-evaluation passive --skill create-commit --case create-commit-conventional-format --model gpt-6-luna --effort medium --trials 1 --jobs 1 --threshold 1 --results-root /absolute/outside-project-results --output /absolute/selection.json
+```
+
+For the deadline control, additionally set `SEVRO_DIAGNOSTIC_TIMEOUT_MS=60000`.
+For runtime controls, append `-- --runtime-config-file /absolute/runtime.json`:
+seedless keeps the original policy except `runtime.seedDirectories=[]`; minimal
+keeps only `format`, `nativeTranscripts: true`, and `hooks.nativeGoal: true`.
+The original diagnostic used the same wrappers with fixed paths; the retained
+preload makes those paths explicit through environment variables.
+
+These one-off controls locate the failure in native sandbox startup and show
+that dependency copying is not necessary for the timeout. Declared tool grants
+increase observed startup on this machine. They do not establish the exact
+native CLI mechanism or a general policy-size scaling law. The product now uses
+a bounded 60-second preflight; the measured saving comes from eliminating the
+second invocation and repeated preparation, not from increasing that deadline.
+The unavailable baseline participant, activation, and grading evidence remain
+unavailable rather than being assigned successful outcomes.
+
+## Snapshot and copy-on-write assessment
+
+The production snapshot owns verified bytes only in runner memory, keyed by the
+policy's seed descriptor. Candidate, peer, checker, and hook writable files never
+alias it. First capture must match the configured digest; later source changes
+do not change that snapshot. Regression coverage changes candidate files and the
+operator source between trials and verifies that the next trial still receives
+the original bytes. Executable modes, internal-link dereferencing, empty
+directories, source protection, offline environment, and role separation remain
+checked.
+
+The benchmark also exercised `COPYFILE_FICLONE_FORCE` on a 32 MiB file. It
+succeeded in about one millisecond in both runs, and changing the destination
+left the source length unchanged. This establishes local copy-on-write support,
+not portable availability or an immutable directory snapshot. The current
+in-memory snapshot deliberately avoids a persistent cache path and its cleanup,
+source-mutation, and cross-process trust requirements. Native clone reuse is not
+implemented; ordinary materialization works on filesystems without cloning.
+Retained policies retain their verified bytes until garbage collection, up to
+the existing per-seed bounds. This memory tradeoff should be measured separately
+for substantially larger dependency sets.
+
+## Native installation reuse assessment
+
+Every actual native installation in both benchmark runs produced configuration
+containing its private home and fixture workspace paths, as retained in the
+`installation-assets` observations. Native install receipts bind an
+`installedPath` inside that particular home's plugin cache; production verifies
+it before accepting explicit invocation. Authorized hook wrappers also bind the
+private runtime, sandbox profile, and execution receipt directory.
+
+A separate [native cache-only reuse experiment](plugin-template-assessment.json)
+installed a local synthetic plugin into a fresh credential-free home. The native
+registration wrote `source_type = "local"` and the original absolute marketplace
+path; the receipt contained the original private `installedPath`. After copying
+only `plugins/cache` into a second fresh home, `codex plugin list --json` returned
+empty `installed` and `available` arrays. To reproduce, use fresh `HOME` and
+`CODEX_HOME`, run `codex plugin marketplace add /absolute/market --json` and
+`codex plugin add probe@sevro-assessment --json`, copy `plugins/cache` to another
+fresh home, and list there. Paths in the raw record use `<assessment>` for the
+throwaway root; no credential file was present.
+
+An entire installed-home template therefore cannot be replayed into a new trial
+without rewriting and reverifying those identities, isolating credentials and
+configuration, and obtaining evidence for the new installed paths. Reusing
+just package files would still require native registration and receipt checks.
+For the measured fixture, warm fresh installation costs about 140 ms versus
+roughly 5.4 seconds for one sandbox startup. No template reuse is implemented.
+Fresh authentication/configuration, native registration and plugin installation,
+receipt validation, hook projection, the combined preflight, and native
+invocation evidence remain per-request work. Fresh-install integration coverage
+is retained.
+
+## Authenticated consumer smoke
+
+The [bounded consumer summary](consumer-smoke.json) records Darrow revision
+`34606ad9e1952b7f4e9c56befa446c18e1a2eb5d`, with no tracked consumer edits. The
+candidate's explicit `tdd-explicit-invocation` case passed execution, shell and
+semantic grading, and complete native explicit-invocation activation. Candidate
+and semantic routes used Codex 0.162.1, `gpt-6-luna`, medium effort, one trial,
+one job, and threshold one:
+
+```sh
+SEVRO_CHECKOUT=/absolute/candidate bun evals/runner/run.ts --harness codex --owner-evaluation passive --skill tdd --case tdd-explicit-invocation --model gpt-6-luna --effort medium --semantic-check-model gpt-6-luna --semantic-check-effort medium --trials 1 --jobs 1 --threshold 1 --results-root /absolute/outside-project-results --output /absolute/selection.json
+```
+
+The earlier implicit `create-commit-conventional-format` smoke completed
+execution and grading on the candidate, including its three shell checks, but
+failed the expected HEAD-change assertion. Its fixture mounted the backend under
+`.agents` without plugin manifests required by the launcher; the participant
+refused the commit. Activation stayed unavailable. The unchanged baseline had
+failed that case before participant execution at sandbox preflight. These
+outcomes are retained honestly; neither the fixture failure nor the later
+single-trial success establishes a matched live latency comparison or repeated
+model reliability. Private native transcripts and credentials remain outside Git.

package/docs/evidence/runtime-startup-30/after.json

@@ -0,0 +1,319 @@
+{
+  "format": "sevro.runtime-startup-benchmark.v1",
+  "sourceRoot": "/Users/bjro/Sources/sevro/.worktrees/perf/30-runtime-startup-overhead",
+  "platform": "darwin",
+  "arch": "arm64",
+  "bun": "1.3.13",
+  "codex": "codex-cli 0.162.1",
+  "case": "synthetic-plugin-response",
+  "model": "synthetic-candidate / synthetic-semantic",
+  "effort": "low",
+  "seed": {
+    "entries": 2049,
+    "bytes": 33554432
+  },
+  "repetitions": 3,
+  "conditions": "Fresh role homes each invocation; first verified snapshot cold, later repetitions warm; OS filesystem caches uncontrolled; no live API/model calls",
+  "rows": [
+    {
+      "repetition": 0,
+      "role": "candidate",
+      "phase": "configuration",
+      "milliseconds": 298.645833,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "runtime.candidate",
+      "milliseconds": 721.155541,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "plugin-command",
+      "milliseconds": 113.721,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "plugin-command",
+      "milliseconds": 70.76491699999997,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "installation-assets",
+      "files": [
+        "cache"
+      ],
+      "configurationContainsPrivateHome": true,
+      "configurationContainsWorkspace": true
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "plugin-install",
+      "milliseconds": 187.57354199999986,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "sandbox-probe",
+      "milliseconds": 5489.6383749999995,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "participant-execution",
+      "milliseconds": 27.323707999999897,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "candidate",
+      "phase": "candidate-total",
+      "milliseconds": 6659.420667,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "semantic",
+      "phase": "runtime.semantic",
+      "milliseconds": 0.3450830000001588,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "semantic",
+      "phase": "plugin-install",
+      "milliseconds": 0.005458999999973457,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "semantic",
+      "phase": "sandbox-probe",
+      "milliseconds": 5548.128959000001,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "semantic",
+      "phase": "participant-execution",
+      "milliseconds": 28.713165999999546,
+      "status": "completed"
+    },
+    {
+      "repetition": 1,
+      "role": "semantic",
+      "phase": "grading",
+      "milliseconds": 5646.800708000001,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "runtime.candidate",
+      "milliseconds": 430.6864170000008,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "plugin-command",
+      "milliseconds": 73.67158399999971,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "plugin-command",
+      "milliseconds": 65.96320800000103,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "installation-assets",
+      "files": [
+        "cache"
+      ],
+      "configurationContainsPrivateHome": true,
+      "configurationContainsWorkspace": true
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "plugin-install",
+      "milliseconds": 141.00866599999972,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "sandbox-probe",
+      "milliseconds": 5404.434915999998,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "participant-execution",
+      "milliseconds": 26.502958000000945,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "candidate",
+      "phase": "candidate-total",
+      "milliseconds": 6199.1395840000005,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "semantic",
+      "phase": "runtime.semantic",
+      "milliseconds": 0.314249999999447,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "semantic",
+      "phase": "plugin-install",
+      "milliseconds": 0.005916999998589745,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "semantic",
+      "phase": "sandbox-probe",
+      "milliseconds": 5359.928749999999,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "semantic",
+      "phase": "participant-execution",
+      "milliseconds": 27.60699999999997,
+      "status": "completed"
+    },
+    {
+      "repetition": 2,
+      "role": "semantic",
+      "phase": "grading",
+      "milliseconds": 5454.279749999998,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "runtime.candidate",
+      "milliseconds": 442.62854199999856,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "plugin-command",
+      "milliseconds": 72.3325420000001,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "plugin-command",
+      "milliseconds": 68.99258300000292,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "installation-assets",
+      "files": [
+        "cache"
+      ],
+      "configurationContainsPrivateHome": true,
+      "configurationContainsWorkspace": true
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "plugin-install",
+      "milliseconds": 142.90399999999863,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "sandbox-probe",
+      "milliseconds": 5447.905040999998,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "participant-execution",
+      "milliseconds": 23.8626669999976,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "candidate",
+      "phase": "candidate-total",
+      "milliseconds": 6256.5165830000005,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "semantic",
+      "phase": "runtime.semantic",
+      "milliseconds": 0.3507499999977881,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "semantic",
+      "phase": "plugin-install",
+      "milliseconds": 0.01341600000159815,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "semantic",
+      "phase": "sandbox-probe",
+      "milliseconds": 5365.139791000001,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "semantic",
+      "phase": "participant-execution",
+      "milliseconds": 28.228125000001455,
+      "status": "completed"
+    },
+    {
+      "repetition": 3,
+      "role": "semantic",
+      "phase": "grading",
+      "milliseconds": 5460.212874999997,
+      "status": "completed"
+    },
+    {
+      "repetition": 4,
+      "role": "semantic",
+      "phase": "cow-copy",
+      "milliseconds": 1.0158329999976559,
+      "status": "completed"
+    },
+    {
+      "phase": "cow-isolation",
+      "sourceUnchanged": true
+    }
+  ]
+}

package/docs/evidence/runtime-startup-30/baseline-timeout.jsonl

@@ -0,0 +1,2 @@
+{"phase":"sandbox-probe","durationMs":10004.7745,"error":"Codex run timed out"}
+{"phase":"host-failure","error":"Codex run timed out"}