@bjoernrochel/sevro 0.1.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. package/LICENSE +159 -0
  2. package/README.md +71 -0
  3. package/docs/advisory-fixture.md +23 -0
  4. package/docs/builtin-graders-v1.md +77 -0
  5. package/docs/cache-ownership-validation.md +49 -0
  6. package/docs/claude-continuation-validation.md +47 -0
  7. package/docs/codex-host.md +134 -0
  8. package/docs/concurrency-validation.md +86 -0
  9. package/docs/configuration-root-validation.md +70 -0
  10. package/docs/development-cli.md +327 -0
  11. package/docs/engine-slice.md +114 -0
  12. package/docs/extension-protocol-v1.md +292 -0
  13. package/docs/fixture-cleanup-validation.md +70 -0
  14. package/docs/identity-v1.md +65 -0
  15. package/docs/native-controls.md +29 -0
  16. package/docs/releases.md +88 -0
  17. package/docs/report-v1.md +32 -0
  18. package/docs/results-v1.md +118 -0
  19. package/examples/basic/graded.json +13 -0
  20. package/examples/basic/host.ts +8 -0
  21. package/examples/basic/prompt-only.json +7 -0
  22. package/package.json +45 -0
  23. package/schemas/cli-result-v1.schema.json +184 -0
  24. package/schemas/extension-v1.schema.json +659 -0
  25. package/schemas/report-v1.schema.json +164 -0
  26. package/schemas/run-evidence-v1.schema.json +510 -0
  27. package/src/advisory-fixture.ts +282 -0
  28. package/src/advisory.ts +105 -0
  29. package/src/cli.ts +837 -0
  30. package/src/engine.ts +2341 -0
  31. package/src/extension-client.ts +257 -0
  32. package/src/extension-session.ts +335 -0
  33. package/src/fixture-bin.ts +87 -0
  34. package/src/fixture-cleanup.ts +38 -0
  35. package/src/fixture-setup.ts +163 -0
  36. package/src/generated-fixture.ts +220 -0
  37. package/src/git-hooks.ts +63 -0
  38. package/src/graders/git-head.ts +176 -0
  39. package/src/graders/output.ts +359 -0
  40. package/src/graders/semantic.ts +120 -0
  41. package/src/graders/shell.ts +277 -0
  42. package/src/hosts/claude-continuation.ts +94 -0
  43. package/src/hosts/claude-credential.ts +73 -0
  44. package/src/hosts/claude-events.ts +128 -0
  45. package/src/hosts/claude-nested-skills.ts +218 -0
  46. package/src/hosts/claude-repository-invocation.ts +239 -0
  47. package/src/hosts/claude-settings.ts +50 -0
  48. package/src/hosts/claude-tool-calls.ts +159 -0
  49. package/src/hosts/claude.ts +511 -0
  50. package/src/hosts/codex-configuration.ts +85 -0
  51. package/src/hosts/codex-events.ts +102 -0
  52. package/src/hosts/codex-native-calls.ts +702 -0
  53. package/src/hosts/codex-profile.ts +78 -0
  54. package/src/hosts/codex-skill-reads.ts +345 -0
  55. package/src/hosts/codex.ts +769 -0
  56. package/src/hosts/isolation-roots.ts +60 -0
  57. package/src/hosts/mac-sandbox.ts +108 -0
  58. package/src/hosts/native-controls.ts +124 -0
  59. package/src/hosts/protected-worktrees.ts +56 -0
  60. package/src/hosts/workspace-fingerprint.ts +55 -0
  61. package/src/identity.ts +156 -0
  62. package/src/instrumentation.ts +106 -0
  63. package/src/preparation.ts +211 -0
  64. package/src/provenance.ts +244 -0
  65. package/src/report.ts +351 -0
  66. package/src/repository-fixture.ts +286 -0
  67. package/src/results.ts +160 -0
  68. package/src/run-owner.ts +276 -0
  69. package/src/schema.ts +28 -0
  70. package/src/storage.ts +23 -0
  71. package/src/trial-scheduler.ts +27 -0
@@ -0,0 +1,70 @@
1
+ # Configuration root validation
2
+
3
+ The public CLI imports one explicit Codex setting from a separate configuration
4
+ root, binds each used role's effective value into retained configuration and
5
+ evaluation identity, and protects that repository's primary and linked worktrees.
6
+ The runtime TOML dependency is pinned to `smol-toml` 1.8.0.
7
+
8
+ Working directory: `/Users/bjro/Sources/sevro`.
9
+
10
+ ## Observed test-first slices
11
+
12
+ 1. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI imports only the Codex limit from its separate configuration root"'`:
13
+ exit `64`, `sevro.invocation.invalid`, because `--config-root` was unsupported.
14
+ 2. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI imports only the Codex limit from its separate configuration root"'`:
15
+ one test passed with eight assertions. The synthetic Codex process requires
16
+ the imported limit in its isolated configuration and rejects an unrelated
17
+ model marker. Changed limits are retained and change evaluation identity.
18
+ 3. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI isolates a separate configuration repository and its linked worktree"'`:
19
+ task exit `1`; the primary configuration secret was denied, but the linked
20
+ worktree secret remained readable by the isolated shell grader.
21
+ 4. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI isolates a separate configuration repository and its linked worktree"'`:
22
+ one test passed with three assertions; both protected-source reads fail.
23
+ 5. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI refuses a dangling Codex configuration link before execution"'`:
24
+ incorrectly returned `0` for a declared unreadable configuration link.
25
+ 6. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI refuses a dangling Codex configuration link before execution"'`:
26
+ one test passed with three assertions; exit `64`, execution `not_run`.
27
+
28
+ Before implementation, the first test's identity assertion was corrected to
29
+ read `evaluationIdentity.digest` from retained evidence rather than an absent
30
+ CLI result field. The second test's check status oracle was also corrected
31
+ from `pass` to the documented `passed` before implementation. Both literal red
32
+ commands were rerun and still failed for the missing product behavior above.
33
+
34
+ Additional guards were added after implementation without a test-first claim.
35
+ They cover missing files and settings, all three Codex roles, malformed TOML,
36
+ non-table agent settings, invalid numeric types and ranges, a non-file config,
37
+ and relative, missing, or non-directory configuration roots. Together the five
38
+ new tests contain 65 assertions. No live model call is involved; macOS isolation
39
+ tests use the installed Codex sandbox and a synthetic candidate executable.
40
+
41
+ ## Test timeout repair
42
+
43
+ The first full `bun test` gate completed 214 tests and timed out the existing
44
+ eight-scenario Claude continuation guard after its default five seconds. Timeout
45
+ cleanup removed its active fixture, producing a subsequent ownership diagnostic.
46
+ The test now has an explicit ten-second bound, with every assertion unchanged.
47
+ `lean-ctx -c 'bun test tests/cli.test.ts -t "Claude continuation never grades failed or unbound native results"'`
48
+ then passed one test with 52 assertions in 3.96 seconds. This is test harness
49
+ maintenance, without a product-behavior red/green claim.
50
+
51
+ ## Independent review repairs
52
+
53
+ The bounded independent review found two medium blocking findings: a dangling
54
+ intermediate `.codex` directory link was treated as absent configuration, and
55
+ an explicitly empty `--config-root` silently selected the project root. The
56
+ original candidate-bound report is retained at
57
+ `/Users/bjro/.darrow/reviews-issue95-configuration-root/8794a1ce3463a83e1434abc110042a76e2397e27951a9a5cbf42214d9bbfc77b/darrow-review.k328difz/review.md`.
58
+
59
+ 1. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI distinguishes absent Codex configuration from a dangling directory link"'`:
60
+ returned `0` for the unreadable declared directory link; absent and ordinary
61
+ empty directories already retained the null host default.
62
+ 2. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI distinguishes absent Codex configuration from a dangling directory link"'`:
63
+ one test passed with seven assertions; the link fails with `64`, execution
64
+ `not_run`, and an absolute configuration-path diagnostic.
65
+ 3. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI rejects an explicitly empty configuration root"'`:
66
+ returned `0` when the supplied empty option should fail.
67
+ 4. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI rejects an explicitly empty configuration root"'`:
68
+ one test passed with four assertions for both separate and equals option
69
+ forms; each returns `64` and execution `not_run`. Omission still defaults
70
+ to the project root through the existing retained-role guards.
@@ -0,0 +1,327 @@
1
+ # Development CLI
2
+
3
+ The local `sevro run` command connects a resolved JSON case or an explicit
4
+ extension to a host route.
5
+ It uses the engine's normal grading and evidence path. This remains a
6
+ development entrypoint while the full extension lifecycle and production
7
+ provenance collection are being built.
8
+
9
+ `sevro report` reads one or more retained JSON results through the
10
+ [versioned report contract](report-v1.md). It emits Markdown by default or
11
+ `sevro.report.v1` with `--json`.
12
+
13
+ An installed package records its name, version, and a digest of its packaged
14
+ runtime and public contract files without reading runner Git metadata. The CLI
15
+ derives the project digest from its revision and dirty patch, or from a bounded
16
+ content snapshot when the project has no Git revision. Result and run-state
17
+ directories are excluded from that snapshot. When running this source checkout
18
+ for coordinated development, pass `--runner-checkout-root` with the absolute
19
+ path to this checkout. Sevro verifies that it is the code actually running and
20
+ records its revision and dirty-patch digest instead of package provenance.
21
+ `bun run test:package-install` verifies the packed package from a separate
22
+ temporary project without runner Git metadata.
23
+
24
+ ```sh
25
+ bun src/cli.ts run --json \
26
+ --case-file /absolute/path/case.json \
27
+ --adapter-module /absolute/path/host-adapter.ts \
28
+ --project-root /absolute/path/project \
29
+ --results-root /absolute/path/results \
30
+ --condition passive --trials 1 --threshold 1
31
+ ```
32
+
33
+ The adapter module exports a default `HostAdapter`. It is executable code
34
+ chosen by the operator. The runner does not load it from a case file or infer
35
+ it from an installed extension. A bundled Codex route is also available:
36
+
37
+ ```sh
38
+ bun src/cli.ts run --json \
39
+ --case-file /absolute/path/case.json \
40
+ --host codex --codex-bin /absolute/path/codex \
41
+ --codex-auth-file /absolute/path/auth.json \
42
+ --model <model> --effort medium \
43
+ --project-root /absolute/path/project \
44
+ --results-root /absolute/path/results \
45
+ --condition passive --trials 1 --threshold 1
46
+ ```
47
+
48
+ The Codex route requires file-based authentication and an installed Codex CLI
49
+ with permission profiles. It copies auth into a private home for each turn and
50
+ verifies the command sandbox before execution. This route currently supports
51
+ only passive conditions. `--host` and `--adapter-module` are exclusive. Use
52
+ `--protected-root` to add private roots for Codex, including when no shell
53
+ checks are selected.
54
+
55
+ ### Trial concurrency
56
+
57
+ `--jobs <positive integer>` bounds simultaneous trials within one case and
58
+ defaults to `3`. Use `--jobs 1` for serial execution. Each trial has an isolated
59
+ workspace and distinct retained artifacts. Completed trials are checkpointed
60
+ before cleanup; public results and checkpoint entries retain trial-number order
61
+ even when completion order differs. The effective job limit is retained in
62
+ redacted configuration and participates in evaluation identity.
63
+
64
+ Cancellation stops admission of later trials, reaches all active host calls,
65
+ and waits for them to retain their results before finalizing the run. Execution
66
+ or grading errors also stop admission while already active trials finish.
67
+ Persistence errors drain active trials before the attempt becomes diagnostic;
68
+ their completed evidence survives, and a trial whose evidence could not be
69
+ persisted keeps its fixture.
70
+
71
+ Host adapters receive the operating system's canonical absolute workspace path.
72
+ This keeps native cwd evidence and path-bound tooling consistent when the
73
+ temporary directory has a logical alias, such as macOS `/var` and `/private/var`.
74
+
75
+ Sevro reserves candidate and semantic workspace roots before execution so each
76
+ sandbox denies later peers as well as active ones. Fixture preparation stays
77
+ within the job limit. Completed candidate contents are removed after retention;
78
+ empty reservations stay until active trials drain, then unused and completed
79
+ roots are removed. A cancellation before any admission retains an interrupted
80
+ run with no trial evidence and exits `130` or `143` without calling a host.
81
+
82
+ Completed fixture cleanup restores owner access to permission-locked candidate
83
+ directories when necessary, then removes their contents. It keeps empty reserved
84
+ roots until active work drains. Permission repair stays in the completed
85
+ workspace and never follows symbolic links to external targets. Evidence is
86
+ retained before cleanup, including when cleanup itself fails.
87
+
88
+ ### Project and configuration roots
89
+
90
+ `--project-root` identifies the evaluated project and extension case discovery.
91
+ Use optional `--config-root /absolute/path/to/configuration` to import Codex
92
+ settings from a separate directory. It defaults to the project root. Sevro
93
+ captures only `agents.max_concurrent_threads_per_session` from
94
+ `.codex/config.toml` once per invocation and applies it to the candidate,
95
+ semantic judge, and advisory reviewer when those roles use Codex. The value
96
+ must be a positive TOML integer within JavaScript's safe integer range. Missing
97
+ files or an absent setting retain the host default; malformed, unreadable,
98
+ oversized, or linked configuration files fail with exit `64` before execution.
99
+ Model, permission, hook, credential, and environment settings are not imported.
100
+
101
+ Retained `configuration.redacted.hostConfiguration` records the effective
102
+ namespaced setting separately for each used role. A changed setting changes
103
+ evaluation identity. The selected configuration root and its primary and linked
104
+ Git worktrees are protected from bundled candidates and isolated shell checks.
105
+ The same worktree protection applies to other declared source and private roots;
106
+ a package without Git metadata remains usable. Result and active-run storage
107
+ stay independent through `--results-root` and `--run-state-root`.
108
+
109
+ A bundled Claude Code candidate route is available on macOS:
110
+
111
+ ```sh
112
+ bun src/cli.ts run --json \
113
+ --case-file /absolute/path/case.json \
114
+ --host claude --claude-bin /absolute/path/claude \
115
+ --model sonnet --effort medium \
116
+ --project-root /absolute/path/project \
117
+ --results-root /absolute/path/results \
118
+ --condition passive --trials 1 --threshold 1
119
+ ```
120
+
121
+ The route reads the local Claude Code login from the macOS keychain and stages
122
+ it inside private per-turn state. Pass `--claude-credential-file` with an
123
+ absolute path to use a separate credential file. It accepts declared Claude
124
+ plugin directories and explicit skill invocations from extension preparation.
125
+ It runs candidate tools under Claude Code's native macOS sandbox with private,
126
+ source, and result paths denied. The route disables hooks and does not expose
127
+ the Write tool. It currently supports passive candidate turns only. Use
128
+ `--protected-root` to exclude additional private paths from the candidate's
129
+ filesystem access. For plugins that run locked UV backends, use
130
+ `--claude-uv-cache-dir /absolute/path/to/curated-cache` to copy a prepared UV
131
+ cache into the trial's Git-private runtime directory before the turn. Use
132
+ `--toolchain-bin-dir /absolute/path/to/bin` for a compatible Python and other
133
+ case tools. The host and isolated shell checks place that directory on `PATH`.
134
+ They keep UV's project environment and Python bytecode outside the assessed
135
+ worktree contents so a check cannot change the candidate fingerprint.
136
+ With a curated UV cache, the Claude candidate's isolated `HOME` also lives
137
+ under the trial's Git-private runtime directory. Isolated checks use their own
138
+ home there. Tools choose their own caches from those homes; Sevro does not
139
+ set repository-specific cache variables or inherit such overrides from the
140
+ caller. Candidate and check homes are separate, while the curated UV cache is
141
+ shared within one trial. Runtime state is removed with the trial fixture.
142
+ The cache and toolchain directories must be outside protected roots. Their
143
+ contents are available to the candidate, so prepare them from public tools and
144
+ dependencies only.
145
+ Use `--claude-project-settings` when an isolated fixture intentionally defines
146
+ project-local `.claude/skills/`. The host still disables hooks in its explicit
147
+ settings and keeps its native sandbox rules. Leave this option off for plugin-only
148
+ cases.
149
+ Without that option, the host does not advertise
150
+ `sevro.claude.repository-invocation` and refuses explicit repository dispatch
151
+ before launching Claude. This avoids recording an unknown command as a usable
152
+ repository-skill route.
153
+ The host reads only bounded Skill metadata from completed child Agent sessions
154
+ in its temporary private configuration, then removes those session files. A
155
+ missing or inconsistent graph leaves `sevro.claude.nested-skills` partial.
156
+
157
+ A case's `followUpPrompt` starts a UUID-bound session and resumes that same
158
+ session after a complete successful initial result. Both results must name
159
+ the requested session. The same isolated workspace, settings, credentials,
160
+ model, and effort apply to both calls. An initial failure prevents resumption;
161
+ a failed or unbound resumed result fails execution and skips grading.
162
+ The `sevro.claude.continuation` observation records the bound session and
163
+ whether visible worktree contents changed before feedback, with partial
164
+ evidence when the fingerprints are unavailable. It retains no file names,
165
+ contents, private launch intent, or message-delivery claim. Combined and
166
+ separate turn event artifacts remain bounded and private. The last response
167
+ is graded, and only complete per-call usage and cost series are summed.
168
+
169
+ For cases with `sevro.semantic` checks, also pass
170
+ `--semantic-adapter-module /absolute/path/semantic-adapter.ts` to load an
171
+ explicit second `HostAdapter` for grading. Alternatively, select the bundled
172
+ Codex route with `--semantic-host codex --semantic-model <model>
173
+ --semantic-effort <effort>`. That route also requires `--codex-bin` and
174
+ `--codex-auth-file`, shared with a Codex candidate route when present. The two
175
+ semantic route options are exclusive. The candidate and semantic routes are
176
+ recorded separately in evidence and evaluation identity. The semantic adapter
177
+ runs after a complete candidate response in its own empty workspace; missing
178
+ or malformed semantic verdicts produce a grading error.
179
+
180
+ For generated Git or repository fixtures, add
181
+ `--advisory-adapter-module /absolute/path/advisory-adapter.ts` to request an
182
+ independent quality review. Alternatively, use `--advisory-host codex
183
+ --advisory-model <model> --advisory-effort <effort>` with the shared Codex binary
184
+ and auth file options. Repeat `--advisory-exclude <fixture-relative-path>` for
185
+ additional evaluator files to omit from the review view. The reviewer sees the
186
+ candidate change in a separate Git workspace, with root `.agents`, `.claude`,
187
+ `.codex`, and Git history withheld. Its structured assessment, usage, and raw
188
+ response are retained separately; reviewer failure or a failing recommendation
189
+ does not change the task verdict.
190
+
191
+ To resolve one case through a versioned extension, replace `--case-file` in
192
+ either command with:
193
+
194
+ ```sh
195
+ --extension-command-file /absolute/path/extension-command.json \
196
+ --extension-source-file /absolute/path/extension-source.ts \
197
+ --case-id example-case
198
+ ```
199
+
200
+ The command file is a JSON argv array such as
201
+ `["/absolute/path/bun", "/absolute/path/extension-source.ts"]`. Repeat
202
+ `--extension-source-file` for the extension's source closure. The runner hashes
203
+ those files and the executable, negotiates `sevro.extension.v1`, resolves the
204
+ selected case, then calls `prepare` and `evaluate` around host execution.
205
+ `--case-file` and `--extension-command-file` are exclusive. This CLI currently
206
+ accepts inline, generated Git, or repository fixtures and additive extension checks. An
207
+ extension may request a host-declared instrumentation capability during
208
+ `prepare`. The host adapter receives the request and must report exactly what
209
+ it applied. An execution-changing request is refused for `--condition passive`;
210
+ an unsupported or unconfirmed request cannot pass. The bundled Codex route
211
+ currently advertises no instrumentation capabilities. Set `executable: true`
212
+ on a preparation artifact to install it with owner execute permission; Sevro
213
+ retains the flag and mode in evidence. Repeat
214
+ `--replace-builtin-grader <sevro.grader-id>` to explicitly replace built-in
215
+ graders declared by the selected case. The case must also declare an advertised
216
+ extension check. The runner validates the replaced declarations but does not
217
+ run them; unselected built-ins stay active. Duplicate or absent grader IDs are
218
+ configuration errors.
219
+ To select an advertised extension task-verdict policy, pass
220
+ `--task-verdict-policy <namespaced-policy-id>` with the extension command. A
221
+ completed, fully graded trial uses the policy's recommendation for its task
222
+ verdict; failed execution, grading errors, and unavailable required evidence
223
+ cannot become a pass. Without this option, the runner uses its default task
224
+ verdict and rejects unsolicited recommendations.
225
+
226
+ If the extension needs configuration, supply both
227
+ `--extension-configuration-file` and
228
+ `--extension-redacted-configuration-file` as absolute paths to JSON objects.
229
+ The latter replaces secret values for the retained configuration digest. Keep
230
+ secrets out of the command argv and the redacted file.
231
+
232
+ An extension may return preparation artifacts that cite source IDs. Declare
233
+ those sources with `--case-source-root /absolute/path/sources` and
234
+ `--case-source-map-file /absolute/path/map.json` together. The map is a JSON
235
+ object from source ID to `file:///` URL. The runner checks that each referenced
236
+ file stays under the declared root and matches the extension's digest. The
237
+ selected case file, extension inputs, and source root are protected from
238
+ candidate execution and isolated shell checks. Each mapped source path is also
239
+ protected, including its primary repository and linked Git worktrees even when
240
+ those worktrees are outside the source root. Bundled candidates, semantic judges,
241
+ advisory reviewers, and isolated shell checks receive these protections.
242
+ The same map can declare a repository directory for a case fixture. A direct
243
+ case file uses `"fixture": {"sourceRef": "fixture-repo"}`; an extension uses
244
+ `{"kind": "repository", "sourceRef": "fixture-repo"}`. The repository must be
245
+ clean and committed. Sevro clones that commit for each trial without hardlinks
246
+ or remotes, and includes the commit in fixture identity. Uncommitted source
247
+ files, hooks, and working-tree state are not copied. Repository fixtures
248
+ may declare bounded `files`, `staged`, and `commitFiles` fields. Sevro applies
249
+ files after cloning, optionally commits them as scaffolding, then stages named
250
+ overlay files. Existing source files may be replaced, but symlinks and `.git/`
251
+ paths cannot be overlay targets. Both Git fixture kinds may declare bounded
252
+ executable `hooks` and `bin` tools by name. Sevro installs them under `.git/`
253
+ after its own fixture commits and before setup or host execution. Setup and
254
+ isolated shell checks receive `.git/fixture-bin` on `PATH`. Host adapters
255
+ receive `fixtureBinDir` and must add it to their native tool environment; the
256
+ bundled Codex adapter also preserves it across login shells. A negotiated
257
+ fixture setup command may then prepare the clone before the host runs.
258
+ Repository submodules and preparation artifacts targeting `.git/` are refused.
259
+
260
+ A direct case or extension can declare a generated Git fixture:
261
+
262
+ ```json
263
+ {
264
+ "kind": "generated",
265
+ "commits": [
266
+ { "message": "chore: initialize", "files": { "README.md": "base\n" } }
267
+ ],
268
+ "files": { "README.md": "edited\n" },
269
+ "staged": ["README.md"]
270
+ }
271
+ ```
272
+
273
+ Sevro creates the declared commits with a fixed local identity and date for
274
+ every trial, then writes the optional working-tree `files`. `staged` may name
275
+ only those overlay files. `commitFiles: true` commits the overlay as scaffolding
276
+ before staging. Paths under `.git/`, collisions, invalid staging, and oversized
277
+ histories fail preflight. Fixture tool and hook declarations enter the
278
+ fixture digest.
279
+
280
+ The case file follows the `ResolvedCase`
281
+ interface in `src/engine.ts`; this slice accepts inline files, generated Git
282
+ history, or a declared repository, built-in output and semantic checks, and
283
+ isolated shell checks. Add `{{sevro.workspace}}` to a case prompt when the host
284
+ needs the trial's absolute fixture path. Sevro resolves it after fixture
285
+ creation for each trial and retains the template in evaluation identity. For
286
+ cases with two participant turns, set `followUpPrompt` to a nonempty second
287
+ prompt. Sevro renders the workspace token in both prompts and requires a host
288
+ that declares `sevro.host.continuation`. The Codex host resumes the original
289
+ thread after a completed first turn and grades the second turn's final message.
290
+ Both bounded event streams are retained as separate artifacts.
291
+
292
+ The Codex host also retains `sevro.codex.continuation`, a bounded observation
293
+ of the resumed thread and whether visible workspace contents stayed unchanged
294
+ until the second prompt. An unreadable or oversized workspace makes that
295
+ comparison partial and its unchanged value null.
296
+ It also records the last validated native-session ordinal before the second
297
+ prompt. Missing or malformed native session evidence leaves this ordinal null;
298
+ the observation retains no prompt, tool arguments, or session content.
299
+
300
+ For shell checks, add
301
+ `--shell-isolation` and repeat `--protected-root /absolute/path` for every
302
+ additional source worktree or private root. The engine always protects the
303
+ selected project, results, runner source, user home, configured host homes,
304
+ and active peer fixtures. Shell checks require macOS `sandbox-exec` in this
305
+ development slice.
306
+ Injected host adapters may return namespaced observations and bounded evidence
307
+ artifacts. A case can list their IDs in `requiredEvidence`; missing or partial
308
+ host evidence produces unavailable grading rather than a passing task. The
309
+ runner retains observations and artifact references for extension grading.
310
+
311
+ `--json` writes exactly one versioned result to stdout, including pre-run
312
+ configuration failures. The process exit code follows `docs/results-v1.md`.
313
+ Add `--dry` to resolve and prepare a case without running the host or graders.
314
+ Its retained trials record `not_run` / `not_requested` / `not_assessed` and exit
315
+ successfully when preparation succeeds. Dry and executed runs have different
316
+ evaluation identities.
317
+ Run evidence and trial files live under `--results-root`. A packaged runner
318
+ needs no Git checkout. This private development checkout records its Git
319
+ revision and dirty content. `--runner-build-digest` and `--project-digest`
320
+ remain optional explicit overrides for test fixtures and coordinated
321
+ development; normal CLI runs derive both values.
322
+ Use `--run-state-root /absolute/path/state` to keep active records and trial
323
+ checkpoints separate from results; it defaults to `--results-root`. Shell
324
+ grading and the bundled Codex route protect this state root from candidate
325
+ commands.
326
+ SIGINT and SIGTERM request cancellation, retain completed trial evidence, and
327
+ return exit codes 130 and 143 respectively after interruption is recorded.
@@ -0,0 +1,114 @@
1
+ # First engine slice
2
+
3
+ `runEvaluation` accepts a resolved inline, generated Git, or repository fixture case and an injected host
4
+ adapter. It is an internal API while the host isolation adapters, extension
5
+ lifecycle, and public CLI are being built. The synthetic adapter in the tests
6
+ exercises the actual engine path without a model call.
7
+
8
+ The engine validates check configuration and fixture paths before execution.
9
+ Repository sources must be declared under an explicit source root, clean, and
10
+ committed. Each trial clones the fixed commit without hardlinks or remotes.
11
+ The source revision contributes to fixture identity; a changed or dirty source
12
+ is refused before it can silently alter a trial. Submodules and preparation
13
+ artifacts targeting `.git/` are refused.
14
+ Generated Git fixtures validate a bounded commit list, optional working-tree
15
+ files, and staged overlay paths before execution. Each trial builds the same
16
+ history with a fixed local identity and date, then applies its own working-tree
17
+ state. The declarative history enters fixture identity; no trial can inherit a
18
+ previous trial's edits.
19
+ The engine runs up to the configured positive `jobs` limit at once, defaulting
20
+ to three. Each admitted trial owns its workspace and artifact names. Completed
21
+ trial evidence is checkpointed before cleanup and retained in trial-number order,
22
+ including when trials complete out of order. Errors stop later admission and
23
+ active work drains before finalization. Use one job when serial ordering is needed;
24
+ custom hosts must support independent concurrent calls when the limit is higher.
25
+ Reserve all candidate and semantic workspace roots before launching a process;
26
+ prepare fixture contents only after admission. Sandbox deny lists include future
27
+ peers. Clear completed candidate contents after retention, keep their empty roots
28
+ until active work drains, and then remove unused and completed reservations.
29
+ Close admission when an execution or required grading error is detected, before
30
+ advisory work, persistence, or cleanup. Already admitted peers still retain evidence.
31
+ For each trial it creates a separate workspace, asks the adapter to run the
32
+ prompt, grades its bounded final message, writes the raw-message reference and
33
+ trial artifact, and only then removes the workspace. An adapter exception
34
+ becomes an execution failure; a later failure leaves earlier trial artifacts
35
+ intact. A failed trial checkpoint retains its fixture and cannot produce a
36
+ successful result.
37
+ Dry preparation still validates the case, runs extension preparation, creates
38
+ the fixture, and retains a trial record. It skips the host and all grading,
39
+ records `not_run` / `not_requested` / `not_assessed`, and has a separate
40
+ evaluation identity from an executed run.
41
+
42
+ The final `run.json` follows the v1 evidence schema and contains separate
43
+ execution, grading, and task states. It records the exact runner build and
44
+ project content digests derived by the CLI or explicitly supplied by the
45
+ caller, plus a computed evaluation identity. Missing optional usage stays
46
+ `null` and incomplete. Results storage
47
+ is explicit and independent of the project root. A packaged runner records its
48
+ package identity without Git; a private development checkout reads its own Git
49
+ revision and hashes tracked and untracked changes for the local runner
50
+ identity. A Git-backed project records its own revision and dirty content;
51
+ a project without a commit retains unknown Git provenance alongside a bounded
52
+ project content snapshot digest. An optional independent
53
+ run-state root holds an active record and a checkpoint pointing to each
54
+ retained trial. An evaluation-identity claim uses a short SQLite transaction
55
+ to admit one verified process owner. A live owner blocks an equivalent run;
56
+ an unverifiable owner is refused, while a confirmed dead owner is marked
57
+ interrupted and its prior attempt files are kept before a new claim starts.
58
+ The active record reaches `complete` only after final run evidence is written.
59
+ Process identity includes the PID, host, and operating-system start time to
60
+ protect against PID reuse. If start time cannot be verified, the run stops
61
+ before host execution.
62
+ The CLI passes SIGINT and SIGTERM through an abort signal. A cooperating host
63
+ stops its work; the engine retains completed trial artifacts, records the
64
+ cancelled trial, finalizes the active attempt as `interrupted`, and exits 130
65
+ or 143 after writing run evidence. An adapter that ignores cancellation cannot
66
+ be force-stopped by the injected-host interface.
67
+
68
+ The host can return bounded, namespaced observations alongside its final
69
+ message and bounded evidence artifacts. The engine retains them before
70
+ extension grading. It passes observations and artifact file references to the
71
+ extension without copying artifact bytes into run JSON. Missing or partial
72
+ required host evidence makes grading unavailable; malformed host evidence
73
+ fails execution. The engine also accepts shell checks when the caller supplies
74
+ protected source roots. They run through the macOS outer sandbox after host
75
+ execution, and their exit observations are retained with the trial. An
76
+ unavailable boundary produces a grading error rather than a passing check.
77
+ Semantic checks use a separately supplied host identity and an empty grader
78
+ workspace. A complete candidate final message is sent to that route with the
79
+ declared propositions. The engine requires one bounded verdict per check,
80
+ retains the raw response and grader artifacts outside the fixture, and records
81
+ the route and per-check observations. Missing candidate output makes semantic
82
+ checks unavailable; failed or malformed grader output makes grading an error.
83
+ Git HEAD checks capture the base commit before the candidate turn and grade
84
+ unchanged HEAD, changed HEAD, or base ancestry afterward. Candidate edits to
85
+ fixture files cannot rewrite the expected base revision.
86
+ An optional negotiated extension can add declared checks. The engine calls its
87
+ `prepare` method before host execution and its `evaluate` method for each
88
+ trial. It retains extension identity, configuration digest, protocol,
89
+ capabilities, grader selection, check evidence references, and metrics.
90
+ Missing declared extension checks remain unavailable; extension errors cannot
91
+ become a passing assessment. Inline preparation artifacts are decoded, checked
92
+ against their SHA-256 digests, mounted into each fixture, and retained outside
93
+ it for later inspection. Source-reference artifacts resolve only through a
94
+ caller-declared ID-to-file-URL map under an explicit case source root. Their
95
+ bytes receive the same size, path, and digest checks before mounting and
96
+ retention.
97
+ An explicitly selected extension task-verdict policy receives its policy ID
98
+ during `evaluate` and must return a recommendation. The engine uses that
99
+ recommendation only when execution and grading completed with all required
100
+ evidence available. It retains the selected policy and trial recommendation;
101
+ missing recommendations are grading errors. An explicitly replaced built-in
102
+ grader still has its declaration validated, but its checks do not run or count
103
+ toward the task verdict. At least one declared extension check must take its
104
+ place. The selection enters comparison identity and retained evidence.
105
+ An extension may request a negotiated host instrumentation capability during
106
+ `prepare`. The engine checks the host-declared capability and whether it
107
+ changes execution before starting a trial. Execution-changing requests are
108
+ refused in a passive condition. The host receives the request and must report
109
+ exactly what it applied; a mismatch or an unconfirmed enforced condition fails
110
+ execution. Run and trial evidence retain requested and applied instrumentation
111
+ separately. The bundled Codex adapter advertises no instrumentation capabilities.
112
+
113
+ Additional host adapters and host-specific instrumentation implementations
114
+ remain in progress.