@bjoernrochel/sevro 0.1.0-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +159 -0
- package/README.md +71 -0
- package/docs/advisory-fixture.md +23 -0
- package/docs/builtin-graders-v1.md +77 -0
- package/docs/cache-ownership-validation.md +49 -0
- package/docs/claude-continuation-validation.md +47 -0
- package/docs/codex-host.md +134 -0
- package/docs/concurrency-validation.md +86 -0
- package/docs/configuration-root-validation.md +70 -0
- package/docs/development-cli.md +327 -0
- package/docs/engine-slice.md +114 -0
- package/docs/extension-protocol-v1.md +292 -0
- package/docs/fixture-cleanup-validation.md +70 -0
- package/docs/identity-v1.md +65 -0
- package/docs/native-controls.md +29 -0
- package/docs/releases.md +88 -0
- package/docs/report-v1.md +32 -0
- package/docs/results-v1.md +118 -0
- package/examples/basic/graded.json +13 -0
- package/examples/basic/host.ts +8 -0
- package/examples/basic/prompt-only.json +7 -0
- package/package.json +45 -0
- package/schemas/cli-result-v1.schema.json +184 -0
- package/schemas/extension-v1.schema.json +659 -0
- package/schemas/report-v1.schema.json +164 -0
- package/schemas/run-evidence-v1.schema.json +510 -0
- package/src/advisory-fixture.ts +282 -0
- package/src/advisory.ts +105 -0
- package/src/cli.ts +837 -0
- package/src/engine.ts +2341 -0
- package/src/extension-client.ts +257 -0
- package/src/extension-session.ts +335 -0
- package/src/fixture-bin.ts +87 -0
- package/src/fixture-cleanup.ts +38 -0
- package/src/fixture-setup.ts +163 -0
- package/src/generated-fixture.ts +220 -0
- package/src/git-hooks.ts +63 -0
- package/src/graders/git-head.ts +176 -0
- package/src/graders/output.ts +359 -0
- package/src/graders/semantic.ts +120 -0
- package/src/graders/shell.ts +277 -0
- package/src/hosts/claude-continuation.ts +94 -0
- package/src/hosts/claude-credential.ts +73 -0
- package/src/hosts/claude-events.ts +128 -0
- package/src/hosts/claude-nested-skills.ts +218 -0
- package/src/hosts/claude-repository-invocation.ts +239 -0
- package/src/hosts/claude-settings.ts +50 -0
- package/src/hosts/claude-tool-calls.ts +159 -0
- package/src/hosts/claude.ts +511 -0
- package/src/hosts/codex-configuration.ts +85 -0
- package/src/hosts/codex-events.ts +102 -0
- package/src/hosts/codex-native-calls.ts +702 -0
- package/src/hosts/codex-profile.ts +78 -0
- package/src/hosts/codex-skill-reads.ts +345 -0
- package/src/hosts/codex.ts +769 -0
- package/src/hosts/isolation-roots.ts +60 -0
- package/src/hosts/mac-sandbox.ts +108 -0
- package/src/hosts/native-controls.ts +124 -0
- package/src/hosts/protected-worktrees.ts +56 -0
- package/src/hosts/workspace-fingerprint.ts +55 -0
- package/src/identity.ts +156 -0
- package/src/instrumentation.ts +106 -0
- package/src/preparation.ts +211 -0
- package/src/provenance.ts +244 -0
- package/src/report.ts +351 -0
- package/src/repository-fixture.ts +286 -0
- package/src/results.ts +160 -0
- package/src/run-owner.ts +276 -0
- package/src/schema.ts +28 -0
- package/src/storage.ts +23 -0
- package/src/trial-scheduler.ts +27 -0
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Configuration root validation
|
|
2
|
+
|
|
3
|
+
The public CLI imports one explicit Codex setting from a separate configuration
|
|
4
|
+
root, binds each used role's effective value into retained configuration and
|
|
5
|
+
evaluation identity, and protects that repository's primary and linked worktrees.
|
|
6
|
+
The runtime TOML dependency is pinned to `smol-toml` 1.8.0.
|
|
7
|
+
|
|
8
|
+
Working directory: `/Users/bjro/Sources/sevro`.
|
|
9
|
+
|
|
10
|
+
## Observed test-first slices
|
|
11
|
+
|
|
12
|
+
1. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI imports only the Codex limit from its separate configuration root"'`:
|
|
13
|
+
exit `64`, `sevro.invocation.invalid`, because `--config-root` was unsupported.
|
|
14
|
+
2. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI imports only the Codex limit from its separate configuration root"'`:
|
|
15
|
+
one test passed with eight assertions. The synthetic Codex process requires
|
|
16
|
+
the imported limit in its isolated configuration and rejects an unrelated
|
|
17
|
+
model marker. Changed limits are retained and change evaluation identity.
|
|
18
|
+
3. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI isolates a separate configuration repository and its linked worktree"'`:
|
|
19
|
+
task exit `1`; the primary configuration secret was denied, but the linked
|
|
20
|
+
worktree secret remained readable by the isolated shell grader.
|
|
21
|
+
4. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI isolates a separate configuration repository and its linked worktree"'`:
|
|
22
|
+
one test passed with three assertions; both protected-source reads fail.
|
|
23
|
+
5. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI refuses a dangling Codex configuration link before execution"'`:
|
|
24
|
+
incorrectly returned `0` for a declared unreadable configuration link.
|
|
25
|
+
6. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI refuses a dangling Codex configuration link before execution"'`:
|
|
26
|
+
one test passed with three assertions; exit `64`, execution `not_run`.
|
|
27
|
+
|
|
28
|
+
Before implementation, the first test's identity assertion was corrected to
|
|
29
|
+
read `evaluationIdentity.digest` from retained evidence rather than an absent
|
|
30
|
+
CLI result field. The second test's check status oracle was also corrected
|
|
31
|
+
from `pass` to the documented `passed` before implementation. Both literal red
|
|
32
|
+
commands were rerun and still failed for the missing product behavior above.
|
|
33
|
+
|
|
34
|
+
Additional guards were added after implementation without a test-first claim.
|
|
35
|
+
They cover missing files and settings, all three Codex roles, malformed TOML,
|
|
36
|
+
non-table agent settings, invalid numeric types and ranges, a non-file config,
|
|
37
|
+
and relative, missing, or non-directory configuration roots. Together the five
|
|
38
|
+
new tests contain 65 assertions. No live model call is involved; macOS isolation
|
|
39
|
+
tests use the installed Codex sandbox and a synthetic candidate executable.
|
|
40
|
+
|
|
41
|
+
## Test timeout repair
|
|
42
|
+
|
|
43
|
+
The first full `bun test` gate completed 214 tests and timed out the existing
|
|
44
|
+
eight-scenario Claude continuation guard after its default five seconds. Timeout
|
|
45
|
+
cleanup removed its active fixture, producing a subsequent ownership diagnostic.
|
|
46
|
+
The test now has an explicit ten-second bound, with every assertion unchanged.
|
|
47
|
+
`lean-ctx -c 'bun test tests/cli.test.ts -t "Claude continuation never grades failed or unbound native results"'`
|
|
48
|
+
then passed one test with 52 assertions in 3.96 seconds. This is test harness
|
|
49
|
+
maintenance, without a product-behavior red/green claim.
|
|
50
|
+
|
|
51
|
+
## Independent review repairs
|
|
52
|
+
|
|
53
|
+
The bounded independent review found two medium blocking findings: a dangling
|
|
54
|
+
intermediate `.codex` directory link was treated as absent configuration, and
|
|
55
|
+
an explicitly empty `--config-root` silently selected the project root. The
|
|
56
|
+
original candidate-bound report is retained at
|
|
57
|
+
`/Users/bjro/.darrow/reviews-issue95-configuration-root/8794a1ce3463a83e1434abc110042a76e2397e27951a9a5cbf42214d9bbfc77b/darrow-review.k328difz/review.md`.
|
|
58
|
+
|
|
59
|
+
1. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI distinguishes absent Codex configuration from a dangling directory link"'`:
|
|
60
|
+
returned `0` for the unreadable declared directory link; absent and ordinary
|
|
61
|
+
empty directories already retained the null host default.
|
|
62
|
+
2. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI distinguishes absent Codex configuration from a dangling directory link"'`:
|
|
63
|
+
one test passed with seven assertions; the link fails with `64`, execution
|
|
64
|
+
`not_run`, and an absolute configuration-path diagnostic.
|
|
65
|
+
3. Red — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI rejects an explicitly empty configuration root"'`:
|
|
66
|
+
returned `0` when the supplied empty option should fail.
|
|
67
|
+
4. Green — `lean-ctx -c 'bun test tests/cli.test.ts -t "CLI rejects an explicitly empty configuration root"'`:
|
|
68
|
+
one test passed with four assertions for both separate and equals option
|
|
69
|
+
forms; each returns `64` and execution `not_run`. Omission still defaults
|
|
70
|
+
to the project root through the existing retained-role guards.
|
|
@@ -0,0 +1,327 @@
|
|
|
1
|
+
# Development CLI
|
|
2
|
+
|
|
3
|
+
The local `sevro run` command connects a resolved JSON case or an explicit
|
|
4
|
+
extension to a host route.
|
|
5
|
+
It uses the engine's normal grading and evidence path. This remains a
|
|
6
|
+
development entrypoint while the full extension lifecycle and production
|
|
7
|
+
provenance collection are being built.
|
|
8
|
+
|
|
9
|
+
`sevro report` reads one or more retained JSON results through the
|
|
10
|
+
[versioned report contract](report-v1.md). It emits Markdown by default or
|
|
11
|
+
`sevro.report.v1` with `--json`.
|
|
12
|
+
|
|
13
|
+
An installed package records its name, version, and a digest of its packaged
|
|
14
|
+
runtime and public contract files without reading runner Git metadata. The CLI
|
|
15
|
+
derives the project digest from its revision and dirty patch, or from a bounded
|
|
16
|
+
content snapshot when the project has no Git revision. Result and run-state
|
|
17
|
+
directories are excluded from that snapshot. When running this source checkout
|
|
18
|
+
for coordinated development, pass `--runner-checkout-root` with the absolute
|
|
19
|
+
path to this checkout. Sevro verifies that it is the code actually running and
|
|
20
|
+
records its revision and dirty-patch digest instead of package provenance.
|
|
21
|
+
`bun run test:package-install` verifies the packed package from a separate
|
|
22
|
+
temporary project without runner Git metadata.
|
|
23
|
+
|
|
24
|
+
```sh
|
|
25
|
+
bun src/cli.ts run --json \
|
|
26
|
+
--case-file /absolute/path/case.json \
|
|
27
|
+
--adapter-module /absolute/path/host-adapter.ts \
|
|
28
|
+
--project-root /absolute/path/project \
|
|
29
|
+
--results-root /absolute/path/results \
|
|
30
|
+
--condition passive --trials 1 --threshold 1
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
The adapter module exports a default `HostAdapter`. It is executable code
|
|
34
|
+
chosen by the operator. The runner does not load it from a case file or infer
|
|
35
|
+
it from an installed extension. A bundled Codex route is also available:
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
bun src/cli.ts run --json \
|
|
39
|
+
--case-file /absolute/path/case.json \
|
|
40
|
+
--host codex --codex-bin /absolute/path/codex \
|
|
41
|
+
--codex-auth-file /absolute/path/auth.json \
|
|
42
|
+
--model <model> --effort medium \
|
|
43
|
+
--project-root /absolute/path/project \
|
|
44
|
+
--results-root /absolute/path/results \
|
|
45
|
+
--condition passive --trials 1 --threshold 1
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The Codex route requires file-based authentication and an installed Codex CLI
|
|
49
|
+
with permission profiles. It copies auth into a private home for each turn and
|
|
50
|
+
verifies the command sandbox before execution. This route currently supports
|
|
51
|
+
only passive conditions. `--host` and `--adapter-module` are exclusive. Use
|
|
52
|
+
`--protected-root` to add private roots for Codex, including when no shell
|
|
53
|
+
checks are selected.
|
|
54
|
+
|
|
55
|
+
### Trial concurrency
|
|
56
|
+
|
|
57
|
+
`--jobs <positive integer>` bounds simultaneous trials within one case and
|
|
58
|
+
defaults to `3`. Use `--jobs 1` for serial execution. Each trial has an isolated
|
|
59
|
+
workspace and distinct retained artifacts. Completed trials are checkpointed
|
|
60
|
+
before cleanup; public results and checkpoint entries retain trial-number order
|
|
61
|
+
even when completion order differs. The effective job limit is retained in
|
|
62
|
+
redacted configuration and participates in evaluation identity.
|
|
63
|
+
|
|
64
|
+
Cancellation stops admission of later trials, reaches all active host calls,
|
|
65
|
+
and waits for them to retain their results before finalizing the run. Execution
|
|
66
|
+
or grading errors also stop admission while already active trials finish.
|
|
67
|
+
Persistence errors drain active trials before the attempt becomes diagnostic;
|
|
68
|
+
their completed evidence survives, and a trial whose evidence could not be
|
|
69
|
+
persisted keeps its fixture.
|
|
70
|
+
|
|
71
|
+
Host adapters receive the operating system's canonical absolute workspace path.
|
|
72
|
+
This keeps native cwd evidence and path-bound tooling consistent when the
|
|
73
|
+
temporary directory has a logical alias, such as macOS `/var` and `/private/var`.
|
|
74
|
+
|
|
75
|
+
Sevro reserves candidate and semantic workspace roots before execution so each
|
|
76
|
+
sandbox denies later peers as well as active ones. Fixture preparation stays
|
|
77
|
+
within the job limit. Completed candidate contents are removed after retention;
|
|
78
|
+
empty reservations stay until active trials drain, then unused and completed
|
|
79
|
+
roots are removed. A cancellation before any admission retains an interrupted
|
|
80
|
+
run with no trial evidence and exits `130` or `143` without calling a host.
|
|
81
|
+
|
|
82
|
+
Completed fixture cleanup restores owner access to permission-locked candidate
|
|
83
|
+
directories when necessary, then removes their contents. It keeps empty reserved
|
|
84
|
+
roots until active work drains. Permission repair stays in the completed
|
|
85
|
+
workspace and never follows symbolic links to external targets. Evidence is
|
|
86
|
+
retained before cleanup, including when cleanup itself fails.
|
|
87
|
+
|
|
88
|
+
### Project and configuration roots
|
|
89
|
+
|
|
90
|
+
`--project-root` identifies the evaluated project and extension case discovery.
|
|
91
|
+
Use optional `--config-root /absolute/path/to/configuration` to import Codex
|
|
92
|
+
settings from a separate directory. It defaults to the project root. Sevro
|
|
93
|
+
captures only `agents.max_concurrent_threads_per_session` from
|
|
94
|
+
`.codex/config.toml` once per invocation and applies it to the candidate,
|
|
95
|
+
semantic judge, and advisory reviewer when those roles use Codex. The value
|
|
96
|
+
must be a positive TOML integer within JavaScript's safe integer range. Missing
|
|
97
|
+
files or an absent setting retain the host default; malformed, unreadable,
|
|
98
|
+
oversized, or linked configuration files fail with exit `64` before execution.
|
|
99
|
+
Model, permission, hook, credential, and environment settings are not imported.
|
|
100
|
+
|
|
101
|
+
Retained `configuration.redacted.hostConfiguration` records the effective
|
|
102
|
+
namespaced setting separately for each used role. A changed setting changes
|
|
103
|
+
evaluation identity. The selected configuration root and its primary and linked
|
|
104
|
+
Git worktrees are protected from bundled candidates and isolated shell checks.
|
|
105
|
+
The same worktree protection applies to other declared source and private roots;
|
|
106
|
+
a package without Git metadata remains usable. Result and active-run storage
|
|
107
|
+
stay independent through `--results-root` and `--run-state-root`.
|
|
108
|
+
|
|
109
|
+
A bundled Claude Code candidate route is available on macOS:
|
|
110
|
+
|
|
111
|
+
```sh
|
|
112
|
+
bun src/cli.ts run --json \
|
|
113
|
+
--case-file /absolute/path/case.json \
|
|
114
|
+
--host claude --claude-bin /absolute/path/claude \
|
|
115
|
+
--model sonnet --effort medium \
|
|
116
|
+
--project-root /absolute/path/project \
|
|
117
|
+
--results-root /absolute/path/results \
|
|
118
|
+
--condition passive --trials 1 --threshold 1
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
The route reads the local Claude Code login from the macOS keychain and stages
|
|
122
|
+
it inside private per-turn state. Pass `--claude-credential-file` with an
|
|
123
|
+
absolute path to use a separate credential file. It accepts declared Claude
|
|
124
|
+
plugin directories and explicit skill invocations from extension preparation.
|
|
125
|
+
It runs candidate tools under Claude Code's native macOS sandbox with private,
|
|
126
|
+
source, and result paths denied. The route disables hooks and does not expose
|
|
127
|
+
the Write tool. It currently supports passive candidate turns only. Use
|
|
128
|
+
`--protected-root` to exclude additional private paths from the candidate's
|
|
129
|
+
filesystem access. For plugins that run locked UV backends, use
|
|
130
|
+
`--claude-uv-cache-dir /absolute/path/to/curated-cache` to copy a prepared UV
|
|
131
|
+
cache into the trial's Git-private runtime directory before the turn. Use
|
|
132
|
+
`--toolchain-bin-dir /absolute/path/to/bin` for a compatible Python and other
|
|
133
|
+
case tools. The host and isolated shell checks place that directory on `PATH`.
|
|
134
|
+
They keep UV's project environment and Python bytecode outside the assessed
|
|
135
|
+
worktree contents so a check cannot change the candidate fingerprint.
|
|
136
|
+
With a curated UV cache, the Claude candidate's isolated `HOME` also lives
|
|
137
|
+
under the trial's Git-private runtime directory. Isolated checks use their own
|
|
138
|
+
home there. Tools choose their own caches from those homes; Sevro does not
|
|
139
|
+
set repository-specific cache variables or inherit such overrides from the
|
|
140
|
+
caller. Candidate and check homes are separate, while the curated UV cache is
|
|
141
|
+
shared within one trial. Runtime state is removed with the trial fixture.
|
|
142
|
+
The cache and toolchain directories must be outside protected roots. Their
|
|
143
|
+
contents are available to the candidate, so prepare them from public tools and
|
|
144
|
+
dependencies only.
|
|
145
|
+
Use `--claude-project-settings` when an isolated fixture intentionally defines
|
|
146
|
+
project-local `.claude/skills/`. The host still disables hooks in its explicit
|
|
147
|
+
settings and keeps its native sandbox rules. Leave this option off for plugin-only
|
|
148
|
+
cases.
|
|
149
|
+
Without that option, the host does not advertise
|
|
150
|
+
`sevro.claude.repository-invocation` and refuses explicit repository dispatch
|
|
151
|
+
before launching Claude. This avoids recording an unknown command as a usable
|
|
152
|
+
repository-skill route.
|
|
153
|
+
The host reads only bounded Skill metadata from completed child Agent sessions
|
|
154
|
+
in its temporary private configuration, then removes those session files. A
|
|
155
|
+
missing or inconsistent graph leaves `sevro.claude.nested-skills` partial.
|
|
156
|
+
|
|
157
|
+
A case's `followUpPrompt` starts a UUID-bound session and resumes that same
|
|
158
|
+
session after a complete successful initial result. Both results must name
|
|
159
|
+
the requested session. The same isolated workspace, settings, credentials,
|
|
160
|
+
model, and effort apply to both calls. An initial failure prevents resumption;
|
|
161
|
+
a failed or unbound resumed result fails execution and skips grading.
|
|
162
|
+
The `sevro.claude.continuation` observation records the bound session and
|
|
163
|
+
whether visible worktree contents changed before feedback, with partial
|
|
164
|
+
evidence when the fingerprints are unavailable. It retains no file names,
|
|
165
|
+
contents, private launch intent, or message-delivery claim. Combined and
|
|
166
|
+
separate turn event artifacts remain bounded and private. The last response
|
|
167
|
+
is graded, and only complete per-call usage and cost series are summed.
|
|
168
|
+
|
|
169
|
+
For cases with `sevro.semantic` checks, also pass
|
|
170
|
+
`--semantic-adapter-module /absolute/path/semantic-adapter.ts` to load an
|
|
171
|
+
explicit second `HostAdapter` for grading. Alternatively, select the bundled
|
|
172
|
+
Codex route with `--semantic-host codex --semantic-model <model>
|
|
173
|
+
--semantic-effort <effort>`. That route also requires `--codex-bin` and
|
|
174
|
+
`--codex-auth-file`, shared with a Codex candidate route when present. The two
|
|
175
|
+
semantic route options are exclusive. The candidate and semantic routes are
|
|
176
|
+
recorded separately in evidence and evaluation identity. The semantic adapter
|
|
177
|
+
runs after a complete candidate response in its own empty workspace; missing
|
|
178
|
+
or malformed semantic verdicts produce a grading error.
|
|
179
|
+
|
|
180
|
+
For generated Git or repository fixtures, add
|
|
181
|
+
`--advisory-adapter-module /absolute/path/advisory-adapter.ts` to request an
|
|
182
|
+
independent quality review. Alternatively, use `--advisory-host codex
|
|
183
|
+
--advisory-model <model> --advisory-effort <effort>` with the shared Codex binary
|
|
184
|
+
and auth file options. Repeat `--advisory-exclude <fixture-relative-path>` for
|
|
185
|
+
additional evaluator files to omit from the review view. The reviewer sees the
|
|
186
|
+
candidate change in a separate Git workspace, with root `.agents`, `.claude`,
|
|
187
|
+
`.codex`, and Git history withheld. Its structured assessment, usage, and raw
|
|
188
|
+
response are retained separately; reviewer failure or a failing recommendation
|
|
189
|
+
does not change the task verdict.
|
|
190
|
+
|
|
191
|
+
To resolve one case through a versioned extension, replace `--case-file` in
|
|
192
|
+
either command with:
|
|
193
|
+
|
|
194
|
+
```sh
|
|
195
|
+
--extension-command-file /absolute/path/extension-command.json \
|
|
196
|
+
--extension-source-file /absolute/path/extension-source.ts \
|
|
197
|
+
--case-id example-case
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
The command file is a JSON argv array such as
|
|
201
|
+
`["/absolute/path/bun", "/absolute/path/extension-source.ts"]`. Repeat
|
|
202
|
+
`--extension-source-file` for the extension's source closure. The runner hashes
|
|
203
|
+
those files and the executable, negotiates `sevro.extension.v1`, resolves the
|
|
204
|
+
selected case, then calls `prepare` and `evaluate` around host execution.
|
|
205
|
+
`--case-file` and `--extension-command-file` are exclusive. This CLI currently
|
|
206
|
+
accepts inline, generated Git, or repository fixtures and additive extension checks. An
|
|
207
|
+
extension may request a host-declared instrumentation capability during
|
|
208
|
+
`prepare`. The host adapter receives the request and must report exactly what
|
|
209
|
+
it applied. An execution-changing request is refused for `--condition passive`;
|
|
210
|
+
an unsupported or unconfirmed request cannot pass. The bundled Codex route
|
|
211
|
+
currently advertises no instrumentation capabilities. Set `executable: true`
|
|
212
|
+
on a preparation artifact to install it with owner execute permission; Sevro
|
|
213
|
+
retains the flag and mode in evidence. Repeat
|
|
214
|
+
`--replace-builtin-grader <sevro.grader-id>` to explicitly replace built-in
|
|
215
|
+
graders declared by the selected case. The case must also declare an advertised
|
|
216
|
+
extension check. The runner validates the replaced declarations but does not
|
|
217
|
+
run them; unselected built-ins stay active. Duplicate or absent grader IDs are
|
|
218
|
+
configuration errors.
|
|
219
|
+
To select an advertised extension task-verdict policy, pass
|
|
220
|
+
`--task-verdict-policy <namespaced-policy-id>` with the extension command. A
|
|
221
|
+
completed, fully graded trial uses the policy's recommendation for its task
|
|
222
|
+
verdict; failed execution, grading errors, and unavailable required evidence
|
|
223
|
+
cannot become a pass. Without this option, the runner uses its default task
|
|
224
|
+
verdict and rejects unsolicited recommendations.
|
|
225
|
+
|
|
226
|
+
If the extension needs configuration, supply both
|
|
227
|
+
`--extension-configuration-file` and
|
|
228
|
+
`--extension-redacted-configuration-file` as absolute paths to JSON objects.
|
|
229
|
+
The latter replaces secret values for the retained configuration digest. Keep
|
|
230
|
+
secrets out of the command argv and the redacted file.
|
|
231
|
+
|
|
232
|
+
An extension may return preparation artifacts that cite source IDs. Declare
|
|
233
|
+
those sources with `--case-source-root /absolute/path/sources` and
|
|
234
|
+
`--case-source-map-file /absolute/path/map.json` together. The map is a JSON
|
|
235
|
+
object from source ID to `file:///` URL. The runner checks that each referenced
|
|
236
|
+
file stays under the declared root and matches the extension's digest. The
|
|
237
|
+
selected case file, extension inputs, and source root are protected from
|
|
238
|
+
candidate execution and isolated shell checks. Each mapped source path is also
|
|
239
|
+
protected, including its primary repository and linked Git worktrees even when
|
|
240
|
+
those worktrees are outside the source root. Bundled candidates, semantic judges,
|
|
241
|
+
advisory reviewers, and isolated shell checks receive these protections.
|
|
242
|
+
The same map can declare a repository directory for a case fixture. A direct
|
|
243
|
+
case file uses `"fixture": {"sourceRef": "fixture-repo"}`; an extension uses
|
|
244
|
+
`{"kind": "repository", "sourceRef": "fixture-repo"}`. The repository must be
|
|
245
|
+
clean and committed. Sevro clones that commit for each trial without hardlinks
|
|
246
|
+
or remotes, and includes the commit in fixture identity. Uncommitted source
|
|
247
|
+
files, hooks, and working-tree state are not copied. Repository fixtures
|
|
248
|
+
may declare bounded `files`, `staged`, and `commitFiles` fields. Sevro applies
|
|
249
|
+
files after cloning, optionally commits them as scaffolding, then stages named
|
|
250
|
+
overlay files. Existing source files may be replaced, but symlinks and `.git/`
|
|
251
|
+
paths cannot be overlay targets. Both Git fixture kinds may declare bounded
|
|
252
|
+
executable `hooks` and `bin` tools by name. Sevro installs them under `.git/`
|
|
253
|
+
after its own fixture commits and before setup or host execution. Setup and
|
|
254
|
+
isolated shell checks receive `.git/fixture-bin` on `PATH`. Host adapters
|
|
255
|
+
receive `fixtureBinDir` and must add it to their native tool environment; the
|
|
256
|
+
bundled Codex adapter also preserves it across login shells. A negotiated
|
|
257
|
+
fixture setup command may then prepare the clone before the host runs.
|
|
258
|
+
Repository submodules and preparation artifacts targeting `.git/` are refused.
|
|
259
|
+
|
|
260
|
+
A direct case or extension can declare a generated Git fixture:
|
|
261
|
+
|
|
262
|
+
```json
|
|
263
|
+
{
|
|
264
|
+
"kind": "generated",
|
|
265
|
+
"commits": [
|
|
266
|
+
{ "message": "chore: initialize", "files": { "README.md": "base\n" } }
|
|
267
|
+
],
|
|
268
|
+
"files": { "README.md": "edited\n" },
|
|
269
|
+
"staged": ["README.md"]
|
|
270
|
+
}
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
Sevro creates the declared commits with a fixed local identity and date for
|
|
274
|
+
every trial, then writes the optional working-tree `files`. `staged` may name
|
|
275
|
+
only those overlay files. `commitFiles: true` commits the overlay as scaffolding
|
|
276
|
+
before staging. Paths under `.git/`, collisions, invalid staging, and oversized
|
|
277
|
+
histories fail preflight. Fixture tool and hook declarations enter the
|
|
278
|
+
fixture digest.
|
|
279
|
+
|
|
280
|
+
The case file follows the `ResolvedCase`
|
|
281
|
+
interface in `src/engine.ts`; this slice accepts inline files, generated Git
|
|
282
|
+
history, or a declared repository, built-in output and semantic checks, and
|
|
283
|
+
isolated shell checks. Add `{{sevro.workspace}}` to a case prompt when the host
|
|
284
|
+
needs the trial's absolute fixture path. Sevro resolves it after fixture
|
|
285
|
+
creation for each trial and retains the template in evaluation identity. For
|
|
286
|
+
cases with two participant turns, set `followUpPrompt` to a nonempty second
|
|
287
|
+
prompt. Sevro renders the workspace token in both prompts and requires a host
|
|
288
|
+
that declares `sevro.host.continuation`. The Codex host resumes the original
|
|
289
|
+
thread after a completed first turn and grades the second turn's final message.
|
|
290
|
+
Both bounded event streams are retained as separate artifacts.
|
|
291
|
+
|
|
292
|
+
The Codex host also retains `sevro.codex.continuation`, a bounded observation
|
|
293
|
+
of the resumed thread and whether visible workspace contents stayed unchanged
|
|
294
|
+
until the second prompt. An unreadable or oversized workspace makes that
|
|
295
|
+
comparison partial and its unchanged value null.
|
|
296
|
+
It also records the last validated native-session ordinal before the second
|
|
297
|
+
prompt. Missing or malformed native session evidence leaves this ordinal null;
|
|
298
|
+
the observation retains no prompt, tool arguments, or session content.
|
|
299
|
+
|
|
300
|
+
For shell checks, add
|
|
301
|
+
`--shell-isolation` and repeat `--protected-root /absolute/path` for every
|
|
302
|
+
additional source worktree or private root. The engine always protects the
|
|
303
|
+
selected project, results, runner source, user home, configured host homes,
|
|
304
|
+
and active peer fixtures. Shell checks require macOS `sandbox-exec` in this
|
|
305
|
+
development slice.
|
|
306
|
+
Injected host adapters may return namespaced observations and bounded evidence
|
|
307
|
+
artifacts. A case can list their IDs in `requiredEvidence`; missing or partial
|
|
308
|
+
host evidence produces unavailable grading rather than a passing task. The
|
|
309
|
+
runner retains observations and artifact references for extension grading.
|
|
310
|
+
|
|
311
|
+
`--json` writes exactly one versioned result to stdout, including pre-run
|
|
312
|
+
configuration failures. The process exit code follows `docs/results-v1.md`.
|
|
313
|
+
Add `--dry` to resolve and prepare a case without running the host or graders.
|
|
314
|
+
Its retained trials record `not_run` / `not_requested` / `not_assessed` and exit
|
|
315
|
+
successfully when preparation succeeds. Dry and executed runs have different
|
|
316
|
+
evaluation identities.
|
|
317
|
+
Run evidence and trial files live under `--results-root`. A packaged runner
|
|
318
|
+
needs no Git checkout. This private development checkout records its Git
|
|
319
|
+
revision and dirty content. `--runner-build-digest` and `--project-digest`
|
|
320
|
+
remain optional explicit overrides for test fixtures and coordinated
|
|
321
|
+
development; normal CLI runs derive both values.
|
|
322
|
+
Use `--run-state-root /absolute/path/state` to keep active records and trial
|
|
323
|
+
checkpoints separate from results; it defaults to `--results-root`. Shell
|
|
324
|
+
grading and the bundled Codex route protect this state root from candidate
|
|
325
|
+
commands.
|
|
326
|
+
SIGINT and SIGTERM request cancellation, retain completed trial evidence, and
|
|
327
|
+
return exit codes 130 and 143 respectively after interruption is recorded.
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# First engine slice
|
|
2
|
+
|
|
3
|
+
`runEvaluation` accepts a resolved inline, generated Git, or repository fixture case and an injected host
|
|
4
|
+
adapter. It is an internal API while the host isolation adapters, extension
|
|
5
|
+
lifecycle, and public CLI are being built. The synthetic adapter in the tests
|
|
6
|
+
exercises the actual engine path without a model call.
|
|
7
|
+
|
|
8
|
+
The engine validates check configuration and fixture paths before execution.
|
|
9
|
+
Repository sources must be declared under an explicit source root, clean, and
|
|
10
|
+
committed. Each trial clones the fixed commit without hardlinks or remotes.
|
|
11
|
+
The source revision contributes to fixture identity; a changed or dirty source
|
|
12
|
+
is refused before it can silently alter a trial. Submodules and preparation
|
|
13
|
+
artifacts targeting `.git/` are refused.
|
|
14
|
+
Generated Git fixtures validate a bounded commit list, optional working-tree
|
|
15
|
+
files, and staged overlay paths before execution. Each trial builds the same
|
|
16
|
+
history with a fixed local identity and date, then applies its own working-tree
|
|
17
|
+
state. The declarative history enters fixture identity; no trial can inherit a
|
|
18
|
+
previous trial's edits.
|
|
19
|
+
The engine runs up to the configured positive `jobs` limit at once, defaulting
|
|
20
|
+
to three. Each admitted trial owns its workspace and artifact names. Completed
|
|
21
|
+
trial evidence is checkpointed before cleanup and retained in trial-number order,
|
|
22
|
+
including when trials complete out of order. Errors stop later admission and
|
|
23
|
+
active work drains before finalization. Use one job when serial ordering is needed;
|
|
24
|
+
custom hosts must support independent concurrent calls when the limit is higher.
|
|
25
|
+
Reserve all candidate and semantic workspace roots before launching a process;
|
|
26
|
+
prepare fixture contents only after admission. Sandbox deny lists include future
|
|
27
|
+
peers. Clear completed candidate contents after retention, keep their empty roots
|
|
28
|
+
until active work drains, and then remove unused and completed reservations.
|
|
29
|
+
Close admission when an execution or required grading error is detected, before
|
|
30
|
+
advisory work, persistence, or cleanup. Already admitted peers still retain evidence.
|
|
31
|
+
For each trial it creates a separate workspace, asks the adapter to run the
|
|
32
|
+
prompt, grades its bounded final message, writes the raw-message reference and
|
|
33
|
+
trial artifact, and only then removes the workspace. An adapter exception
|
|
34
|
+
becomes an execution failure; a later failure leaves earlier trial artifacts
|
|
35
|
+
intact. A failed trial checkpoint retains its fixture and cannot produce a
|
|
36
|
+
successful result.
|
|
37
|
+
Dry preparation still validates the case, runs extension preparation, creates
|
|
38
|
+
the fixture, and retains a trial record. It skips the host and all grading,
|
|
39
|
+
records `not_run` / `not_requested` / `not_assessed`, and has a separate
|
|
40
|
+
evaluation identity from an executed run.
|
|
41
|
+
|
|
42
|
+
The final `run.json` follows the v1 evidence schema and contains separate
|
|
43
|
+
execution, grading, and task states. It records the exact runner build and
|
|
44
|
+
project content digests derived by the CLI or explicitly supplied by the
|
|
45
|
+
caller, plus a computed evaluation identity. Missing optional usage stays
|
|
46
|
+
`null` and incomplete. Results storage
|
|
47
|
+
is explicit and independent of the project root. A packaged runner records its
|
|
48
|
+
package identity without Git; a private development checkout reads its own Git
|
|
49
|
+
revision and hashes tracked and untracked changes for the local runner
|
|
50
|
+
identity. A Git-backed project records its own revision and dirty content;
|
|
51
|
+
a project without a commit retains unknown Git provenance alongside a bounded
|
|
52
|
+
project content snapshot digest. An optional independent
|
|
53
|
+
run-state root holds an active record and a checkpoint pointing to each
|
|
54
|
+
retained trial. An evaluation-identity claim uses a short SQLite transaction
|
|
55
|
+
to admit one verified process owner. A live owner blocks an equivalent run;
|
|
56
|
+
an unverifiable owner is refused, while a confirmed dead owner is marked
|
|
57
|
+
interrupted and its prior attempt files are kept before a new claim starts.
|
|
58
|
+
The active record reaches `complete` only after final run evidence is written.
|
|
59
|
+
Process identity includes the PID, host, and operating-system start time to
|
|
60
|
+
protect against PID reuse. If start time cannot be verified, the run stops
|
|
61
|
+
before host execution.
|
|
62
|
+
The CLI passes SIGINT and SIGTERM through an abort signal. A cooperating host
|
|
63
|
+
stops its work; the engine retains completed trial artifacts, records the
|
|
64
|
+
cancelled trial, finalizes the active attempt as `interrupted`, and exits 130
|
|
65
|
+
or 143 after writing run evidence. An adapter that ignores cancellation cannot
|
|
66
|
+
be force-stopped by the injected-host interface.
|
|
67
|
+
|
|
68
|
+
The host can return bounded, namespaced observations alongside its final
|
|
69
|
+
message and bounded evidence artifacts. The engine retains them before
|
|
70
|
+
extension grading. It passes observations and artifact file references to the
|
|
71
|
+
extension without copying artifact bytes into run JSON. Missing or partial
|
|
72
|
+
required host evidence makes grading unavailable; malformed host evidence
|
|
73
|
+
fails execution. The engine also accepts shell checks when the caller supplies
|
|
74
|
+
protected source roots. They run through the macOS outer sandbox after host
|
|
75
|
+
execution, and their exit observations are retained with the trial. An
|
|
76
|
+
unavailable boundary produces a grading error rather than a passing check.
|
|
77
|
+
Semantic checks use a separately supplied host identity and an empty grader
|
|
78
|
+
workspace. A complete candidate final message is sent to that route with the
|
|
79
|
+
declared propositions. The engine requires one bounded verdict per check,
|
|
80
|
+
retains the raw response and grader artifacts outside the fixture, and records
|
|
81
|
+
the route and per-check observations. Missing candidate output makes semantic
|
|
82
|
+
checks unavailable; failed or malformed grader output makes grading an error.
|
|
83
|
+
Git HEAD checks capture the base commit before the candidate turn and grade
|
|
84
|
+
unchanged HEAD, changed HEAD, or base ancestry afterward. Candidate edits to
|
|
85
|
+
fixture files cannot rewrite the expected base revision.
|
|
86
|
+
An optional negotiated extension can add declared checks. The engine calls its
|
|
87
|
+
`prepare` method before host execution and its `evaluate` method for each
|
|
88
|
+
trial. It retains extension identity, configuration digest, protocol,
|
|
89
|
+
capabilities, grader selection, check evidence references, and metrics.
|
|
90
|
+
Missing declared extension checks remain unavailable; extension errors cannot
|
|
91
|
+
become a passing assessment. Inline preparation artifacts are decoded, checked
|
|
92
|
+
against their SHA-256 digests, mounted into each fixture, and retained outside
|
|
93
|
+
it for later inspection. Source-reference artifacts resolve only through a
|
|
94
|
+
caller-declared ID-to-file-URL map under an explicit case source root. Their
|
|
95
|
+
bytes receive the same size, path, and digest checks before mounting and
|
|
96
|
+
retention.
|
|
97
|
+
An explicitly selected extension task-verdict policy receives its policy ID
|
|
98
|
+
during `evaluate` and must return a recommendation. The engine uses that
|
|
99
|
+
recommendation only when execution and grading completed with all required
|
|
100
|
+
evidence available. It retains the selected policy and trial recommendation;
|
|
101
|
+
missing recommendations are grading errors. An explicitly replaced built-in
|
|
102
|
+
grader still has its declaration validated, but its checks do not run or count
|
|
103
|
+
toward the task verdict. At least one declared extension check must take its
|
|
104
|
+
place. The selection enters comparison identity and retained evidence.
|
|
105
|
+
An extension may request a negotiated host instrumentation capability during
|
|
106
|
+
`prepare`. The engine checks the host-declared capability and whether it
|
|
107
|
+
changes execution before starting a trial. Execution-changing requests are
|
|
108
|
+
refused in a passive condition. The host receives the request and must report
|
|
109
|
+
exactly what it applied; a mismatch or an unconfirmed enforced condition fails
|
|
110
|
+
execution. Run and trial evidence retain requested and applied instrumentation
|
|
111
|
+
separately. The bundled Codex adapter advertises no instrumentation capabilities.
|
|
112
|
+
|
|
113
|
+
Additional host adapters and host-specific instrumentation implementations
|
|
114
|
+
remain in progress.
|