@bjoernrochel/sevro 0.1.0-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +159 -0
- package/README.md +71 -0
- package/docs/advisory-fixture.md +23 -0
- package/docs/builtin-graders-v1.md +77 -0
- package/docs/cache-ownership-validation.md +49 -0
- package/docs/claude-continuation-validation.md +47 -0
- package/docs/codex-host.md +134 -0
- package/docs/concurrency-validation.md +86 -0
- package/docs/configuration-root-validation.md +70 -0
- package/docs/development-cli.md +327 -0
- package/docs/engine-slice.md +114 -0
- package/docs/extension-protocol-v1.md +292 -0
- package/docs/fixture-cleanup-validation.md +70 -0
- package/docs/identity-v1.md +65 -0
- package/docs/native-controls.md +29 -0
- package/docs/releases.md +88 -0
- package/docs/report-v1.md +32 -0
- package/docs/results-v1.md +118 -0
- package/examples/basic/graded.json +13 -0
- package/examples/basic/host.ts +8 -0
- package/examples/basic/prompt-only.json +7 -0
- package/package.json +45 -0
- package/schemas/cli-result-v1.schema.json +184 -0
- package/schemas/extension-v1.schema.json +659 -0
- package/schemas/report-v1.schema.json +164 -0
- package/schemas/run-evidence-v1.schema.json +510 -0
- package/src/advisory-fixture.ts +282 -0
- package/src/advisory.ts +105 -0
- package/src/cli.ts +837 -0
- package/src/engine.ts +2341 -0
- package/src/extension-client.ts +257 -0
- package/src/extension-session.ts +335 -0
- package/src/fixture-bin.ts +87 -0
- package/src/fixture-cleanup.ts +38 -0
- package/src/fixture-setup.ts +163 -0
- package/src/generated-fixture.ts +220 -0
- package/src/git-hooks.ts +63 -0
- package/src/graders/git-head.ts +176 -0
- package/src/graders/output.ts +359 -0
- package/src/graders/semantic.ts +120 -0
- package/src/graders/shell.ts +277 -0
- package/src/hosts/claude-continuation.ts +94 -0
- package/src/hosts/claude-credential.ts +73 -0
- package/src/hosts/claude-events.ts +128 -0
- package/src/hosts/claude-nested-skills.ts +218 -0
- package/src/hosts/claude-repository-invocation.ts +239 -0
- package/src/hosts/claude-settings.ts +50 -0
- package/src/hosts/claude-tool-calls.ts +159 -0
- package/src/hosts/claude.ts +511 -0
- package/src/hosts/codex-configuration.ts +85 -0
- package/src/hosts/codex-events.ts +102 -0
- package/src/hosts/codex-native-calls.ts +702 -0
- package/src/hosts/codex-profile.ts +78 -0
- package/src/hosts/codex-skill-reads.ts +345 -0
- package/src/hosts/codex.ts +769 -0
- package/src/hosts/isolation-roots.ts +60 -0
- package/src/hosts/mac-sandbox.ts +108 -0
- package/src/hosts/native-controls.ts +124 -0
- package/src/hosts/protected-worktrees.ts +56 -0
- package/src/hosts/workspace-fingerprint.ts +55 -0
- package/src/identity.ts +156 -0
- package/src/instrumentation.ts +106 -0
- package/src/preparation.ts +211 -0
- package/src/provenance.ts +244 -0
- package/src/report.ts +351 -0
- package/src/repository-fixture.ts +286 -0
- package/src/results.ts +160 -0
- package/src/run-owner.ts +276 -0
- package/src/schema.ts +28 -0
- package/src/storage.ts +23 -0
- package/src/trial-scheduler.ts +27 -0
|
@@ -0,0 +1,292 @@
|
|
|
1
|
+
# Extension protocol v1
|
|
2
|
+
|
|
3
|
+
This is the target public contract for `sevro.extension.v1`. Extensions are
|
|
4
|
+
explicitly configured development tools. Installing one does not change
|
|
5
|
+
built-in grading or grant it control of fixture isolation, execution,
|
|
6
|
+
persistence, or cleanup. The executable is trusted by the operator; a process
|
|
7
|
+
boundary alone is not a sandbox.
|
|
8
|
+
|
|
9
|
+
[`extension-v1.schema.json`](../schemas/extension-v1.schema.json) validates the
|
|
10
|
+
versioned request and response shapes. The engine also validates request IDs,
|
|
11
|
+
negotiated capabilities, path containment, decoded content, and the source and
|
|
12
|
+
configuration digests at runtime.
|
|
13
|
+
|
|
14
|
+
## Transport
|
|
15
|
+
|
|
16
|
+
The runner receives an extension command as an argv array and never interprets
|
|
17
|
+
it through a shell. For each method it starts a fresh process, writes exactly
|
|
18
|
+
one UTF-8 JSON request to stdin, closes stdin, and reads exactly one UTF-8 JSON
|
|
19
|
+
response from stdout before process exit. The request and response are each
|
|
20
|
+
limited to 8 MiB. Extra stdout, malformed JSON, an unmatched request ID, a
|
|
21
|
+
nonzero exit, timeout, or an oversized message is a protocol error. Stderr is
|
|
22
|
+
diagnostic only, bounded to 64 KiB in retained evidence; it cannot supply a
|
|
23
|
+
result. The runner cancels the extension process group on interruption.
|
|
24
|
+
|
|
25
|
+
Every request has this envelope. `describe` uses the stable
|
|
26
|
+
`sevro.discovery.v1` bootstrap protocol; subsequent methods use the selected
|
|
27
|
+
extension protocol:
|
|
28
|
+
|
|
29
|
+
```json
|
|
30
|
+
{
|
|
31
|
+
"protocol": "sevro.discovery.v1",
|
|
32
|
+
"id": "request-1",
|
|
33
|
+
"method": "describe",
|
|
34
|
+
"params": {}
|
|
35
|
+
}
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
A response echoes `protocol`, `id`, and `method`, then supplies exactly one of
|
|
39
|
+
`result` or `error`:
|
|
40
|
+
|
|
41
|
+
```json
|
|
42
|
+
{
|
|
43
|
+
"protocol": "sevro.discovery.v1",
|
|
44
|
+
"id": "request-1",
|
|
45
|
+
"method": "describe",
|
|
46
|
+
"result": {
|
|
47
|
+
"extension": { "id": "example.policy", "version": "1.0.0" },
|
|
48
|
+
"protocols": ["sevro.extension.v1"],
|
|
49
|
+
"requiredCapabilities": [],
|
|
50
|
+
"optionalCapabilities": [],
|
|
51
|
+
"graders": [],
|
|
52
|
+
"taskVerdictPolicies": []
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
An error has `{ "code": "namespaced.code", "message": "bounded text" }`.
|
|
58
|
+
An error response, process failure, or invalid response never becomes a
|
|
59
|
+
successful grade. The engine retains the method, error class, and bounded
|
|
60
|
+
diagnostic without copying secrets or unbounded extension output.
|
|
61
|
+
|
|
62
|
+
## Negotiation and operations
|
|
63
|
+
|
|
64
|
+
The engine calls `describe` first through the discovery protocol. The
|
|
65
|
+
extension identifies its stable ID and version, supported protocol versions,
|
|
66
|
+
required and optional capability IDs,
|
|
67
|
+
grader IDs it supplies, and task-verdict policy IDs it supplies. Capability IDs
|
|
68
|
+
are namespaced strings. The engine chooses one compatible protocol, verifies
|
|
69
|
+
all required capabilities against its own and the selected host's capability
|
|
70
|
+
registry, and records the negotiated set. An unknown required capability is an
|
|
71
|
+
error. Unknown optional capabilities remain unavailable and cannot be used.
|
|
72
|
+
The engine, not the extension, classifies whether instrumentation changes host
|
|
73
|
+
execution. Future extension protocols still need a compatible discovery
|
|
74
|
+
response or an explicitly selected bridge; an unsupported protocol cannot be
|
|
75
|
+
guessed from a failed extension call.
|
|
76
|
+
|
|
77
|
+
`resolve` receives the selected project root, selectors, and extension
|
|
78
|
+
configuration. The CLI also supplies `host` with the candidate adapter's `id`,
|
|
79
|
+
`model`, `effort`, and available `capabilities`. The engine capability
|
|
80
|
+
`sevro.case.host-route` guarantees this context; extensions that depend on it
|
|
81
|
+
must negotiate that capability. These are selected route facts for preparing
|
|
82
|
+
cases, not observations of completed execution. Semantic and advisory judge
|
|
83
|
+
routes are not candidate context. Older v1 callers may omit `host` when this
|
|
84
|
+
capability is unavailable.
|
|
85
|
+
|
|
86
|
+
It returns neutral case descriptions with stable IDs, prompts,
|
|
87
|
+
an optional `followUpPrompt` for a second turn in the same host session,
|
|
88
|
+
fixture and built-in check declarations, required evidence declarations, and
|
|
89
|
+
namespaced extension data. Paths in case descriptions are resolved and
|
|
90
|
+
validated by the engine against the declared project and case-source roots.
|
|
91
|
+
An extension must negotiate `sevro.host.continuation` before resolving a
|
|
92
|
+
continuation case, and the selected host must declare the same capability.
|
|
93
|
+
The extension cannot add a hidden source root by returning an arbitrary path.
|
|
94
|
+
Fixtures may contain inline files, a declared clean repository source, or a
|
|
95
|
+
bounded generated Git history. Both Git fixture kinds support optional
|
|
96
|
+
working-tree files, staged paths, committed scaffolding, bounded Git hooks,
|
|
97
|
+
and fixture executables. Sevro validates overlay paths, refuses symlinks and
|
|
98
|
+
repository metadata in repository clones, installs hooks and executables
|
|
99
|
+
after its own commits, and includes these declarations in fixture identity. Repository-specific policy and
|
|
100
|
+
custom setup remain in the extension.
|
|
101
|
+
|
|
102
|
+
An extension that negotiates `sevro.fixture.setup` may return an optional
|
|
103
|
+
`fixtureSetup` from `prepare`. It names an absolute executable and bounded argv,
|
|
104
|
+
plus optional environment variables. Sevro runs that trusted command once per
|
|
105
|
+
trial in the Git fixture after source history and working-tree files are built,
|
|
106
|
+
before extension artifacts are mounted or the host starts. The command is never
|
|
107
|
+
interpreted through a shell by Sevro; an extension may explicitly select a
|
|
108
|
+
shell executable. `{{sevro.project}}` and `{{sevro.workspace}}` in environment
|
|
109
|
+
values resolve to the declared roots at execution time. The symbolic values and
|
|
110
|
+
command enter fixture and comparison identity, while retained configuration
|
|
111
|
+
records only their digest. Setup has a two-minute limit, inherits no credential
|
|
112
|
+
environment variables, and its output is discarded. Failure or interruption
|
|
113
|
+
stops the run before host execution; it cannot become a passing assessment.
|
|
114
|
+
Absolute path references in this protocol use `file:///` URLs so Windows and
|
|
115
|
+
Unix hosts share one serialized form. Fixture-relative paths use `/` separators
|
|
116
|
+
and cannot escape their declared root.
|
|
117
|
+
|
|
118
|
+
`prepare` receives one resolved case, the chosen host and capabilities, the
|
|
119
|
+
configured passive or enforced condition, and extension data. It returns
|
|
120
|
+
fixture preparation artifacts and requested instrumentation IDs with
|
|
121
|
+
parameters. The engine validates these requests before host execution and
|
|
122
|
+
passes them to the host. It rejects unknown or unsupported IDs and rejects any
|
|
123
|
+
execution-changing instrumentation in a passive trial. Requested and applied
|
|
124
|
+
instrumentation are recorded separately. Host adapters declare supported IDs
|
|
125
|
+
and whether each changes execution. A requested ID must be among the
|
|
126
|
+
negotiated capabilities. The host receives the validated request and must
|
|
127
|
+
report the same applied IDs and parameters; a mismatch fails execution. An
|
|
128
|
+
execution-changing request also requires the host to confirm an enforced
|
|
129
|
+
condition. Instrumentation parameters enter retained evidence and must not
|
|
130
|
+
contain secrets.
|
|
131
|
+
|
|
132
|
+
A Codex extension may also negotiate `sevro.codex.plugin-marketplace` and
|
|
133
|
+
return `codexMarketplace` from `prepare` with `artifactRoot`,
|
|
134
|
+
`marketplaceName`, and nonempty `pluginNames`. Every package file must be a
|
|
135
|
+
Git-excluded preparation artifact below that fixture-relative root, including
|
|
136
|
+
`.claude-plugin/marketplace.json`. Sevro checks the declared names, package
|
|
137
|
+
paths, and local manifest sources before invoking the Codex plugin CLI. The
|
|
138
|
+
declaration and artifact digests enter evaluation identity. The Codex host
|
|
139
|
+
installs the marketplace and named plugins in its private home for each trial;
|
|
140
|
+
no package is installed from an undeclared artifact. This fixture setup is
|
|
141
|
+
available in passive trials because it does not add enforcement
|
|
142
|
+
instrumentation.
|
|
143
|
+
An extension that also negotiates `sevro.codex.explicit-invocation` may declare
|
|
144
|
+
`codexSkillInvocation` with a plugin and skill in that package. The resolved
|
|
145
|
+
prompts must contain `{{sevro.codex.skill_invocation}}` exactly once across
|
|
146
|
+
both turns. Sevro
|
|
147
|
+
renders the installed `$plugin:skill` token separately for each trial and the
|
|
148
|
+
Codex host retains a bounded dispatch receipt only after a completed turn.
|
|
149
|
+
Missing or repeated tokens, absent packaged skills, and unsupported hosts fail
|
|
150
|
+
before candidate execution. Supporting skill reads remain separate evidence.
|
|
151
|
+
|
|
152
|
+
For a repository skill, negotiate `sevro.codex.repository-invocation` and
|
|
153
|
+
declare `codexRepositorySkillInvocation` with `skillName`. It is mutually
|
|
154
|
+
exclusive with the plugin invocation declarations. Prepare the exact
|
|
155
|
+
`.agents/skills/<skillName>/SKILL.md` as a Git-excluded artifact, and place
|
|
156
|
+
`{{sevro.skill_invocation}}` exactly once across the prompts. Sevro renders
|
|
157
|
+
`$skillName`, preserves repository scope in the host request, and records the
|
|
158
|
+
declaration separately in configuration and evaluation identity. Missing mounts,
|
|
159
|
+
repeated tokens, or absent negotiation fail before candidate execution. The
|
|
160
|
+
Codex host verifies the repository mount and emits its normal bounded explicit
|
|
161
|
+
dispatch receipt after a completed turn. No plugin package is required.
|
|
162
|
+
|
|
163
|
+
For a Claude repository skill, negotiate `sevro.claude.repository-invocation`
|
|
164
|
+
and declare `claudeRepositorySkillInvocation` with `skillName`. Mount the exact
|
|
165
|
+
`.claude/skills/<skillName>/SKILL.md` as a Git-excluded preparation artifact.
|
|
166
|
+
The bundled host advertises this capability only with
|
|
167
|
+
`--claude-project-settings`; otherwise it rejects dispatch before execution.
|
|
168
|
+
The neutral invocation placeholder must lead the initial prompt and occur once;
|
|
169
|
+
Sevro renders `/skillName`. This declaration is mutually exclusive with all
|
|
170
|
+
other invocation declarations and enters retained configuration and identity.
|
|
171
|
+
The host validates the bounded mount without following symbolic links. Its
|
|
172
|
+
`sevro.claude.repository-invocation` observation binds one native command, exact
|
|
173
|
+
arguments, and the complete mounted body before the first assistant entry in
|
|
174
|
+
the completed session. It retains only acceptance and ordered skill metadata.
|
|
175
|
+
Missing or ambiguous session evidence yields a partial observation; later Skill
|
|
176
|
+
calls cannot repair an unverified explicit repository dispatch.
|
|
177
|
+
|
|
178
|
+
A Claude extension may negotiate `sevro.claude.plugin-dirs` and return
|
|
179
|
+
`claudePluginDirs` with nonempty fixture-relative `artifactRoots`. Roots must
|
|
180
|
+
be unique and must not overlap. Every file below each root must be a
|
|
181
|
+
Git-excluded preparation artifact, and each root must contain
|
|
182
|
+
`.claude-plugin/plugin.json`. Sevro passes only the declared roots and their
|
|
183
|
+
verified artifact paths to a capable host. The declaration and artifact
|
|
184
|
+
digests enter evaluation identity. The host remains responsible for loading
|
|
185
|
+
the plugin directories and isolating its credentials from candidate tools.
|
|
186
|
+
An extension that also negotiates `sevro.claude.explicit-invocation` may
|
|
187
|
+
declare `claudeSkillInvocation` for a skill present in one of those packages.
|
|
188
|
+
The resolved prompts must contain `{{sevro.skill_invocation}}` exactly once
|
|
189
|
+
across both turns. Sevro renders the `/plugin:skill` token for each trial and
|
|
190
|
+
passes the same token to the Claude host. The neutral placeholder also works
|
|
191
|
+
with `codexSkillInvocation`; the older
|
|
192
|
+
`{{sevro.codex.skill_invocation}}` placeholder remains supported for Codex.
|
|
193
|
+
|
|
194
|
+
An artifact for a generated or cloned Git fixture may set `gitExclude: true`.
|
|
195
|
+
Sevro mounts its verified bytes, adds only that exact artifact path to the
|
|
196
|
+
fixture's `.git/info/exclude`, and retains the flag in artifact evidence and
|
|
197
|
+
fixture identity. The flag is invalid for a non-Git fixture. It lets evaluator
|
|
198
|
+
assets such as project skills remain visible to the host without appearing as
|
|
199
|
+
candidate changes in Git status checks.
|
|
200
|
+
Artifact mounts refuse existing files and any parent that is a symlink or
|
|
201
|
+
non-directory, including paths created by fixture setup or a cloned repository.
|
|
202
|
+
An artifact may set `executable: true` to install its verified bytes with owner
|
|
203
|
+
execute permission. The flag enters fixture identity and retained artifact
|
|
204
|
+
references; absent or false leaves the file readable without execute permission.
|
|
205
|
+
|
|
206
|
+
`evaluate` receives one completed or failed trial's bounded host observations,
|
|
207
|
+
their source and completeness, built-in check results, artifact references,
|
|
208
|
+
and the extension's namespaced case data. It returns namespaced checks and
|
|
209
|
+
metrics. A check is `passed`, `failed`, or `unavailable`; it names the evidence
|
|
210
|
+
used. A missing required observation remains unavailable. The extension may
|
|
211
|
+
return a task-verdict recommendation only when an explicitly configured
|
|
212
|
+
task-verdict policy names that extension policy. In that case, `evaluate`
|
|
213
|
+
receives `selectedTaskVerdictPolicy` and must return a recommendation. The
|
|
214
|
+
engine uses it only for a completed trial with completed grading and available
|
|
215
|
+
required evidence. It retains the selected policy and each trial's
|
|
216
|
+
recommendation. An omitted recommendation is a grading error. A policy may
|
|
217
|
+
change a failed check's task verdict, but cannot conceal execution failure,
|
|
218
|
+
grader error, or unavailable required evidence.
|
|
219
|
+
Host adapters can supply additional namespaced observations. The engine bounds
|
|
220
|
+
and retains those records, then passes them to `evaluate`. Required observation
|
|
221
|
+
IDs must be present with `complete` evidence before a task can pass.
|
|
222
|
+
Host adapters can also supply bounded evidence artifacts. The engine persists
|
|
223
|
+
their bytes outside the candidate workspace before `evaluate`, then passes
|
|
224
|
+
file URLs and digests through the existing artifact references. A required host
|
|
225
|
+
artifact must be present before a task can pass.
|
|
226
|
+
|
|
227
|
+
The methods are stateless. Every request carries the configuration and prior
|
|
228
|
+
extension data it needs. The runner verifies extension identity and source
|
|
229
|
+
digest for the run; a changed executable or configuration cannot silently
|
|
230
|
+
continue the same run.
|
|
231
|
+
|
|
232
|
+
The session client in [`extension-session.ts`](../src/extension-session.ts)
|
|
233
|
+
implements discovery, `resolve`, `prepare`, and `evaluate` as separate process
|
|
234
|
+
calls. Its caller declares the extension source-file closure; the client also
|
|
235
|
+
includes the executable and hashes that closure before discovery and before
|
|
236
|
+
and after later calls. The redacted configuration and command argv affect the
|
|
237
|
+
configuration digest. Command argv must not contain credentials. A passing
|
|
238
|
+
extension check must cite available, complete evidence. The engine now accepts
|
|
239
|
+
inline and declared clean-repository cases with additive extension checks, runs `prepare` before the host,
|
|
240
|
+
and persists check outcomes, evidence references, and metrics before fixture
|
|
241
|
+
cleanup. It applies and retains bounded inline preparation artifacts after
|
|
242
|
+
checking their path and digest. Source references require a caller-declared
|
|
243
|
+
ID-to-file-URL map under a case source root and receive the same checks.
|
|
244
|
+
The bundled Codex host currently advertises no instrumentation capabilities.
|
|
245
|
+
|
|
246
|
+
## Grading and replacement
|
|
247
|
+
|
|
248
|
+
The runner provides built-in shell, regex, JSON/schema, combined output,
|
|
249
|
+
semantic-output, and
|
|
250
|
+
advisory quality graders. A run can use those without any extension. An
|
|
251
|
+
extension's graders add checks and metrics by default. The run configuration
|
|
252
|
+
must explicitly name each built-in grader it replaces, or explicitly select
|
|
253
|
+
one advertised task-verdict policy. Unknown or duplicate replacement IDs are
|
|
254
|
+
errors. Retained evidence lists every active grader, its identity and route,
|
|
255
|
+
and any replaced default.
|
|
256
|
+
|
|
257
|
+
An extension may also return namespaced `domainOutcomes` with passed, failed,
|
|
258
|
+
or unavailable status, evidence references, and bounded domain data. Sevro
|
|
259
|
+
validates their evidence and retains them separately from task checks. They do
|
|
260
|
+
not change the task verdict. Older evidence without this field leaves domain
|
|
261
|
+
outcomes unknown.
|
|
262
|
+
|
|
263
|
+
When a built-in grader is selected for replacement, the engine validates its
|
|
264
|
+
declared check configuration but does not run its checks. It requires at least
|
|
265
|
+
one declared, advertised extension check, and grades those checks instead.
|
|
266
|
+
Unselected built-in checks stay active. The replacement list enters extension
|
|
267
|
+
configuration and evaluation identity; evidence retains the selected defaults
|
|
268
|
+
and omits them from active graders and trial check results. A replaced shell or
|
|
269
|
+
semantic grader does not require its usual isolation or host route because it
|
|
270
|
+
does not execute.
|
|
271
|
+
|
|
272
|
+
An extension cannot replace execution status, grading status, or the evidence
|
|
273
|
+
availability rules. It cannot turn an error or missing required evidence into
|
|
274
|
+
a successful task assessment. A prompt without success criteria produces
|
|
275
|
+
execution evidence and a task verdict of `not_assessed`.
|
|
276
|
+
|
|
277
|
+
## Lifecycle
|
|
278
|
+
|
|
279
|
+
The engine orders each trial as follows:
|
|
280
|
+
|
|
281
|
+
1. Negotiate protocol and capabilities; resolve the case.
|
|
282
|
+
2. Prepare the fixture and extension inputs; validate instrumentation against
|
|
283
|
+
the host and passive/enforced condition.
|
|
284
|
+
3. Execute or resume the host and collect bounded observations.
|
|
285
|
+
4. Run built-in checks, then extension grading, then compute separate
|
|
286
|
+
execution, grading, and task states.
|
|
287
|
+
5. Persist complete trial evidence before fixture cleanup.
|
|
288
|
+
|
|
289
|
+
On cancellation or failure, completed trials remain persisted and partial
|
|
290
|
+
attempts retain an explicit incomplete or diagnostic state. The engine owns
|
|
291
|
+
process-group cancellation and cleanup even when an extension fails. A dry
|
|
292
|
+
run records preparation but no observed behavioral success.
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Permission-locked fixture cleanup
|
|
2
|
+
|
|
3
|
+
The Darrow extraction comparison exposed a cleanup difference: its legacy
|
|
4
|
+
fixture destroyer removes permission-locked directories, while Sevro retained
|
|
5
|
+
them after a completed trial. Sevro now restores owner access inside completed
|
|
6
|
+
candidate workspaces when needed. It clears their contents after trial evidence
|
|
7
|
+
is retained, keeps empty reservations until active peers drain, and then removes
|
|
8
|
+
the completed roots. Directory traversal uses link metadata and skips symbolic
|
|
9
|
+
links; external targets retain their permissions and contents.
|
|
10
|
+
|
|
11
|
+
## Public CLI regression slices
|
|
12
|
+
|
|
13
|
+
The public seam is `sevro run --case-file ... --adapter-module ...`, as documented
|
|
14
|
+
in [Development CLI](development-cli.md). Each controlled adapter uses real
|
|
15
|
+
filesystem permissions and returns its workspace path as a bounded observation.
|
|
16
|
+
The test reads retained public evidence and verifies that the workspace is gone.
|
|
17
|
+
A preliminary internal-engine test was diagnostic only; it is not part of this
|
|
18
|
+
TDD trace and was removed before implementation.
|
|
19
|
+
|
|
20
|
+
Red — `bun test tests/cli.test.ts --test-name-pattern 'CLI removes a permission-locked candidate directory'`: completed evidence existed, but the mode-000 child directory kept the workspace present.
|
|
21
|
+
|
|
22
|
+
Green — `bun test tests/cli.test.ts --test-name-pattern 'CLI removes a permission-locked candidate directory'`: the workspace was removed and retained evidence still agreed with the CLI result.
|
|
23
|
+
|
|
24
|
+
Red — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(root\)'`: a mode-000 workspace root remained present after completion.
|
|
25
|
+
|
|
26
|
+
Green — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(root\)'`: the workspace root was removed after evidence retention.
|
|
27
|
+
|
|
28
|
+
Red — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(readable root\)'`: the readable mode-500 root lacked write access and remained present.
|
|
29
|
+
|
|
30
|
+
Green — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(readable root\)'`: restoring owner write access allowed cleanup; the external link target's mode and bytes were unchanged.
|
|
31
|
+
|
|
32
|
+
Each final example also links to an external directory with a read-only file.
|
|
33
|
+
All three check that its directory mode, file mode, and file contents stay intact.
|
|
34
|
+
These are deterministic filesystem and lifecycle checks, with no live model call.
|
|
35
|
+
|
|
36
|
+
## Final source gates
|
|
37
|
+
|
|
38
|
+
`bun test` passed all 243 tests and 1399 assertions across 38 files in 93.83
|
|
39
|
+
seconds. The gate includes parallel admission, peer isolation, cancellation,
|
|
40
|
+
failed persistence, extension lifecycle, and generated/repository fixture tests.
|
|
41
|
+
`bun run typecheck` and `bun run format:check` pass.
|
|
42
|
+
The full log and subsequent installed-archive checks are retained under
|
|
43
|
+
`/Users/bjro/.darrow/issue95-fixture-cleanup/`. The tests ran on macOS with Bun
|
|
44
|
+
1.3.13; they do not establish another platform or native model route.
|
|
45
|
+
|
|
46
|
+
## Canonical workspace follow-up
|
|
47
|
+
|
|
48
|
+
Darrow's private fixture builder also checked that its returned path was
|
|
49
|
+
canonical. An initial diagnostic assertion in the cleanup examples exposed
|
|
50
|
+
Sevro's logical temporary-directory alias; those diagnostic additions were
|
|
51
|
+
removed before the isolated regression. The public canonical-workspace test
|
|
52
|
+
records the operating system's `realpath` through a controlled host and compares
|
|
53
|
+
it with the received workspace in retained evidence. The full source gate above
|
|
54
|
+
predates this follow-up.
|
|
55
|
+
|
|
56
|
+
The public CLI contract was clarified before implementation. Candidate and
|
|
57
|
+
semantic workspace allocations now resolve the new directory through `realpath`
|
|
58
|
+
before reservation, preparation, or host execution.
|
|
59
|
+
|
|
60
|
+
Red — `bun test tests/cli.test.ts --test-name-pattern 'CLI supplies a canonical candidate workspace'`:
|
|
61
|
+
exit `1`; the received `/var/...` path differed from the operating system's
|
|
62
|
+
`/private/var/...` path. One test failed with four assertions in 222 milliseconds.
|
|
63
|
+
|
|
64
|
+
Green — `bun test tests/cli.test.ts --test-name-pattern 'CLI supplies a canonical candidate workspace'`:
|
|
65
|
+
exit `0`; one test and four assertions passed in 228 milliseconds after the fix.
|
|
66
|
+
|
|
67
|
+
The final `bun test` gate passed 244 tests and 1403 assertions across 38 files in
|
|
68
|
+
94.77 seconds. Typecheck and formatting pass. The source log is retained at
|
|
69
|
+
`/Users/bjro/.darrow/issue95-fixture-mounting/sevro-full-gate.log`. These remain
|
|
70
|
+
deterministic macOS/Bun checks, without a live model or additional platform claim.
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Evaluation identity v1
|
|
2
|
+
|
|
3
|
+
`sevro.identity.v1` binds retained evidence to the inputs that can change an
|
|
4
|
+
assessment. Its `dimensions` object has exactly the fields required by
|
|
5
|
+
[`run-evidence-v1.schema.json`](../schemas/run-evidence-v1.schema.json). The
|
|
6
|
+
runner computes each `*Digest` from the relevant normalized content. It must
|
|
7
|
+
not use a temporary path, run ID, or timestamp as an input.
|
|
8
|
+
|
|
9
|
+
| Dimension | Content represented |
|
|
10
|
+
| ------------------------------------------ | ---------------------------------------------------------------------------------------------- |
|
|
11
|
+
| `runnerBuildDigest` | Exact installed package build or local runner source build. |
|
|
12
|
+
| `projectDigest` | Evaluated project revision and dirty content, or a content snapshot without Git. |
|
|
13
|
+
| `configurationDigest` | Evaluation settings with credentials replaced by secret-presence markers. |
|
|
14
|
+
| `extensionDigest`, `extensionProtocol` | Extension source closure and negotiated protocol, or both `null`. |
|
|
15
|
+
| `caseDigest`, `fixtureDigest` | Resolved case declaration and fixture contents. |
|
|
16
|
+
| `checksDigest`, `requiredEvidenceDigest` | Declared checks and evidence requirements. |
|
|
17
|
+
| `evaluatorDigest`, `graderDigest` | Evaluator policy, grader versions, active selection, and replacements. |
|
|
18
|
+
| `instrumentationDigest` | Requested instrumentation and expected application; trial evidence records actual application. |
|
|
19
|
+
| `routeDigest` | Candidate, semantic, and advisory host/model/effort routes. |
|
|
20
|
+
| `condition`, `trialCount`, `passThreshold` | Declared condition and assessment sampling rule. |
|
|
21
|
+
|
|
22
|
+
Each structured component digest is SHA-256 of its [RFC 8785 JSON
|
|
23
|
+
Canonicalization Scheme](https://www.rfc-editor.org/rfc/rfc8785.html) UTF-8
|
|
24
|
+
bytes. The runner rejects values outside canonical JSON, including non-finite
|
|
25
|
+
numbers and unpaired Unicode surrogates. The final evaluation digest is:
|
|
26
|
+
|
|
27
|
+
```text
|
|
28
|
+
SHA-256(UTF-8("sevro.identity.v1\n") || UTF-8(JCS(dimensions)))
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The CLI derives `runnerBuildDigest` from sorted packaged file paths and their
|
|
32
|
+
SHA-256 bytes under `src/`, `schemas/`, `docs/`, and `examples/`, plus `README.md` and
|
|
33
|
+
`package.json`. For a Git project, `projectDigest` binds the revision and dirty
|
|
34
|
+
patch digest without the checkout path. Without a revision it hashes a bounded
|
|
35
|
+
file snapshot, excluding `.git`, results, and run-state storage. A caller may
|
|
36
|
+
provide explicit 64-character digests for controlled development runs.
|
|
37
|
+
|
|
38
|
+
The prefix separates this digest from component digests and future identity
|
|
39
|
+
versions. An exact-repeat comparison requires equal identities. An ablation may
|
|
40
|
+
deliberately differ on `condition` or instrumentation; its report must name
|
|
41
|
+
those dimensions and verify that the intended controls match. Historical
|
|
42
|
+
artifacts without a dimension cannot be silently assigned a current identity.
|
|
43
|
+
|
|
44
|
+
Credential values, raw transcripts, and temporary output locations never enter
|
|
45
|
+
the stored dimensions. A credential's presence can be recorded in a redacted
|
|
46
|
+
configuration; its bytes are never retained in evidence or its digest input.
|
|
47
|
+
The effective positive trial job limit is retained as `configuration.redacted.jobs`
|
|
48
|
+
and enters `configurationDigest`. Changing parallelism changes evaluation identity,
|
|
49
|
+
even when the condition, host route, trial count, and assessment threshold match.
|
|
50
|
+
|
|
51
|
+
Extension runs retain the operator-supplied redacted extension configuration
|
|
52
|
+
under `configuration.redacted.extensionConfiguration`, alongside
|
|
53
|
+
`extensionConfigurationDigest`. The retained copy is the same snapshot used
|
|
54
|
+
for extension configuration identity. Private configuration stays outside run
|
|
55
|
+
evidence. Consumers can compare the visible configuration with their declared
|
|
56
|
+
inputs without reconstructing the engine's component digest inputs.
|
|
57
|
+
|
|
58
|
+
Hosts may supply explicit, redacted `configuration` values. Sevro retains them
|
|
59
|
+
under `configuration.redacted.hostConfiguration`, keyed by the used candidate,
|
|
60
|
+
semantic, or advisory role, and includes them in `configurationDigest`. Custom
|
|
61
|
+
adapter authors must keep secrets out of this object. Bundled Codex roles record
|
|
62
|
+
`sevro.codex.agent-concurrency-limit` as the captured positive integer or `null`
|
|
63
|
+
for the host default. Configuration paths and unrelated TOML bytes are not
|
|
64
|
+
imported settings; a relocated configuration with the same effective values
|
|
65
|
+
keeps this component unchanged.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Native control observation v1
|
|
2
|
+
|
|
3
|
+
The bundled Codex and Claude hosts advertise `sevro.host.native-controls` and
|
|
4
|
+
retain an observation with that ID. It exposes host facts for extension policy:
|
|
5
|
+
|
|
6
|
+
- `method: native_control_calls`;
|
|
7
|
+
- ordered `calls` containing only `ordinal`, `namespace`, and `name`;
|
|
8
|
+
- `acceptedAgentCount`, a verified Codex acceptance count or `null` on Claude;
|
|
9
|
+
- `submittedExecCalls`, the Codex submitted-code count or `null` on Claude; and
|
|
10
|
+
- `truncated`, which is `false` for a complete record and may be unknown (`null`)
|
|
11
|
+
for an incomplete Codex record.
|
|
12
|
+
|
|
13
|
+
The observation's source is the executing host, not the extension. It keeps no
|
|
14
|
+
tool arguments, task messages, command bodies, or results. Call labels prove an
|
|
15
|
+
invocation attempt, not success. Extensions decide which calls are prohibited.
|
|
16
|
+
An unknown acceptance count cannot establish agent acceptance.
|
|
17
|
+
|
|
18
|
+
Codex projects labels and counts from its bound native-session observation and
|
|
19
|
+
preserves its completeness. Incomplete counts remain `null`. Submitted code remains explicit: extensions cannot
|
|
20
|
+
infer absence of controls inside code that this observer did not inspect.
|
|
21
|
+
Claude records every tool-use label from the bounded stream, with namespace
|
|
22
|
+
`claude`, without filtering to Skill or Agent. It requires a completed successful
|
|
23
|
+
turn, valid unique tool-use IDs and names, and a non-truncated list of at most
|
|
24
|
+
128 calls. Malformed, duplicate, failed, or excessive streams remain partial.
|
|
25
|
+
Tool-looking text in arguments or output does not become a native call.
|
|
26
|
+
|
|
27
|
+
Consumers must verify the source, completeness, shape, ordering, and counts
|
|
28
|
+
before using an empty call list to establish absence. Missing or partial
|
|
29
|
+
evidence cannot establish a negative assertion.
|
package/docs/releases.md
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# Releases
|
|
2
|
+
|
|
3
|
+
Sevro releases are public npm packages named `@bjoernrochel/sevro`. The public
|
|
4
|
+
command remains `sevro`, and npm archives use `bjoernrochel-sevro-<version>.tgz`.
|
|
5
|
+
A release version is an
|
|
6
|
+
exact semantic version; development versions containing `-dev` are not release
|
|
7
|
+
candidates. Darrow pins the published version in development dependencies and
|
|
8
|
+
its lockfile. Marketplace plugins do not depend on Sevro.
|
|
9
|
+
|
|
10
|
+
## Candidate contract
|
|
11
|
+
|
|
12
|
+
Prepare one candidate from the intended release tag and package metadata. The
|
|
13
|
+
tag must equal `v<package version>`. Package metadata must identify the public
|
|
14
|
+
repository, registry, distribution tag, and an owner-selected license with its
|
|
15
|
+
nonempty `LICENSE` file. Prereleases cannot use npm's `latest` distribution tag.
|
|
16
|
+
|
|
17
|
+
Preparation creates a new absolute output directory and never replaces an
|
|
18
|
+
existing directory. It packs with lifecycle scripts disabled, retains the npm
|
|
19
|
+
file inventory, and writes `release.json` and `SHA256SUMS` beside the tarball.
|
|
20
|
+
The record identifies the package, version, release tag, distribution tag,
|
|
21
|
+
tarball SHA-256 and npm SHA-512 integrity, and the preparation runtimes.
|
|
22
|
+
Preparation does not publish a package or create a Git tag.
|
|
23
|
+
|
|
24
|
+
The package installation check accepts `--tarball <absolute file>` to verify
|
|
25
|
+
that exact candidate without repacking. The supplied tarball remains intact.
|
|
26
|
+
Its installed name and version must match the source metadata. Installation,
|
|
27
|
+
CLI execution, generic grading, prompt-only execution, release provenance,
|
|
28
|
+
reporting, and supported deterministic native fixtures run without source Git
|
|
29
|
+
metadata in the installed package.
|
|
30
|
+
|
|
31
|
+
## Maintainer workflow
|
|
32
|
+
|
|
33
|
+
1. Select the license, update `package.json` to the intended release version,
|
|
34
|
+
and commit those changes. Preserve the existing public schemas and document
|
|
35
|
+
behavior changes. For the first candidate, use `0.1.0-rc.1` with npm tag
|
|
36
|
+
`next`; a prerelease does not establish full Darrow migration completion.
|
|
37
|
+
2. Run frozen installation, typecheck, formatting, and the full test suite.
|
|
38
|
+
3. Prepare and test the exact candidate:
|
|
39
|
+
|
|
40
|
+
```sh
|
|
41
|
+
bun run release:prepare --tag v0.1.0-rc.1 --output /absolute/new/release-directory
|
|
42
|
+
bun run test:package-install --tarball /absolute/new/release-directory/bjoernrochel-sevro-0.1.0-rc.1.tgz
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
4. Run Darrow's public integration against that same tarball:
|
|
46
|
+
|
|
47
|
+
```sh
|
|
48
|
+
SEVRO_PACKAGE_TARBALL=/absolute/new/release-directory/bjoernrochel-sevro-0.1.0-rc.1.tgz bun run test:eval-runner-sevro-package
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
5. Review the changes, file inventory, retained checks, and compatibility
|
|
52
|
+
limitations before authorizing publication. Keep the reviewed tarball and
|
|
53
|
+
its checksums. Publish that artifact rather than repacking after review.
|
|
54
|
+
6. After publication, verify registry integrity, install the exact version in
|
|
55
|
+
Darrow, and run its frozen installed compatibility gate. Retain the prior
|
|
56
|
+
version's evidence for updates. Roll back through a new dependency/lockfile
|
|
57
|
+
commit and rerun the installed gate; never relabel failed-version evidence.
|
|
58
|
+
|
|
59
|
+
## Manual release workflow
|
|
60
|
+
|
|
61
|
+
`.github/workflows/release.yml` runs only through explicit dispatch against a
|
|
62
|
+
release tag. Its default prepares and checks the artifact. `publish: true`
|
|
63
|
+
additionally publishes that verified artifact using npm trusted publishing.
|
|
64
|
+
No push or pull-request event publishes a package.
|
|
65
|
+
|
|
66
|
+
Before the first publication, an npm owner must establish the package and its
|
|
67
|
+
publishing authorization. The initial reviewed tarball can be published
|
|
68
|
+
manually with an authenticated npm account:
|
|
69
|
+
|
|
70
|
+
```sh
|
|
71
|
+
npm publish /absolute/reviewed/bjoernrochel-sevro-0.1.0-rc.1.tgz --ignore-scripts --access public --tag next
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
After the package exists, configure its trusted publisher for GitHub owner
|
|
75
|
+
`BjRo`, repository `sevro`, and workflow filename `release.yml`. The workflow
|
|
76
|
+
uses a GitHub-hosted runner, Node 24, npm 11.20.0, and job-scoped `id-token: write`.
|
|
77
|
+
The package repository URL must match this public repository. See npm's
|
|
78
|
+
[trusted publishing documentation](https://docs.npmjs.com/trusted-publishers/).
|
|
79
|
+
The workflow checks the downloaded artifact's SHA-256 before publishing and
|
|
80
|
+
retains its release record. Configuring an owner or workflow does not itself
|
|
81
|
+
authorize this coding session to publish or push.
|
|
82
|
+
|
|
83
|
+
The first local candidate is `0.1.0-rc.1`, using the owner-selected
|
|
84
|
+
`BUSL-1.1` license. Its `LICENSE` preserves Darrow's terms and parameters,
|
|
85
|
+
changing only the Licensed Work name and description to Sevro. Neither local
|
|
86
|
+
checks nor a workflow file claim a remote CI run or publication. Darrow's exact
|
|
87
|
+
registry pin, default caller switch, remaining workflow migrations, focused live validation, and
|
|
88
|
+
generic runner removal remain separate extraction gates.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Generic report v1
|
|
2
|
+
|
|
3
|
+
`sevro report` reads one or more retained `sevro.cli-result.v1` files and their
|
|
4
|
+
referenced `sevro.run-evidence.v1` files. It validates both public schemas and
|
|
5
|
+
checks run identity, exit category, case count, and trial references before
|
|
6
|
+
reporting. Paths must be absolute and distinct.
|
|
7
|
+
|
|
8
|
+
```sh
|
|
9
|
+
sevro report --result-file /absolute/path/to/result.json
|
|
10
|
+
sevro report --json --result-file /absolute/path/to/first.json \
|
|
11
|
+
--result-file /absolute/path/to/second.json
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
The default output is Markdown. `--json` emits one `sevro.report.v1` document,
|
|
15
|
+
defined by [`report-v1.schema.json`](../schemas/report-v1.schema.json). Invalid
|
|
16
|
+
inputs return exit 64 with a diagnostic on stderr. Reporting does not change
|
|
17
|
+
the exit category of any input run; each row retains its original code.
|
|
18
|
+
Pre-run failures with no case rows remain visible in the input-run section.
|
|
19
|
+
|
|
20
|
+
Each case row keeps task verdict, execution, and grading separate. Task pass
|
|
21
|
+
rate is measured only when every trial completed execution and grading with an
|
|
22
|
+
assessed task verdict. Candidate duration sums the measured host call time
|
|
23
|
+
across trials. Tokens and cost are totals only when every trial has complete
|
|
24
|
+
usage and that measurement is present. Dry trials, incomplete usage, absent
|
|
25
|
+
provider cost, and historical results without duration show `null` in JSON and
|
|
26
|
+
`unknown` in Markdown. Missing values never become zero. The report retains
|
|
27
|
+
exact candidate, semantic, and advisory routes and lists domain outcomes
|
|
28
|
+
separately from the task verdict.
|
|
29
|
+
|
|
30
|
+
Rows are descriptive, not matched comparisons. Darrow's skill activation,
|
|
31
|
+
ablation matching, and benchmark interpretation remain repository policy and
|
|
32
|
+
require their own report layer over these generic results.
|