@bjoernrochel/sevro 0.1.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. package/LICENSE +159 -0
  2. package/README.md +71 -0
  3. package/docs/advisory-fixture.md +23 -0
  4. package/docs/builtin-graders-v1.md +77 -0
  5. package/docs/cache-ownership-validation.md +49 -0
  6. package/docs/claude-continuation-validation.md +47 -0
  7. package/docs/codex-host.md +134 -0
  8. package/docs/concurrency-validation.md +86 -0
  9. package/docs/configuration-root-validation.md +70 -0
  10. package/docs/development-cli.md +327 -0
  11. package/docs/engine-slice.md +114 -0
  12. package/docs/extension-protocol-v1.md +292 -0
  13. package/docs/fixture-cleanup-validation.md +70 -0
  14. package/docs/identity-v1.md +65 -0
  15. package/docs/native-controls.md +29 -0
  16. package/docs/releases.md +88 -0
  17. package/docs/report-v1.md +32 -0
  18. package/docs/results-v1.md +118 -0
  19. package/examples/basic/graded.json +13 -0
  20. package/examples/basic/host.ts +8 -0
  21. package/examples/basic/prompt-only.json +7 -0
  22. package/package.json +45 -0
  23. package/schemas/cli-result-v1.schema.json +184 -0
  24. package/schemas/extension-v1.schema.json +659 -0
  25. package/schemas/report-v1.schema.json +164 -0
  26. package/schemas/run-evidence-v1.schema.json +510 -0
  27. package/src/advisory-fixture.ts +282 -0
  28. package/src/advisory.ts +105 -0
  29. package/src/cli.ts +837 -0
  30. package/src/engine.ts +2341 -0
  31. package/src/extension-client.ts +257 -0
  32. package/src/extension-session.ts +335 -0
  33. package/src/fixture-bin.ts +87 -0
  34. package/src/fixture-cleanup.ts +38 -0
  35. package/src/fixture-setup.ts +163 -0
  36. package/src/generated-fixture.ts +220 -0
  37. package/src/git-hooks.ts +63 -0
  38. package/src/graders/git-head.ts +176 -0
  39. package/src/graders/output.ts +359 -0
  40. package/src/graders/semantic.ts +120 -0
  41. package/src/graders/shell.ts +277 -0
  42. package/src/hosts/claude-continuation.ts +94 -0
  43. package/src/hosts/claude-credential.ts +73 -0
  44. package/src/hosts/claude-events.ts +128 -0
  45. package/src/hosts/claude-nested-skills.ts +218 -0
  46. package/src/hosts/claude-repository-invocation.ts +239 -0
  47. package/src/hosts/claude-settings.ts +50 -0
  48. package/src/hosts/claude-tool-calls.ts +159 -0
  49. package/src/hosts/claude.ts +511 -0
  50. package/src/hosts/codex-configuration.ts +85 -0
  51. package/src/hosts/codex-events.ts +102 -0
  52. package/src/hosts/codex-native-calls.ts +702 -0
  53. package/src/hosts/codex-profile.ts +78 -0
  54. package/src/hosts/codex-skill-reads.ts +345 -0
  55. package/src/hosts/codex.ts +769 -0
  56. package/src/hosts/isolation-roots.ts +60 -0
  57. package/src/hosts/mac-sandbox.ts +108 -0
  58. package/src/hosts/native-controls.ts +124 -0
  59. package/src/hosts/protected-worktrees.ts +56 -0
  60. package/src/hosts/workspace-fingerprint.ts +55 -0
  61. package/src/identity.ts +156 -0
  62. package/src/instrumentation.ts +106 -0
  63. package/src/preparation.ts +211 -0
  64. package/src/provenance.ts +244 -0
  65. package/src/report.ts +351 -0
  66. package/src/repository-fixture.ts +286 -0
  67. package/src/results.ts +160 -0
  68. package/src/run-owner.ts +276 -0
  69. package/src/schema.ts +28 -0
  70. package/src/storage.ts +23 -0
  71. package/src/trial-scheduler.ts +27 -0
@@ -0,0 +1,292 @@
1
+ # Extension protocol v1
2
+
3
+ This is the target public contract for `sevro.extension.v1`. Extensions are
4
+ explicitly configured development tools. Installing one does not change
5
+ built-in grading or grant it control of fixture isolation, execution,
6
+ persistence, or cleanup. The executable is trusted by the operator; a process
7
+ boundary alone is not a sandbox.
8
+
9
+ [`extension-v1.schema.json`](../schemas/extension-v1.schema.json) validates the
10
+ versioned request and response shapes. The engine also validates request IDs,
11
+ negotiated capabilities, path containment, decoded content, and the source and
12
+ configuration digests at runtime.
13
+
14
+ ## Transport
15
+
16
+ The runner receives an extension command as an argv array and never interprets
17
+ it through a shell. For each method it starts a fresh process, writes exactly
18
+ one UTF-8 JSON request to stdin, closes stdin, and reads exactly one UTF-8 JSON
19
+ response from stdout before process exit. The request and response are each
20
+ limited to 8 MiB. Extra stdout, malformed JSON, an unmatched request ID, a
21
+ nonzero exit, timeout, or an oversized message is a protocol error. Stderr is
22
+ diagnostic only, bounded to 64 KiB in retained evidence; it cannot supply a
23
+ result. The runner cancels the extension process group on interruption.
24
+
25
+ Every request has this envelope. `describe` uses the stable
26
+ `sevro.discovery.v1` bootstrap protocol; subsequent methods use the selected
27
+ extension protocol:
28
+
29
+ ```json
30
+ {
31
+ "protocol": "sevro.discovery.v1",
32
+ "id": "request-1",
33
+ "method": "describe",
34
+ "params": {}
35
+ }
36
+ ```
37
+
38
+ A response echoes `protocol`, `id`, and `method`, then supplies exactly one of
39
+ `result` or `error`:
40
+
41
+ ```json
42
+ {
43
+ "protocol": "sevro.discovery.v1",
44
+ "id": "request-1",
45
+ "method": "describe",
46
+ "result": {
47
+ "extension": { "id": "example.policy", "version": "1.0.0" },
48
+ "protocols": ["sevro.extension.v1"],
49
+ "requiredCapabilities": [],
50
+ "optionalCapabilities": [],
51
+ "graders": [],
52
+ "taskVerdictPolicies": []
53
+ }
54
+ }
55
+ ```
56
+
57
+ An error has `{ "code": "namespaced.code", "message": "bounded text" }`.
58
+ An error response, process failure, or invalid response never becomes a
59
+ successful grade. The engine retains the method, error class, and bounded
60
+ diagnostic without copying secrets or unbounded extension output.
61
+
62
+ ## Negotiation and operations
63
+
64
+ The engine calls `describe` first through the discovery protocol. The
65
+ extension identifies its stable ID and version, supported protocol versions,
66
+ required and optional capability IDs,
67
+ grader IDs it supplies, and task-verdict policy IDs it supplies. Capability IDs
68
+ are namespaced strings. The engine chooses one compatible protocol, verifies
69
+ all required capabilities against its own and the selected host's capability
70
+ registry, and records the negotiated set. An unknown required capability is an
71
+ error. Unknown optional capabilities remain unavailable and cannot be used.
72
+ The engine, not the extension, classifies whether instrumentation changes host
73
+ execution. Future extension protocols still need a compatible discovery
74
+ response or an explicitly selected bridge; an unsupported protocol cannot be
75
+ guessed from a failed extension call.
76
+
77
+ `resolve` receives the selected project root, selectors, and extension
78
+ configuration. The CLI also supplies `host` with the candidate adapter's `id`,
79
+ `model`, `effort`, and available `capabilities`. The engine capability
80
+ `sevro.case.host-route` guarantees this context; extensions that depend on it
81
+ must negotiate that capability. These are selected route facts for preparing
82
+ cases, not observations of completed execution. Semantic and advisory judge
83
+ routes are not candidate context. Older v1 callers may omit `host` when this
84
+ capability is unavailable.
85
+
86
+ It returns neutral case descriptions with stable IDs, prompts,
87
+ an optional `followUpPrompt` for a second turn in the same host session,
88
+ fixture and built-in check declarations, required evidence declarations, and
89
+ namespaced extension data. Paths in case descriptions are resolved and
90
+ validated by the engine against the declared project and case-source roots.
91
+ An extension must negotiate `sevro.host.continuation` before resolving a
92
+ continuation case, and the selected host must declare the same capability.
93
+ The extension cannot add a hidden source root by returning an arbitrary path.
94
+ Fixtures may contain inline files, a declared clean repository source, or a
95
+ bounded generated Git history. Both Git fixture kinds support optional
96
+ working-tree files, staged paths, committed scaffolding, bounded Git hooks,
97
+ and fixture executables. Sevro validates overlay paths, refuses symlinks and
98
+ repository metadata in repository clones, installs hooks and executables
99
+ after its own commits, and includes these declarations in fixture identity. Repository-specific policy and
100
+ custom setup remain in the extension.
101
+
102
+ An extension that negotiates `sevro.fixture.setup` may return an optional
103
+ `fixtureSetup` from `prepare`. It names an absolute executable and bounded argv,
104
+ plus optional environment variables. Sevro runs that trusted command once per
105
+ trial in the Git fixture after source history and working-tree files are built,
106
+ before extension artifacts are mounted or the host starts. The command is never
107
+ interpreted through a shell by Sevro; an extension may explicitly select a
108
+ shell executable. `{{sevro.project}}` and `{{sevro.workspace}}` in environment
109
+ values resolve to the declared roots at execution time. The symbolic values and
110
+ command enter fixture and comparison identity, while retained configuration
111
+ records only their digest. Setup has a two-minute limit, inherits no credential
112
+ environment variables, and its output is discarded. Failure or interruption
113
+ stops the run before host execution; it cannot become a passing assessment.
114
+ Absolute path references in this protocol use `file:///` URLs so Windows and
115
+ Unix hosts share one serialized form. Fixture-relative paths use `/` separators
116
+ and cannot escape their declared root.
117
+
118
+ `prepare` receives one resolved case, the chosen host and capabilities, the
119
+ configured passive or enforced condition, and extension data. It returns
120
+ fixture preparation artifacts and requested instrumentation IDs with
121
+ parameters. The engine validates these requests before host execution and
122
+ passes them to the host. It rejects unknown or unsupported IDs and rejects any
123
+ execution-changing instrumentation in a passive trial. Requested and applied
124
+ instrumentation are recorded separately. Host adapters declare supported IDs
125
+ and whether each changes execution. A requested ID must be among the
126
+ negotiated capabilities. The host receives the validated request and must
127
+ report the same applied IDs and parameters; a mismatch fails execution. An
128
+ execution-changing request also requires the host to confirm an enforced
129
+ condition. Instrumentation parameters enter retained evidence and must not
130
+ contain secrets.
131
+
132
+ A Codex extension may also negotiate `sevro.codex.plugin-marketplace` and
133
+ return `codexMarketplace` from `prepare` with `artifactRoot`,
134
+ `marketplaceName`, and nonempty `pluginNames`. Every package file must be a
135
+ Git-excluded preparation artifact below that fixture-relative root, including
136
+ `.claude-plugin/marketplace.json`. Sevro checks the declared names, package
137
+ paths, and local manifest sources before invoking the Codex plugin CLI. The
138
+ declaration and artifact digests enter evaluation identity. The Codex host
139
+ installs the marketplace and named plugins in its private home for each trial;
140
+ no package is installed from an undeclared artifact. This fixture setup is
141
+ available in passive trials because it does not add enforcement
142
+ instrumentation.
143
+ An extension that also negotiates `sevro.codex.explicit-invocation` may declare
144
+ `codexSkillInvocation` with a plugin and skill in that package. The resolved
145
+ prompts must contain `{{sevro.codex.skill_invocation}}` exactly once across
146
+ both turns. Sevro
147
+ renders the installed `$plugin:skill` token separately for each trial and the
148
+ Codex host retains a bounded dispatch receipt only after a completed turn.
149
+ Missing or repeated tokens, absent packaged skills, and unsupported hosts fail
150
+ before candidate execution. Supporting skill reads remain separate evidence.
151
+
152
+ For a repository skill, negotiate `sevro.codex.repository-invocation` and
153
+ declare `codexRepositorySkillInvocation` with `skillName`. It is mutually
154
+ exclusive with the plugin invocation declarations. Prepare the exact
155
+ `.agents/skills/<skillName>/SKILL.md` as a Git-excluded artifact, and place
156
+ `{{sevro.skill_invocation}}` exactly once across the prompts. Sevro renders
157
+ `$skillName`, preserves repository scope in the host request, and records the
158
+ declaration separately in configuration and evaluation identity. Missing mounts,
159
+ repeated tokens, or absent negotiation fail before candidate execution. The
160
+ Codex host verifies the repository mount and emits its normal bounded explicit
161
+ dispatch receipt after a completed turn. No plugin package is required.
162
+
163
+ For a Claude repository skill, negotiate `sevro.claude.repository-invocation`
164
+ and declare `claudeRepositorySkillInvocation` with `skillName`. Mount the exact
165
+ `.claude/skills/<skillName>/SKILL.md` as a Git-excluded preparation artifact.
166
+ The bundled host advertises this capability only with
167
+ `--claude-project-settings`; otherwise it rejects dispatch before execution.
168
+ The neutral invocation placeholder must lead the initial prompt and occur once;
169
+ Sevro renders `/skillName`. This declaration is mutually exclusive with all
170
+ other invocation declarations and enters retained configuration and identity.
171
+ The host validates the bounded mount without following symbolic links. Its
172
+ `sevro.claude.repository-invocation` observation binds one native command, exact
173
+ arguments, and the complete mounted body before the first assistant entry in
174
+ the completed session. It retains only acceptance and ordered skill metadata.
175
+ Missing or ambiguous session evidence yields a partial observation; later Skill
176
+ calls cannot repair an unverified explicit repository dispatch.
177
+
178
+ A Claude extension may negotiate `sevro.claude.plugin-dirs` and return
179
+ `claudePluginDirs` with nonempty fixture-relative `artifactRoots`. Roots must
180
+ be unique and must not overlap. Every file below each root must be a
181
+ Git-excluded preparation artifact, and each root must contain
182
+ `.claude-plugin/plugin.json`. Sevro passes only the declared roots and their
183
+ verified artifact paths to a capable host. The declaration and artifact
184
+ digests enter evaluation identity. The host remains responsible for loading
185
+ the plugin directories and isolating its credentials from candidate tools.
186
+ An extension that also negotiates `sevro.claude.explicit-invocation` may
187
+ declare `claudeSkillInvocation` for a skill present in one of those packages.
188
+ The resolved prompts must contain `{{sevro.skill_invocation}}` exactly once
189
+ across both turns. Sevro renders the `/plugin:skill` token for each trial and
190
+ passes the same token to the Claude host. The neutral placeholder also works
191
+ with `codexSkillInvocation`; the older
192
+ `{{sevro.codex.skill_invocation}}` placeholder remains supported for Codex.
193
+
194
+ An artifact for a generated or cloned Git fixture may set `gitExclude: true`.
195
+ Sevro mounts its verified bytes, adds only that exact artifact path to the
196
+ fixture's `.git/info/exclude`, and retains the flag in artifact evidence and
197
+ fixture identity. The flag is invalid for a non-Git fixture. It lets evaluator
198
+ assets such as project skills remain visible to the host without appearing as
199
+ candidate changes in Git status checks.
200
+ Artifact mounts refuse existing files and any parent that is a symlink or
201
+ non-directory, including paths created by fixture setup or a cloned repository.
202
+ An artifact may set `executable: true` to install its verified bytes with owner
203
+ execute permission. The flag enters fixture identity and retained artifact
204
+ references; absent or false leaves the file readable without execute permission.
205
+
206
+ `evaluate` receives one completed or failed trial's bounded host observations,
207
+ their source and completeness, built-in check results, artifact references,
208
+ and the extension's namespaced case data. It returns namespaced checks and
209
+ metrics. A check is `passed`, `failed`, or `unavailable`; it names the evidence
210
+ used. A missing required observation remains unavailable. The extension may
211
+ return a task-verdict recommendation only when an explicitly configured
212
+ task-verdict policy names that extension policy. In that case, `evaluate`
213
+ receives `selectedTaskVerdictPolicy` and must return a recommendation. The
214
+ engine uses it only for a completed trial with completed grading and available
215
+ required evidence. It retains the selected policy and each trial's
216
+ recommendation. An omitted recommendation is a grading error. A policy may
217
+ change a failed check's task verdict, but cannot conceal execution failure,
218
+ grader error, or unavailable required evidence.
219
+ Host adapters can supply additional namespaced observations. The engine bounds
220
+ and retains those records, then passes them to `evaluate`. Required observation
221
+ IDs must be present with `complete` evidence before a task can pass.
222
+ Host adapters can also supply bounded evidence artifacts. The engine persists
223
+ their bytes outside the candidate workspace before `evaluate`, then passes
224
+ file URLs and digests through the existing artifact references. A required host
225
+ artifact must be present before a task can pass.
226
+
227
+ The methods are stateless. Every request carries the configuration and prior
228
+ extension data it needs. The runner verifies extension identity and source
229
+ digest for the run; a changed executable or configuration cannot silently
230
+ continue the same run.
231
+
232
+ The session client in [`extension-session.ts`](../src/extension-session.ts)
233
+ implements discovery, `resolve`, `prepare`, and `evaluate` as separate process
234
+ calls. Its caller declares the extension source-file closure; the client also
235
+ includes the executable and hashes that closure before discovery and before
236
+ and after later calls. The redacted configuration and command argv affect the
237
+ configuration digest. Command argv must not contain credentials. A passing
238
+ extension check must cite available, complete evidence. The engine now accepts
239
+ inline and declared clean-repository cases with additive extension checks, runs `prepare` before the host,
240
+ and persists check outcomes, evidence references, and metrics before fixture
241
+ cleanup. It applies and retains bounded inline preparation artifacts after
242
+ checking their path and digest. Source references require a caller-declared
243
+ ID-to-file-URL map under a case source root and receive the same checks.
244
+ The bundled Codex host currently advertises no instrumentation capabilities.
245
+
246
+ ## Grading and replacement
247
+
248
+ The runner provides built-in shell, regex, JSON/schema, combined output,
249
+ semantic-output, and
250
+ advisory quality graders. A run can use those without any extension. An
251
+ extension's graders add checks and metrics by default. The run configuration
252
+ must explicitly name each built-in grader it replaces, or explicitly select
253
+ one advertised task-verdict policy. Unknown or duplicate replacement IDs are
254
+ errors. Retained evidence lists every active grader, its identity and route,
255
+ and any replaced default.
256
+
257
+ An extension may also return namespaced `domainOutcomes` with passed, failed,
258
+ or unavailable status, evidence references, and bounded domain data. Sevro
259
+ validates their evidence and retains them separately from task checks. They do
260
+ not change the task verdict. Older evidence without this field leaves domain
261
+ outcomes unknown.
262
+
263
+ When a built-in grader is selected for replacement, the engine validates its
264
+ declared check configuration but does not run its checks. It requires at least
265
+ one declared, advertised extension check, and grades those checks instead.
266
+ Unselected built-in checks stay active. The replacement list enters extension
267
+ configuration and evaluation identity; evidence retains the selected defaults
268
+ and omits them from active graders and trial check results. A replaced shell or
269
+ semantic grader does not require its usual isolation or host route because it
270
+ does not execute.
271
+
272
+ An extension cannot replace execution status, grading status, or the evidence
273
+ availability rules. It cannot turn an error or missing required evidence into
274
+ a successful task assessment. A prompt without success criteria produces
275
+ execution evidence and a task verdict of `not_assessed`.
276
+
277
+ ## Lifecycle
278
+
279
+ The engine orders each trial as follows:
280
+
281
+ 1. Negotiate protocol and capabilities; resolve the case.
282
+ 2. Prepare the fixture and extension inputs; validate instrumentation against
283
+ the host and passive/enforced condition.
284
+ 3. Execute or resume the host and collect bounded observations.
285
+ 4. Run built-in checks, then extension grading, then compute separate
286
+ execution, grading, and task states.
287
+ 5. Persist complete trial evidence before fixture cleanup.
288
+
289
+ On cancellation or failure, completed trials remain persisted and partial
290
+ attempts retain an explicit incomplete or diagnostic state. The engine owns
291
+ process-group cancellation and cleanup even when an extension fails. A dry
292
+ run records preparation but no observed behavioral success.
@@ -0,0 +1,70 @@
1
+ # Permission-locked fixture cleanup
2
+
3
+ The Darrow extraction comparison exposed a cleanup difference: its legacy
4
+ fixture destroyer removes permission-locked directories, while Sevro retained
5
+ them after a completed trial. Sevro now restores owner access inside completed
6
+ candidate workspaces when needed. It clears their contents after trial evidence
7
+ is retained, keeps empty reservations until active peers drain, and then removes
8
+ the completed roots. Directory traversal uses link metadata and skips symbolic
9
+ links; external targets retain their permissions and contents.
10
+
11
+ ## Public CLI regression slices
12
+
13
+ The public seam is `sevro run --case-file ... --adapter-module ...`, as documented
14
+ in [Development CLI](development-cli.md). Each controlled adapter uses real
15
+ filesystem permissions and returns its workspace path as a bounded observation.
16
+ The test reads retained public evidence and verifies that the workspace is gone.
17
+ A preliminary internal-engine test was diagnostic only; it is not part of this
18
+ TDD trace and was removed before implementation.
19
+
20
+ Red — `bun test tests/cli.test.ts --test-name-pattern 'CLI removes a permission-locked candidate directory'`: completed evidence existed, but the mode-000 child directory kept the workspace present.
21
+
22
+ Green — `bun test tests/cli.test.ts --test-name-pattern 'CLI removes a permission-locked candidate directory'`: the workspace was removed and retained evidence still agreed with the CLI result.
23
+
24
+ Red — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(root\)'`: a mode-000 workspace root remained present after completion.
25
+
26
+ Green — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(root\)'`: the workspace root was removed after evidence retention.
27
+
28
+ Red — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(readable root\)'`: the readable mode-500 root lacked write access and remained present.
29
+
30
+ Green — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(readable root\)'`: restoring owner write access allowed cleanup; the external link target's mode and bytes were unchanged.
31
+
32
+ Each final example also links to an external directory with a read-only file.
33
+ All three check that its directory mode, file mode, and file contents stay intact.
34
+ These are deterministic filesystem and lifecycle checks, with no live model call.
35
+
36
+ ## Final source gates
37
+
38
+ `bun test` passed all 243 tests and 1399 assertions across 38 files in 93.83
39
+ seconds. The gate includes parallel admission, peer isolation, cancellation,
40
+ failed persistence, extension lifecycle, and generated/repository fixture tests.
41
+ `bun run typecheck` and `bun run format:check` pass.
42
+ The full log and subsequent installed-archive checks are retained under
43
+ `/Users/bjro/.darrow/issue95-fixture-cleanup/`. The tests ran on macOS with Bun
44
+ 1.3.13; they do not establish another platform or native model route.
45
+
46
+ ## Canonical workspace follow-up
47
+
48
+ Darrow's private fixture builder also checked that its returned path was
49
+ canonical. An initial diagnostic assertion in the cleanup examples exposed
50
+ Sevro's logical temporary-directory alias; those diagnostic additions were
51
+ removed before the isolated regression. The public canonical-workspace test
52
+ records the operating system's `realpath` through a controlled host and compares
53
+ it with the received workspace in retained evidence. The full source gate above
54
+ predates this follow-up.
55
+
56
+ The public CLI contract was clarified before implementation. Candidate and
57
+ semantic workspace allocations now resolve the new directory through `realpath`
58
+ before reservation, preparation, or host execution.
59
+
60
+ Red — `bun test tests/cli.test.ts --test-name-pattern 'CLI supplies a canonical candidate workspace'`:
61
+ exit `1`; the received `/var/...` path differed from the operating system's
62
+ `/private/var/...` path. One test failed with four assertions in 222 milliseconds.
63
+
64
+ Green — `bun test tests/cli.test.ts --test-name-pattern 'CLI supplies a canonical candidate workspace'`:
65
+ exit `0`; one test and four assertions passed in 228 milliseconds after the fix.
66
+
67
+ The final `bun test` gate passed 244 tests and 1403 assertions across 38 files in
68
+ 94.77 seconds. Typecheck and formatting pass. The source log is retained at
69
+ `/Users/bjro/.darrow/issue95-fixture-mounting/sevro-full-gate.log`. These remain
70
+ deterministic macOS/Bun checks, without a live model or additional platform claim.
@@ -0,0 +1,65 @@
1
+ # Evaluation identity v1
2
+
3
+ `sevro.identity.v1` binds retained evidence to the inputs that can change an
4
+ assessment. Its `dimensions` object has exactly the fields required by
5
+ [`run-evidence-v1.schema.json`](../schemas/run-evidence-v1.schema.json). The
6
+ runner computes each `*Digest` from the relevant normalized content. It must
7
+ not use a temporary path, run ID, or timestamp as an input.
8
+
9
+ | Dimension | Content represented |
10
+ | ------------------------------------------ | ---------------------------------------------------------------------------------------------- |
11
+ | `runnerBuildDigest` | Exact installed package build or local runner source build. |
12
+ | `projectDigest` | Evaluated project revision and dirty content, or a content snapshot without Git. |
13
+ | `configurationDigest` | Evaluation settings with credentials replaced by secret-presence markers. |
14
+ | `extensionDigest`, `extensionProtocol` | Extension source closure and negotiated protocol, or both `null`. |
15
+ | `caseDigest`, `fixtureDigest` | Resolved case declaration and fixture contents. |
16
+ | `checksDigest`, `requiredEvidenceDigest` | Declared checks and evidence requirements. |
17
+ | `evaluatorDigest`, `graderDigest` | Evaluator policy, grader versions, active selection, and replacements. |
18
+ | `instrumentationDigest` | Requested instrumentation and expected application; trial evidence records actual application. |
19
+ | `routeDigest` | Candidate, semantic, and advisory host/model/effort routes. |
20
+ | `condition`, `trialCount`, `passThreshold` | Declared condition and assessment sampling rule. |
21
+
22
+ Each structured component digest is SHA-256 of its [RFC 8785 JSON
23
+ Canonicalization Scheme](https://www.rfc-editor.org/rfc/rfc8785.html) UTF-8
24
+ bytes. The runner rejects values outside canonical JSON, including non-finite
25
+ numbers and unpaired Unicode surrogates. The final evaluation digest is:
26
+
27
+ ```text
28
+ SHA-256(UTF-8("sevro.identity.v1\n") || UTF-8(JCS(dimensions)))
29
+ ```
30
+
31
+ The CLI derives `runnerBuildDigest` from sorted packaged file paths and their
32
+ SHA-256 bytes under `src/`, `schemas/`, `docs/`, and `examples/`, plus `README.md` and
33
+ `package.json`. For a Git project, `projectDigest` binds the revision and dirty
34
+ patch digest without the checkout path. Without a revision it hashes a bounded
35
+ file snapshot, excluding `.git`, results, and run-state storage. A caller may
36
+ provide explicit 64-character digests for controlled development runs.
37
+
38
+ The prefix separates this digest from component digests and future identity
39
+ versions. An exact-repeat comparison requires equal identities. An ablation may
40
+ deliberately differ on `condition` or instrumentation; its report must name
41
+ those dimensions and verify that the intended controls match. Historical
42
+ artifacts without a dimension cannot be silently assigned a current identity.
43
+
44
+ Credential values, raw transcripts, and temporary output locations never enter
45
+ the stored dimensions. A credential's presence can be recorded in a redacted
46
+ configuration; its bytes are never retained in evidence or its digest input.
47
+ The effective positive trial job limit is retained as `configuration.redacted.jobs`
48
+ and enters `configurationDigest`. Changing parallelism changes evaluation identity,
49
+ even when the condition, host route, trial count, and assessment threshold match.
50
+
51
+ Extension runs retain the operator-supplied redacted extension configuration
52
+ under `configuration.redacted.extensionConfiguration`, alongside
53
+ `extensionConfigurationDigest`. The retained copy is the same snapshot used
54
+ for extension configuration identity. Private configuration stays outside run
55
+ evidence. Consumers can compare the visible configuration with their declared
56
+ inputs without reconstructing the engine's component digest inputs.
57
+
58
+ Hosts may supply explicit, redacted `configuration` values. Sevro retains them
59
+ under `configuration.redacted.hostConfiguration`, keyed by the used candidate,
60
+ semantic, or advisory role, and includes them in `configurationDigest`. Custom
61
+ adapter authors must keep secrets out of this object. Bundled Codex roles record
62
+ `sevro.codex.agent-concurrency-limit` as the captured positive integer or `null`
63
+ for the host default. Configuration paths and unrelated TOML bytes are not
64
+ imported settings; a relocated configuration with the same effective values
65
+ keeps this component unchanged.
@@ -0,0 +1,29 @@
1
+ # Native control observation v1
2
+
3
+ The bundled Codex and Claude hosts advertise `sevro.host.native-controls` and
4
+ retain an observation with that ID. It exposes host facts for extension policy:
5
+
6
+ - `method: native_control_calls`;
7
+ - ordered `calls` containing only `ordinal`, `namespace`, and `name`;
8
+ - `acceptedAgentCount`, a verified Codex acceptance count or `null` on Claude;
9
+ - `submittedExecCalls`, the Codex submitted-code count or `null` on Claude; and
10
+ - `truncated`, which is `false` for a complete record and may be unknown (`null`)
11
+ for an incomplete Codex record.
12
+
13
+ The observation's source is the executing host, not the extension. It keeps no
14
+ tool arguments, task messages, command bodies, or results. Call labels prove an
15
+ invocation attempt, not success. Extensions decide which calls are prohibited.
16
+ An unknown acceptance count cannot establish agent acceptance.
17
+
18
+ Codex projects labels and counts from its bound native-session observation and
19
+ preserves its completeness. Incomplete counts remain `null`. Submitted code remains explicit: extensions cannot
20
+ infer absence of controls inside code that this observer did not inspect.
21
+ Claude records every tool-use label from the bounded stream, with namespace
22
+ `claude`, without filtering to Skill or Agent. It requires a completed successful
23
+ turn, valid unique tool-use IDs and names, and a non-truncated list of at most
24
+ 128 calls. Malformed, duplicate, failed, or excessive streams remain partial.
25
+ Tool-looking text in arguments or output does not become a native call.
26
+
27
+ Consumers must verify the source, completeness, shape, ordering, and counts
28
+ before using an empty call list to establish absence. Missing or partial
29
+ evidence cannot establish a negative assertion.
@@ -0,0 +1,88 @@
1
+ # Releases
2
+
3
+ Sevro releases are public npm packages named `@bjoernrochel/sevro`. The public
4
+ command remains `sevro`, and npm archives use `bjoernrochel-sevro-<version>.tgz`.
5
+ A release version is an
6
+ exact semantic version; development versions containing `-dev` are not release
7
+ candidates. Darrow pins the published version in development dependencies and
8
+ its lockfile. Marketplace plugins do not depend on Sevro.
9
+
10
+ ## Candidate contract
11
+
12
+ Prepare one candidate from the intended release tag and package metadata. The
13
+ tag must equal `v<package version>`. Package metadata must identify the public
14
+ repository, registry, distribution tag, and an owner-selected license with its
15
+ nonempty `LICENSE` file. Prereleases cannot use npm's `latest` distribution tag.
16
+
17
+ Preparation creates a new absolute output directory and never replaces an
18
+ existing directory. It packs with lifecycle scripts disabled, retains the npm
19
+ file inventory, and writes `release.json` and `SHA256SUMS` beside the tarball.
20
+ The record identifies the package, version, release tag, distribution tag,
21
+ tarball SHA-256 and npm SHA-512 integrity, and the preparation runtimes.
22
+ Preparation does not publish a package or create a Git tag.
23
+
24
+ The package installation check accepts `--tarball <absolute file>` to verify
25
+ that exact candidate without repacking. The supplied tarball remains intact.
26
+ Its installed name and version must match the source metadata. Installation,
27
+ CLI execution, generic grading, prompt-only execution, release provenance,
28
+ reporting, and supported deterministic native fixtures run without source Git
29
+ metadata in the installed package.
30
+
31
+ ## Maintainer workflow
32
+
33
+ 1. Select the license, update `package.json` to the intended release version,
34
+ and commit those changes. Preserve the existing public schemas and document
35
+ behavior changes. For the first candidate, use `0.1.0-rc.1` with npm tag
36
+ `next`; a prerelease does not establish full Darrow migration completion.
37
+ 2. Run frozen installation, typecheck, formatting, and the full test suite.
38
+ 3. Prepare and test the exact candidate:
39
+
40
+ ```sh
41
+ bun run release:prepare --tag v0.1.0-rc.1 --output /absolute/new/release-directory
42
+ bun run test:package-install --tarball /absolute/new/release-directory/bjoernrochel-sevro-0.1.0-rc.1.tgz
43
+ ```
44
+
45
+ 4. Run Darrow's public integration against that same tarball:
46
+
47
+ ```sh
48
+ SEVRO_PACKAGE_TARBALL=/absolute/new/release-directory/bjoernrochel-sevro-0.1.0-rc.1.tgz bun run test:eval-runner-sevro-package
49
+ ```
50
+
51
+ 5. Review the changes, file inventory, retained checks, and compatibility
52
+ limitations before authorizing publication. Keep the reviewed tarball and
53
+ its checksums. Publish that artifact rather than repacking after review.
54
+ 6. After publication, verify registry integrity, install the exact version in
55
+ Darrow, and run its frozen installed compatibility gate. Retain the prior
56
+ version's evidence for updates. Roll back through a new dependency/lockfile
57
+ commit and rerun the installed gate; never relabel failed-version evidence.
58
+
59
+ ## Manual release workflow
60
+
61
+ `.github/workflows/release.yml` runs only through explicit dispatch against a
62
+ release tag. Its default prepares and checks the artifact. `publish: true`
63
+ additionally publishes that verified artifact using npm trusted publishing.
64
+ No push or pull-request event publishes a package.
65
+
66
+ Before the first publication, an npm owner must establish the package and its
67
+ publishing authorization. The initial reviewed tarball can be published
68
+ manually with an authenticated npm account:
69
+
70
+ ```sh
71
+ npm publish /absolute/reviewed/bjoernrochel-sevro-0.1.0-rc.1.tgz --ignore-scripts --access public --tag next
72
+ ```
73
+
74
+ After the package exists, configure its trusted publisher for GitHub owner
75
+ `BjRo`, repository `sevro`, and workflow filename `release.yml`. The workflow
76
+ uses a GitHub-hosted runner, Node 24, npm 11.20.0, and job-scoped `id-token: write`.
77
+ The package repository URL must match this public repository. See npm's
78
+ [trusted publishing documentation](https://docs.npmjs.com/trusted-publishers/).
79
+ The workflow checks the downloaded artifact's SHA-256 before publishing and
80
+ retains its release record. Configuring an owner or workflow does not itself
81
+ authorize this coding session to publish or push.
82
+
83
+ The first local candidate is `0.1.0-rc.1`, using the owner-selected
84
+ `BUSL-1.1` license. Its `LICENSE` preserves Darrow's terms and parameters,
85
+ changing only the Licensed Work name and description to Sevro. Neither local
86
+ checks nor a workflow file claim a remote CI run or publication. Darrow's exact
87
+ registry pin, default caller switch, remaining workflow migrations, focused live validation, and
88
+ generic runner removal remain separate extraction gates.
@@ -0,0 +1,32 @@
1
+ # Generic report v1
2
+
3
+ `sevro report` reads one or more retained `sevro.cli-result.v1` files and their
4
+ referenced `sevro.run-evidence.v1` files. It validates both public schemas and
5
+ checks run identity, exit category, case count, and trial references before
6
+ reporting. Paths must be absolute and distinct.
7
+
8
+ ```sh
9
+ sevro report --result-file /absolute/path/to/result.json
10
+ sevro report --json --result-file /absolute/path/to/first.json \
11
+ --result-file /absolute/path/to/second.json
12
+ ```
13
+
14
+ The default output is Markdown. `--json` emits one `sevro.report.v1` document,
15
+ defined by [`report-v1.schema.json`](../schemas/report-v1.schema.json). Invalid
16
+ inputs return exit 64 with a diagnostic on stderr. Reporting does not change
17
+ the exit category of any input run; each row retains its original code.
18
+ Pre-run failures with no case rows remain visible in the input-run section.
19
+
20
+ Each case row keeps task verdict, execution, and grading separate. Task pass
21
+ rate is measured only when every trial completed execution and grading with an
22
+ assessed task verdict. Candidate duration sums the measured host call time
23
+ across trials. Tokens and cost are totals only when every trial has complete
24
+ usage and that measurement is present. Dry trials, incomplete usage, absent
25
+ provider cost, and historical results without duration show `null` in JSON and
26
+ `unknown` in Markdown. Missing values never become zero. The report retains
27
+ exact candidate, semantic, and advisory routes and lists domain outcomes
28
+ separately from the task verdict.
29
+
30
+ Rows are descriptive, not matched comparisons. Darrow's skill activation,
31
+ ablation matching, and benchmark interpretation remain repository policy and
32
+ require their own report layer over these generic results.