@bjoernrochel/sevro 0.1.0-rc.7 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +22 -6
- package/README.md +5 -1
- package/docs/README.md +19 -15
- package/docs/architecture.md +138 -17
- package/docs/assets/README.md +5 -4
- package/docs/builtin-graders-v1.md +17 -3
- package/docs/codex-host.md +155 -49
- package/docs/contracts.md +2 -1
- package/docs/creating-cases.md +9 -2
- package/docs/development-cli.md +187 -19
- package/docs/documentation-quality.md +12 -2
- package/docs/evaluation-workflow.md +110 -0
- package/docs/extension-protocol-v1.md +26 -3
- package/docs/getting-started.md +23 -2
- package/docs/guide-evaluation.md +33 -1
- package/docs/identity-v1.md +25 -2
- package/docs/installing.md +11 -11
- package/docs/native-controls.md +76 -5
- package/docs/native-hosts.md +196 -7
- package/docs/reading-results.md +102 -0
- package/docs/recipes.md +489 -0
- package/docs/releases.md +31 -12
- package/docs/report-v1.md +29 -1
- package/docs/results-v1.md +141 -4
- package/docs/run-settings-v1.md +147 -0
- package/docs/runtime-v1.md +307 -18
- package/docs/troubleshooting.md +12 -12
- package/docs/typescript-quality.md +89 -26
- package/examples/recipes/candidate.ts +26 -0
- package/examples/recipes/cases/coding-change.json +21 -0
- package/examples/recipes/cases/host-tools.json +21 -0
- package/examples/recipes/cases/mise.json +21 -0
- package/examples/recipes/cases/node.json +21 -0
- package/examples/recipes/cases/python-uv.json +21 -0
- package/examples/recipes/cases/semantic.json +17 -0
- package/examples/recipes/coding-change/check.ts +23 -0
- package/examples/recipes/extension-artifacts.ts +65 -0
- package/examples/recipes/extension-methods.ts +174 -0
- package/examples/recipes/extension.ts +37 -0
- package/examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md +7 -0
- package/examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md +7 -0
- package/examples/recipes/repository-skill/README.md +3 -0
- package/examples/recipes/run-inputs.ts +188 -0
- package/examples/recipes/run-options.ts +131 -0
- package/examples/recipes/run.ts +63 -0
- package/examples/recipes/runtime/empty.json +1 -0
- package/examples/recipes/runtime/goals.json +1 -0
- package/examples/recipes/runtime/mise.json +6 -0
- package/examples/recipes/runtime/path.json +1 -0
- package/examples/recipes/runtime/plugins.json +4 -0
- package/examples/recipes/runtime/python-uv.json +10 -0
- package/examples/recipes/semantic-judge.ts +29 -0
- package/examples/templates/coding/adapter.ts +17 -0
- package/examples/templates/coding/case.json +15 -0
- package/examples/templates/repository-skill/.agents/skills/startup-guide/SKILL.md +6 -0
- package/examples/templates/repository-skill/extension-data.ts +64 -0
- package/examples/templates/repository-skill/extension.ts +31 -0
- package/examples/templates/response/adapter.ts +12 -0
- package/examples/templates/response/case.json +13 -0
- package/package.json +11 -5
- package/schemas/report-v1.schema.json +9 -0
- package/schemas/run-evidence-v1.schema.json +2 -1
- package/schemas/runtime-v1.schema.json +63 -7
- package/src/advisory.ts +2 -2
- package/src/candidate-artifacts.ts +206 -0
- package/src/candidate-codex-state.ts +185 -0
- package/src/case-validation.ts +73 -50
- package/src/cli/cancellation.ts +20 -0
- package/src/cli/command.ts +87 -0
- package/src/cli/completion.ts +15 -0
- package/src/cli/demo.ts +60 -0
- package/src/cli/evaluation.ts +55 -0
- package/src/cli/extension.ts +79 -0
- package/src/cli/help.ts +198 -0
- package/src/cli/hosts.ts +55 -0
- package/src/cli/init-settings.ts +30 -0
- package/src/cli/init.ts +111 -0
- package/src/{cli-invocation.ts → cli/invocation.ts} +214 -218
- package/src/cli/loading.ts +151 -0
- package/src/cli/preparation.ts +154 -0
- package/src/cli/presentation.ts +93 -0
- package/src/cli/progress.ts +6 -0
- package/src/cli/run-manifest.ts +128 -0
- package/src/cli/run-options.ts +403 -0
- package/src/cli/run-overrides.ts +69 -0
- package/src/cli/settings.ts +154 -0
- package/src/cli/summary.ts +43 -0
- package/src/cli.ts +13 -591
- package/src/contract-types.ts +94 -0
- package/src/engine.ts +13 -5
- package/src/evaluation/advisory.ts +150 -0
- package/src/{evaluation-check-plan.ts → evaluation/check-plan.ts} +103 -67
- package/src/evaluation/collection.ts +88 -0
- package/src/evaluation/context.ts +10 -0
- package/src/{evaluation-grading-host.ts → evaluation/grading-host.ts} +5 -5
- package/src/evaluation/grading.ts +235 -0
- package/src/{evaluation-invocation-plan.ts → evaluation/invocation-plan.ts} +147 -58
- package/src/{evaluation-plan.ts → evaluation/plan.ts} +208 -141
- package/src/evaluation/progress.ts +19 -0
- package/src/evaluation/resources.ts +241 -0
- package/src/evaluation/run.ts +206 -0
- package/src/{evaluation-semantic.ts → evaluation/semantic.ts} +133 -115
- package/src/evaluation/trial.ts +800 -0
- package/src/evaluation/types.ts +207 -0
- package/src/evidence/host-results.ts +85 -0
- package/src/evidence/retained-artifacts.ts +18 -0
- package/src/evidence/trial-writer.ts +203 -0
- package/src/evidence/usage.ts +17 -0
- package/src/extension-client.ts +34 -96
- package/src/extension-session.ts +24 -48
- package/src/{advisory-fixture.ts → fixtures/advisory.ts} +15 -76
- package/src/{fixture-bin.ts → fixtures/bin.ts} +1 -1
- package/src/{fixture-cleanup.ts → fixtures/cleanup.ts} +13 -7
- package/src/fixtures/create.ts +183 -0
- package/src/{generated-fixture.ts → fixtures/generated.ts} +10 -53
- package/src/{git-hooks.ts → fixtures/git-hooks.ts} +1 -1
- package/src/{preparation.ts → fixtures/preparation.ts} +6 -16
- package/src/{repository-fixture.ts → fixtures/repository.ts} +30 -64
- package/src/{fixture-setup.ts → fixtures/setup.ts} +33 -51
- package/src/{fixture-tools.ts → fixtures/tools.ts} +1 -1
- package/src/fixtures/workspace-state.ts +212 -0
- package/src/generated/cli-result.d.cts +2 -1
- package/src/generated/extension.d.cts +2 -1
- package/src/generated/report.cjs +1 -1
- package/src/generated/report.d.cts +2 -1
- package/src/generated/run-evidence.cjs +1 -1
- package/src/generated/run-evidence.d.cts +2 -1
- package/src/generated/runtime.cjs +1 -1
- package/src/generated/runtime.d.cts +2 -1
- package/src/git.ts +93 -0
- package/src/graders/git-head.ts +29 -84
- package/src/graders/shell.ts +57 -68
- package/src/host-diagnostic-redaction.ts +52 -0
- package/src/host-diagnostics.ts +331 -0
- package/src/hosts/claude-continuation.ts +57 -40
- package/src/hosts/claude-credential.ts +160 -63
- package/src/hosts/claude-events.ts +34 -18
- package/src/hosts/claude-keychain.ts +78 -21
- package/src/hosts/claude-login-recovery.ts +85 -0
- package/src/hosts/claude-login-refresh.ts +178 -0
- package/src/hosts/claude-login-store.ts +162 -0
- package/src/hosts/claude-login.ts +107 -0
- package/src/hosts/claude-native-goal.ts +15 -12
- package/src/hosts/claude-process.ts +10 -50
- package/src/hosts/claude-refresh-lock.ts +81 -0
- package/src/hosts/claude-repository-invocation.ts +9 -34
- package/src/hosts/claude-settings.ts +91 -14
- package/src/hosts/claude-skill-activations.ts +169 -0
- package/src/hosts/claude-state.ts +46 -16
- package/src/hosts/claude-tool-calls.ts +13 -10
- package/src/hosts/claude.ts +122 -73
- package/src/hosts/codex-app-server.ts +78 -52
- package/src/hosts/codex-configuration.ts +5 -10
- package/src/hosts/codex-exec-controls.ts +516 -0
- package/src/hosts/codex-marketplace.ts +12 -10
- package/src/hosts/codex-native-calls.ts +15 -122
- package/src/hosts/codex-process.ts +10 -54
- package/src/hosts/codex-profile.ts +50 -21
- package/src/hosts/codex-session-files.ts +1 -9
- package/src/hosts/codex-state.ts +141 -19
- package/src/hosts/codex.ts +221 -179
- package/src/hosts/isolation-common.ts +15 -11
- package/src/hosts/isolation-roots.ts +13 -20
- package/src/hosts/linux-sandbox.ts +27 -3
- package/src/hosts/mac-sandbox.ts +111 -18
- package/src/hosts/native-controls.ts +72 -22
- package/src/hosts/native-request.ts +55 -0
- package/src/hosts/process-lifecycle.ts +221 -45
- package/src/hosts/protected-worktrees.ts +1 -1
- package/src/hosts/skill-hook-catalog.ts +131 -0
- package/src/hosts/skill-hook-handler.ts +374 -0
- package/src/hosts/skill-hooks.ts +577 -0
- package/src/instrumentation.ts +7 -8
- package/src/native-transcript-state.ts +22 -3
- package/src/native-transcript-view.ts +52 -13
- package/src/native-transcripts.ts +34 -15
- package/src/paths.ts +22 -0
- package/src/process-lifecycle.ts +150 -0
- package/src/provenance.ts +27 -7
- package/src/prune.ts +300 -0
- package/src/report.ts +63 -36
- package/src/resolved-case.ts +1 -1
- package/src/results-lock.ts +42 -0
- package/src/results.ts +14 -18
- package/src/run-owner.ts +4 -4
- package/src/{runtime-config.ts → runtime/config.ts} +182 -44
- package/src/{runtime-discovery-apple.ts → runtime/discovery-apple.ts} +3 -3
- package/src/{runtime-discovery-providers.ts → runtime/discovery-providers.ts} +3 -9
- package/src/{runtime-discovery.ts → runtime/discovery.ts} +4 -4
- package/src/{runtime-hook-processes.ts → runtime/hook-processes.ts} +1 -1
- package/src/{runtime-hooks.ts → runtime/hooks.ts} +33 -22
- package/src/{runtime-paths.ts → runtime/paths.ts} +6 -18
- package/src/{runtime-query.ts → runtime/query.ts} +5 -5
- package/src/{runtime-seeds.ts → runtime/seeds.ts} +3 -3
- package/src/runtime/shared-cache.ts +130 -0
- package/src/{runtime-state.ts → runtime/state.ts} +28 -11
- package/src/schema-types.ts +73 -161
- package/src/shell.ts +4 -0
- package/src/transcript-retention.ts +146 -0
- package/src/value-guards.ts +22 -0
- package/docs/assets/sevro-logo.png +0 -0
- package/docs/assets/sevro-symbol.svg +0 -1
- package/docs/assets/sevro-wordmark.svg +0 -1
- package/docs/cache-ownership-validation.md +0 -49
- package/docs/claude-continuation-validation.md +0 -47
- package/docs/concurrency-validation.md +0 -86
- package/docs/configuration-root-validation.md +0 -70
- package/docs/engine-slice.md +0 -114
- package/docs/evidence/runtime-startup-30/README.md +0 -185
- package/docs/evidence/runtime-startup-30/after.json +0 -319
- package/docs/evidence/runtime-startup-30/baseline-timeout.jsonl +0 -2
- package/docs/evidence/runtime-startup-30/before.json +0 -403
- package/docs/evidence/runtime-startup-30/consumer-smoke.json +0 -63
- package/docs/evidence/runtime-startup-30/full-policy-diagnostic.jsonl +0 -3
- package/docs/evidence/runtime-startup-30/minimal-policy-diagnostic.jsonl +0 -3
- package/docs/evidence/runtime-startup-30/plugin-template-assessment.json +0 -20
- package/docs/evidence/runtime-startup-30/seedless-timeout.jsonl +0 -2
- package/docs/evidence.md +0 -17
- package/docs/fixture-cleanup-validation.md +0 -70
- package/docs/guide-verification.json +0 -239
- package/src/evaluation-context.ts +0 -257
- package/src/evaluation-fixture.ts +0 -313
- package/src/evaluation-run.ts +0 -198
- package/src/evaluation-trial.ts +0 -946
- package/src/evaluation-types.ts +0 -149
- package/src/hosts/claude-nested-skills.ts +0 -272
- package/src/hosts/codex-command-output.ts +0 -510
- package/src/hosts/codex-executor-command.ts +0 -291
- package/src/hosts/codex-skill-reads.ts +0 -609
- /package/src/{trial-scheduler.ts → evaluation/scheduler.ts} +0 -0
|
@@ -1,403 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"format": "sevro.runtime-startup-benchmark.v1",
|
|
3
|
-
"sourceRoot": "/Users/bjro/Sources/sevro",
|
|
4
|
-
"platform": "darwin",
|
|
5
|
-
"arch": "arm64",
|
|
6
|
-
"bun": "1.3.13",
|
|
7
|
-
"codex": "codex-cli 0.162.1",
|
|
8
|
-
"case": "synthetic-plugin-response",
|
|
9
|
-
"model": "synthetic-candidate / synthetic-semantic",
|
|
10
|
-
"effort": "low",
|
|
11
|
-
"seed": {
|
|
12
|
-
"entries": 2049,
|
|
13
|
-
"bytes": 33554432
|
|
14
|
-
},
|
|
15
|
-
"repetitions": 3,
|
|
16
|
-
"conditions": "Fresh role homes each invocation; first verified snapshot cold, later repetitions warm; OS filesystem caches uncontrolled; no live API/model calls",
|
|
17
|
-
"rows": [
|
|
18
|
-
{
|
|
19
|
-
"repetition": 0,
|
|
20
|
-
"role": "candidate",
|
|
21
|
-
"phase": "configuration",
|
|
22
|
-
"milliseconds": 261.92554200000006,
|
|
23
|
-
"status": "completed"
|
|
24
|
-
},
|
|
25
|
-
{
|
|
26
|
-
"repetition": 1,
|
|
27
|
-
"role": "candidate",
|
|
28
|
-
"phase": "runtime.candidate",
|
|
29
|
-
"milliseconds": 1969.3474580000002,
|
|
30
|
-
"status": "completed"
|
|
31
|
-
},
|
|
32
|
-
{
|
|
33
|
-
"repetition": 1,
|
|
34
|
-
"role": "candidate",
|
|
35
|
-
"phase": "plugin-command",
|
|
36
|
-
"milliseconds": 300.13179200000013,
|
|
37
|
-
"status": "completed"
|
|
38
|
-
},
|
|
39
|
-
{
|
|
40
|
-
"repetition": 1,
|
|
41
|
-
"role": "candidate",
|
|
42
|
-
"phase": "plugin-command",
|
|
43
|
-
"milliseconds": 81.15104199999996,
|
|
44
|
-
"status": "completed"
|
|
45
|
-
},
|
|
46
|
-
{
|
|
47
|
-
"repetition": 1,
|
|
48
|
-
"role": "candidate",
|
|
49
|
-
"phase": "installation-assets",
|
|
50
|
-
"files": [
|
|
51
|
-
"cache"
|
|
52
|
-
],
|
|
53
|
-
"configurationContainsPrivateHome": true,
|
|
54
|
-
"configurationContainsWorkspace": true
|
|
55
|
-
},
|
|
56
|
-
{
|
|
57
|
-
"repetition": 1,
|
|
58
|
-
"role": "candidate",
|
|
59
|
-
"phase": "plugin-install",
|
|
60
|
-
"milliseconds": 385.09974999999986,
|
|
61
|
-
"status": "completed"
|
|
62
|
-
},
|
|
63
|
-
{
|
|
64
|
-
"repetition": 1,
|
|
65
|
-
"role": "candidate",
|
|
66
|
-
"phase": "runtime.hooks",
|
|
67
|
-
"milliseconds": 1225.7079159999994,
|
|
68
|
-
"status": "completed"
|
|
69
|
-
},
|
|
70
|
-
{
|
|
71
|
-
"repetition": 1,
|
|
72
|
-
"role": "candidate",
|
|
73
|
-
"phase": "sandbox-probe",
|
|
74
|
-
"milliseconds": 5528.486166999999,
|
|
75
|
-
"status": "completed"
|
|
76
|
-
},
|
|
77
|
-
{
|
|
78
|
-
"repetition": 1,
|
|
79
|
-
"role": "candidate",
|
|
80
|
-
"phase": "sandbox-probe",
|
|
81
|
-
"milliseconds": 5409.168874999999,
|
|
82
|
-
"status": "completed"
|
|
83
|
-
},
|
|
84
|
-
{
|
|
85
|
-
"repetition": 1,
|
|
86
|
-
"role": "candidate",
|
|
87
|
-
"phase": "participant-execution",
|
|
88
|
-
"milliseconds": 27.37004100000013,
|
|
89
|
-
"status": "completed"
|
|
90
|
-
},
|
|
91
|
-
{
|
|
92
|
-
"repetition": 1,
|
|
93
|
-
"role": "candidate",
|
|
94
|
-
"phase": "candidate-total",
|
|
95
|
-
"milliseconds": 14924.6165,
|
|
96
|
-
"status": "completed"
|
|
97
|
-
},
|
|
98
|
-
{
|
|
99
|
-
"repetition": 1,
|
|
100
|
-
"role": "semantic",
|
|
101
|
-
"phase": "runtime.semantic",
|
|
102
|
-
"milliseconds": 1174.1918750000004,
|
|
103
|
-
"status": "completed"
|
|
104
|
-
},
|
|
105
|
-
{
|
|
106
|
-
"repetition": 1,
|
|
107
|
-
"role": "semantic",
|
|
108
|
-
"phase": "plugin-install",
|
|
109
|
-
"milliseconds": 0.014666000002762303,
|
|
110
|
-
"status": "completed"
|
|
111
|
-
},
|
|
112
|
-
{
|
|
113
|
-
"repetition": 1,
|
|
114
|
-
"role": "semantic",
|
|
115
|
-
"phase": "runtime.hooks",
|
|
116
|
-
"milliseconds": 1104.8465419999993,
|
|
117
|
-
"status": "completed"
|
|
118
|
-
},
|
|
119
|
-
{
|
|
120
|
-
"repetition": 1,
|
|
121
|
-
"role": "semantic",
|
|
122
|
-
"phase": "sandbox-probe",
|
|
123
|
-
"milliseconds": 5485.405999999999,
|
|
124
|
-
"status": "completed"
|
|
125
|
-
},
|
|
126
|
-
{
|
|
127
|
-
"repetition": 1,
|
|
128
|
-
"role": "semantic",
|
|
129
|
-
"phase": "sandbox-probe",
|
|
130
|
-
"milliseconds": 5369.393208999998,
|
|
131
|
-
"status": "completed"
|
|
132
|
-
},
|
|
133
|
-
{
|
|
134
|
-
"repetition": 1,
|
|
135
|
-
"role": "semantic",
|
|
136
|
-
"phase": "participant-execution",
|
|
137
|
-
"milliseconds": 27.99250000000029,
|
|
138
|
-
"status": "completed"
|
|
139
|
-
},
|
|
140
|
-
{
|
|
141
|
-
"repetition": 1,
|
|
142
|
-
"role": "semantic",
|
|
143
|
-
"phase": "grading",
|
|
144
|
-
"milliseconds": 13494.667042000001,
|
|
145
|
-
"status": "completed"
|
|
146
|
-
},
|
|
147
|
-
{
|
|
148
|
-
"repetition": 2,
|
|
149
|
-
"role": "candidate",
|
|
150
|
-
"phase": "runtime.candidate",
|
|
151
|
-
"milliseconds": 1357.1145419999993,
|
|
152
|
-
"status": "completed"
|
|
153
|
-
},
|
|
154
|
-
{
|
|
155
|
-
"repetition": 2,
|
|
156
|
-
"role": "candidate",
|
|
157
|
-
"phase": "plugin-command",
|
|
158
|
-
"milliseconds": 72.52966600000218,
|
|
159
|
-
"status": "completed"
|
|
160
|
-
},
|
|
161
|
-
{
|
|
162
|
-
"repetition": 2,
|
|
163
|
-
"role": "candidate",
|
|
164
|
-
"phase": "plugin-command",
|
|
165
|
-
"milliseconds": 67.09337500000038,
|
|
166
|
-
"status": "completed"
|
|
167
|
-
},
|
|
168
|
-
{
|
|
169
|
-
"repetition": 2,
|
|
170
|
-
"role": "candidate",
|
|
171
|
-
"phase": "installation-assets",
|
|
172
|
-
"files": [
|
|
173
|
-
"cache"
|
|
174
|
-
],
|
|
175
|
-
"configurationContainsPrivateHome": true,
|
|
176
|
-
"configurationContainsWorkspace": true
|
|
177
|
-
},
|
|
178
|
-
{
|
|
179
|
-
"repetition": 2,
|
|
180
|
-
"role": "candidate",
|
|
181
|
-
"phase": "plugin-install",
|
|
182
|
-
"milliseconds": 141.13416599999982,
|
|
183
|
-
"status": "completed"
|
|
184
|
-
},
|
|
185
|
-
{
|
|
186
|
-
"repetition": 2,
|
|
187
|
-
"role": "candidate",
|
|
188
|
-
"phase": "runtime.hooks",
|
|
189
|
-
"milliseconds": 1464.9581660000003,
|
|
190
|
-
"status": "completed"
|
|
191
|
-
},
|
|
192
|
-
{
|
|
193
|
-
"repetition": 2,
|
|
194
|
-
"role": "candidate",
|
|
195
|
-
"phase": "sandbox-probe",
|
|
196
|
-
"milliseconds": 5689.016333,
|
|
197
|
-
"status": "completed"
|
|
198
|
-
},
|
|
199
|
-
{
|
|
200
|
-
"repetition": 2,
|
|
201
|
-
"role": "candidate",
|
|
202
|
-
"phase": "sandbox-probe",
|
|
203
|
-
"milliseconds": 5342.566417000002,
|
|
204
|
-
"status": "completed"
|
|
205
|
-
},
|
|
206
|
-
{
|
|
207
|
-
"repetition": 2,
|
|
208
|
-
"role": "candidate",
|
|
209
|
-
"phase": "participant-execution",
|
|
210
|
-
"milliseconds": 28.610333000004175,
|
|
211
|
-
"status": "completed"
|
|
212
|
-
},
|
|
213
|
-
{
|
|
214
|
-
"repetition": 2,
|
|
215
|
-
"role": "candidate",
|
|
216
|
-
"phase": "candidate-total",
|
|
217
|
-
"milliseconds": 14369.195708,
|
|
218
|
-
"status": "completed"
|
|
219
|
-
},
|
|
220
|
-
{
|
|
221
|
-
"repetition": 2,
|
|
222
|
-
"role": "semantic",
|
|
223
|
-
"phase": "runtime.semantic",
|
|
224
|
-
"milliseconds": 1235.6402080000043,
|
|
225
|
-
"status": "completed"
|
|
226
|
-
},
|
|
227
|
-
{
|
|
228
|
-
"repetition": 2,
|
|
229
|
-
"role": "semantic",
|
|
230
|
-
"phase": "plugin-install",
|
|
231
|
-
"milliseconds": 0.014084000002185348,
|
|
232
|
-
"status": "completed"
|
|
233
|
-
},
|
|
234
|
-
{
|
|
235
|
-
"repetition": 2,
|
|
236
|
-
"role": "semantic",
|
|
237
|
-
"phase": "runtime.hooks",
|
|
238
|
-
"milliseconds": 1087.1219169999968,
|
|
239
|
-
"status": "completed"
|
|
240
|
-
},
|
|
241
|
-
{
|
|
242
|
-
"repetition": 2,
|
|
243
|
-
"role": "semantic",
|
|
244
|
-
"phase": "sandbox-probe",
|
|
245
|
-
"milliseconds": 5409.527834,
|
|
246
|
-
"status": "completed"
|
|
247
|
-
},
|
|
248
|
-
{
|
|
249
|
-
"repetition": 2,
|
|
250
|
-
"role": "semantic",
|
|
251
|
-
"phase": "sandbox-probe",
|
|
252
|
-
"milliseconds": 5393.348124999997,
|
|
253
|
-
"status": "completed"
|
|
254
|
-
},
|
|
255
|
-
{
|
|
256
|
-
"repetition": 2,
|
|
257
|
-
"role": "semantic",
|
|
258
|
-
"phase": "participant-execution",
|
|
259
|
-
"milliseconds": 44.382833999996365,
|
|
260
|
-
"status": "completed"
|
|
261
|
-
},
|
|
262
|
-
{
|
|
263
|
-
"repetition": 2,
|
|
264
|
-
"role": "semantic",
|
|
265
|
-
"phase": "grading",
|
|
266
|
-
"milliseconds": 13533.550041999995,
|
|
267
|
-
"status": "completed"
|
|
268
|
-
},
|
|
269
|
-
{
|
|
270
|
-
"repetition": 3,
|
|
271
|
-
"role": "candidate",
|
|
272
|
-
"phase": "runtime.candidate",
|
|
273
|
-
"milliseconds": 1273.4362919999985,
|
|
274
|
-
"status": "completed"
|
|
275
|
-
},
|
|
276
|
-
{
|
|
277
|
-
"repetition": 3,
|
|
278
|
-
"role": "candidate",
|
|
279
|
-
"phase": "plugin-command",
|
|
280
|
-
"milliseconds": 75.75125000000116,
|
|
281
|
-
"status": "completed"
|
|
282
|
-
},
|
|
283
|
-
{
|
|
284
|
-
"repetition": 3,
|
|
285
|
-
"role": "candidate",
|
|
286
|
-
"phase": "plugin-command",
|
|
287
|
-
"milliseconds": 65.14912499999627,
|
|
288
|
-
"status": "completed"
|
|
289
|
-
},
|
|
290
|
-
{
|
|
291
|
-
"repetition": 3,
|
|
292
|
-
"role": "candidate",
|
|
293
|
-
"phase": "installation-assets",
|
|
294
|
-
"files": [
|
|
295
|
-
"cache"
|
|
296
|
-
],
|
|
297
|
-
"configurationContainsPrivateHome": true,
|
|
298
|
-
"configurationContainsWorkspace": true
|
|
299
|
-
},
|
|
300
|
-
{
|
|
301
|
-
"repetition": 3,
|
|
302
|
-
"role": "candidate",
|
|
303
|
-
"phase": "plugin-install",
|
|
304
|
-
"milliseconds": 143.153999999995,
|
|
305
|
-
"status": "completed"
|
|
306
|
-
},
|
|
307
|
-
{
|
|
308
|
-
"repetition": 3,
|
|
309
|
-
"role": "candidate",
|
|
310
|
-
"phase": "runtime.hooks",
|
|
311
|
-
"milliseconds": 1234.849000000002,
|
|
312
|
-
"status": "completed"
|
|
313
|
-
},
|
|
314
|
-
{
|
|
315
|
-
"repetition": 3,
|
|
316
|
-
"role": "candidate",
|
|
317
|
-
"phase": "sandbox-probe",
|
|
318
|
-
"milliseconds": 5916.286457999995,
|
|
319
|
-
"status": "completed"
|
|
320
|
-
},
|
|
321
|
-
{
|
|
322
|
-
"repetition": 3,
|
|
323
|
-
"role": "candidate",
|
|
324
|
-
"phase": "sandbox-probe",
|
|
325
|
-
"milliseconds": 5706.173332999999,
|
|
326
|
-
"status": "completed"
|
|
327
|
-
},
|
|
328
|
-
{
|
|
329
|
-
"repetition": 3,
|
|
330
|
-
"role": "candidate",
|
|
331
|
-
"phase": "participant-execution",
|
|
332
|
-
"milliseconds": 38.983124999998836,
|
|
333
|
-
"status": "completed"
|
|
334
|
-
},
|
|
335
|
-
{
|
|
336
|
-
"repetition": 3,
|
|
337
|
-
"role": "candidate",
|
|
338
|
-
"phase": "candidate-total",
|
|
339
|
-
"milliseconds": 14666.299250000004,
|
|
340
|
-
"status": "completed"
|
|
341
|
-
},
|
|
342
|
-
{
|
|
343
|
-
"repetition": 3,
|
|
344
|
-
"role": "semantic",
|
|
345
|
-
"phase": "runtime.semantic",
|
|
346
|
-
"milliseconds": 2261.5522090000013,
|
|
347
|
-
"status": "completed"
|
|
348
|
-
},
|
|
349
|
-
{
|
|
350
|
-
"repetition": 3,
|
|
351
|
-
"role": "semantic",
|
|
352
|
-
"phase": "plugin-install",
|
|
353
|
-
"milliseconds": 0.016417000006185845,
|
|
354
|
-
"status": "completed"
|
|
355
|
-
},
|
|
356
|
-
{
|
|
357
|
-
"repetition": 3,
|
|
358
|
-
"role": "semantic",
|
|
359
|
-
"phase": "runtime.hooks",
|
|
360
|
-
"milliseconds": 1363.224583999996,
|
|
361
|
-
"status": "completed"
|
|
362
|
-
},
|
|
363
|
-
{
|
|
364
|
-
"repetition": 3,
|
|
365
|
-
"role": "semantic",
|
|
366
|
-
"phase": "sandbox-probe",
|
|
367
|
-
"milliseconds": 5592.143207999994,
|
|
368
|
-
"status": "completed"
|
|
369
|
-
},
|
|
370
|
-
{
|
|
371
|
-
"repetition": 3,
|
|
372
|
-
"role": "semantic",
|
|
373
|
-
"phase": "sandbox-probe",
|
|
374
|
-
"milliseconds": 5439.415540999995,
|
|
375
|
-
"status": "completed"
|
|
376
|
-
},
|
|
377
|
-
{
|
|
378
|
-
"repetition": 3,
|
|
379
|
-
"role": "semantic",
|
|
380
|
-
"phase": "participant-execution",
|
|
381
|
-
"milliseconds": 27.27187499999127,
|
|
382
|
-
"status": "completed"
|
|
383
|
-
},
|
|
384
|
-
{
|
|
385
|
-
"repetition": 3,
|
|
386
|
-
"role": "semantic",
|
|
387
|
-
"phase": "grading",
|
|
388
|
-
"milliseconds": 15030.493917,
|
|
389
|
-
"status": "completed"
|
|
390
|
-
},
|
|
391
|
-
{
|
|
392
|
-
"repetition": 4,
|
|
393
|
-
"role": "semantic",
|
|
394
|
-
"phase": "cow-copy",
|
|
395
|
-
"milliseconds": 1.0108749999926658,
|
|
396
|
-
"status": "completed"
|
|
397
|
-
},
|
|
398
|
-
{
|
|
399
|
-
"phase": "cow-isolation",
|
|
400
|
-
"sourceUnchanged": true
|
|
401
|
-
}
|
|
402
|
-
]
|
|
403
|
-
}
|
|
@@ -1,63 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"format": "sevro.consumer-smoke.v1",
|
|
3
|
-
"date": "2026-10-11",
|
|
4
|
-
"consumer": {
|
|
5
|
-
"repository": "https://github.com/BjRo/darrow",
|
|
6
|
-
"revision": "34606ad9e1952b7f4e9c56befa446c18e1a2eb5d",
|
|
7
|
-
"trackedChanges": false
|
|
8
|
-
},
|
|
9
|
-
"environment": {
|
|
10
|
-
"platform": "darwin",
|
|
11
|
-
"arch": "arm64",
|
|
12
|
-
"bun": "1.3.13",
|
|
13
|
-
"codex": "0.162.1",
|
|
14
|
-
"candidateModel": "gpt-6-luna",
|
|
15
|
-
"candidateEffort": "medium",
|
|
16
|
-
"semanticModel": "gpt-6-luna",
|
|
17
|
-
"semanticEffort": "medium"
|
|
18
|
-
},
|
|
19
|
-
"sampling": {
|
|
20
|
-
"trials": 1,
|
|
21
|
-
"jobs": 1,
|
|
22
|
-
"threshold": 1
|
|
23
|
-
},
|
|
24
|
-
"runs": [
|
|
25
|
-
{
|
|
26
|
-
"runner": "unchanged c205482b25d4d52a395c70434f2b392f6acfb279",
|
|
27
|
-
"case": "create-commit-conventional-format",
|
|
28
|
-
"execution": "failed",
|
|
29
|
-
"task": "not_assessed",
|
|
30
|
-
"activation": "unavailable",
|
|
31
|
-
"participantReached": false,
|
|
32
|
-
"diagnostic": "First sandbox preflight exceeded its 10-second deadline."
|
|
33
|
-
},
|
|
34
|
-
{
|
|
35
|
-
"runner": "issue-30 candidate",
|
|
36
|
-
"case": "create-commit-conventional-format",
|
|
37
|
-
"execution": "completed",
|
|
38
|
-
"grading": "completed",
|
|
39
|
-
"task": "failed",
|
|
40
|
-
"shellChecksPassed": 3,
|
|
41
|
-
"activation": "unavailable",
|
|
42
|
-
"limitation": "Implicit repository fixture omitted plugin manifests required by its backend launcher. Candidate correctly refused to create the requested commit; head-change assertion failed."
|
|
43
|
-
},
|
|
44
|
-
{
|
|
45
|
-
"runner": "issue-30 candidate",
|
|
46
|
-
"case": "tdd-explicit-invocation",
|
|
47
|
-
"execution": "completed",
|
|
48
|
-
"grading": "completed",
|
|
49
|
-
"task": "passed",
|
|
50
|
-
"shellCheck": "passed",
|
|
51
|
-
"semanticCheck": "passed",
|
|
52
|
-
"activation": "passed",
|
|
53
|
-
"activationCompleteness": "complete",
|
|
54
|
-
"activationEvidence": "Native Codex explicit-invocation receipt"
|
|
55
|
-
}
|
|
56
|
-
],
|
|
57
|
-
"limits": [
|
|
58
|
-
"Only one authenticated trial per listed run.",
|
|
59
|
-
"Different cases are not a matched live performance comparison.",
|
|
60
|
-
"No live API/model latency saving is inferred.",
|
|
61
|
-
"Private transcripts, prompts, tool arguments, and credentials are omitted."
|
|
62
|
-
]
|
|
63
|
-
}
|
|
@@ -1,20 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"codex": "codex-cli 0.162.1",
|
|
3
|
-
"registered": {
|
|
4
|
-
"code": 0,
|
|
5
|
-
"out": "{\n \"marketplaceName\": \"sevro-assessment\",\n \"installedRoot\": \"<assessment>/market\",\n \"alreadyAdded\": false\n}\n",
|
|
6
|
-
"err": ""
|
|
7
|
-
},
|
|
8
|
-
"installed": {
|
|
9
|
-
"code": 0,
|
|
10
|
-
"out": "{\n \"pluginId\": \"probe@sevro-assessment\",\n \"name\": \"probe\",\n \"marketplaceName\": \"sevro-assessment\",\n \"version\": \"1.0.0\",\n \"installedPath\": \"<assessment>/home/plugins/cache/sevro-assessment/probe/1.0.0\",\n \"authPolicy\": \"ON_INSTALL\"\n}\n",
|
|
11
|
-
"err": ""
|
|
12
|
-
},
|
|
13
|
-
"configuration": "[marketplaces.sevro-assessment]\nsource_type = \"local\"\nsource = \"<assessment>/market\"\n\n[plugins.\"probe@sevro-assessment\"]\nenabled = true\n",
|
|
14
|
-
"cacheOnlyFreshHomeList": {
|
|
15
|
-
"code": 0,
|
|
16
|
-
"out": "{\n \"installed\": [],\n \"available\": []\n}\n",
|
|
17
|
-
"err": ""
|
|
18
|
-
},
|
|
19
|
-
"conditions": "Local synthetic plugin; fresh homes without credentials; installed files copied to a second home; no registration copied"
|
|
20
|
-
}
|
package/docs/evidence.md
DELETED
|
@@ -1,17 +0,0 @@
|
|
|
1
|
-
# Historical validation evidence
|
|
2
|
-
|
|
3
|
-
These records preserve specific observations and limits. They are not current
|
|
4
|
-
public contracts, fresh acceptance runs, or proof of broader platform support.
|
|
5
|
-
Keep their observations intact:
|
|
6
|
-
|
|
7
|
-
- [Cache ownership](cache-ownership-validation.md)
|
|
8
|
-
- [Claude continuation](claude-continuation-validation.md)
|
|
9
|
-
- [Concurrency](concurrency-validation.md)
|
|
10
|
-
- [Configuration roots](configuration-root-validation.md)
|
|
11
|
-
- [Fixture cleanup](fixture-cleanup-validation.md)
|
|
12
|
-
|
|
13
|
-
The [first engine slice](engine-slice.md) also records bounded implementation
|
|
14
|
-
context. [Current contracts](contracts.md) and [architecture](architecture.md)
|
|
15
|
-
describe present interfaces. Coverage discovery belongs to [issue #1](https://github.com/BjRo/sevro/issues/1);
|
|
16
|
-
targeted prototypes do not prove full-suite coverage.
|
|
17
|
-
Return to the [hub](README.md).
|
|
@@ -1,70 +0,0 @@
|
|
|
1
|
-
# Permission-locked fixture cleanup
|
|
2
|
-
|
|
3
|
-
The Darrow extraction comparison exposed a cleanup difference: its legacy
|
|
4
|
-
fixture destroyer removes permission-locked directories, while Sevro retained
|
|
5
|
-
them after a completed trial. Sevro now restores owner access inside completed
|
|
6
|
-
candidate workspaces when needed. It clears their contents after trial evidence
|
|
7
|
-
is retained, keeps empty reservations until active peers drain, and then removes
|
|
8
|
-
the completed roots. Directory traversal uses link metadata and skips symbolic
|
|
9
|
-
links; external targets retain their permissions and contents.
|
|
10
|
-
|
|
11
|
-
## Public CLI regression slices
|
|
12
|
-
|
|
13
|
-
The public seam is `sevro run --case-file ... --adapter-module ...`, as documented
|
|
14
|
-
in [Development CLI](development-cli.md). Each controlled adapter uses real
|
|
15
|
-
filesystem permissions and returns its workspace path as a bounded observation.
|
|
16
|
-
The test reads retained public evidence and verifies that the workspace is gone.
|
|
17
|
-
A preliminary internal-engine test was diagnostic only; it is not part of this
|
|
18
|
-
TDD trace and was removed before implementation.
|
|
19
|
-
|
|
20
|
-
Red — `bun test tests/cli.test.ts --test-name-pattern 'CLI removes a permission-locked candidate directory'`: completed evidence existed, but the mode-000 child directory kept the workspace present.
|
|
21
|
-
|
|
22
|
-
Green — `bun test tests/cli.test.ts --test-name-pattern 'CLI removes a permission-locked candidate directory'`: the workspace was removed and retained evidence still agreed with the CLI result.
|
|
23
|
-
|
|
24
|
-
Red — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(root\)'`: a mode-000 workspace root remained present after completion.
|
|
25
|
-
|
|
26
|
-
Green — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(root\)'`: the workspace root was removed after evidence retention.
|
|
27
|
-
|
|
28
|
-
Red — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(readable root\)'`: the readable mode-500 root lacked write access and remained present.
|
|
29
|
-
|
|
30
|
-
Green — `bun test tests/cli.test.ts --test-name-pattern 'permission-locked candidate directory \(readable root\)'`: restoring owner write access allowed cleanup; the external link target's mode and bytes were unchanged.
|
|
31
|
-
|
|
32
|
-
Each final example also links to an external directory with a read-only file.
|
|
33
|
-
All three check that its directory mode, file mode, and file contents stay intact.
|
|
34
|
-
These are deterministic filesystem and lifecycle checks, with no live model call.
|
|
35
|
-
|
|
36
|
-
## Final source gates
|
|
37
|
-
|
|
38
|
-
`bun test` passed all 243 tests and 1399 assertions across 38 files in 93.83
|
|
39
|
-
seconds. The gate includes parallel admission, peer isolation, cancellation,
|
|
40
|
-
failed persistence, extension lifecycle, and generated/repository fixture tests.
|
|
41
|
-
`bun run typecheck` and `bun run format:check` pass.
|
|
42
|
-
The full log and subsequent installed-archive checks are retained under
|
|
43
|
-
`/Users/bjro/.darrow/issue95-fixture-cleanup/`. The tests ran on macOS with Bun
|
|
44
|
-
1.3.13; they do not establish another platform or native model route.
|
|
45
|
-
|
|
46
|
-
## Canonical workspace follow-up
|
|
47
|
-
|
|
48
|
-
Darrow's private fixture builder also checked that its returned path was
|
|
49
|
-
canonical. An initial diagnostic assertion in the cleanup examples exposed
|
|
50
|
-
Sevro's logical temporary-directory alias; those diagnostic additions were
|
|
51
|
-
removed before the isolated regression. The public canonical-workspace test
|
|
52
|
-
records the operating system's `realpath` through a controlled host and compares
|
|
53
|
-
it with the received workspace in retained evidence. The full source gate above
|
|
54
|
-
predates this follow-up.
|
|
55
|
-
|
|
56
|
-
The public CLI contract was clarified before implementation. Candidate and
|
|
57
|
-
semantic workspace allocations now resolve the new directory through `realpath`
|
|
58
|
-
before reservation, preparation, or host execution.
|
|
59
|
-
|
|
60
|
-
Red — `bun test tests/cli.test.ts --test-name-pattern 'CLI supplies a canonical candidate workspace'`:
|
|
61
|
-
exit `1`; the received `/var/...` path differed from the operating system's
|
|
62
|
-
`/private/var/...` path. One test failed with four assertions in 222 milliseconds.
|
|
63
|
-
|
|
64
|
-
Green — `bun test tests/cli.test.ts --test-name-pattern 'CLI supplies a canonical candidate workspace'`:
|
|
65
|
-
exit `0`; one test and four assertions passed in 228 milliseconds after the fix.
|
|
66
|
-
|
|
67
|
-
The final `bun test` gate passed 244 tests and 1403 assertions across 38 files in
|
|
68
|
-
94.77 seconds. Typecheck and formatting pass. The source log is retained at
|
|
69
|
-
`/Users/bjro/.darrow/issue95-fixture-mounting/sevro-full-gate.log`. These remain
|
|
70
|
-
deterministic macOS/Bun checks, without a live model or additional platform claim.
|