@bjoernrochel/sevro 0.1.0-rc.8 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (582) hide show
  1. package/CONTRIBUTING.md +52 -6
  2. package/README.md +12 -7
  3. package/dist/examples/basic/graded.json +13 -0
  4. package/dist/examples/basic/host.js +9 -0
  5. package/dist/examples/basic/host.js.map +1 -0
  6. package/dist/examples/basic/prompt-only.json +7 -0
  7. package/dist/examples/recipes/candidate.js +23 -0
  8. package/dist/examples/recipes/candidate.js.map +1 -0
  9. package/dist/examples/recipes/cases/coding-change.json +21 -0
  10. package/dist/examples/recipes/cases/host-tools.json +21 -0
  11. package/dist/examples/recipes/cases/mise.json +21 -0
  12. package/dist/examples/recipes/cases/node.json +21 -0
  13. package/dist/examples/recipes/cases/python-uv.json +21 -0
  14. package/dist/examples/recipes/cases/semantic.json +17 -0
  15. package/dist/examples/recipes/coding-change/check.js +16 -0
  16. package/dist/examples/recipes/coding-change/check.js.map +1 -0
  17. package/dist/examples/recipes/executable.js +20 -0
  18. package/dist/examples/recipes/executable.js.map +1 -0
  19. package/dist/examples/recipes/extension-artifacts.js +56 -0
  20. package/dist/examples/recipes/extension-artifacts.js.map +1 -0
  21. package/dist/examples/recipes/extension-methods.js +167 -0
  22. package/dist/examples/recipes/extension-methods.js.map +1 -0
  23. package/dist/examples/recipes/extension.js +28 -0
  24. package/dist/examples/recipes/extension.js.map +1 -0
  25. package/dist/examples/recipes/modules.js +9 -0
  26. package/dist/examples/recipes/modules.js.map +1 -0
  27. package/dist/examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md +7 -0
  28. package/dist/examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md +7 -0
  29. package/dist/examples/recipes/repository-skill/README.md +3 -0
  30. package/dist/examples/recipes/run-inputs.js +144 -0
  31. package/dist/examples/recipes/run-inputs.js.map +1 -0
  32. package/dist/examples/recipes/run-options.js +103 -0
  33. package/dist/examples/recipes/run-options.js.map +1 -0
  34. package/dist/examples/recipes/run.js +62 -0
  35. package/dist/examples/recipes/run.js.map +1 -0
  36. package/dist/examples/recipes/runtime/empty.json +1 -0
  37. package/dist/examples/recipes/runtime/goals.json +1 -0
  38. package/dist/examples/recipes/runtime/mise.json +6 -0
  39. package/dist/examples/recipes/runtime/path.json +1 -0
  40. package/dist/examples/recipes/runtime/plugins.json +4 -0
  41. package/dist/examples/recipes/runtime/python-uv.json +10 -0
  42. package/dist/examples/recipes/semantic-judge.js +27 -0
  43. package/dist/examples/recipes/semantic-judge.js.map +1 -0
  44. package/dist/examples/templates/coding/adapter.js +14 -0
  45. package/dist/examples/templates/coding/adapter.js.map +1 -0
  46. package/dist/examples/templates/coding/case.json +15 -0
  47. package/dist/examples/templates/repository-skill/.agents/skills/startup-guide/SKILL.md +6 -0
  48. package/dist/examples/templates/repository-skill/extension-data.js +61 -0
  49. package/dist/examples/templates/repository-skill/extension-data.js.map +1 -0
  50. package/dist/examples/templates/repository-skill/extension.js +29 -0
  51. package/dist/examples/templates/repository-skill/extension.js.map +1 -0
  52. package/dist/examples/templates/response/adapter.js +13 -0
  53. package/dist/examples/templates/response/adapter.js.map +1 -0
  54. package/dist/examples/templates/response/case.json +13 -0
  55. package/dist/src/advisory.js +86 -0
  56. package/dist/src/advisory.js.map +1 -0
  57. package/dist/src/candidate-artifacts.js +139 -0
  58. package/dist/src/candidate-artifacts.js.map +1 -0
  59. package/dist/src/candidate-codex-state.js +110 -0
  60. package/dist/src/candidate-codex-state.js.map +1 -0
  61. package/dist/src/case-validation.js +92 -0
  62. package/dist/src/case-validation.js.map +1 -0
  63. package/dist/src/cli/cancellation.js +21 -0
  64. package/dist/src/cli/cancellation.js.map +1 -0
  65. package/dist/src/cli/command.js +60 -0
  66. package/dist/src/cli/command.js.map +1 -0
  67. package/dist/src/cli/completion.js +14 -0
  68. package/dist/src/cli/completion.js.map +1 -0
  69. package/dist/src/cli/demo.js +45 -0
  70. package/dist/src/cli/demo.js.map +1 -0
  71. package/dist/src/cli/evaluation.js +40 -0
  72. package/dist/src/cli/evaluation.js.map +1 -0
  73. package/dist/src/cli/extension.js +54 -0
  74. package/dist/src/cli/extension.js.map +1 -0
  75. package/dist/src/cli/help.js +190 -0
  76. package/dist/src/cli/help.js.map +1 -0
  77. package/dist/src/cli/hosts.js +33 -0
  78. package/dist/src/cli/hosts.js.map +1 -0
  79. package/dist/src/cli/init-settings.js +29 -0
  80. package/dist/src/cli/init-settings.js.map +1 -0
  81. package/dist/src/cli/init.js +82 -0
  82. package/dist/src/cli/init.js.map +1 -0
  83. package/dist/src/cli/invocation.js +431 -0
  84. package/dist/src/cli/invocation.js.map +1 -0
  85. package/dist/src/cli/loading.js +110 -0
  86. package/dist/src/cli/loading.js.map +1 -0
  87. package/dist/src/cli/preparation.js +88 -0
  88. package/dist/src/cli/preparation.js.map +1 -0
  89. package/dist/src/cli/presentation.js +61 -0
  90. package/dist/src/cli/presentation.js.map +1 -0
  91. package/dist/src/cli/progress.js +5 -0
  92. package/dist/src/cli/progress.js.map +1 -0
  93. package/dist/src/cli/run-manifest.js +108 -0
  94. package/dist/src/cli/run-manifest.js.map +1 -0
  95. package/dist/src/cli/run-options.js +367 -0
  96. package/dist/src/cli/run-options.js.map +1 -0
  97. package/dist/src/cli/run-overrides.js +51 -0
  98. package/dist/src/cli/run-overrides.js.map +1 -0
  99. package/dist/src/cli/settings.js +127 -0
  100. package/dist/src/cli/settings.js.map +1 -0
  101. package/dist/src/cli/summary.js +28 -0
  102. package/dist/src/cli/summary.js.map +1 -0
  103. package/dist/src/cli.js +20 -0
  104. package/dist/src/cli.js.map +1 -0
  105. package/dist/src/contract-types.js +2 -0
  106. package/dist/src/contract-types.js.map +1 -0
  107. package/dist/src/engine.js +4 -0
  108. package/dist/src/engine.js.map +1 -0
  109. package/dist/src/evaluation/advisory.js +85 -0
  110. package/dist/src/evaluation/advisory.js.map +1 -0
  111. package/dist/src/evaluation/check-plan.js +292 -0
  112. package/dist/src/evaluation/check-plan.js.map +1 -0
  113. package/dist/src/evaluation/collection.js +61 -0
  114. package/dist/src/evaluation/collection.js.map +1 -0
  115. package/dist/src/evaluation/context.js +2 -0
  116. package/dist/src/evaluation/context.js.map +1 -0
  117. package/dist/src/evaluation/grading-host.js +28 -0
  118. package/dist/src/evaluation/grading-host.js.map +1 -0
  119. package/dist/src/evaluation/grading.js +154 -0
  120. package/dist/src/evaluation/grading.js.map +1 -0
  121. package/dist/src/evaluation/invocation-plan.js +312 -0
  122. package/dist/src/evaluation/invocation-plan.js.map +1 -0
  123. package/dist/src/evaluation/plan.js +376 -0
  124. package/dist/src/evaluation/plan.js.map +1 -0
  125. package/dist/src/evaluation/progress.js +14 -0
  126. package/dist/src/evaluation/progress.js.map +1 -0
  127. package/dist/src/evaluation/resources.js +159 -0
  128. package/dist/src/evaluation/resources.js.map +1 -0
  129. package/dist/src/evaluation/run.js +148 -0
  130. package/dist/src/evaluation/run.js.map +1 -0
  131. package/dist/src/evaluation/scheduler.js +27 -0
  132. package/dist/src/evaluation/scheduler.js.map +1 -0
  133. package/dist/src/evaluation/semantic.js +183 -0
  134. package/dist/src/evaluation/semantic.js.map +1 -0
  135. package/dist/src/evaluation/trial.js +593 -0
  136. package/dist/src/evaluation/trial.js.map +1 -0
  137. package/dist/src/evaluation/types.js +20 -0
  138. package/dist/src/evaluation/types.js.map +1 -0
  139. package/dist/src/evidence/host-results.js +67 -0
  140. package/dist/src/evidence/host-results.js.map +1 -0
  141. package/dist/src/evidence/retained-artifacts.js +17 -0
  142. package/dist/src/evidence/retained-artifacts.js.map +1 -0
  143. package/dist/src/evidence/trial-writer.js +131 -0
  144. package/dist/src/evidence/trial-writer.js.map +1 -0
  145. package/dist/src/evidence/usage.js +14 -0
  146. package/dist/src/evidence/usage.js.map +1 -0
  147. package/dist/src/executable.js +20 -0
  148. package/dist/src/executable.js.map +1 -0
  149. package/dist/src/extension-client.js +212 -0
  150. package/dist/src/extension-client.js.map +1 -0
  151. package/dist/src/extension-session.js +236 -0
  152. package/dist/src/extension-session.js.map +1 -0
  153. package/dist/src/fixtures/advisory.js +206 -0
  154. package/dist/src/fixtures/advisory.js.map +1 -0
  155. package/dist/src/fixtures/bin.js +41 -0
  156. package/dist/src/fixtures/bin.js.map +1 -0
  157. package/dist/src/fixtures/cleanup.js +43 -0
  158. package/dist/src/fixtures/cleanup.js.map +1 -0
  159. package/dist/src/fixtures/create.js +94 -0
  160. package/dist/src/fixtures/create.js.map +1 -0
  161. package/dist/src/fixtures/generated.js +181 -0
  162. package/dist/src/fixtures/generated.js.map +1 -0
  163. package/dist/src/fixtures/git-hooks.js +16 -0
  164. package/dist/src/fixtures/git-hooks.js.map +1 -0
  165. package/dist/src/fixtures/preparation.js +201 -0
  166. package/dist/src/fixtures/preparation.js.map +1 -0
  167. package/dist/src/fixtures/repository.js +255 -0
  168. package/dist/src/fixtures/repository.js.map +1 -0
  169. package/dist/src/fixtures/setup.js +175 -0
  170. package/dist/src/fixtures/setup.js.map +1 -0
  171. package/dist/src/fixtures/tools.js +57 -0
  172. package/dist/src/fixtures/tools.js.map +1 -0
  173. package/dist/src/fixtures/workspace-state.js +155 -0
  174. package/dist/src/fixtures/workspace-state.js.map +1 -0
  175. package/dist/src/generated/cli-result.cjs +2 -0
  176. package/dist/src/generated/extension.cjs +2 -0
  177. package/dist/src/generated/report.cjs +2 -0
  178. package/dist/src/generated/run-evidence.cjs +2 -0
  179. package/dist/src/generated/runtime.cjs +2 -0
  180. package/dist/src/git.js +72 -0
  181. package/dist/src/git.js.map +1 -0
  182. package/dist/src/graders/artifact.js +68 -0
  183. package/dist/src/graders/artifact.js.map +1 -0
  184. package/dist/src/graders/git-head.js +112 -0
  185. package/dist/src/graders/git-head.js.map +1 -0
  186. package/dist/src/graders/output.js +345 -0
  187. package/dist/src/graders/output.js.map +1 -0
  188. package/dist/src/graders/semantic.js +132 -0
  189. package/dist/src/graders/semantic.js.map +1 -0
  190. package/dist/src/graders/shell.js +321 -0
  191. package/dist/src/graders/shell.js.map +1 -0
  192. package/dist/src/host-diagnostic-redaction.js +52 -0
  193. package/dist/src/host-diagnostic-redaction.js.map +1 -0
  194. package/dist/src/host-diagnostics.js +248 -0
  195. package/dist/src/host-diagnostics.js.map +1 -0
  196. package/dist/src/hosts/claude-continuation.js +89 -0
  197. package/dist/src/hosts/claude-continuation.js.map +1 -0
  198. package/dist/src/hosts/claude-credential.js +121 -0
  199. package/dist/src/hosts/claude-credential.js.map +1 -0
  200. package/dist/src/hosts/claude-events.js +136 -0
  201. package/dist/src/hosts/claude-events.js.map +1 -0
  202. package/dist/src/hosts/claude-keychain.js +100 -0
  203. package/dist/src/hosts/claude-keychain.js.map +1 -0
  204. package/dist/src/hosts/claude-login-recovery.js +67 -0
  205. package/dist/src/hosts/claude-login-recovery.js.map +1 -0
  206. package/dist/src/hosts/claude-login-refresh.js +146 -0
  207. package/dist/src/hosts/claude-login-refresh.js.map +1 -0
  208. package/dist/src/hosts/claude-login-store.js +124 -0
  209. package/dist/src/hosts/claude-login-store.js.map +1 -0
  210. package/dist/src/hosts/claude-login.js +74 -0
  211. package/dist/src/hosts/claude-login.js.map +1 -0
  212. package/dist/src/hosts/claude-native-goal.js +138 -0
  213. package/dist/src/hosts/claude-native-goal.js.map +1 -0
  214. package/dist/src/hosts/claude-process.js +10 -0
  215. package/dist/src/hosts/claude-process.js.map +1 -0
  216. package/dist/src/hosts/claude-refresh-lock.js +76 -0
  217. package/dist/src/hosts/claude-refresh-lock.js.map +1 -0
  218. package/dist/src/hosts/claude-repository-invocation.js +229 -0
  219. package/dist/src/hosts/claude-repository-invocation.js.map +1 -0
  220. package/dist/src/hosts/claude-settings.js +156 -0
  221. package/dist/src/hosts/claude-settings.js.map +1 -0
  222. package/dist/src/hosts/claude-skill-activations.js +131 -0
  223. package/dist/src/hosts/claude-skill-activations.js.map +1 -0
  224. package/dist/src/hosts/claude-state.js +237 -0
  225. package/dist/src/hosts/claude-state.js.map +1 -0
  226. package/dist/src/hosts/claude-tool-calls.js +150 -0
  227. package/dist/src/hosts/claude-tool-calls.js.map +1 -0
  228. package/dist/src/hosts/claude.js +373 -0
  229. package/dist/src/hosts/claude.js.map +1 -0
  230. package/dist/src/hosts/codex-app-server.js +764 -0
  231. package/dist/src/hosts/codex-app-server.js.map +1 -0
  232. package/dist/src/hosts/codex-configuration.js +91 -0
  233. package/dist/src/hosts/codex-configuration.js.map +1 -0
  234. package/dist/src/hosts/codex-events.js +104 -0
  235. package/dist/src/hosts/codex-events.js.map +1 -0
  236. package/dist/src/hosts/codex-exec-controls.js +434 -0
  237. package/dist/src/hosts/codex-exec-controls.js.map +1 -0
  238. package/dist/src/hosts/codex-invocation.js +54 -0
  239. package/dist/src/hosts/codex-invocation.js.map +1 -0
  240. package/dist/src/hosts/codex-marketplace.js +185 -0
  241. package/dist/src/hosts/codex-marketplace.js.map +1 -0
  242. package/dist/src/hosts/codex-native-calls.js +516 -0
  243. package/dist/src/hosts/codex-native-calls.js.map +1 -0
  244. package/dist/src/hosts/codex-process.js +10 -0
  245. package/dist/src/hosts/codex-process.js.map +1 -0
  246. package/dist/src/hosts/codex-profile.js +206 -0
  247. package/dist/src/hosts/codex-profile.js.map +1 -0
  248. package/dist/src/hosts/codex-session-files.js +89 -0
  249. package/dist/src/hosts/codex-session-files.js.map +1 -0
  250. package/dist/src/hosts/codex-state.js +310 -0
  251. package/dist/src/hosts/codex-state.js.map +1 -0
  252. package/dist/src/hosts/codex.js +527 -0
  253. package/dist/src/hosts/codex.js.map +1 -0
  254. package/dist/src/hosts/isolation-common.js +43 -0
  255. package/dist/src/hosts/isolation-common.js.map +1 -0
  256. package/dist/src/hosts/isolation-roots.js +64 -0
  257. package/dist/src/hosts/isolation-roots.js.map +1 -0
  258. package/dist/src/hosts/isolation.js +12 -0
  259. package/dist/src/hosts/isolation.js.map +1 -0
  260. package/dist/src/hosts/linux-sandbox.js +127 -0
  261. package/dist/src/hosts/linux-sandbox.js.map +1 -0
  262. package/dist/src/hosts/mac-sandbox.js +137 -0
  263. package/dist/src/hosts/mac-sandbox.js.map +1 -0
  264. package/dist/src/hosts/native-controls.js +151 -0
  265. package/dist/src/hosts/native-controls.js.map +1 -0
  266. package/dist/src/hosts/native-request.js +33 -0
  267. package/dist/src/hosts/native-request.js.map +1 -0
  268. package/dist/src/hosts/process-lifecycle.js +167 -0
  269. package/dist/src/hosts/process-lifecycle.js.map +1 -0
  270. package/dist/src/hosts/protected-worktrees.js +81 -0
  271. package/dist/src/hosts/protected-worktrees.js.map +1 -0
  272. package/dist/src/hosts/skill-hook-catalog.js +100 -0
  273. package/dist/src/hosts/skill-hook-catalog.js.map +1 -0
  274. package/dist/src/hosts/skill-hook-handler.js +228 -0
  275. package/dist/src/hosts/skill-hook-handler.js.map +1 -0
  276. package/dist/src/hosts/skill-hooks.js +423 -0
  277. package/dist/src/hosts/skill-hooks.js.map +1 -0
  278. package/dist/src/hosts/workspace-fingerprint.js +72 -0
  279. package/dist/src/hosts/workspace-fingerprint.js.map +1 -0
  280. package/dist/src/identity.js +167 -0
  281. package/dist/src/identity.js.map +1 -0
  282. package/dist/src/instrumentation.js +83 -0
  283. package/dist/src/instrumentation.js.map +1 -0
  284. package/dist/src/native-transcript-state.js +85 -0
  285. package/dist/src/native-transcript-state.js.map +1 -0
  286. package/dist/src/native-transcript-view.js +65 -0
  287. package/dist/src/native-transcript-view.js.map +1 -0
  288. package/dist/src/native-transcripts.js +208 -0
  289. package/dist/src/native-transcripts.js.map +1 -0
  290. package/dist/src/paths.js +13 -0
  291. package/dist/src/paths.js.map +1 -0
  292. package/dist/src/process-lifecycle.js +101 -0
  293. package/dist/src/process-lifecycle.js.map +1 -0
  294. package/dist/src/provenance.js +263 -0
  295. package/dist/src/provenance.js.map +1 -0
  296. package/dist/src/prune.js +228 -0
  297. package/dist/src/prune.js.map +1 -0
  298. package/dist/src/report.js +276 -0
  299. package/dist/src/report.js.map +1 -0
  300. package/dist/src/resolved-case.js +29 -0
  301. package/dist/src/resolved-case.js.map +1 -0
  302. package/dist/src/results-lock.js +40 -0
  303. package/dist/src/results-lock.js.map +1 -0
  304. package/dist/src/results.js +127 -0
  305. package/dist/src/results.js.map +1 -0
  306. package/dist/src/run-owner.js +283 -0
  307. package/dist/src/run-owner.js.map +1 -0
  308. package/dist/src/runner-paths.js +11 -0
  309. package/dist/src/runner-paths.js.map +1 -0
  310. package/dist/src/runtime/config.js +323 -0
  311. package/dist/src/runtime/config.js.map +1 -0
  312. package/dist/src/runtime/discovery-apple.js +65 -0
  313. package/dist/src/runtime/discovery-apple.js.map +1 -0
  314. package/dist/src/runtime/discovery-providers.js +37 -0
  315. package/dist/src/runtime/discovery-providers.js.map +1 -0
  316. package/dist/src/runtime/discovery.js +40 -0
  317. package/dist/src/runtime/discovery.js.map +1 -0
  318. package/dist/src/runtime/hook-processes.js +81 -0
  319. package/dist/src/runtime/hook-processes.js.map +1 -0
  320. package/dist/src/runtime/hooks.js +304 -0
  321. package/dist/src/runtime/hooks.js.map +1 -0
  322. package/dist/src/runtime/paths.js +244 -0
  323. package/dist/src/runtime/paths.js.map +1 -0
  324. package/dist/src/runtime/query.js +76 -0
  325. package/dist/src/runtime/query.js.map +1 -0
  326. package/dist/src/runtime/seeds.js +98 -0
  327. package/dist/src/runtime/seeds.js.map +1 -0
  328. package/dist/src/runtime/shared-cache.js +81 -0
  329. package/dist/src/runtime/shared-cache.js.map +1 -0
  330. package/dist/src/runtime/state.js +106 -0
  331. package/dist/src/runtime/state.js.map +1 -0
  332. package/dist/src/runtime-process.js +148 -0
  333. package/dist/src/runtime-process.js.map +1 -0
  334. package/dist/src/schema-types.js +2 -0
  335. package/dist/src/schema-types.js.map +1 -0
  336. package/dist/src/schema.js +16 -0
  337. package/dist/src/schema.js.map +1 -0
  338. package/dist/src/shell.js +5 -0
  339. package/dist/src/shell.js.map +1 -0
  340. package/dist/src/sqlite-lock.js +18 -0
  341. package/dist/src/sqlite-lock.js.map +1 -0
  342. package/dist/src/storage.js +21 -0
  343. package/dist/src/storage.js.map +1 -0
  344. package/dist/src/transcript-retention.js +109 -0
  345. package/dist/src/transcript-retention.js.map +1 -0
  346. package/dist/src/value-guards.js +26 -0
  347. package/dist/src/value-guards.js.map +1 -0
  348. package/docs/README.md +19 -15
  349. package/docs/architecture.md +141 -17
  350. package/docs/assets/README.md +5 -4
  351. package/docs/builtin-graders-v1.md +9 -2
  352. package/docs/codex-host.md +155 -57
  353. package/docs/contracts.md +2 -1
  354. package/docs/creating-cases.md +10 -4
  355. package/docs/development-cli.md +197 -24
  356. package/docs/documentation-quality.md +16 -5
  357. package/docs/evaluation-workflow.md +110 -0
  358. package/docs/extension-protocol-v1.md +26 -3
  359. package/docs/getting-started.md +29 -7
  360. package/docs/guide-evaluation.md +34 -2
  361. package/docs/identity-v1.md +29 -3
  362. package/docs/installing.md +35 -17
  363. package/docs/licensing.md +2 -2
  364. package/docs/native-controls.md +76 -5
  365. package/docs/native-hosts.md +197 -8
  366. package/docs/reading-results.md +102 -0
  367. package/docs/recipes.md +490 -0
  368. package/docs/releases.md +48 -13
  369. package/docs/report-v1.md +29 -1
  370. package/docs/results-v1.md +122 -4
  371. package/docs/run-settings-v1.md +147 -0
  372. package/docs/runtime-v1.md +255 -22
  373. package/docs/specs/repository-guide.md +1 -1
  374. package/docs/troubleshooting.md +12 -12
  375. package/docs/typescript-quality.md +106 -26
  376. package/examples/recipes/candidate.ts +26 -0
  377. package/examples/recipes/cases/coding-change.json +21 -0
  378. package/examples/recipes/cases/host-tools.json +21 -0
  379. package/examples/recipes/cases/mise.json +21 -0
  380. package/examples/recipes/cases/node.json +21 -0
  381. package/examples/recipes/cases/python-uv.json +21 -0
  382. package/examples/recipes/cases/semantic.json +17 -0
  383. package/examples/recipes/coding-change/check.ts +23 -0
  384. package/examples/recipes/executable.ts +19 -0
  385. package/examples/recipes/extension-artifacts.ts +65 -0
  386. package/examples/recipes/extension-methods.ts +174 -0
  387. package/examples/recipes/extension.ts +38 -0
  388. package/examples/recipes/modules.ts +9 -0
  389. package/examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md +7 -0
  390. package/examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md +7 -0
  391. package/examples/recipes/repository-skill/README.md +3 -0
  392. package/examples/recipes/run-inputs.ts +183 -0
  393. package/examples/recipes/run-options.ts +132 -0
  394. package/examples/recipes/run.ts +77 -0
  395. package/examples/recipes/runtime/empty.json +1 -0
  396. package/examples/recipes/runtime/goals.json +1 -0
  397. package/examples/recipes/runtime/mise.json +6 -0
  398. package/examples/recipes/runtime/path.json +1 -0
  399. package/examples/recipes/runtime/plugins.json +4 -0
  400. package/examples/recipes/runtime/python-uv.json +10 -0
  401. package/examples/recipes/semantic-judge.ts +29 -0
  402. package/examples/templates/coding/adapter.ts +17 -0
  403. package/examples/templates/coding/case.json +15 -0
  404. package/examples/templates/repository-skill/.agents/skills/startup-guide/SKILL.md +6 -0
  405. package/examples/templates/repository-skill/extension-data.ts +64 -0
  406. package/examples/templates/repository-skill/extension.ts +32 -0
  407. package/examples/templates/response/adapter.ts +12 -0
  408. package/examples/templates/response/case.json +13 -0
  409. package/package.json +23 -9
  410. package/schemas/report-v1.schema.json +9 -0
  411. package/schemas/run-evidence-v1.schema.json +2 -1
  412. package/schemas/runtime-v1.schema.json +43 -7
  413. package/src/advisory.ts +2 -2
  414. package/src/candidate-artifacts.ts +42 -11
  415. package/src/candidate-codex-state.ts +185 -0
  416. package/src/case-validation.ts +73 -50
  417. package/src/cli/cancellation.ts +20 -0
  418. package/src/cli/command.ts +87 -0
  419. package/src/cli/completion.ts +15 -0
  420. package/src/cli/demo.ts +61 -0
  421. package/src/cli/evaluation.ts +55 -0
  422. package/src/cli/extension.ts +79 -0
  423. package/src/cli/help.ts +199 -0
  424. package/src/cli/hosts.ts +55 -0
  425. package/src/cli/init-settings.ts +31 -0
  426. package/src/cli/init.ts +112 -0
  427. package/src/{cli-invocation.ts → cli/invocation.ts} +215 -218
  428. package/src/cli/loading.ts +151 -0
  429. package/src/cli/preparation.ts +154 -0
  430. package/src/cli/presentation.ts +93 -0
  431. package/src/cli/progress.ts +6 -0
  432. package/src/cli/run-manifest.ts +128 -0
  433. package/src/cli/run-options.ts +403 -0
  434. package/src/cli/run-overrides.ts +69 -0
  435. package/src/cli/settings.ts +154 -0
  436. package/src/cli/summary.ts +43 -0
  437. package/src/cli.ts +14 -592
  438. package/src/contract-types.ts +94 -0
  439. package/src/engine.ts +9 -5
  440. package/src/evaluation/advisory.ts +150 -0
  441. package/src/{evaluation-check-plan.ts → evaluation/check-plan.ts} +103 -67
  442. package/src/evaluation/collection.ts +88 -0
  443. package/src/evaluation/context.ts +10 -0
  444. package/src/{evaluation-grading-host.ts → evaluation/grading-host.ts} +5 -5
  445. package/src/evaluation/grading.ts +235 -0
  446. package/src/{evaluation-invocation-plan.ts → evaluation/invocation-plan.ts} +147 -58
  447. package/src/{evaluation-plan.ts → evaluation/plan.ts} +208 -141
  448. package/src/evaluation/progress.ts +19 -0
  449. package/src/evaluation/resources.ts +241 -0
  450. package/src/evaluation/run.ts +206 -0
  451. package/src/{evaluation-semantic.ts → evaluation/semantic.ts} +77 -87
  452. package/src/evaluation/trial.ts +800 -0
  453. package/src/{evaluation-types.ts → evaluation/types.ts} +79 -29
  454. package/src/evidence/host-results.ts +85 -0
  455. package/src/evidence/retained-artifacts.ts +18 -0
  456. package/src/evidence/trial-writer.ts +203 -0
  457. package/src/evidence/usage.ts +17 -0
  458. package/src/executable.ts +19 -0
  459. package/src/extension-client.ts +40 -108
  460. package/src/extension-session.ts +24 -48
  461. package/src/{advisory-fixture.ts → fixtures/advisory.ts} +12 -72
  462. package/src/{fixture-bin.ts → fixtures/bin.ts} +1 -1
  463. package/src/{fixture-cleanup.ts → fixtures/cleanup.ts} +13 -7
  464. package/src/fixtures/create.ts +183 -0
  465. package/src/{generated-fixture.ts → fixtures/generated.ts} +10 -53
  466. package/src/{git-hooks.ts → fixtures/git-hooks.ts} +1 -1
  467. package/src/{preparation.ts → fixtures/preparation.ts} +6 -16
  468. package/src/{repository-fixture.ts → fixtures/repository.ts} +30 -64
  469. package/src/{fixture-setup.ts → fixtures/setup.ts} +36 -53
  470. package/src/{fixture-tools.ts → fixtures/tools.ts} +1 -1
  471. package/src/{workspace-state.ts → fixtures/workspace-state.ts} +5 -5
  472. package/src/generated/cli-result.d.cts +2 -1
  473. package/src/generated/extension.d.cts +2 -1
  474. package/src/generated/report.cjs +1 -1
  475. package/src/generated/report.d.cts +2 -1
  476. package/src/generated/run-evidence.cjs +1 -1
  477. package/src/generated/run-evidence.d.cts +2 -1
  478. package/src/generated/runtime.cjs +1 -1
  479. package/src/generated/runtime.d.cts +2 -1
  480. package/src/git.ts +94 -0
  481. package/src/graders/git-head.ts +31 -85
  482. package/src/graders/shell.ts +54 -71
  483. package/src/host-diagnostic-redaction.ts +52 -0
  484. package/src/host-diagnostics.ts +331 -0
  485. package/src/hosts/claude-continuation.ts +57 -40
  486. package/src/hosts/claude-credential.ts +160 -63
  487. package/src/hosts/claude-events.ts +34 -18
  488. package/src/hosts/claude-keychain.ts +79 -21
  489. package/src/hosts/claude-login-recovery.ts +85 -0
  490. package/src/hosts/claude-login-refresh.ts +179 -0
  491. package/src/hosts/claude-login-store.ts +162 -0
  492. package/src/hosts/claude-login.ts +107 -0
  493. package/src/hosts/claude-native-goal.ts +15 -12
  494. package/src/hosts/claude-process.ts +10 -50
  495. package/src/hosts/claude-refresh-lock.ts +82 -0
  496. package/src/hosts/claude-repository-invocation.ts +9 -34
  497. package/src/hosts/claude-settings.ts +46 -9
  498. package/src/hosts/claude-skill-activations.ts +169 -0
  499. package/src/hosts/claude-state.ts +28 -15
  500. package/src/hosts/claude-tool-calls.ts +13 -10
  501. package/src/hosts/claude.ts +121 -81
  502. package/src/hosts/codex-app-server.ts +78 -52
  503. package/src/hosts/codex-configuration.ts +5 -10
  504. package/src/hosts/codex-exec-controls.ts +516 -0
  505. package/src/hosts/codex-marketplace.ts +12 -10
  506. package/src/hosts/codex-native-calls.ts +15 -122
  507. package/src/hosts/codex-process.ts +10 -183
  508. package/src/hosts/codex-profile.ts +36 -21
  509. package/src/hosts/codex-session-files.ts +1 -9
  510. package/src/hosts/codex-state.ts +134 -18
  511. package/src/hosts/codex.ts +210 -183
  512. package/src/hosts/isolation-common.ts +8 -11
  513. package/src/hosts/isolation-roots.ts +5 -4
  514. package/src/hosts/linux-sandbox.ts +15 -2
  515. package/src/hosts/mac-sandbox.ts +87 -18
  516. package/src/hosts/native-controls.ts +72 -22
  517. package/src/hosts/native-request.ts +55 -0
  518. package/src/hosts/process-lifecycle.ts +215 -49
  519. package/src/hosts/protected-worktrees.ts +3 -2
  520. package/src/hosts/skill-hook-catalog.ts +131 -0
  521. package/src/hosts/skill-hook-handler.ts +374 -0
  522. package/src/hosts/skill-hooks.ts +585 -0
  523. package/src/instrumentation.ts +7 -8
  524. package/src/native-transcript-state.ts +22 -3
  525. package/src/native-transcript-view.ts +52 -13
  526. package/src/native-transcripts.ts +26 -11
  527. package/src/paths.ts +22 -0
  528. package/src/process-lifecycle.ts +150 -0
  529. package/src/provenance.ts +34 -13
  530. package/src/prune.ts +300 -0
  531. package/src/report.ts +63 -36
  532. package/src/resolved-case.ts +1 -1
  533. package/src/results-lock.ts +42 -0
  534. package/src/results.ts +14 -18
  535. package/src/run-owner.ts +15 -9
  536. package/src/runner-paths.ts +21 -0
  537. package/src/{runtime-config.ts → runtime/config.ts} +160 -46
  538. package/src/{runtime-discovery-apple.ts → runtime/discovery-apple.ts} +3 -3
  539. package/src/{runtime-discovery-providers.ts → runtime/discovery-providers.ts} +3 -9
  540. package/src/{runtime-discovery.ts → runtime/discovery.ts} +4 -4
  541. package/src/{runtime-hook-processes.ts → runtime/hook-processes.ts} +3 -2
  542. package/src/{runtime-hooks.ts → runtime/hooks.ts} +40 -28
  543. package/src/{runtime-paths.ts → runtime/paths.ts} +5 -19
  544. package/src/{runtime-query.ts → runtime/query.ts} +5 -5
  545. package/src/{runtime-seeds.ts → runtime/seeds.ts} +3 -3
  546. package/src/runtime/shared-cache.ts +130 -0
  547. package/src/{runtime-state.ts → runtime/state.ts} +28 -11
  548. package/src/runtime-process.ts +194 -0
  549. package/src/schema-types.ts +73 -161
  550. package/src/shell.ts +4 -0
  551. package/src/sqlite-lock.ts +25 -0
  552. package/src/transcript-retention.ts +146 -0
  553. package/src/value-guards.ts +22 -0
  554. package/docs/assets/sevro-logo.png +0 -0
  555. package/docs/assets/sevro-symbol.svg +0 -1
  556. package/docs/assets/sevro-wordmark.svg +0 -1
  557. package/docs/cache-ownership-validation.md +0 -49
  558. package/docs/claude-continuation-validation.md +0 -47
  559. package/docs/concurrency-validation.md +0 -86
  560. package/docs/configuration-root-validation.md +0 -70
  561. package/docs/engine-slice.md +0 -114
  562. package/docs/evidence/runtime-startup-30/README.md +0 -185
  563. package/docs/evidence/runtime-startup-30/after.json +0 -319
  564. package/docs/evidence/runtime-startup-30/baseline-timeout.jsonl +0 -2
  565. package/docs/evidence/runtime-startup-30/before.json +0 -403
  566. package/docs/evidence/runtime-startup-30/consumer-smoke.json +0 -63
  567. package/docs/evidence/runtime-startup-30/full-policy-diagnostic.jsonl +0 -3
  568. package/docs/evidence/runtime-startup-30/minimal-policy-diagnostic.jsonl +0 -3
  569. package/docs/evidence/runtime-startup-30/plugin-template-assessment.json +0 -20
  570. package/docs/evidence/runtime-startup-30/seedless-timeout.jsonl +0 -2
  571. package/docs/evidence.md +0 -17
  572. package/docs/fixture-cleanup-validation.md +0 -73
  573. package/docs/guide-verification.json +0 -239
  574. package/src/evaluation-context.ts +0 -259
  575. package/src/evaluation-fixture.ts +0 -302
  576. package/src/evaluation-run.ts +0 -200
  577. package/src/evaluation-trial.ts +0 -1016
  578. package/src/hosts/claude-nested-skills.ts +0 -272
  579. package/src/hosts/codex-command-output.ts +0 -510
  580. package/src/hosts/codex-executor-command.ts +0 -291
  581. package/src/hosts/codex-skill-reads.ts +0 -783
  582. /package/src/{trial-scheduler.ts → evaluation/scheduler.ts} +0 -0
@@ -0,0 +1,490 @@
1
+ # Run complete evaluation recipes
2
+
3
+ Copy a recipe, run its positive and negative controls, then select a native
4
+ model. The complete projects live in [examples/recipes](../examples/recipes).
5
+ They include cases, fixtures, evaluator checks, runtime files, adapters, and a
6
+ small extension.
7
+
8
+ ## Prepare a recipe project
9
+
10
+ Use Node.js 24.13.0 or newer, Git, and `rg`. Bun is optional for the installed
11
+ recipes; the same JavaScript files also run with Bun 1.3.13. Shell checks need macOS
12
+ `sandbox-exec` or Ubuntu `bubblewrap`; native Claude also needs `socat` on
13
+ Ubuntu. Python/UV and mise add the prerequisites stated below.
14
+ [Install Sevro](installing.md), then set the recipe path from the directory
15
+ containing its local installation:
16
+
17
+ ```sh
18
+ export SEVRO_RECIPES="$PWD/node_modules/@bjoernrochel/sevro/dist/examples/recipes"
19
+ ```
20
+
21
+ Run commands below from an otherwise empty working directory to keep generated
22
+ outputs together. The launcher uses
23
+ the CLI beside its original package; it does not require a global installation.
24
+
25
+ To adapt a separate copy, preserve the small project's whole directory:
26
+
27
+ ```sh
28
+ export SEVRO_RECIPE_CLI="$(cd "$SEVRO_RECIPES/../../src" && pwd)/cli.js"
29
+ cp -R "$SEVRO_RECIPES" ./my-recipes
30
+ export SEVRO_RECIPES="$PWD/my-recipes"
31
+ ```
32
+
33
+ Each command saves `result.json`, the materialized case or extension command,
34
+ and retained evidence under `.sevro-recipes/<recipe>/<route>/<mode>/` in your
35
+ current directory. The project snapshot is separate from recipe inputs and
36
+ results. Stderr names the exact result path and effective settings; stdout is
37
+ one `sevro.cli-result.v1` JSON document. Follow its absolute `evidencePath`.
38
+ The launcher supplies explicit source, project, config, runtime, isolation,
39
+ condition, trial and threshold options; inspect [run.ts](../examples/recipes/run.ts)
40
+ and the saved inputs when adapting a plain CLI invocation. Existing results
41
+ remain until you remove their directory.
42
+
43
+ For every deterministic positive control, expect exit `0`, execution
44
+ `completed`, grading `completed`, task `passed`. Each separate `--fail` command
45
+ deliberately exits `1` with execution/grading `completed` and task `failed`.
46
+ Run negative controls separately when using a shell with `set -e`. A missing
47
+ tool is a prerequisite error, not a passing or intentionally failing control.
48
+
49
+ `candidate.js` (built from `candidate.ts`) is an explicitly trusted deterministic fixture editor. It changes
50
+ the arithmetic implementation for the coding recipe and supplies a fixed answer
51
+ for the other controls. It does not simulate native tool permissions: the real
52
+ isolated shell graders execute the stated tools and assertions. Native routes
53
+ run the actual task with the same fixture and runtime policy.
54
+
55
+ For any recipe that offers a deterministic control, replace `deterministic`
56
+ with `codex` or `claude` to use a real participant. First install/authenticate
57
+ the selected [native CLI](native-hosts.md). For Codex, set the model to an
58
+ available identifier and supply your private authentication file path:
59
+
60
+ ```sh
61
+ export SEVRO_RECIPE_MODEL='replace-with-an-available-Codex-model'
62
+ export SEVRO_RECIPE_CODEX_AUTH="${CODEX_HOME:-$HOME/.codex}/auth.json"
63
+ node "$SEVRO_RECIPES/run.js" coding-change codex --dry
64
+ node "$SEVRO_RECIPES/run.js" coding-change codex
65
+ ```
66
+
67
+ For Claude, use its supported credential route and a Claude model identifier:
68
+
69
+ ```sh
70
+ export SEVRO_RECIPE_MODEL='replace-with-an-available-Claude-model'
71
+ node "$SEVRO_RECIPES/run.js" coding-change claude --dry
72
+ node "$SEVRO_RECIPES/run.js" coding-change claude
73
+ ```
74
+
75
+ Substitute the other recipe name to run its native task. `--dry` prepares and
76
+ retains inputs without calling a model; remove it for execution. Add `--fail`
77
+ to a live command for its deliberate negative control. Native model outcomes
78
+ must be observed; a model can ignore a negative-control instruction. The
79
+ semantic recipe additionally requires its judge route described below.
80
+
81
+ ## Fix a coding bug with hidden evaluator checks
82
+
83
+ Files: [coding-change case](../examples/recipes/cases/coding-change.json),
84
+ [typed evaluator test](../examples/recipes/coding-change/check.ts),
85
+ [tool configuration](../examples/recipes/runtime/path.json), and
86
+ [candidate](../examples/recipes/candidate.ts). Prerequisite: Node on `PATH`.
87
+ The fixture contains a broken `math.cjs` and a task README. The task asks the
88
+ participant to implement addition. The evaluator checks positive, negative and
89
+ zero arguments. Its source stays outside the candidate fixture: the launcher
90
+ reads the compiled evaluator test and embeds its JavaScript in the
91
+ evaluator-owned shell command before execution.
92
+
93
+ <!-- sevro-example:recipe-coding-change-pass -->
94
+
95
+ ```sh
96
+ node "$SEVRO_RECIPES/run.js" coding-change deterministic
97
+ ```
98
+
99
+ The isolated check first asserts that the original evaluator test is unreadable,
100
+ then runs Node against the changed fixture. The passing shell observation
101
+ retains the result and stdout digest. The negative control leaves subtraction
102
+ in place, so the arithmetic assertion fails:
103
+
104
+ <!-- sevro-example:recipe-coding-change-fail -->
105
+
106
+ ```sh
107
+ node "$SEVRO_RECIPES/run.js" coding-change deterministic --fail
108
+ ```
109
+
110
+ Use a native route below to evaluate an actual coding agent. Adding a participant
111
+ test to the fixture would make it visible; it would not replace this hidden
112
+ evaluator-owned check. The launcher selects `--shell-isolation` and protects
113
+ the recipe source; declare additional private worktrees through the
114
+ [CLI's protected roots](development-cli.md).
115
+
116
+ ## Add host-tool access, one capability at a time
117
+
118
+ The smallest useful runtime file is
119
+ [runtime/empty.json](../examples/recipes/runtime/empty.json):
120
+
121
+ ```json
122
+ { "format": "sevro.runtime.v1" }
123
+ ```
124
+
125
+ It selects the existing isolation defaults without importing tools or hooks.
126
+ Add only a declared non-secret `PATH` to reuse installed tools:
127
+
128
+ ```json
129
+ {
130
+ "format": "sevro.runtime.v1",
131
+ "environment": { "inherit": ["PATH"] }
132
+ }
133
+ ```
134
+
135
+ Sevro filters credential-related PATH entries, discovers bounded installation
136
+ support trees, grants read-only tool access, and keeps private writable home
137
+ and temporary directories. It does not grant the full operator home. Runtime
138
+ values enter retained evidence; do not put secrets in this file. See
139
+ [runtime details](runtime-v1.md#environment-and-host-tools).
140
+
141
+ ### Minimal tool: ripgrep
142
+
143
+ Files: [host-tools case](../examples/recipes/cases/host-tools.json) and
144
+ [PATH runtime](../examples/recipes/runtime/path.json). Prerequisite: standalone `rg` on
145
+ `PATH`, installed outside Codex/Claude credential directories (for example,
146
+ `brew install ripgrep` on macOS or `apt-get install ripgrep` on Ubuntu).
147
+ Sevro deliberately filters a Codex session's injected `~/.codex/.../rg` path;
148
+ that embedded executable alone does not satisfy this recipe. The fixture's `app.env` contains `port=3000`; an actual isolated `rg`
149
+ process must find exactly that line.
150
+
151
+ <!-- sevro-example:recipe-host-tools-pass -->
152
+
153
+ ```sh
154
+ node "$SEVRO_RECIPES/run.js" host-tools deterministic
155
+ ```
156
+
157
+ The negative fixture contains port `9000`, so that same search fails:
158
+
159
+ <!-- sevro-example:recipe-host-tools-fail -->
160
+
161
+ ```sh
162
+ node "$SEVRO_RECIPES/run.js" host-tools deterministic --fail
163
+ ```
164
+
165
+ ### Node
166
+
167
+ Files: [Node case](../examples/recipes/cases/node.json) and the same PATH runtime.
168
+ Prerequisite: Node 24 on `PATH`. The fixture contains `app.json`. A real isolated
169
+ Node process parses it, asserts the port, and prints `3000`.
170
+
171
+ <!-- sevro-example:recipe-node-pass -->
172
+
173
+ ```sh
174
+ node "$SEVRO_RECIPES/run.js" node deterministic
175
+ ```
176
+
177
+ The negative fixture sets port `9000`, causing the Node assertion to fail:
178
+
179
+ <!-- sevro-example:recipe-node-fail -->
180
+
181
+ ```sh
182
+ node "$SEVRO_RECIPES/run.js" node deterministic --fail
183
+ ```
184
+
185
+ ### Python with UV
186
+
187
+ Files: [Python/UV case](../examples/recipes/cases/python-uv.json) and
188
+ [UV runtime](../examples/recipes/runtime/python-uv.json). Install `uv` and an
189
+ existing Python 3 first; CI uses UV `0.11.21`. Both executables must be on
190
+ `PATH`. The launcher selects `python3`; optionally export `RECIPE_PYTHON` to
191
+ the absolute existing interpreter you want. Put that interpreter's installation
192
+ bin on `PATH` too, so its bounded support installation can be discovered.
193
+
194
+ This adds a private writable `UV_CACHE_DIR` and disables Python downloads.
195
+ `uv run --offline --no-project` runs the selected existing interpreter, with no
196
+ dependency resolution or network. Python parses the fixture JSON, asserts its
197
+ port and prints `3000`:
198
+
199
+ <!-- sevro-example:recipe-python-uv-pass -->
200
+
201
+ ```sh
202
+ node "$SEVRO_RECIPES/run.js" python-uv deterministic
203
+ ```
204
+
205
+ The negative fixture changes the port to `9000`; Python raises an assertion:
206
+
207
+ <!-- sevro-example:recipe-python-uv-fail -->
208
+
209
+ ```sh
210
+ node "$SEVRO_RECIPES/run.js" python-uv deterministic --fail
211
+ ```
212
+
213
+ For real dependencies, prepare a public cache once outside the evaluated project
214
+ and declare a [shared cache](runtime-v1.md#reuse-a-shared-dependency-cache), then
215
+ use your locked offline install command. Select only the roles using UV; add
216
+ `checks` only when the grader needs it. The same directory can serve candidates,
217
+ hooks, and UV-based graders across trials and runs without copying the cache.
218
+ Use [private seeds](runtime-v1.md#private-writable-state) when isolated dependency
219
+ bytes matter more than disk reuse. This dependency-free recipe needs no cache seed.
220
+
221
+ ### An existing mise installation
222
+
223
+ Files: [mise case](../examples/recipes/cases/mise.json),
224
+ [runtime](../examples/recipes/runtime/mise.json), and the selection in
225
+ [run-options.ts](../examples/recipes/run-options.ts). Install mise first; CI
226
+ uses `2026.3.13`. Prepare its exact tool before evaluation:
227
+
228
+ ```sh
229
+ mise install node@24.13.0
230
+ mise where node@24.13.0
231
+ ```
232
+
233
+ The launcher asks `mise where node@24.13.0`, prepends that installation's `bin`
234
+ to the declared PATH, then runs its Node inside isolation. The evaluator checks
235
+ the exact Node version and fixture port and prints `3000`. This route executes
236
+ the resolved versioned tool; it does not run a mutable mise shim inside the
237
+ sandbox or grant mise's whole data directory.
238
+
239
+ <!-- sevro-example:recipe-mise-pass -->
240
+
241
+ ```sh
242
+ node "$SEVRO_RECIPES/run.js" mise deterministic
243
+ ```
244
+
245
+ The negative fixture changes the port to `9000`, failing the same Node check:
246
+
247
+ <!-- sevro-example:recipe-mise-fail -->
248
+
249
+ ```sh
250
+ node "$SEVRO_RECIPES/run.js" mise deterministic --fail
251
+ ```
252
+
253
+ An attempted isolated `mise exec` with read-only operator data returned
254
+ `Operation not permitted` on macOS. If your shim needs extra discovery or
255
+ writable state, resolve the existing installation before launch as above.
256
+ Do not solve that failure by granting the operator's whole home.
257
+
258
+ ## Evaluate a repository skill with Codex or Claude
259
+
260
+ Files: the complete [sample repository](../examples/recipes/repository-skill/README.md),
261
+ its [Codex skill](../examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md),
262
+ its [Claude skill](../examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md),
263
+ and the [extension](../examples/recipes/extension.ts). The extension builds a
264
+ fresh committed Git fixture and mounts the selected skill as a Git-excluded
265
+ artifact. The README is participant-visible; checks and extension source stay
266
+ outside. Copy these actual skill bodies and their supporting files when
267
+ substituting your own skill.
268
+
269
+ Install the [native host prerequisites](native-hosts.md). Dry preparation needs
270
+ the selected CLI on PATH, but no model login or quota. It negotiates repository
271
+ invocation, mounts and hashes the actual skill, validates the case and runtime,
272
+ and retains the artifacts. Expect exit `0`, execution `not_run`, grading
273
+ `not_requested`, task `not_assessed`:
274
+
275
+ <!-- sevro-example:recipe-repository-skill-codex-dry -->
276
+
277
+ ```sh
278
+ node "$SEVRO_RECIPES/run.js" repository-skill codex --dry
279
+ ```
280
+
281
+ <!-- sevro-example:recipe-repository-skill-claude-dry -->
282
+
283
+ ```sh
284
+ node "$SEVRO_RECIPES/run.js" repository-skill claude --dry
285
+ ```
286
+
287
+ For a real Codex trial, export `SEVRO_RECIPE_MODEL` to an available Codex model
288
+ and `SEVRO_RECIPE_CODEX_AUTH` to your private absolute `auth.json` path, then
289
+ remove `--dry`. For Claude, export `SEVRO_RECIPE_MODEL` to an available Claude
290
+ model and authenticate through one of its documented credential routes, then
291
+ remove `--dry`. Credentials stay outside recipe/project inputs and argv.
292
+ These calls consume account quota.
293
+
294
+ ```sh
295
+ node "$SEVRO_RECIPES/run.js" repository-skill codex
296
+ node "$SEVRO_RECIPES/run.js" repository-skill codex --fail
297
+ ```
298
+
299
+ With `SEVRO_RECIPE_MODEL` set for Claude instead:
300
+
301
+ ```sh
302
+ node "$SEVRO_RECIPES/run.js" repository-skill claude
303
+ node "$SEVRO_RECIPES/run.js" repository-skill claude --fail
304
+ ```
305
+
306
+ The neutral invocation placeholder becomes `$startup-guide` for Codex and
307
+ `/startup-guide` at the start of Claude's prompt. Claude uses the required
308
+ `--claude-project-settings`; its runtime leaves goals/hooks disabled. Passing
309
+ requires answer content (command and port), the prepared-file assertion, and a
310
+ complete explicit-invocation receipt. That receipt proves the bounded dispatch,
311
+ not general grounding or quality across other questions.
312
+
313
+ The deliberate failure is the same command without `--dry`, with `--fail`
314
+ added. It explicitly requests the wrong response; expected task verdict is
315
+ `failed`, exit `1`. An actual model can disregard a negative-control prompt;
316
+ inspect its observed answer and receipt if it unexpectedly passes. Dry
317
+ `--fail` only validates that negative-control preparation; it does not observe
318
+ model failure.
319
+
320
+ ## Add a separate semantic judge
321
+
322
+ Files: [semantic case](../examples/recipes/cases/semantic.json), candidate
323
+ adapter and [protocol judge](../examples/recipes/semantic-judge.ts). The case
324
+ asks for the startup explanation and declares a proposition requiring the
325
+ command and port. These credential-free controls exercise routing, source
326
+ construction, verdict parsing, and retained semantic evidence:
327
+
328
+ <!-- sevro-example:recipe-semantic-pass -->
329
+
330
+ ```sh
331
+ node "$SEVRO_RECIPES/run.js" semantic deterministic
332
+ ```
333
+
334
+ <!-- sevro-example:recipe-semantic-fail -->
335
+
336
+ ```sh
337
+ node "$SEVRO_RECIPES/run.js" semantic deterministic --fail
338
+ ```
339
+
340
+ The protocol judge derives pass/fail from the actual candidate-response field
341
+ using an exact fixture oracle. It is not a model and does not establish
342
+ qualitative judgment quality. The negative response gives the wrong command
343
+ and port; semantic grading completes with a failing proposition.
344
+
345
+ For an actual model candidate and judge, set the native variables above and
346
+ export `SEVRO_RECIPE_JUDGE_MODEL` to an available Codex judge model:
347
+
348
+ ```sh
349
+ node "$SEVRO_RECIPES/run.js" semantic codex --dry
350
+ node "$SEVRO_RECIPES/run.js" semantic codex
351
+ ```
352
+
353
+ `semantic claude` selects a Claude candidate with a separate Codex judge;
354
+ it needs both CLIs and the Codex auth path. A correct response is expected to
355
+ pass; add `--fail` for the deliberately wrong-response task and an expected
356
+ failed verdict. Live verdicts remain observations, not guarantees.
357
+
358
+ This response-only recipe makes one candidate model call and one additional
359
+ judge call per completed trial. The judge receives all its propositions in an
360
+ empty temporary workspace. Separate document sources can add further calls;
361
+ missing candidate output can avoid a judge call. Judge usage and raw structured
362
+ verdicts are retained separately. A malformed or incomplete judge response is
363
+ a grading error, not a failed proposition or a passing task. Read
364
+ [semantic grading](builtin-graders-v1.md) before adapting its oracle.
365
+
366
+ ## Run the complete extension lifecycle
367
+
368
+ Files: [extension.ts](../examples/recipes/extension.ts),
369
+ [methods](../examples/recipes/extension-methods.ts), and
370
+ [artifact preparation](../examples/recipes/extension-artifacts.ts). Prerequisites:
371
+ Node or Bun, Git and the platform's shell isolation tool. No model is required:
372
+
373
+ <!-- sevro-example:recipe-lifecycle-pass -->
374
+
375
+ ```sh
376
+ node "$SEVRO_RECIPES/run.js" lifecycle deterministic
377
+ ```
378
+
379
+ `describe` advertises version/capabilities/grader; `resolve` returns the task
380
+ and committed fixture; `prepare` supplies hashed Git-excluded `startup.txt`;
381
+ the candidate runs; an isolated built-in check reads the mounted bytes; then
382
+ `evaluate` checks the complete final-message evidence for port `3000` and cites
383
+ that observation. The default task policy combines these checks. Evidence is
384
+ saved before cleanup. Every extension method runs in a fresh process, so prior
385
+ process globals are not state. Stdout contains only its matching protocol
386
+ response; diagnostics belong on stderr.
387
+
388
+ <!-- sevro-example:recipe-lifecycle-fail -->
389
+
390
+ ```sh
391
+ node "$SEVRO_RECIPES/run.js" lifecycle deterministic --fail
392
+ ```
393
+
394
+ The wrong final answer fails the built-in output assertion and extension check,
395
+ while preparation's shell check still passes. The extension cannot turn missing
396
+ evidence or a failed exchange into success. Inspect the saved argv command and
397
+ source closure, including its skill resources, to adapt it into your own
398
+ [trusted extension](extension-protocol-v1.md).
399
+
400
+ ## Pair plugins, goals, and hooks with preparation
401
+
402
+ Files: the same extension and artifact builder, plus
403
+ [goals runtime](../examples/recipes/runtime/goals.json) and
404
+ [selected-plugin runtime](../examples/recipes/runtime/plugins.json).
405
+ Prerequisites: a native CLI with plugin and native-goal support; use the
406
+ [checked host versions](typescript-quality.md#environment-and-ci). The plugin
407
+ manifests, skill and SessionStart command hook are all prepared artifacts, not
408
+ an undeclared global installation. The Codex package uses a local marketplace;
409
+ Claude uses declared plugin directories. The extension explicitly invokes the
410
+ prepared `startup:startup-guide` skill.
411
+
412
+ First add only native goals to the empty runtime and validate preparation:
413
+
414
+ <!-- sevro-example:recipe-goals-codex-dry -->
415
+
416
+ ```sh
417
+ node "$SEVRO_RECIPES/run.js" plugins codex --dry --goals-only
418
+ ```
419
+
420
+ <!-- sevro-example:recipe-goals-claude-dry -->
421
+
422
+ ```sh
423
+ node "$SEVRO_RECIPES/run.js" plugins claude --dry --goals-only
424
+ ```
425
+
426
+ Then add `hooks.plugins: ["startup"]` so only this prepared plugin's command
427
+ handlers are authorized. Codex selects app-server for its native goal route;
428
+ Claude uses plugin-only settings, so do not add `--claude-project-settings`:
429
+
430
+ <!-- sevro-example:recipe-plugins-codex-dry -->
431
+
432
+ ```sh
433
+ node "$SEVRO_RECIPES/run.js" plugins codex --dry
434
+ ```
435
+
436
+ <!-- sevro-example:recipe-plugins-claude-dry -->
437
+
438
+ ```sh
439
+ node "$SEVRO_RECIPES/run.js" plugins claude --dry
440
+ ```
441
+
442
+ All four dry commands expect `not_run`/`not_requested`/`not_assessed`, exit `0`.
443
+ They validate preparation and retain its actual hashed files. They do not
444
+ install a plugin into a live native host, execute its hook, or observe a goal.
445
+ `--goals-only` is deliberately a preparation step; the final recipe's hook
446
+ assertion requires the subsequent full policy.
447
+
448
+ With the native model/credentials configured, remove `--dry` from the full
449
+ plugin command. Expect `passed`, exit `0`: the selected SessionStart hook
450
+ writes `hook-ran.txt`, an isolated shell assertion reads `hook ran`, and the
451
+ skill answer supplies the startup command/port. The hook runs credential-free
452
+ with network denied and only the prepared plugin subtree readable.
453
+ `hooks.nativeGoal` enables the native goal facility; inspect retained
454
+ `sevro.host.native-goal` observations for actual creation/status. Enabling the
455
+ facility or passing this startup task alone does not prove goal completion.
456
+
457
+ ```sh
458
+ node "$SEVRO_RECIPES/run.js" plugins codex
459
+ node "$SEVRO_RECIPES/run.js" plugins codex --fail
460
+ ```
461
+
462
+ Or, with the Claude model/credential route configured:
463
+
464
+ ```sh
465
+ node "$SEVRO_RECIPES/run.js" plugins claude
466
+ node "$SEVRO_RECIPES/run.js" plugins claude --fail
467
+ ```
468
+
469
+ For a deliberate failure, run that live command with `--fail`; the requested
470
+ wrong response should fail the answer checks while the hook assertion passes.
471
+ Removing `startup` from the runtime's allowed plugins is a separate diagnostic:
472
+ the command hook is filtered and the hook-file assertion fails. Native/model
473
+ behavior must be observed before claiming either live outcome.
474
+
475
+ ## What tutorial verification proves
476
+
477
+ `bun run test:docs-examples` packs this candidate, copies the packaged recipe
478
+ project to a fresh consumer directory, and executes every marked deterministic
479
+ positive/negative command. It runs real isolated `rg`, Node, UV/Python and
480
+ mise-selected Node checks, the hidden arithmetic test, semantic protocol
481
+ control, and the entire extension lifecycle. Missing UV, Python, mise or its
482
+ declared Node installation fails with an actionable prerequisite error; no
483
+ recipe is silently skipped. CI provisions these tools explicitly.
484
+
485
+ The verifier also dry-prepares every recipe with both native hosts, including
486
+ both skill/plugin variants, negative-control inputs and the goals-only step,
487
+ and verifies retained artifact digests. These are not authenticated model,
488
+ goal, plugin-installation or hook-execution trials. Run the live commands to
489
+ measure those routes. The separate published-consumer check executes the
490
+ installed-package tutorial using the exact current version and license metadata.
package/docs/releases.md CHANGED
@@ -12,6 +12,41 @@ Published `rc.1` and `rc.2` retain their original BSL grants. Always select a ne
12
12
  version for publication; a local development tarball does not replace an
13
13
  already published artifact.
14
14
 
15
+ <a id="unreleased-node-runtime-support"></a>
16
+
17
+ ## Node runtime support
18
+
19
+ The package ships compiled JavaScript for Node.js 24.13.0 or newer and Bun
20
+ 1.3.13. Development retains Bun. Builds populate `dist/`; the package phase
21
+ checks both runtimes and includes Node workflows with Bun absent from PATH.
22
+ `runnerBuildDigest` includes compiled output for installed execution, so this
23
+ changes build identities. Existing evidence keeps its original identity.
24
+ Release preparation builds first, then packs with lifecycle scripts disabled;
25
+ this adds no release test rerun.
26
+
27
+ <a id="unreleased-smaller-npm-package"></a>
28
+
29
+ ## Package contents and size
30
+
31
+ The npm package omits logo images, historical validation and usability pages,
32
+ and raw evidence, including the Darrow migration patch and old guide verification
33
+ record. All remain in the repository; packaged pages link to their online copies.
34
+ The current `LICENSE`, contribution terms, branding permissions and retained
35
+ earlier-release license text still ship. No licensing or branding grants change.
36
+
37
+ During the package-size cleanup, `npm pack --dry-run --ignore-scripts --json` measured a
38
+ development tarball at about 0.39 MB compressed, down from 3,545,849 bytes, and
39
+ about 1.86 MB unpacked, down from 6,632,891 bytes (about 89% and 72% smaller).
40
+ These historical development-build measurements do not guarantee the size of
41
+ every release.
42
+
43
+ `runnerBuildDigest` hashes packaged paths and bytes under `docs/` as well as the
44
+ runtime and other package inputs. This packaging change therefore causes a
45
+ one-time digest change even without runner behavior changes. Existing evidence
46
+ keeps its original identity; exact-repeat comparisons across this boundary
47
+ will differ on `runnerBuildDigest`. The identity format and hashing rules stay
48
+ unchanged.
49
+
15
50
  ## Candidate contract
16
51
 
17
52
  Prepare one candidate from the intended release tag and package metadata. The
@@ -22,7 +57,8 @@ including a release candidate. Use `next` for a preview that should require an
22
57
  explicit opt-in. Preparation accepts either tag for prerelease and stable versions.
23
58
 
24
59
  Preparation creates a new absolute output directory and never replaces an
25
- existing directory. It packs with lifecycle scripts disabled, retains the npm
60
+ existing directory. `release:prepare` first builds the JavaScript artifact,
61
+ then packs with lifecycle scripts disabled, retains the npm
26
62
  file inventory, and writes `release.json` and `SHA256SUMS` beside the tarball.
27
63
  The record identifies the package, version, release tag, distribution tag,
28
64
  tarball SHA-256 and npm SHA-512 integrity, and the preparation runtimes.
@@ -51,9 +87,11 @@ with their own CI before starting the release.
51
87
  2. Merge the release PR after its static checks and reused CI evidence pass.
52
88
  Release CI compares the candidate against a successful ancestor run on
53
89
  `main`, permitting only version/distribution-tag changes and the exact
54
- release documentation and workflow files inventoried in
55
- [the CI selector](../.github/scripts/release-ci.py). All other tracked inputs
56
- must match. The original run must have completed the full quality gate;
90
+ release documentation files inventoried in
91
+ [the CI selector](../.github/scripts/release-ci.py). Changes to the CI workflow
92
+ or selector require the full quality gate. All other tracked inputs must
93
+ match. The original run must have completed all four quality phases
94
+ and their gate steps, or the historical complete quality gate;
57
95
  another release's reused status is insufficient. CI retains the source SHA
58
96
  and run URL in `release-ci.json`. Missing evidence fails the release check;
59
97
  it does not trigger another test run. Ordinary code changes and explicit
@@ -104,25 +142,22 @@ retained checksum immediately before publication:
104
142
  ```sh
105
143
  cd /absolute/reviewed
106
144
  shasum -a 256 -c SHA256SUMS
107
- npm publish ./bjoernrochel-sevro-0.1.0-rc.1.tgz --ignore-scripts --access public --tag latest
145
+ npm publish "./bjoernrochel-sevro-${SEVRO_RELEASE_VERSION}.tgz" --ignore-scripts --access public --tag latest
108
146
  ```
109
147
 
110
- Replace the example version and distribution tag with the reviewed candidate's
111
- values. The package repository URL must match this public repository. Preparing
112
- a tarball or opening a pull request does not authorize publication.
148
+ Replace the distribution tag with the reviewed candidate's value. The package
149
+ repository URL must match this public repository. Preparing a tarball or
150
+ opening a pull request does not authorize publication.
113
151
 
114
152
  An existing published candidate can become the default without republishing:
115
153
 
116
154
  ```sh
117
- npm dist-tag add @bjoernrochel/sevro@0.1.0-rc.2 latest
155
+ npm dist-tag add "@bjoernrochel/sevro@${SEVRO_RELEASE_VERSION}" latest
118
156
  npm view @bjoernrochel/sevro dist-tags --json
119
157
  ```
120
158
 
121
159
  Promotion changes the registry tag; it does not update the published tarball.
122
160
 
123
- The first published candidate was `0.1.0-rc.1`, using the owner-selected
124
- `BUSL-1.1` license. Its `LICENSE` preserves Darrow's terms and parameters,
125
- changing only the Licensed Work name and description to Sevro. Local checks do
126
- not claim a remote CI run or publication. Darrow's exact
161
+ Local checks do not claim a remote CI run or publication. Darrow's exact
127
162
  registry pin, default caller switch, remaining workflow migrations, focused live validation, and
128
163
  generic runner removal remain separate extraction gates.
package/docs/report-v1.md CHANGED
@@ -3,10 +3,14 @@
3
3
  `sevro report` reads one or more retained `sevro.cli-result.v1` files and their
4
4
  referenced `sevro.run-evidence.v1` files. It validates both public schemas and
5
5
  checks run identity, exit category, case count, and trial references before
6
- reporting. Paths must be absolute and distinct.
6
+ reporting. Programmatic `createReport` paths must be absolute and distinct.
7
+ CLI `--result-file` paths may be relative to the invoking directory; they are
8
+ canonicalized before duplicate detection and loading.
7
9
 
8
10
  ```sh
9
11
  sevro report --result-file /absolute/path/to/result.json
12
+ sevro report --latest
13
+ sevro report --latest --results-root ./other-results
10
14
  sevro report --json --result-file /absolute/path/to/first.json \
11
15
  --result-file /absolute/path/to/second.json
12
16
  ```
@@ -17,6 +21,30 @@ inputs return exit 64 with a diagnostic on stderr. Reporting does not change
17
21
  the exit category of any input run; each row retains its original code.
18
22
  Pre-run failures with no case rows remain visible in the input-run section.
19
23
 
24
+ `--latest` selects
25
+ `<cwd>/.sevro/results/latest.json`; `--results-root` changes that root and requires
26
+ `--latest`. It is exclusive with `--result-file`. Each retained CLI evaluation
27
+ saves a `result.json` beside its run evidence and atomically updates latest,
28
+ including dry and failed runs. Pre-run failures do not replace latest. Missing
29
+ latest or referenced evidence is an error. See
30
+ [the evaluation workflow](evaluation-workflow.md).
31
+
32
+ [`sevro prune`](reading-results.md#prune-retained-results) maintains this index
33
+ when removing retained evidence. After pruning, latest selects the newest
34
+ remaining completed result; if none remain, the index is removed and `--latest`
35
+ reports a missing-input diagnostic. Explicit reports still require their selected
36
+ results and evidence to exist.
37
+
38
+ After [`sevro prune --native-transcripts`](reading-results.md#remove-native-transcript-bundles),
39
+ reports accept absent native bundles only with the matching
40
+ [pruning receipt](results-v1.md#native-transcript-bundles). Present bundles must
41
+ still match their recorded hashes; unexplained missing bytes remain an error.
42
+ New report rows include optional `nativeTranscripts` with nonnegative integer
43
+ `retained` and `pruned` bundle counts across the case's trials and host roles.
44
+ Markdown shows the same counts beside retained evidence. Older report documents
45
+ without this field remain valid. Pruning does not change verdicts, routes,
46
+ identity, usage, or duration.
47
+
20
48
  Each case row keeps task verdict, execution, and grading separate. Task pass
21
49
  rate is measured only when every trial completed execution and grading with an
22
50
  assessed task verdict. Candidate duration sums the measured host call time