@bjoernrochel/sevro 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +34 -4
- package/README.md +7 -6
- package/dist/examples/basic/graded.json +13 -0
- package/dist/examples/basic/host.js +9 -0
- package/dist/examples/basic/host.js.map +1 -0
- package/dist/examples/basic/prompt-only.json +7 -0
- package/dist/examples/recipes/candidate.js +23 -0
- package/dist/examples/recipes/candidate.js.map +1 -0
- package/dist/examples/recipes/cases/coding-change.json +21 -0
- package/dist/examples/recipes/cases/host-tools.json +21 -0
- package/dist/examples/recipes/cases/mise.json +21 -0
- package/dist/examples/recipes/cases/node.json +21 -0
- package/dist/examples/recipes/cases/python-uv.json +21 -0
- package/dist/examples/recipes/cases/semantic.json +17 -0
- package/dist/examples/recipes/coding-change/check.js +16 -0
- package/dist/examples/recipes/coding-change/check.js.map +1 -0
- package/dist/examples/recipes/executable.js +20 -0
- package/dist/examples/recipes/executable.js.map +1 -0
- package/dist/examples/recipes/extension-artifacts.js +56 -0
- package/dist/examples/recipes/extension-artifacts.js.map +1 -0
- package/dist/examples/recipes/extension-methods.js +167 -0
- package/dist/examples/recipes/extension-methods.js.map +1 -0
- package/dist/examples/recipes/extension.js +28 -0
- package/dist/examples/recipes/extension.js.map +1 -0
- package/dist/examples/recipes/modules.js +9 -0
- package/dist/examples/recipes/modules.js.map +1 -0
- package/dist/examples/recipes/repository-skill/.agents/skills/startup-guide/SKILL.md +7 -0
- package/dist/examples/recipes/repository-skill/.claude/skills/startup-guide/SKILL.md +7 -0
- package/dist/examples/recipes/repository-skill/README.md +3 -0
- package/dist/examples/recipes/run-inputs.js +144 -0
- package/dist/examples/recipes/run-inputs.js.map +1 -0
- package/dist/examples/recipes/run-options.js +103 -0
- package/dist/examples/recipes/run-options.js.map +1 -0
- package/dist/examples/recipes/run.js +62 -0
- package/dist/examples/recipes/run.js.map +1 -0
- package/dist/examples/recipes/runtime/empty.json +1 -0
- package/dist/examples/recipes/runtime/goals.json +1 -0
- package/dist/examples/recipes/runtime/mise.json +6 -0
- package/dist/examples/recipes/runtime/path.json +1 -0
- package/dist/examples/recipes/runtime/plugins.json +4 -0
- package/dist/examples/recipes/runtime/python-uv.json +10 -0
- package/dist/examples/recipes/semantic-judge.js +27 -0
- package/dist/examples/recipes/semantic-judge.js.map +1 -0
- package/dist/examples/templates/coding/adapter.js +14 -0
- package/dist/examples/templates/coding/adapter.js.map +1 -0
- package/dist/examples/templates/coding/case.json +15 -0
- package/dist/examples/templates/repository-skill/.agents/skills/startup-guide/SKILL.md +6 -0
- package/dist/examples/templates/repository-skill/extension-data.js +61 -0
- package/dist/examples/templates/repository-skill/extension-data.js.map +1 -0
- package/dist/examples/templates/repository-skill/extension.js +29 -0
- package/dist/examples/templates/repository-skill/extension.js.map +1 -0
- package/dist/examples/templates/response/adapter.js +13 -0
- package/dist/examples/templates/response/adapter.js.map +1 -0
- package/dist/examples/templates/response/case.json +13 -0
- package/dist/src/advisory.js +86 -0
- package/dist/src/advisory.js.map +1 -0
- package/dist/src/candidate-artifacts.js +139 -0
- package/dist/src/candidate-artifacts.js.map +1 -0
- package/dist/src/candidate-codex-state.js +110 -0
- package/dist/src/candidate-codex-state.js.map +1 -0
- package/dist/src/case-validation.js +92 -0
- package/dist/src/case-validation.js.map +1 -0
- package/dist/src/cli/cancellation.js +21 -0
- package/dist/src/cli/cancellation.js.map +1 -0
- package/dist/src/cli/command.js +60 -0
- package/dist/src/cli/command.js.map +1 -0
- package/dist/src/cli/completion.js +14 -0
- package/dist/src/cli/completion.js.map +1 -0
- package/dist/src/cli/demo.js +45 -0
- package/dist/src/cli/demo.js.map +1 -0
- package/dist/src/cli/evaluation.js +40 -0
- package/dist/src/cli/evaluation.js.map +1 -0
- package/dist/src/cli/extension.js +54 -0
- package/dist/src/cli/extension.js.map +1 -0
- package/dist/src/cli/help.js +190 -0
- package/dist/src/cli/help.js.map +1 -0
- package/dist/src/cli/hosts.js +33 -0
- package/dist/src/cli/hosts.js.map +1 -0
- package/dist/src/cli/init-settings.js +29 -0
- package/dist/src/cli/init-settings.js.map +1 -0
- package/dist/src/cli/init.js +82 -0
- package/dist/src/cli/init.js.map +1 -0
- package/dist/src/cli/invocation.js +431 -0
- package/dist/src/cli/invocation.js.map +1 -0
- package/dist/src/cli/loading.js +110 -0
- package/dist/src/cli/loading.js.map +1 -0
- package/dist/src/cli/preparation.js +88 -0
- package/dist/src/cli/preparation.js.map +1 -0
- package/dist/src/cli/presentation.js +61 -0
- package/dist/src/cli/presentation.js.map +1 -0
- package/dist/src/cli/progress.js +5 -0
- package/dist/src/cli/progress.js.map +1 -0
- package/dist/src/cli/run-manifest.js +108 -0
- package/dist/src/cli/run-manifest.js.map +1 -0
- package/dist/src/cli/run-options.js +367 -0
- package/dist/src/cli/run-options.js.map +1 -0
- package/dist/src/cli/run-overrides.js +51 -0
- package/dist/src/cli/run-overrides.js.map +1 -0
- package/dist/src/cli/settings.js +127 -0
- package/dist/src/cli/settings.js.map +1 -0
- package/dist/src/cli/summary.js +28 -0
- package/dist/src/cli/summary.js.map +1 -0
- package/dist/src/cli.js +20 -0
- package/dist/src/cli.js.map +1 -0
- package/dist/src/contract-types.js +2 -0
- package/dist/src/contract-types.js.map +1 -0
- package/dist/src/engine.js +4 -0
- package/dist/src/engine.js.map +1 -0
- package/dist/src/evaluation/advisory.js +85 -0
- package/dist/src/evaluation/advisory.js.map +1 -0
- package/dist/src/evaluation/check-plan.js +292 -0
- package/dist/src/evaluation/check-plan.js.map +1 -0
- package/dist/src/evaluation/collection.js +61 -0
- package/dist/src/evaluation/collection.js.map +1 -0
- package/dist/src/evaluation/context.js +2 -0
- package/dist/src/evaluation/context.js.map +1 -0
- package/dist/src/evaluation/grading-host.js +28 -0
- package/dist/src/evaluation/grading-host.js.map +1 -0
- package/dist/src/evaluation/grading.js +154 -0
- package/dist/src/evaluation/grading.js.map +1 -0
- package/dist/src/evaluation/invocation-plan.js +312 -0
- package/dist/src/evaluation/invocation-plan.js.map +1 -0
- package/dist/src/evaluation/plan.js +376 -0
- package/dist/src/evaluation/plan.js.map +1 -0
- package/dist/src/evaluation/progress.js +14 -0
- package/dist/src/evaluation/progress.js.map +1 -0
- package/dist/src/evaluation/resources.js +159 -0
- package/dist/src/evaluation/resources.js.map +1 -0
- package/dist/src/evaluation/run.js +148 -0
- package/dist/src/evaluation/run.js.map +1 -0
- package/dist/src/evaluation/scheduler.js +27 -0
- package/dist/src/evaluation/scheduler.js.map +1 -0
- package/dist/src/evaluation/semantic.js +183 -0
- package/dist/src/evaluation/semantic.js.map +1 -0
- package/dist/src/evaluation/trial.js +593 -0
- package/dist/src/evaluation/trial.js.map +1 -0
- package/dist/src/evaluation/types.js +20 -0
- package/dist/src/evaluation/types.js.map +1 -0
- package/dist/src/evidence/host-results.js +67 -0
- package/dist/src/evidence/host-results.js.map +1 -0
- package/dist/src/evidence/retained-artifacts.js +17 -0
- package/dist/src/evidence/retained-artifacts.js.map +1 -0
- package/dist/src/evidence/trial-writer.js +131 -0
- package/dist/src/evidence/trial-writer.js.map +1 -0
- package/dist/src/evidence/usage.js +14 -0
- package/dist/src/evidence/usage.js.map +1 -0
- package/dist/src/executable.js +20 -0
- package/dist/src/executable.js.map +1 -0
- package/dist/src/extension-client.js +212 -0
- package/dist/src/extension-client.js.map +1 -0
- package/dist/src/extension-session.js +236 -0
- package/dist/src/extension-session.js.map +1 -0
- package/dist/src/fixtures/advisory.js +206 -0
- package/dist/src/fixtures/advisory.js.map +1 -0
- package/dist/src/fixtures/bin.js +41 -0
- package/dist/src/fixtures/bin.js.map +1 -0
- package/dist/src/fixtures/cleanup.js +43 -0
- package/dist/src/fixtures/cleanup.js.map +1 -0
- package/dist/src/fixtures/create.js +94 -0
- package/dist/src/fixtures/create.js.map +1 -0
- package/dist/src/fixtures/generated.js +181 -0
- package/dist/src/fixtures/generated.js.map +1 -0
- package/dist/src/fixtures/git-hooks.js +16 -0
- package/dist/src/fixtures/git-hooks.js.map +1 -0
- package/dist/src/fixtures/preparation.js +201 -0
- package/dist/src/fixtures/preparation.js.map +1 -0
- package/dist/src/fixtures/repository.js +255 -0
- package/dist/src/fixtures/repository.js.map +1 -0
- package/dist/src/fixtures/setup.js +175 -0
- package/dist/src/fixtures/setup.js.map +1 -0
- package/dist/src/fixtures/tools.js +57 -0
- package/dist/src/fixtures/tools.js.map +1 -0
- package/dist/src/fixtures/workspace-state.js +155 -0
- package/dist/src/fixtures/workspace-state.js.map +1 -0
- package/dist/src/generated/cli-result.cjs +2 -0
- package/dist/src/generated/extension.cjs +2 -0
- package/dist/src/generated/report.cjs +2 -0
- package/dist/src/generated/run-evidence.cjs +2 -0
- package/dist/src/generated/runtime.cjs +2 -0
- package/dist/src/git.js +72 -0
- package/dist/src/git.js.map +1 -0
- package/dist/src/graders/artifact.js +68 -0
- package/dist/src/graders/artifact.js.map +1 -0
- package/dist/src/graders/git-head.js +112 -0
- package/dist/src/graders/git-head.js.map +1 -0
- package/dist/src/graders/output.js +345 -0
- package/dist/src/graders/output.js.map +1 -0
- package/dist/src/graders/semantic.js +132 -0
- package/dist/src/graders/semantic.js.map +1 -0
- package/dist/src/graders/shell.js +321 -0
- package/dist/src/graders/shell.js.map +1 -0
- package/dist/src/host-diagnostic-redaction.js +52 -0
- package/dist/src/host-diagnostic-redaction.js.map +1 -0
- package/dist/src/host-diagnostics.js +248 -0
- package/dist/src/host-diagnostics.js.map +1 -0
- package/dist/src/hosts/claude-continuation.js +89 -0
- package/dist/src/hosts/claude-continuation.js.map +1 -0
- package/dist/src/hosts/claude-credential.js +121 -0
- package/dist/src/hosts/claude-credential.js.map +1 -0
- package/dist/src/hosts/claude-events.js +136 -0
- package/dist/src/hosts/claude-events.js.map +1 -0
- package/dist/src/hosts/claude-keychain.js +100 -0
- package/dist/src/hosts/claude-keychain.js.map +1 -0
- package/dist/src/hosts/claude-login-recovery.js +67 -0
- package/dist/src/hosts/claude-login-recovery.js.map +1 -0
- package/dist/src/hosts/claude-login-refresh.js +146 -0
- package/dist/src/hosts/claude-login-refresh.js.map +1 -0
- package/dist/src/hosts/claude-login-store.js +124 -0
- package/dist/src/hosts/claude-login-store.js.map +1 -0
- package/dist/src/hosts/claude-login.js +74 -0
- package/dist/src/hosts/claude-login.js.map +1 -0
- package/dist/src/hosts/claude-native-goal.js +138 -0
- package/dist/src/hosts/claude-native-goal.js.map +1 -0
- package/dist/src/hosts/claude-process.js +10 -0
- package/dist/src/hosts/claude-process.js.map +1 -0
- package/dist/src/hosts/claude-refresh-lock.js +76 -0
- package/dist/src/hosts/claude-refresh-lock.js.map +1 -0
- package/dist/src/hosts/claude-repository-invocation.js +229 -0
- package/dist/src/hosts/claude-repository-invocation.js.map +1 -0
- package/dist/src/hosts/claude-settings.js +156 -0
- package/dist/src/hosts/claude-settings.js.map +1 -0
- package/dist/src/hosts/claude-skill-activations.js +131 -0
- package/dist/src/hosts/claude-skill-activations.js.map +1 -0
- package/dist/src/hosts/claude-state.js +237 -0
- package/dist/src/hosts/claude-state.js.map +1 -0
- package/dist/src/hosts/claude-tool-calls.js +150 -0
- package/dist/src/hosts/claude-tool-calls.js.map +1 -0
- package/dist/src/hosts/claude.js +373 -0
- package/dist/src/hosts/claude.js.map +1 -0
- package/dist/src/hosts/codex-app-server.js +764 -0
- package/dist/src/hosts/codex-app-server.js.map +1 -0
- package/dist/src/hosts/codex-configuration.js +91 -0
- package/dist/src/hosts/codex-configuration.js.map +1 -0
- package/dist/src/hosts/codex-events.js +104 -0
- package/dist/src/hosts/codex-events.js.map +1 -0
- package/dist/src/hosts/codex-exec-controls.js +434 -0
- package/dist/src/hosts/codex-exec-controls.js.map +1 -0
- package/dist/src/hosts/codex-invocation.js +54 -0
- package/dist/src/hosts/codex-invocation.js.map +1 -0
- package/dist/src/hosts/codex-marketplace.js +185 -0
- package/dist/src/hosts/codex-marketplace.js.map +1 -0
- package/dist/src/hosts/codex-native-calls.js +516 -0
- package/dist/src/hosts/codex-native-calls.js.map +1 -0
- package/dist/src/hosts/codex-process.js +10 -0
- package/dist/src/hosts/codex-process.js.map +1 -0
- package/dist/src/hosts/codex-profile.js +206 -0
- package/dist/src/hosts/codex-profile.js.map +1 -0
- package/dist/src/hosts/codex-session-files.js +89 -0
- package/dist/src/hosts/codex-session-files.js.map +1 -0
- package/dist/src/hosts/codex-state.js +310 -0
- package/dist/src/hosts/codex-state.js.map +1 -0
- package/dist/src/hosts/codex.js +527 -0
- package/dist/src/hosts/codex.js.map +1 -0
- package/dist/src/hosts/isolation-common.js +43 -0
- package/dist/src/hosts/isolation-common.js.map +1 -0
- package/dist/src/hosts/isolation-roots.js +64 -0
- package/dist/src/hosts/isolation-roots.js.map +1 -0
- package/dist/src/hosts/isolation.js +12 -0
- package/dist/src/hosts/isolation.js.map +1 -0
- package/dist/src/hosts/linux-sandbox.js +127 -0
- package/dist/src/hosts/linux-sandbox.js.map +1 -0
- package/dist/src/hosts/mac-sandbox.js +137 -0
- package/dist/src/hosts/mac-sandbox.js.map +1 -0
- package/dist/src/hosts/native-controls.js +151 -0
- package/dist/src/hosts/native-controls.js.map +1 -0
- package/dist/src/hosts/native-request.js +33 -0
- package/dist/src/hosts/native-request.js.map +1 -0
- package/dist/src/hosts/process-lifecycle.js +167 -0
- package/dist/src/hosts/process-lifecycle.js.map +1 -0
- package/dist/src/hosts/protected-worktrees.js +81 -0
- package/dist/src/hosts/protected-worktrees.js.map +1 -0
- package/dist/src/hosts/skill-hook-catalog.js +100 -0
- package/dist/src/hosts/skill-hook-catalog.js.map +1 -0
- package/dist/src/hosts/skill-hook-handler.js +228 -0
- package/dist/src/hosts/skill-hook-handler.js.map +1 -0
- package/dist/src/hosts/skill-hooks.js +423 -0
- package/dist/src/hosts/skill-hooks.js.map +1 -0
- package/dist/src/hosts/workspace-fingerprint.js +72 -0
- package/dist/src/hosts/workspace-fingerprint.js.map +1 -0
- package/dist/src/identity.js +167 -0
- package/dist/src/identity.js.map +1 -0
- package/dist/src/instrumentation.js +83 -0
- package/dist/src/instrumentation.js.map +1 -0
- package/dist/src/native-transcript-state.js +85 -0
- package/dist/src/native-transcript-state.js.map +1 -0
- package/dist/src/native-transcript-view.js +65 -0
- package/dist/src/native-transcript-view.js.map +1 -0
- package/dist/src/native-transcripts.js +208 -0
- package/dist/src/native-transcripts.js.map +1 -0
- package/dist/src/paths.js +13 -0
- package/dist/src/paths.js.map +1 -0
- package/dist/src/process-lifecycle.js +101 -0
- package/dist/src/process-lifecycle.js.map +1 -0
- package/dist/src/provenance.js +263 -0
- package/dist/src/provenance.js.map +1 -0
- package/dist/src/prune.js +228 -0
- package/dist/src/prune.js.map +1 -0
- package/dist/src/report.js +276 -0
- package/dist/src/report.js.map +1 -0
- package/dist/src/resolved-case.js +29 -0
- package/dist/src/resolved-case.js.map +1 -0
- package/dist/src/results-lock.js +40 -0
- package/dist/src/results-lock.js.map +1 -0
- package/dist/src/results.js +127 -0
- package/dist/src/results.js.map +1 -0
- package/dist/src/run-owner.js +283 -0
- package/dist/src/run-owner.js.map +1 -0
- package/dist/src/runner-paths.js +11 -0
- package/dist/src/runner-paths.js.map +1 -0
- package/dist/src/runtime/config.js +323 -0
- package/dist/src/runtime/config.js.map +1 -0
- package/dist/src/runtime/discovery-apple.js +65 -0
- package/dist/src/runtime/discovery-apple.js.map +1 -0
- package/dist/src/runtime/discovery-providers.js +37 -0
- package/dist/src/runtime/discovery-providers.js.map +1 -0
- package/dist/src/runtime/discovery.js +40 -0
- package/dist/src/runtime/discovery.js.map +1 -0
- package/dist/src/runtime/hook-processes.js +81 -0
- package/dist/src/runtime/hook-processes.js.map +1 -0
- package/dist/src/runtime/hooks.js +304 -0
- package/dist/src/runtime/hooks.js.map +1 -0
- package/dist/src/runtime/paths.js +244 -0
- package/dist/src/runtime/paths.js.map +1 -0
- package/dist/src/runtime/query.js +76 -0
- package/dist/src/runtime/query.js.map +1 -0
- package/dist/src/runtime/seeds.js +98 -0
- package/dist/src/runtime/seeds.js.map +1 -0
- package/dist/src/runtime/shared-cache.js +81 -0
- package/dist/src/runtime/shared-cache.js.map +1 -0
- package/dist/src/runtime/state.js +106 -0
- package/dist/src/runtime/state.js.map +1 -0
- package/dist/src/runtime-process.js +148 -0
- package/dist/src/runtime-process.js.map +1 -0
- package/dist/src/schema-types.js +2 -0
- package/dist/src/schema-types.js.map +1 -0
- package/dist/src/schema.js +16 -0
- package/dist/src/schema.js.map +1 -0
- package/dist/src/shell.js +5 -0
- package/dist/src/shell.js.map +1 -0
- package/dist/src/sqlite-lock.js +18 -0
- package/dist/src/sqlite-lock.js.map +1 -0
- package/dist/src/storage.js +21 -0
- package/dist/src/storage.js.map +1 -0
- package/dist/src/transcript-retention.js +109 -0
- package/dist/src/transcript-retention.js.map +1 -0
- package/dist/src/value-guards.js +26 -0
- package/dist/src/value-guards.js.map +1 -0
- package/docs/architecture.md +5 -2
- package/docs/creating-cases.md +2 -3
- package/docs/development-cli.md +12 -7
- package/docs/documentation-quality.md +4 -3
- package/docs/evaluation-workflow.md +8 -8
- package/docs/getting-started.md +7 -6
- package/docs/guide-evaluation.md +1 -1
- package/docs/identity-v1.md +4 -1
- package/docs/installing.md +31 -13
- package/docs/licensing.md +2 -2
- package/docs/native-hosts.md +1 -1
- package/docs/recipes.md +41 -40
- package/docs/releases.md +20 -4
- package/docs/run-settings-v1.md +3 -3
- package/docs/specs/repository-guide.md +1 -1
- package/docs/troubleshooting.md +1 -1
- package/docs/typescript-quality.md +18 -1
- package/examples/recipes/executable.ts +19 -0
- package/examples/recipes/extension-artifacts.ts +1 -1
- package/examples/recipes/extension.ts +2 -1
- package/examples/recipes/modules.ts +9 -0
- package/examples/recipes/run-inputs.ts +14 -19
- package/examples/recipes/run-options.ts +9 -8
- package/examples/recipes/run.ts +21 -7
- package/examples/templates/repository-skill/extension-data.ts +1 -1
- package/examples/templates/repository-skill/extension.ts +2 -1
- package/package.json +14 -6
- package/src/cli/demo.ts +3 -2
- package/src/cli/help.ts +4 -3
- package/src/cli/init-settings.ts +5 -4
- package/src/cli/init.ts +9 -8
- package/src/cli/invocation.ts +2 -1
- package/src/cli/preparation.ts +2 -2
- package/src/cli.ts +1 -1
- package/src/executable.ts +19 -0
- package/src/extension-client.ts +6 -12
- package/src/fixtures/setup.ts +3 -2
- package/src/git.ts +2 -1
- package/src/graders/git-head.ts +3 -2
- package/src/graders/shell.ts +4 -3
- package/src/hosts/claude-keychain.ts +3 -2
- package/src/hosts/claude-login-refresh.ts +3 -2
- package/src/hosts/claude-refresh-lock.ts +2 -1
- package/src/hosts/isolation-roots.ts +3 -2
- package/src/hosts/process-lifecycle.ts +4 -6
- package/src/hosts/protected-worktrees.ts +2 -1
- package/src/hosts/skill-hook-handler.ts +1 -1
- package/src/hosts/skill-hooks.ts +10 -2
- package/src/provenance.ts +7 -6
- package/src/results-lock.ts +4 -4
- package/src/run-owner.ts +15 -9
- package/src/runner-paths.ts +21 -0
- package/src/runtime/hook-processes.ts +2 -1
- package/src/runtime/hooks.ts +9 -7
- package/src/runtime-process.ts +194 -0
- package/src/sqlite-lock.ts +25 -0
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"schema-types.js","sourceRoot":"","sources":["schema-types.ts"],"names":[],"mappings":"","sourcesContent":["import type { AdvisoryAssessment } from \"./advisory\";\nimport type {\n ArtifactReference,\n Assessment,\n CheckOutcome,\n Diagnostic,\n DomainOutcome,\n EvidenceRoute,\n ExecutionStatus,\n ExitCode,\n GradingStatus,\n MetricEvidence,\n ObservationCompleteness,\n ObservationEvidence,\n OutcomeStatus,\n RawResultReference,\n TaskVerdict,\n UsageEvidence,\n} from \"./contract-types\";\nimport type { IdentityDimensions } from \"./identity\";\nimport type { InstrumentationRequest } from \"./instrumentation\";\n\n/** Wire assessments additionally allow optional serialized error codes. */\nexport type WireAssessment = {\n execution: Assessment[\"execution\"] & { errorCode?: string };\n grading: Assessment[\"grading\"] & { errorCode?: string };\n task: Assessment[\"task\"];\n};\n\n/** Validated v1 wire data also represents failures before run allocation. */\nexport type CliResultData = WireAssessment & {\n format: \"sevro.cli-result.v1\";\n runId: string | null;\n exitCode: ExitCode;\n evidencePath: string | null;\n cases: Array<CliResultDataCase>;\n diagnostic?: Diagnostic;\n};\nexport type CliResultDataTrial = WireAssessment & {\n trial: number;\n checks: Array<CheckOutcome>;\n domainOutcomes?: Array<DomainOutcome>;\n artifactPath: string | null;\n};\nexport type CliResultDataCase = WireAssessment & {\n caseId: string;\n trials: Array<CliResultDataTrial>;\n};\n\n/** Structural contracts for data validated by the versioned JSON schemas. */\nexport type RunEvidenceData = {\n format: \"sevro.run-evidence.v1\";\n runId: string;\n evaluationIdentity: {\n algorithm: \"sevro.identity.v1\";\n digest: RunEvidenceDataDigest;\n dimensions: IdentityDimensions;\n };\n configuration: {\n digest: RunEvidenceDataDigest;\n redacted: Record<string, unknown>;\n resolved?: Record<string, unknown>;\n };\n runner: RunEvidenceDataRunner;\n project: {\n root: RunEvidenceDataFileUrl;\n revision: string | null;\n dirtyPatchDigest: RunEvidenceDataDigest | null;\n };\n extension: null | RunEvidenceDataExtension;\n condition: {\n requested: \"passive\" | \"enforced\";\n actual: \"passive\" | \"enforced\" | \"unknown\";\n requestedInstrumentation: InstrumentationRequest[];\n appliedInstrumentation: InstrumentationRequest[];\n };\n graders: {\n active: Array<RunEvidenceDataGrader>;\n replacedDefaults: Array<string>;\n };\n routes: Array<EvidenceRoute>;\n result: CliResultData;\n trials: Array<RunEvidenceDataTrialEvidence>;\n diagnostic?: Diagnostic;\n};\ntype RunEvidenceDataDigest = string;\ntype RunEvidenceDataFileUrl = string;\ntype RunEvidenceDataRunner =\n | {\n source: \"package\";\n packageName: string;\n version: string;\n buildDigest: RunEvidenceDataDigest;\n }\n | {\n source: \"checkout\";\n root: RunEvidenceDataFileUrl;\n revision: string;\n dirtyPatchDigest: RunEvidenceDataDigest | null;\n buildDigest: RunEvidenceDataDigest;\n };\ntype RunEvidenceDataExtension = {\n id: string;\n version: string;\n sourceDigest: RunEvidenceDataDigest;\n configurationDigest: RunEvidenceDataDigest;\n protocol: \"sevro.extension.v1\";\n capabilities: Array<string>;\n replacements: { graders: Array<string>; taskVerdictPolicy: string | null };\n};\ntype RunEvidenceDataGrader = {\n id: string;\n source: \"builtin\" | \"extension\";\n version: string;\n};\nexport type AdvisoryReviewData = {\n status: \"completed\" | \"failed\" | \"not_run\";\n assessment: AdvisoryAssessment | null;\n usage: UsageEvidence;\n rawResult: RawResultReference;\n};\nexport type RunEvidenceDataTrialEvidence = {\n caseId: string;\n trial: number;\n executionMode: \"executed\" | \"dry\" | \"unknown\";\n candidateDurationMs?: number | null;\n condition: {\n requested: \"passive\" | \"enforced\";\n actual: \"passive\" | \"enforced\" | \"unknown\";\n appliedInstrumentation: InstrumentationRequest[];\n };\n observationCompleteness: ObservationCompleteness;\n observations: Array<ObservationEvidence>;\n metrics: Array<MetricEvidence>;\n domainOutcomes?: Array<DomainOutcome>;\n taskVerdictPolicy: null | {\n id: string;\n recommendation: TaskVerdict | null;\n };\n advisoryReview?: AdvisoryReviewData | null;\n routes: Array<EvidenceRoute>;\n usage: UsageEvidence;\n rawResult: RawResultReference;\n artifactRefs: Array<ArtifactReference>;\n};\nexport type ReportData = {\n format: \"sevro.report.v1\";\n inputs: Array<ReportDataInput>;\n rows: Array<ReportDataRow>;\n summary: {\n cases: number;\n passed: number;\n failed: number;\n notAssessed: number;\n };\n};\ntype ReportDataPath = string;\ntype ReportDataDigest = string;\n/** The report wire schema deliberately allows either state family in rows. */\ntype ReportDataState = ExecutionStatus | GradingStatus;\ntype ReportDataInput = {\n resultFile: ReportDataPath;\n execution: ExecutionStatus;\n grading: GradingStatus;\n task: TaskVerdict;\n exitCode: ExitCode;\n caseCount: number;\n};\ntype ReportDataRoute = Omit<EvidenceRoute, \"role\">;\ntype ReportDataOutcome = {\n trial: number;\n id: string;\n status: OutcomeStatus;\n};\ntype ReportDataRow = {\n nativeTranscripts?: { retained: number; pruned: number };\n resultFile: ReportDataPath;\n evidencePath: string | null;\n runId: string | null;\n caseId: string;\n execution: ReportDataState;\n grading: ReportDataState;\n task: TaskVerdict;\n exitCode: ExitCode;\n trialCount: number;\n taskPassRate: number | null;\n candidateDurationMs: number | null;\n inputTokens: number | null;\n outputTokens: number | null;\n costUsd: number | null;\n runnerBuildDigest: ReportDataDigest | null;\n evaluationDigest: ReportDataDigest | null;\n requestedCondition: \"passive\" | \"enforced\" | null;\n actualCondition: \"passive\" | \"enforced\" | \"unknown\" | null;\n candidateRoute: ReportDataRoute | null;\n routes: Array<EvidenceRoute>;\n domainOutcomes: Array<ReportDataOutcome>;\n};\n"]}
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
import cliValidator from "./generated/cli-result.cjs";
|
|
2
|
+
import runValidator from "./generated/run-evidence.cjs";
|
|
3
|
+
import reportValidator from "./generated/report.cjs";
|
|
4
|
+
export function assertCliResult(value) {
|
|
5
|
+
if (!cliValidator(value))
|
|
6
|
+
throw new Error("invalid Sevro CLI result");
|
|
7
|
+
}
|
|
8
|
+
export function assertRunEvidence(value) {
|
|
9
|
+
if (!runValidator(value))
|
|
10
|
+
throw new Error("invalid Sevro retained evidence");
|
|
11
|
+
}
|
|
12
|
+
export function assertReport(value) {
|
|
13
|
+
if (!reportValidator(value))
|
|
14
|
+
throw new Error("invalid Sevro report");
|
|
15
|
+
}
|
|
16
|
+
//# sourceMappingURL=schema.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"schema.js","sourceRoot":"","sources":["schema.ts"],"names":[],"mappings":"AAKA,OAAO,YAAY,mCAAmC;AACtD,OAAO,YAAY,qCAAqC;AACxD,OAAO,eAAe,+BAA+B;AAErD,MAAM,UAAU,eAAe,CAC7B,KAAc;IAEd,IAAI,CAAC,YAAY,CAAC,KAAK,CAAC;QAAE,MAAM,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;AACxE,CAAC;AAED,MAAM,UAAU,iBAAiB,CAC/B,KAAc;IAEd,IAAI,CAAC,YAAY,CAAC,KAAK,CAAC;QAAE,MAAM,IAAI,KAAK,CAAC,iCAAiC,CAAC,CAAC;AAC/E,CAAC;AAED,MAAM,UAAU,YAAY,CAAC,KAAc;IACzC,IAAI,CAAC,eAAe,CAAC,KAAK,CAAC;QAAE,MAAM,IAAI,KAAK,CAAC,sBAAsB,CAAC,CAAC;AACvE,CAAC","sourcesContent":["import type {\n CliResultData,\n RunEvidenceData,\n ReportData,\n} from \"./schema-types\";\nimport cliValidator from \"./generated/cli-result.cjs\";\nimport runValidator from \"./generated/run-evidence.cjs\";\nimport reportValidator from \"./generated/report.cjs\";\n\nexport function assertCliResult(\n value: unknown,\n): asserts value is CliResultData {\n if (!cliValidator(value)) throw new Error(\"invalid Sevro CLI result\");\n}\n\nexport function assertRunEvidence(\n value: unknown,\n): asserts value is RunEvidenceData {\n if (!runValidator(value)) throw new Error(\"invalid Sevro retained evidence\");\n}\n\nexport function assertReport(value: unknown): asserts value is ReportData {\n if (!reportValidator(value)) throw new Error(\"invalid Sevro report\");\n}\n"]}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"shell.js","sourceRoot":"","sources":["shell.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,MAAM,UAAU,UAAU,CAAC,KAAa;IACtC,OAAO,IAAI,KAAK,CAAC,UAAU,CAAC,GAAG,EAAE,OAAO,CAAC,GAAG,CAAC;AAC/C,CAAC","sourcesContent":["/** Encode one literal argument for a POSIX shell, including empty strings. */\nexport function shellQuote(value: string): string {\n return `'${value.replaceAll(\"'\", `'\"'\"'`)}'`;\n}\n"]}
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
import { createRequire } from "node:module";
|
|
2
|
+
const require = createRequire(import.meta.url);
|
|
3
|
+
/** Both drivers use SQLite file locks, including across Node and Bun runners. */
|
|
4
|
+
export function openLockDatabase(path) {
|
|
5
|
+
if (process.versions.bun) {
|
|
6
|
+
const { Database } = require("bun:sqlite");
|
|
7
|
+
const database = new Database(path);
|
|
8
|
+
return {
|
|
9
|
+
exec: (sql) => database.run(sql),
|
|
10
|
+
close() {
|
|
11
|
+
database.close();
|
|
12
|
+
},
|
|
13
|
+
};
|
|
14
|
+
}
|
|
15
|
+
const { DatabaseSync } = require("node:sqlite");
|
|
16
|
+
return new DatabaseSync(path);
|
|
17
|
+
}
|
|
18
|
+
//# sourceMappingURL=sqlite-lock.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"sqlite-lock.js","sourceRoot":"","sources":["sqlite-lock.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,oBAAoB;AAE5C,MAAM,OAAO,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAO/C,iFAAiF;AACjF,MAAM,UAAU,gBAAgB,CAAC,IAAY;IAC3C,IAAI,OAAO,CAAC,QAAQ,CAAC,GAAG,EAAE,CAAC;QACzB,MAAM,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC,YAAY,CAAgC,CAAC;QAC1E,MAAM,QAAQ,GAAG,IAAI,QAAQ,CAAC,IAAI,CAAC,CAAC;QACpC,OAAO;YACL,IAAI,EAAE,CAAC,GAAG,EAAE,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,GAAG,CAAC;YAChC,KAAK;gBACH,QAAQ,CAAC,KAAK,EAAE,CAAC;YACnB,CAAC;SACF,CAAC;IACJ,CAAC;IACD,MAAM,EAAE,YAAY,EAAE,GACpB,OAAO,CAAC,aAAa,CAAiC,CAAC;IACzD,OAAO,IAAI,YAAY,CAAC,IAAI,CAAC,CAAC;AAChC,CAAC","sourcesContent":["import { createRequire } from \"node:module\";\n\nconst require = createRequire(import.meta.url);\n\ninterface LockDatabase {\n exec(sql: string): unknown;\n close(): void;\n}\n\n/** Both drivers use SQLite file locks, including across Node and Bun runners. */\nexport function openLockDatabase(path: string): LockDatabase {\n if (process.versions.bun) {\n const { Database } = require(\"bun:sqlite\") as typeof import(\"bun:sqlite\");\n const database = new Database(path);\n return {\n exec: (sql) => database.run(sql),\n close() {\n database.close();\n },\n };\n }\n const { DatabaseSync } =\n require(\"node:sqlite\") as typeof import(\"node:sqlite\");\n return new DatabaseSync(path);\n}\n"]}
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
import { mkdir, rename, unlink, writeFile } from "node:fs/promises";
|
|
2
|
+
import { basename, dirname, join } from "node:path";
|
|
3
|
+
import { randomUUID } from "node:crypto";
|
|
4
|
+
/** Write a private JSON artifact and publish it with one atomic rename. */
|
|
5
|
+
export async function atomicWriteJson(path, value) {
|
|
6
|
+
const parent = dirname(path);
|
|
7
|
+
await mkdir(parent, { recursive: true, mode: 0o700 });
|
|
8
|
+
const temporary = join(parent, `.${basename(path)}.${randomUUID()}.tmp`);
|
|
9
|
+
try {
|
|
10
|
+
await writeFile(temporary, `${JSON.stringify(value, null, 2)}\n`, {
|
|
11
|
+
flag: "wx",
|
|
12
|
+
mode: 0o600,
|
|
13
|
+
});
|
|
14
|
+
await rename(temporary, path);
|
|
15
|
+
}
|
|
16
|
+
catch (error) {
|
|
17
|
+
await unlink(temporary).catch(() => { });
|
|
18
|
+
throw error;
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
//# sourceMappingURL=storage.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"storage.js","sourceRoot":"","sources":["storage.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,SAAS,EAAE,yBAAyB;AACpE,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE,kBAAkB;AACpD,OAAO,EAAE,UAAU,EAAE,oBAAoB;AAEzC,2EAA2E;AAC3E,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,IAAY,EACZ,KAAc;IAEd,MAAM,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC7B,MAAM,KAAK,CAAC,MAAM,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC;IACtD,MAAM,SAAS,GAAG,IAAI,CAAC,MAAM,EAAE,IAAI,QAAQ,CAAC,IAAI,CAAC,IAAI,UAAU,EAAE,MAAM,CAAC,CAAC;IACzE,IAAI,CAAC;QACH,MAAM,SAAS,CAAC,SAAS,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE;YAChE,IAAI,EAAE,IAAI;YACV,IAAI,EAAE,KAAK;SACZ,CAAC,CAAC;QACH,MAAM,MAAM,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;IAChC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,MAAM,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;QACxC,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC","sourcesContent":["import { mkdir, rename, unlink, writeFile } from \"node:fs/promises\";\nimport { basename, dirname, join } from \"node:path\";\nimport { randomUUID } from \"node:crypto\";\n\n/** Write a private JSON artifact and publish it with one atomic rename. */\nexport async function atomicWriteJson(\n path: string,\n value: unknown,\n): Promise<void> {\n const parent = dirname(path);\n await mkdir(parent, { recursive: true, mode: 0o700 });\n const temporary = join(parent, `.${basename(path)}.${randomUUID()}.tmp`);\n try {\n await writeFile(temporary, `${JSON.stringify(value, null, 2)}\\n`, {\n flag: \"wx\",\n mode: 0o600,\n });\n await rename(temporary, path);\n } catch (error) {\n await unlink(temporary).catch(() => {});\n throw error;\n }\n}\n"]}
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
import { readFile, rm } from "node:fs/promises";
|
|
2
|
+
import { dirname, join } from "node:path";
|
|
3
|
+
import { fileURLToPath } from "node:url";
|
|
4
|
+
import { canonicalJson, sha256 } from "./identity.js";
|
|
5
|
+
import { retainedFile } from "./results-lock.js";
|
|
6
|
+
import { atomicWriteJson } from "./storage.js";
|
|
7
|
+
function isNativeTranscript(artifact) {
|
|
8
|
+
return /^(?:sevro\.(?:semantic(?:\.[a-f0-9]{64})?|advisory)\.)?sevro\.native-transcripts\.bundle$/.test(artifact.id);
|
|
9
|
+
}
|
|
10
|
+
function transcriptArtifacts(evidence) {
|
|
11
|
+
return evidence.trials.flatMap((trial) => trial.artifactRefs.filter(isNativeTranscript).map((artifact) => ({
|
|
12
|
+
caseId: trial.caseId,
|
|
13
|
+
trial: trial.trial,
|
|
14
|
+
...artifact,
|
|
15
|
+
})));
|
|
16
|
+
}
|
|
17
|
+
function pruningRecord(evidence) {
|
|
18
|
+
return {
|
|
19
|
+
format: "sevro.pruned-native-transcripts.v1",
|
|
20
|
+
runId: evidence.runId,
|
|
21
|
+
artifacts: transcriptArtifacts(evidence),
|
|
22
|
+
};
|
|
23
|
+
}
|
|
24
|
+
function pruningPath(evidence) {
|
|
25
|
+
if (evidence.result.evidencePath === null)
|
|
26
|
+
throw new Error("native transcript retention requires a retained evidence path");
|
|
27
|
+
return join(dirname(evidence.result.evidencePath), "native-transcripts-pruned.json");
|
|
28
|
+
}
|
|
29
|
+
function transcriptPath(evidence, artifact) {
|
|
30
|
+
const path = fileURLToPath(artifact.path);
|
|
31
|
+
if (dirname(path) !== dirname(pruningPath(evidence)))
|
|
32
|
+
throw new Error("native transcript artifact is outside its retained result directory");
|
|
33
|
+
if (metadataPaths(evidence).includes(path))
|
|
34
|
+
throw new Error("native transcript artifact aliases retained result metadata");
|
|
35
|
+
if (otherArtifactPaths(evidence).includes(path))
|
|
36
|
+
throw new Error("native transcript artifact aliases other retained evidence");
|
|
37
|
+
return path;
|
|
38
|
+
}
|
|
39
|
+
function otherArtifactPaths(evidence) {
|
|
40
|
+
return evidence.trials
|
|
41
|
+
.flatMap((trial) => [
|
|
42
|
+
trial.rawResult.path,
|
|
43
|
+
trial.advisoryReview?.rawResult.path,
|
|
44
|
+
...trial.artifactRefs
|
|
45
|
+
.filter((ref) => !isNativeTranscript(ref))
|
|
46
|
+
.map((ref) => ref.path),
|
|
47
|
+
])
|
|
48
|
+
.filter((path) => path != null)
|
|
49
|
+
.map((path) => fileURLToPath(path));
|
|
50
|
+
}
|
|
51
|
+
function metadataPaths(evidence) {
|
|
52
|
+
const receipt = pruningPath(evidence);
|
|
53
|
+
return [
|
|
54
|
+
receipt,
|
|
55
|
+
evidence.result.evidencePath,
|
|
56
|
+
join(dirname(receipt), "result.json"),
|
|
57
|
+
join(dirname(receipt), "latest.json"),
|
|
58
|
+
...evidence.result.cases.flatMap((selected) => selected.trials.map((trial) => trial.artifactPath)),
|
|
59
|
+
];
|
|
60
|
+
}
|
|
61
|
+
async function transcriptPresent(evidence, artifact) {
|
|
62
|
+
const path = transcriptPath(evidence, artifact);
|
|
63
|
+
if (!(await retainedFile(path)))
|
|
64
|
+
return false;
|
|
65
|
+
if (sha256(await readFile(path)) !== artifact.sha256)
|
|
66
|
+
throw new Error("retained native transcript evidence hash differs");
|
|
67
|
+
return true;
|
|
68
|
+
}
|
|
69
|
+
/** Keep capture evidence immutable; a separate receipt records intentional removal. */
|
|
70
|
+
export async function pruneNativeTranscripts(evidence) {
|
|
71
|
+
const artifacts = transcriptArtifacts(evidence);
|
|
72
|
+
if (!artifacts.length)
|
|
73
|
+
return 0;
|
|
74
|
+
await atomicWriteJson(pruningPath(evidence), pruningRecord(evidence));
|
|
75
|
+
let removed = 0;
|
|
76
|
+
for (const artifact of artifacts) {
|
|
77
|
+
const path = transcriptPath(evidence, artifact);
|
|
78
|
+
if (!(await retainedFile(path)))
|
|
79
|
+
continue;
|
|
80
|
+
await rm(path);
|
|
81
|
+
removed++;
|
|
82
|
+
}
|
|
83
|
+
return removed;
|
|
84
|
+
}
|
|
85
|
+
/** Missing native bytes require an exact receipt bound to their original references. */
|
|
86
|
+
async function hasPruningRecord(evidence) {
|
|
87
|
+
const path = pruningPath(evidence);
|
|
88
|
+
const marked = await retainedFile(path);
|
|
89
|
+
if (marked) {
|
|
90
|
+
const receipt = JSON.parse(await readFile(path, "utf8"));
|
|
91
|
+
if (canonicalJson(receipt) !== canonicalJson(pruningRecord(evidence)))
|
|
92
|
+
throw new Error("native transcript pruning record differs from retained evidence");
|
|
93
|
+
}
|
|
94
|
+
return marked !== null;
|
|
95
|
+
}
|
|
96
|
+
export async function nativeTranscriptRetention(evidence) {
|
|
97
|
+
if (!evidence)
|
|
98
|
+
return [];
|
|
99
|
+
const marked = await hasPruningRecord(evidence);
|
|
100
|
+
const states = [];
|
|
101
|
+
for (const artifact of transcriptArtifacts(evidence)) {
|
|
102
|
+
const present = await transcriptPresent(evidence, artifact);
|
|
103
|
+
if (!present && !marked)
|
|
104
|
+
throw new Error("retained native transcript evidence is missing without a pruning record");
|
|
105
|
+
states.push({ caseId: artifact.caseId, pruned: !present });
|
|
106
|
+
}
|
|
107
|
+
return states;
|
|
108
|
+
}
|
|
109
|
+
//# sourceMappingURL=transcript-retention.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"transcript-retention.js","sourceRoot":"","sources":["transcript-retention.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,yBAAyB;AAChD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,kBAAkB;AAC1C,OAAO,EAAE,aAAa,EAAE,iBAAiB;AAEzC,OAAO,EAAE,aAAa,EAAE,MAAM,EAAE,sBAAmB;AAEnD,OAAO,EAAE,YAAY,EAAE,0BAAuB;AAC9C,OAAO,EAAE,eAAe,EAAE,qBAAkB;AAE5C,SAAS,kBAAkB,CAAC,QAA2B;IACrD,OAAO,2FAA2F,CAAC,IAAI,CACrG,QAAQ,CAAC,EAAE,CACZ,CAAC;AACJ,CAAC;AAED,SAAS,mBAAmB,CAAC,QAAyB;IACpD,OAAO,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,EAAE,CACvC,KAAK,CAAC,YAAY,CAAC,MAAM,CAAC,kBAAkB,CAAC,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;QAC/D,MAAM,EAAE,KAAK,CAAC,MAAM;QACpB,KAAK,EAAE,KAAK,CAAC,KAAK;QAClB,GAAG,QAAQ;KACZ,CAAC,CAAC,CACJ,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,QAAyB;IAC9C,OAAO;QACL,MAAM,EAAE,oCAAoC;QAC5C,KAAK,EAAE,QAAQ,CAAC,KAAK;QACrB,SAAS,EAAE,mBAAmB,CAAC,QAAQ,CAAC;KACzC,CAAC;AACJ,CAAC;AAED,SAAS,WAAW,CAAC,QAAyB;IAC5C,IAAI,QAAQ,CAAC,MAAM,CAAC,YAAY,KAAK,IAAI;QACvC,MAAM,IAAI,KAAK,CACb,+DAA+D,CAChE,CAAC;IACJ,OAAO,IAAI,CACT,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,YAAY,CAAC,EACrC,gCAAgC,CACjC,CAAC;AACJ,CAAC;AAED,SAAS,cAAc,CACrB,QAAyB,EACzB,QAA2B;IAE3B,MAAM,IAAI,GAAG,aAAa,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC;IAC1C,IAAI,OAAO,CAAC,IAAI,CAAC,KAAK,OAAO,CAAC,WAAW,CAAC,QAAQ,CAAC,CAAC;QAClD,MAAM,IAAI,KAAK,CACb,qEAAqE,CACtE,CAAC;IACJ,IAAI,aAAa,CAAC,QAAQ,CAAC,CAAC,QAAQ,CAAC,IAAI,CAAC;QACxC,MAAM,IAAI,KAAK,CACb,6DAA6D,CAC9D,CAAC;IACJ,IAAI,kBAAkB,CAAC,QAAQ,CAAC,CAAC,QAAQ,CAAC,IAAI,CAAC;QAC7C,MAAM,IAAI,KAAK,CACb,4DAA4D,CAC7D,CAAC;IACJ,OAAO,IAAI,CAAC;AACd,CAAC;AAED,SAAS,kBAAkB,CAAC,QAAyB;IACnD,OAAO,QAAQ,CAAC,MAAM;SACnB,OAAO,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC;QAClB,KAAK,CAAC,SAAS,CAAC,IAAI;QACpB,KAAK,CAAC,cAAc,EAAE,SAAS,CAAC,IAAI;QACpC,GAAG,KAAK,CAAC,YAAY;aAClB,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,kBAAkB,CAAC,GAAG,CAAC,CAAC;aACzC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC;KAC1B,CAAC;SACD,MAAM,CAAC,CAAC,IAAI,EAAkB,EAAE,CAAC,IAAI,IAAI,IAAI,CAAC;SAC9C,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,CAAC;AACxC,CAAC;AAED,SAAS,aAAa,CAAC,QAAyB;IAC9C,MAAM,OAAO,GAAG,WAAW,CAAC,QAAQ,CAAC,CAAC;IACtC,OAAO;QACL,OAAO;QACP,QAAQ,CAAC,MAAM,CAAC,YAAY;QAC5B,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,aAAa,CAAC;QACrC,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,aAAa,CAAC;QACrC,GAAG,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,EAAE,CAC5C,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,YAAY,CAAC,CACnD;KACF,CAAC;AACJ,CAAC;AAED,KAAK,UAAU,iBAAiB,CAC9B,QAAyB,EACzB,QAA2B;IAE3B,MAAM,IAAI,GAAG,cAAc,CAAC,QAAQ,EAAE,QAAQ,CAAC,CAAC;IAChD,IAAI,CAAC,CAAC,MAAM,YAAY,CAAC,IAAI,CAAC,CAAC;QAAE,OAAO,KAAK,CAAC;IAC9C,IAAI,MAAM,CAAC,MAAM,QAAQ,CAAC,IAAI,CAAC,CAAC,KAAK,QAAQ,CAAC,MAAM;QAClD,MAAM,IAAI,KAAK,CAAC,kDAAkD,CAAC,CAAC;IACtE,OAAO,IAAI,CAAC;AACd,CAAC;AAED,uFAAuF;AACvF,MAAM,CAAC,KAAK,UAAU,sBAAsB,CAAC,QAAyB;IACpE,MAAM,SAAS,GAAG,mBAAmB,CAAC,QAAQ,CAAC,CAAC;IAChD,IAAI,CAAC,SAAS,CAAC,MAAM;QAAE,OAAO,CAAC,CAAC;IAChC,MAAM,eAAe,CAAC,WAAW,CAAC,QAAQ,CAAC,EAAE,aAAa,CAAC,QAAQ,CAAC,CAAC,CAAC;IACtE,IAAI,OAAO,GAAG,CAAC,CAAC;IAChB,KAAK,MAAM,QAAQ,IAAI,SAAS,EAAE,CAAC;QACjC,MAAM,IAAI,GAAG,cAAc,CAAC,QAAQ,EAAE,QAAQ,CAAC,CAAC;QAChD,IAAI,CAAC,CAAC,MAAM,YAAY,CAAC,IAAI,CAAC,CAAC;YAAE,SAAS;QAC1C,MAAM,EAAE,CAAC,IAAI,CAAC,CAAC;QACf,OAAO,EAAE,CAAC;IACZ,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC;AAED,wFAAwF;AACxF,KAAK,UAAU,gBAAgB,CAAC,QAAyB;IACvD,MAAM,IAAI,GAAG,WAAW,CAAC,QAAQ,CAAC,CAAC;IACnC,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,IAAI,CAAC,CAAC;IACxC,IAAI,MAAM,EAAE,CAAC;QACX,MAAM,OAAO,GAAY,IAAI,CAAC,KAAK,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC;QAClE,IAAI,aAAa,CAAC,OAAO,CAAC,KAAK,aAAa,CAAC,aAAa,CAAC,QAAQ,CAAC,CAAC;YACnE,MAAM,IAAI,KAAK,CACb,iEAAiE,CAClE,CAAC;IACN,CAAC;IACD,OAAO,MAAM,KAAK,IAAI,CAAC;AACzB,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,yBAAyB,CAC7C,QAAgC;IAEhC,IAAI,CAAC,QAAQ;QAAE,OAAO,EAAE,CAAC;IACzB,MAAM,MAAM,GAAG,MAAM,gBAAgB,CAAC,QAAQ,CAAC,CAAC;IAChD,MAAM,MAAM,GAAG,EAAE,CAAC;IAClB,KAAK,MAAM,QAAQ,IAAI,mBAAmB,CAAC,QAAQ,CAAC,EAAE,CAAC;QACrD,MAAM,OAAO,GAAG,MAAM,iBAAiB,CAAC,QAAQ,EAAE,QAAQ,CAAC,CAAC;QAC5D,IAAI,CAAC,OAAO,IAAI,CAAC,MAAM;YACrB,MAAM,IAAI,KAAK,CACb,yEAAyE,CAC1E,CAAC;QACJ,MAAM,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,QAAQ,CAAC,MAAM,EAAE,MAAM,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;IAC7D,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC","sourcesContent":["import { readFile, rm } from \"node:fs/promises\";\nimport { dirname, join } from \"node:path\";\nimport { fileURLToPath } from \"node:url\";\nimport type { RunEvidenceData } from \"./schema-types\";\nimport { canonicalJson, sha256 } from \"./identity\";\nimport type { ArtifactReference } from \"./contract-types\";\nimport { retainedFile } from \"./results-lock\";\nimport { atomicWriteJson } from \"./storage\";\n\nfunction isNativeTranscript(artifact: ArtifactReference) {\n return /^(?:sevro\\.(?:semantic(?:\\.[a-f0-9]{64})?|advisory)\\.)?sevro\\.native-transcripts\\.bundle$/.test(\n artifact.id,\n );\n}\n\nfunction transcriptArtifacts(evidence: RunEvidenceData) {\n return evidence.trials.flatMap((trial) =>\n trial.artifactRefs.filter(isNativeTranscript).map((artifact) => ({\n caseId: trial.caseId,\n trial: trial.trial,\n ...artifact,\n })),\n );\n}\n\nfunction pruningRecord(evidence: RunEvidenceData) {\n return {\n format: \"sevro.pruned-native-transcripts.v1\",\n runId: evidence.runId,\n artifacts: transcriptArtifacts(evidence),\n };\n}\n\nfunction pruningPath(evidence: RunEvidenceData) {\n if (evidence.result.evidencePath === null)\n throw new Error(\n \"native transcript retention requires a retained evidence path\",\n );\n return join(\n dirname(evidence.result.evidencePath),\n \"native-transcripts-pruned.json\",\n );\n}\n\nfunction transcriptPath(\n evidence: RunEvidenceData,\n artifact: ArtifactReference,\n) {\n const path = fileURLToPath(artifact.path);\n if (dirname(path) !== dirname(pruningPath(evidence)))\n throw new Error(\n \"native transcript artifact is outside its retained result directory\",\n );\n if (metadataPaths(evidence).includes(path))\n throw new Error(\n \"native transcript artifact aliases retained result metadata\",\n );\n if (otherArtifactPaths(evidence).includes(path))\n throw new Error(\n \"native transcript artifact aliases other retained evidence\",\n );\n return path;\n}\n\nfunction otherArtifactPaths(evidence: RunEvidenceData) {\n return evidence.trials\n .flatMap((trial) => [\n trial.rawResult.path,\n trial.advisoryReview?.rawResult.path,\n ...trial.artifactRefs\n .filter((ref) => !isNativeTranscript(ref))\n .map((ref) => ref.path),\n ])\n .filter((path): path is string => path != null)\n .map((path) => fileURLToPath(path));\n}\n\nfunction metadataPaths(evidence: RunEvidenceData) {\n const receipt = pruningPath(evidence);\n return [\n receipt,\n evidence.result.evidencePath,\n join(dirname(receipt), \"result.json\"),\n join(dirname(receipt), \"latest.json\"),\n ...evidence.result.cases.flatMap((selected) =>\n selected.trials.map((trial) => trial.artifactPath),\n ),\n ];\n}\n\nasync function transcriptPresent(\n evidence: RunEvidenceData,\n artifact: ArtifactReference,\n) {\n const path = transcriptPath(evidence, artifact);\n if (!(await retainedFile(path))) return false;\n if (sha256(await readFile(path)) !== artifact.sha256)\n throw new Error(\"retained native transcript evidence hash differs\");\n return true;\n}\n\n/** Keep capture evidence immutable; a separate receipt records intentional removal. */\nexport async function pruneNativeTranscripts(evidence: RunEvidenceData) {\n const artifacts = transcriptArtifacts(evidence);\n if (!artifacts.length) return 0;\n await atomicWriteJson(pruningPath(evidence), pruningRecord(evidence));\n let removed = 0;\n for (const artifact of artifacts) {\n const path = transcriptPath(evidence, artifact);\n if (!(await retainedFile(path))) continue;\n await rm(path);\n removed++;\n }\n return removed;\n}\n\n/** Missing native bytes require an exact receipt bound to their original references. */\nasync function hasPruningRecord(evidence: RunEvidenceData) {\n const path = pruningPath(evidence);\n const marked = await retainedFile(path);\n if (marked) {\n const receipt: unknown = JSON.parse(await readFile(path, \"utf8\"));\n if (canonicalJson(receipt) !== canonicalJson(pruningRecord(evidence)))\n throw new Error(\n \"native transcript pruning record differs from retained evidence\",\n );\n }\n return marked !== null;\n}\n\nexport async function nativeTranscriptRetention(\n evidence: RunEvidenceData | null,\n) {\n if (!evidence) return [];\n const marked = await hasPruningRecord(evidence);\n const states = [];\n for (const artifact of transcriptArtifacts(evidence)) {\n const present = await transcriptPresent(evidence, artifact);\n if (!present && !marked)\n throw new Error(\n \"retained native transcript evidence is missing without a pruning record\",\n );\n states.push({ caseId: artifact.caseId, pruned: !present });\n }\n return states;\n}\n"]}
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
/** Narrow JSON containers before accessing untrusted members. */
|
|
2
|
+
export function isRecord(value) {
|
|
3
|
+
return value !== null && typeof value === "object" && !Array.isArray(value);
|
|
4
|
+
}
|
|
5
|
+
export function requireRecord(value, message) {
|
|
6
|
+
if (!isRecord(value))
|
|
7
|
+
throw new Error(message);
|
|
8
|
+
return value;
|
|
9
|
+
}
|
|
10
|
+
export function validQualifiedId(value) {
|
|
11
|
+
return (typeof value === "string" &&
|
|
12
|
+
/^[a-z][a-z0-9]*(?:[.-][a-z0-9]+)+$/.test(value));
|
|
13
|
+
}
|
|
14
|
+
export function containsControl(value) {
|
|
15
|
+
return Array.from(value).some((character) => character.charCodeAt(0) < 0x20 || character.charCodeAt(0) === 0x7f);
|
|
16
|
+
}
|
|
17
|
+
export function isUnknownArray(value) {
|
|
18
|
+
return Array.isArray(value);
|
|
19
|
+
}
|
|
20
|
+
export function isStringArray(value) {
|
|
21
|
+
return (isUnknownArray(value) && value.every((item) => typeof item === "string"));
|
|
22
|
+
}
|
|
23
|
+
export function isOptionalNonblankString(value) {
|
|
24
|
+
return (value === undefined || (typeof value === "string" && Boolean(value.trim())));
|
|
25
|
+
}
|
|
26
|
+
//# sourceMappingURL=value-guards.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"value-guards.js","sourceRoot":"","sources":["value-guards.ts"],"names":[],"mappings":"AAAA,iEAAiE;AACjE,MAAM,UAAU,QAAQ,CAAC,KAAc;IACrC,OAAO,KAAK,KAAK,IAAI,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC9E,CAAC;AAED,MAAM,UAAU,aAAa,CAC3B,KAAc,EACd,OAAe;IAEf,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC;QAAE,MAAM,IAAI,KAAK,CAAC,OAAO,CAAC,CAAC;IAC/C,OAAO,KAAK,CAAC;AACf,CAAC;AAED,MAAM,UAAU,gBAAgB,CAAC,KAAc;IAC7C,OAAO,CACL,OAAO,KAAK,KAAK,QAAQ;QACzB,oCAAoC,CAAC,IAAI,CAAC,KAAK,CAAC,CACjD,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,eAAe,CAAC,KAAa;IAC3C,OAAO,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,CAC3B,CAAC,SAAS,EAAE,EAAE,CACZ,SAAS,CAAC,UAAU,CAAC,CAAC,CAAC,GAAG,IAAI,IAAI,SAAS,CAAC,UAAU,CAAC,CAAC,CAAC,KAAK,IAAI,CACrE,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,KAAc;IAC3C,OAAO,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC9B,CAAC;AAED,MAAM,UAAU,aAAa,CAAC,KAAc;IAC1C,OAAO,CACL,cAAc,CAAC,KAAK,CAAC,IAAI,KAAK,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,OAAO,IAAI,KAAK,QAAQ,CAAC,CACzE,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,wBAAwB,CAAC,KAAc;IACrD,OAAO,CACL,KAAK,KAAK,SAAS,IAAI,CAAC,OAAO,KAAK,KAAK,QAAQ,IAAI,OAAO,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,CAC5E,CAAC;AACJ,CAAC","sourcesContent":["/** Narrow JSON containers before accessing untrusted members. */\nexport function isRecord(value: unknown): value is Record<string, unknown> {\n return value !== null && typeof value === \"object\" && !Array.isArray(value);\n}\n\nexport function requireRecord(\n value: unknown,\n message: string,\n): Record<string, unknown> {\n if (!isRecord(value)) throw new Error(message);\n return value;\n}\n\nexport function validQualifiedId(value: unknown): value is string {\n return (\n typeof value === \"string\" &&\n /^[a-z][a-z0-9]*(?:[.-][a-z0-9]+)+$/.test(value)\n );\n}\n\nexport function containsControl(value: string): boolean {\n return Array.from(value).some(\n (character) =>\n character.charCodeAt(0) < 0x20 || character.charCodeAt(0) === 0x7f,\n );\n}\n\nexport function isUnknownArray(value: unknown): value is unknown[] {\n return Array.isArray(value);\n}\n\nexport function isStringArray(value: unknown): value is string[] {\n return (\n isUnknownArray(value) && value.every((item) => typeof item === \"string\")\n );\n}\n\nexport function isOptionalNonblankString(value: unknown): boolean {\n return (\n value === undefined || (typeof value === \"string\" && Boolean(value.trim()))\n );\n}\n"]}
|
package/docs/architecture.md
CHANGED
|
@@ -4,7 +4,7 @@ Sevro owns generic execution/evidence contracts. Projects own cases and policy.
|
|
|
4
4
|
This lets Darrow and other integrations use the same runner without importing
|
|
5
5
|
their benchmark policy into it.
|
|
6
6
|
|
|
7
|
-
This page explains
|
|
7
|
+
This page explains Sevro's implementation and points to acceptance
|
|
8
8
|
tests for changes. [Current contracts](contracts.md) define public behavior;
|
|
9
9
|
[historical validation](https://github.com/BjRo/sevro/blob/main/docs/evidence.md) records bounded past observations. The
|
|
10
10
|
[engine slice](https://github.com/BjRo/sevro/blob/main/docs/engine-slice.md) retains its earlier scope and detailed lifecycle
|
|
@@ -12,7 +12,10 @@ explanation. Implementation paths and test scenarios here are navigation, not
|
|
|
12
12
|
evidence that a particular candidate passed validation.
|
|
13
13
|
|
|
14
14
|
[`engine.ts`](../src/engine.ts) remains the programmatic facade, and
|
|
15
|
-
[`cli.ts`](../src/cli.ts)
|
|
15
|
+
[`cli.ts`](../src/cli.ts) builds to the installed `dist/src/cli.js` entrypoint.
|
|
16
|
+
The package runs on Node.js and Bun; contributor commands still use Bun.
|
|
17
|
+
[Process adapters](../src/runtime-process.ts) normalize child processes and
|
|
18
|
+
Web Streams. [SQLite locking](../src/sqlite-lock.ts) selects the runtime's native driver, preserving shared file-lock semantics. The CLI
|
|
16
19
|
loads and assembles evaluation options; evaluation modules coordinate the run
|
|
17
20
|
and its trials.
|
|
18
21
|
|
package/docs/creating-cases.md
CHANGED
|
@@ -114,9 +114,8 @@ The [grader reference](builtin-graders-v1.md) defines their limits.
|
|
|
114
114
|
To grade skill selection or verify explicit invocation, use an extension that
|
|
115
115
|
prepares skill mounts and interprets complete native host evidence. The
|
|
116
116
|
[extension protocol](extension-protocol-v1.md) defines repository and plugin
|
|
117
|
-
invocations; the
|
|
118
|
-
approach.
|
|
119
|
-
this checkout can contain [changes beyond its published version](installing.md#published-package-and-checkout).
|
|
117
|
+
invocations; the repository's [guide evals](guide-evaluation.md) demonstrate the
|
|
118
|
+
approach.
|
|
120
119
|
|
|
121
120
|
## Reusable project policy
|
|
122
121
|
|
package/docs/development-cli.md
CHANGED
|
@@ -12,9 +12,14 @@ grading, evidence, and extension lifecycle.
|
|
|
12
12
|
and maintains the latest report index. Bare pruning keeps the newest ten results;
|
|
13
13
|
`--keep`, `--older-than`, and `--latest` select explicit retention policies.
|
|
14
14
|
|
|
15
|
-
Examples assume [Sevro and
|
|
15
|
+
Examples assume [Sevro and Node are on `PATH`](installing.md#make-the-command-available).
|
|
16
16
|
For a contributor checkout, use `bun src/cli.ts` in place of `sevro`.
|
|
17
17
|
|
|
18
|
+
Use JavaScript adapter modules for portable Node and Bun execution. Bun can
|
|
19
|
+
load TypeScript source adapters directly. Node's native TypeScript loading has
|
|
20
|
+
syntax and dependency-directory restrictions; compile adapters that need full
|
|
21
|
+
TypeScript support before selecting them with `--adapter-module`.
|
|
22
|
+
|
|
18
23
|
Use `sevro --help`, `sevro run --help`, `sevro report --help`, or
|
|
19
24
|
`sevro prune --help` for examples,
|
|
20
25
|
required options, and defaults without preparing a run. `sevro --version` prints
|
|
@@ -34,7 +39,7 @@ configuration are independent of the caller's files and runtime policy.
|
|
|
34
39
|
An ordinary run needs a case source and candidate route. For example:
|
|
35
40
|
|
|
36
41
|
```sh
|
|
37
|
-
sevro run --case-file ./case.json --adapter-module ./host-adapter.
|
|
42
|
+
sevro run --case-file ./case.json --adapter-module ./host-adapter.js
|
|
38
43
|
```
|
|
39
44
|
|
|
40
45
|
| Setting | Default |
|
|
@@ -108,9 +113,9 @@ An installed package records its name, version, and a digest of its packaged
|
|
|
108
113
|
runtime and public contract files without reading runner Git metadata. The CLI
|
|
109
114
|
derives the project digest from its revision and dirty patch, or from a bounded
|
|
110
115
|
content snapshot when the project has no Git revision. Result and run-state
|
|
111
|
-
directories are excluded from that snapshot and untracked Git identity. When running
|
|
116
|
+
directories are excluded from that snapshot and untracked Git identity. When running from source
|
|
112
117
|
for coordinated development, pass `--runner-checkout-root` with the absolute
|
|
113
|
-
path to
|
|
118
|
+
path to the running Sevro checkout. Sevro verifies that it is the code actually running and
|
|
114
119
|
records its revision and dirty-patch digest instead of package provenance.
|
|
115
120
|
`bun run test:package-install` verifies the packed package from a separate
|
|
116
121
|
temporary project without runner Git metadata.
|
|
@@ -118,7 +123,7 @@ temporary project without runner Git metadata.
|
|
|
118
123
|
```sh
|
|
119
124
|
sevro run --json \
|
|
120
125
|
--case-file /absolute/path/case.json \
|
|
121
|
-
--adapter-module /absolute/path/host-adapter.
|
|
126
|
+
--adapter-module /absolute/path/host-adapter.js \
|
|
122
127
|
--project-root /absolute/path/project \
|
|
123
128
|
--results-root /absolute/path/results \
|
|
124
129
|
--condition passive --trials 1 --threshold 1
|
|
@@ -355,7 +360,7 @@ separate turn event artifacts remain bounded and private. The last response
|
|
|
355
360
|
is graded, and only complete per-call usage and cost series are summed.
|
|
356
361
|
|
|
357
362
|
For cases with `sevro.semantic` checks, also pass
|
|
358
|
-
`--semantic-adapter-module /absolute/path/semantic-adapter.
|
|
363
|
+
`--semantic-adapter-module /absolute/path/semantic-adapter.js` to load an
|
|
359
364
|
explicit second `HostAdapter` for grading. Alternatively, select the bundled
|
|
360
365
|
Codex route with `--semantic-host codex --semantic-model <model>
|
|
361
366
|
--semantic-effort <effort>`. That route also requires `--codex-bin` and
|
|
@@ -367,7 +372,7 @@ runs after a complete candidate response in its own empty workspace; missing
|
|
|
367
372
|
or malformed semantic verdicts produce a grading error.
|
|
368
373
|
|
|
369
374
|
For generated Git or repository fixtures, add
|
|
370
|
-
`--advisory-adapter-module /absolute/path/advisory-adapter.
|
|
375
|
+
`--advisory-adapter-module /absolute/path/advisory-adapter.js` to request an
|
|
371
376
|
independent quality review. Alternatively, use `--advisory-host codex
|
|
372
377
|
--advisory-model <model> --advisory-effort <effort>` with the shared Codex binary
|
|
373
378
|
and auth file options. Repeat `--advisory-exclude <fixture-relative-path>` for
|
|
@@ -12,8 +12,9 @@ It validates the guide inventory's source/static destinations and case coverage.
|
|
|
12
12
|
It ignores code contents as links and excludes inactive fixture snapshots.
|
|
13
13
|
Local checks need no network. Run `bun run test:docs-examples` for the exact
|
|
14
14
|
marked tutorial commands in fresh consumer/contributor directories, including
|
|
15
|
-
expected verdicts and saved evidence. Candidate
|
|
16
|
-
|
|
15
|
+
expected verdicts and saved evidence. Candidate commands run against compiled JavaScript. The released-package
|
|
16
|
+
check uses the tutorial bundled in that published version; the two results
|
|
17
|
+
are reported separately. These commands create and clean temporary directories.
|
|
17
18
|
The [recipe verifier](../scripts/documentation-recipes.ts) runs the new packaged
|
|
18
19
|
projects separately from the current published version. It executes actual
|
|
19
20
|
isolated tool/check positive and negative controls and both native hosts' dry
|
|
@@ -23,7 +24,7 @@ are mandatory, including UV and mise's declared Node installation.
|
|
|
23
24
|
|
|
24
25
|
`package.json.version` is the authoritative current version. `bun run docs:sync`
|
|
25
26
|
updates only the marked [Current release](installing.md#current-release) section
|
|
26
|
-
with its exact installation
|
|
27
|
+
with its exact npm and Bun installation commands. Keep its markers and heading in place;
|
|
27
28
|
the command refuses missing, duplicate, or reversed markers and invalid versions.
|
|
28
29
|
The rest of the page stays ordinary Markdown. `check:docs` compares this section
|
|
29
30
|
without writing; release preparation and package-install validation also reject
|
|
@@ -5,7 +5,7 @@ result with the installed `sevro` command.
|
|
|
5
5
|
|
|
6
6
|
## Try the demo and create a project
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Use Node.js 24.13.0 or newer, or Bun 1.3.13. [Install Sevro](installing.md) and
|
|
9
9
|
[make its command available on `PATH`](installing.md#make-the-command-available),
|
|
10
10
|
then run:
|
|
11
11
|
|
|
@@ -52,7 +52,7 @@ sevro report --latest
|
|
|
52
52
|
The first run's adapter returns exactly `ready`; its regex assertion must pass.
|
|
53
53
|
The negative control returns `wrong`: expect exit `1`, completed execution and
|
|
54
54
|
grading, task `failed`, and the named `ready` check's reason. Then edit `case.json`
|
|
55
|
-
and `adapter.ts
|
|
55
|
+
and `adapter.js` (`adapter.ts` when initialized from contributor sources), prepare again and rerun. Explicit flags override saved settings;
|
|
56
56
|
`--trials 3` repeats the task without rewriting `run.json`.
|
|
57
57
|
|
|
58
58
|
Progress uses stderr at actual preparation, candidate, grading and evidence
|
|
@@ -88,16 +88,16 @@ Reports label bundles intentionally removed by that command as pruned.
|
|
|
88
88
|
|
|
89
89
|
## Choose another starter
|
|
90
90
|
|
|
91
|
-
| Template | Real assertions | Prerequisites and interpretation
|
|
92
|
-
| ------------------ | ------------------------------------------------------------------------------------------------------------------ |
|
|
93
|
-
| `response` | Exact `ready` response; negative control returns `wrong` | Bun; no model credentials |
|
|
94
|
-
| `coding` | Edit `add.sh`; isolated checks test positive, negative and zero addition | Bun plus `sandbox-exec` on macOS or `bubblewrap`/`socat` on Linux; negative control leaves subtraction bug |
|
|
95
|
-
| `repository-skill` | Actual `.agents/skills/startup-guide/SKILL.md` mount; explicit invocation evidence; correct start command and port | Bun, Git, Codex, native isolation, explicit model/effort and readable auth file; dry is unassessed |
|
|
91
|
+
| Template | Real assertions | Prerequisites and interpretation |
|
|
92
|
+
| ------------------ | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ |
|
|
93
|
+
| `response` | Exact `ready` response; negative control returns `wrong` | Node or Bun; no model credentials |
|
|
94
|
+
| `coding` | Edit `add.sh`; isolated checks test positive, negative and zero addition | Node or Bun plus `sandbox-exec` on macOS or `bubblewrap`/`socat` on Linux; negative control leaves subtraction bug |
|
|
95
|
+
| `repository-skill` | Actual `.agents/skills/startup-guide/SKILL.md` mount; explicit invocation evidence; correct start command and port | Node or Bun, Git, Codex, native isolation, explicit model/effort and readable auth file; dry is unassessed |
|
|
96
96
|
|
|
97
97
|
From a new parent directory, use `sevro init --template coding --directory ./coding-eval`
|
|
98
98
|
or `sevro init --template repository-skill --directory ./skill-eval`. Read its generated
|
|
99
99
|
README for controls and dry/live commands. Repository-skill `command.json` records
|
|
100
|
-
absolute
|
|
100
|
+
the selected runtime executable and absolute extension path; regenerate it after moving the project.
|
|
101
101
|
|
|
102
102
|
For model evaluations, replace a saved adapter route with explicit `--host`,
|
|
103
103
|
`--model`, `--effort` and host authentication options. Candidate, semantic and
|
package/docs/getting-started.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Your first evaluation
|
|
2
2
|
|
|
3
|
-
Run a deterministic case and inspect its evidence.
|
|
3
|
+
Run a deterministic case and inspect its evidence. Use Node.js 24.13.0 or newer, or Bun 1.3.13; no
|
|
4
4
|
Darrow checkout, Codex/Claude login, or model credentials are required. The
|
|
5
5
|
adapter returns `ready`; Sevro grades that response against `^ready$`.
|
|
6
6
|
This teaches the runner lifecycle, not the quality of an actual model.
|
|
@@ -33,15 +33,16 @@ Prepare a new directory:
|
|
|
33
33
|
```sh
|
|
34
34
|
mkdir sevro-demo
|
|
35
35
|
cd sevro-demo
|
|
36
|
-
|
|
36
|
+
npm init -y
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
-
Install the [current release](installing.md#current-release)
|
|
40
|
-
here to run the example. See [licensing](licensing.md) for the current terms and
|
|
39
|
+
Install the [current release](installing.md#current-release)
|
|
40
|
+
there, then return here to run the example. See [licensing](licensing.md) for the current terms and
|
|
41
41
|
the licenses retained by historical releases.
|
|
42
42
|
|
|
43
43
|
Make the local installation available on `PATH` for this shell, then call
|
|
44
|
-
`sevro` directly
|
|
44
|
+
`sevro` directly with Node on `PATH`. Bun users can use the
|
|
45
|
+
[explicit Bun invocation](installing.md#make-the-command-available).
|
|
45
46
|
|
|
46
47
|
<!-- sevro-example:consumer -->
|
|
47
48
|
|
|
@@ -50,7 +51,7 @@ export PATH="$PWD/node_modules/.bin:$PATH"
|
|
|
50
51
|
mkdir -p project
|
|
51
52
|
sevro run --json \
|
|
52
53
|
--case-file "$PWD/node_modules/@bjoernrochel/sevro/examples/basic/graded.json" \
|
|
53
|
-
--adapter-module "$PWD/node_modules/@bjoernrochel/sevro/examples/basic/host.
|
|
54
|
+
--adapter-module "$PWD/node_modules/@bjoernrochel/sevro/dist/examples/basic/host.js" \
|
|
54
55
|
--project-root "$PWD/project" --results-root "$PWD/sevro-results" \
|
|
55
56
|
--condition passive --trials 1 --threshold 1 > sevro-result.json
|
|
56
57
|
```
|
package/docs/guide-evaluation.md
CHANGED
|
@@ -5,7 +5,7 @@ and a small extension for guide-specific evidence and fixtures. The inventory an
|
|
|
5
5
|
cases remain under `.agents/skills/sevro-guide/evals/`. The
|
|
6
6
|
[extension modules](../scripts/guide/README.md) do not implement a second runner.
|
|
7
7
|
|
|
8
|
-
From
|
|
8
|
+
From the Sevro repository root, validate fixture preparation without launching a model:
|
|
9
9
|
|
|
10
10
|
```sh
|
|
11
11
|
bun run eval:guide --case-id orientation --host codex \
|
package/docs/identity-v1.md
CHANGED
|
@@ -30,7 +30,10 @@ SHA-256(UTF-8("sevro.identity.v1\n") || UTF-8(JCS(dimensions)))
|
|
|
30
30
|
|
|
31
31
|
The CLI derives `runnerBuildDigest` from sorted packaged file paths and their
|
|
32
32
|
SHA-256 bytes under `src/`, `schemas/`, `docs/`, and `examples/`, plus `README.md` and
|
|
33
|
-
`package.json`.
|
|
33
|
+
`package.json`. Installed JavaScript execution also hashes `dist/`, including
|
|
34
|
+
the compiled runtime, examples and source maps. This binds evidence to executed
|
|
35
|
+
output; raw checkout execution excludes generated builds. The same installed
|
|
36
|
+
artifact has the same build digest under Node and Bun. For a Git project, `projectDigest` binds the revision and dirty
|
|
34
37
|
patch digest without the checkout path. Without a revision it hashes a bounded
|
|
35
38
|
file snapshot, excluding `.git`, results, and run-state storage. Results and
|
|
36
39
|
run-state storage are also excluded from untracked Git identity, so repeated
|
package/docs/installing.md
CHANGED
|
@@ -1,8 +1,13 @@
|
|
|
1
1
|
# Install, update, or remove Sevro
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
<a id="node-support-preview"></a>
|
|
4
|
+
|
|
5
|
+
## Runtime requirements
|
|
6
|
+
|
|
7
|
+
The installed package contains compiled JavaScript and runs with Node.js
|
|
8
|
+
24.13.0 or newer, or Bun 1.3.13. Node users do not need Bun for installation
|
|
9
|
+
or execution. Development and release builds continue to use Bun. Native
|
|
10
|
+
isolation targets macOS and Ubuntu 24.04 x64. See the [bounded platform evidence](native-hosts.md#validated-platform-scope);
|
|
6
11
|
Windows remains unverified.
|
|
7
12
|
|
|
8
13
|
## Install locally
|
|
@@ -10,7 +15,7 @@ Windows remains unverified.
|
|
|
10
15
|
```sh
|
|
11
16
|
mkdir sevro-demo
|
|
12
17
|
cd sevro-demo
|
|
13
|
-
|
|
18
|
+
npm init -y
|
|
14
19
|
```
|
|
15
20
|
|
|
16
21
|
Then install the [current release](#current-release) in that directory.
|
|
@@ -23,10 +28,16 @@ Local installation records the selected version in the project lockfile.
|
|
|
23
28
|
|
|
24
29
|
## Current release
|
|
25
30
|
|
|
26
|
-
The current release is `1.
|
|
31
|
+
The current release is `1.1.0`. Install this exact version with npm:
|
|
27
32
|
|
|
28
33
|
```sh
|
|
29
|
-
|
|
34
|
+
npm install --save-exact @bjoernrochel/sevro@1.1.0
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Or with Bun:
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
bun add --exact @bjoernrochel/sevro@1.1.0
|
|
30
41
|
```
|
|
31
42
|
|
|
32
43
|
<!-- sevro-current-release:end -->
|
|
@@ -42,20 +53,24 @@ export PATH="$PWD/node_modules/.bin:$PATH"
|
|
|
42
53
|
|
|
43
54
|
This selects the locally installed version for the current shell, including
|
|
44
55
|
after changing directories. If your installed `sevro` is already on `PATH`,
|
|
45
|
-
no adjustment is needed.
|
|
46
|
-
|
|
56
|
+
no adjustment is needed. The executable uses `#!/usr/bin/env node`, so Node
|
|
57
|
+
must be on `PATH` for a direct `sevro` invocation. For Bun execution, use
|
|
58
|
+
`bun run --bun sevro` from the installation directory, or run
|
|
59
|
+
`bun ./node_modules/@bjoernrochel/sevro/dist/src/cli.js` with the same arguments.
|
|
60
|
+
Bun-only users can initialize with `bun init -y` and install with `bun add`.
|
|
47
61
|
|
|
48
62
|
## Published package and checkout
|
|
49
63
|
|
|
50
64
|
See [licensing](licensing.md) for Sevro's source-available terms and historical
|
|
51
65
|
licensing transitions.
|
|
52
66
|
|
|
53
|
-
The npm package contains
|
|
67
|
+
The npm package contains compiled runtime and example modules under `dist/`,
|
|
68
|
+
alongside TypeScript sources, schemas, examples, documentation,
|
|
54
69
|
LICENSE, and contribution guidance. It excludes tests, developer scripts, Git
|
|
55
70
|
metadata, and repository guide mounts. Use a source checkout for the guide.
|
|
56
71
|
|
|
57
|
-
|
|
58
|
-
|
|
72
|
+
Use the exact release tag and package integrity when establishing provenance;
|
|
73
|
+
see the [release workflow](releases.md).
|
|
59
74
|
|
|
60
75
|
## Work from source
|
|
61
76
|
|
|
@@ -65,7 +80,8 @@ cd sevro
|
|
|
65
80
|
bun install --frozen-lockfile
|
|
66
81
|
```
|
|
67
82
|
|
|
68
|
-
Run the [
|
|
83
|
+
Run the [contributor tutorial](getting-started.md#from-a-contributor-checkout).
|
|
84
|
+
To build and install a development archive, follow [Contributing](../CONTRIBUTING.md#install-a-development-archive).
|
|
69
85
|
`bun run test:package-install` packs and installs a local tarball in temporary
|
|
70
86
|
directories; it does not publish. [Contributing](../CONTRIBUTING.md) lists checks.
|
|
71
87
|
|
|
@@ -81,8 +97,10 @@ Rollback uses the prior exact version and another verification run.
|
|
|
81
97
|
## Remove
|
|
82
98
|
|
|
83
99
|
```sh
|
|
84
|
-
|
|
100
|
+
npm uninstall @bjoernrochel/sevro
|
|
85
101
|
```
|
|
86
102
|
|
|
103
|
+
With Bun, use `bun remove @bjoernrochel/sevro`.
|
|
104
|
+
|
|
87
105
|
This updates the dependency and lockfile, not saved cases, results, or model
|
|
88
106
|
logins. Remove those separately only when wanted.
|
package/docs/licensing.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Sevro licensing
|
|
2
2
|
|
|
3
|
-
[LICENSE](../LICENSE)
|
|
3
|
+
[LICENSE](../LICENSE) defines Sevro's licensing terms. Sevro Source Available
|
|
4
4
|
License 1.0 allows free use in all contexts, modification, and free redistribution
|
|
5
5
|
under the same terms. Commercial work, paid education, research, CI, and
|
|
6
6
|
consulting are permitted.
|
|
@@ -23,7 +23,7 @@ it does not promise that alternative licenses will never be offered.
|
|
|
23
23
|
|
|
24
24
|
Published `0.1.0-rc.1` and `0.1.0-rc.2` retain BUSL-1.1, their original
|
|
25
25
|
production grant, and their per-version change date. Each changes to MPL-2.0
|
|
26
|
-
two years after its first public availability.
|
|
26
|
+
two years after its first public availability. The current license does not revoke
|
|
27
27
|
or rewrite earlier grants. The [retained rc.2 license](licenses/BUSL-1.1-0.1.0-rc.2.txt)
|
|
28
28
|
records those terms.
|
|
29
29
|
|
package/docs/native-hosts.md
CHANGED
|
@@ -35,7 +35,7 @@ observations are also available. Unsupported instrumentation is refused.
|
|
|
35
35
|
|
|
36
36
|
Select `--host claude`, an absolute `--claude-bin`, and the desired model/effort.
|
|
37
37
|
The route supports passive candidate turns. Explicit `--claude-credential-file`
|
|
38
|
-
takes precedence. Otherwise
|
|
38
|
+
takes precedence. Otherwise Sevro forwards inherited API/OAuth
|
|
39
39
|
credentials, then uses the saved login from the macOS Keychain or saved credential
|
|
40
40
|
file. Trials receive only an access token; the parent refreshes a saved login and
|
|
41
41
|
writes it back, so evaluations leave your normal Claude login working. See the
|