@bjoernrochel/sevro 0.1.0-rc.1

package/docs/results-v1.md

@@ -0,0 +1,118 @@
+# Result and evidence contract v1
+
+The public CLI accepts `--json`. In that mode stdout contains one JSON result
+document and no progress text; diagnostics go to stderr. Human output names
+the same execution, grading, and task states. Every result names its schema
+version. A run with retained evidence names its absolute evidence path;
+pre-run configuration failures can return `null`.
+Each domain outcome also gets one human-readable line with its case, trial,
+outcome ID, and status. Domain outcomes remain independent of the task verdict
+and exit code.
+
+## Separate states
+
+Each trial and aggregate result carries:
+
+| Field              | Values                                               | Meaning                                                                           |
+| ------------------ | ---------------------------------------------------- | --------------------------------------------------------------------------------- |
+| `execution.status` | `completed`, `failed`, `cancelled`, `not_run`        | Whether host execution completed. Dry preparation is `not_run`.                   |
+| `grading.status`   | `completed`, `error`, `unavailable`, `not_requested` | Whether required grading completed. A prompt without criteria is `not_requested`. |
+| `task.verdict`     | `passed`, `failed`, `not_assessed`                   | Judgment against declared criteria, independent of process completion.            |
+
+With successful host execution and no success criteria, the result is
+`completed` / `not_requested` / `not_assessed`. A failed check after successful
+execution is `completed` / `completed` / `failed`. An execution failure,
+grader error, or unavailable required evidence cannot yield `passed`. A dry
+run is `not_run` / `not_requested` / `not_assessed` even when preparation checks
+pass. Missing optional cost or usage remains `null` or explicitly incomplete;
+it is never converted to zero.
+
+Each new trial records `candidateDurationMs` from a monotonic clock around the
+candidate host call and observation validation. Dry preparation records `null`;
+older artifacts without this field have unknown duration.
+
+For multiple trials, a case applies its declared pass threshold only after all
+required trials are assessed. An unassessed trial makes the case
+`not_assessed`; otherwise its pass rate determines `passed` or `failed`. The
+run verdict is `failed` if any case fails, otherwise `not_assessed` if any case
+is unassessed, otherwise `passed`. A failed run verdict can coexist with an
+execution or grading error from another case. Execution and grading aggregate
+states expose errors or unavailability instead of averaging them away.
+
+## Exit codes
+
+The CLI uses this precedence when several outcomes occur in one invocation:
+
+| Code  | Outcome                                                                                                                                                                             |
+| ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `0`   | No execution or grading error and no failed assessment. Includes prompt-only and successful dry runs whose task verdict is `not_assessed`; output distinguishes them from `passed`. |
+| `1`   | At least one completed task assessment failed.                                                                                                                                      |
+| `2`   | Host execution failed.                                                                                                                                                              |
+| `3`   | A required grader or extension failed.                                                                                                                                              |
+| `4`   | Required evidence was unavailable.                                                                                                                                                  |
+| `64`  | Invalid invocation or configuration before a run.                                                                                                                                   |
+| `70`  | Internal runner or persistence failure.                                                                                                                                             |
+| `130` | Interrupted by SIGINT.                                                                                                                                                              |
+| `143` | Interrupted by SIGTERM.                                                                                                                                                             |
+
+Interruption and internal failure outrank execution failure, which outranks
+grading error, unavailable evidence, and task failure. The exit code is a
+summary; the JSON result retains every affected trial's exact status.
+
+## Identity and provenance
+
+New retained evidence separately identifies:
+
+- the Sevro package name, exact version and build digest, or an explicit local
+  checkout revision and dirty-patch digest;
+- the evaluated project's revision and dirty-patch digest, without assuming it
+  is the runner checkout;
+- the extension ID, version, source/content digest, configuration digest,
+  negotiated protocol and capabilities, active graders, replaced defaults,
+  selected task-verdict policy, and requested and applied instrumentation;
+- each candidate, semantic grader, and advisory judge's exact host, model,
+  effort, raw-result provenance, and observation completeness; and
+- the declared passive/enforced condition, case inputs, check definitions,
+  required evidence, and execution mode.
+
+The evaluation identity includes evaluator version and content, configuration,
+grader selection and replacements, required evidence, instrumentation, host
+routes, case inputs, fixture, checks, trial count, and threshold. A comparison
+cannot claim matched conditions when those dimensions differ. Timestamps,
+temporary paths, run IDs, and output locations do not change evaluation
+identity. Historical Darrow artifacts keep their original schema version;
+fields absent from them remain unknown unless explicit historical provenance
+establishes the value.
+The pre-run instrumentation digest binds the requested list and its expected
+application. Trial evidence records the host-reported application; a mismatch
+fails execution and cannot be treated as a matched successful condition.
+When a task-verdict policy is selected, each trial records its recommendation
+separately from the final task verdict. A missing recommendation cannot produce
+a passing assessment.
+An optional advisory review records `completed`, `failed`, or `not_run` with
+its structured assessment, independent usage, and raw-response provenance.
+Its recommendation and any reviewer failure do not change execution, grading,
+task verdict, or exit code. A failure to retain review evidence remains a
+persistence error.
+
+[`identity-v1.md`](identity-v1.md) defines the canonical digest and each
+comparison dimension.
+
+The versioned machine-readable contracts are
+[`cli-result-v1.schema.json`](../schemas/cli-result-v1.schema.json) and
+[`run-evidence-v1.schema.json`](../schemas/run-evidence-v1.schema.json). A
+schema-valid document still needs engine validation for cross-record equality,
+canonical identity computation, path containment, secret redaction, and
+referenced artifact availability.
+
+The engine persists each completed trial atomically before fixture cleanup.
+An interrupted or failed run keeps completed trials plus an explicit partial
+attempt or diagnostic record. Failed persistence is an internal error and
+cannot be presented as a successful assessment.
+
+A host adapter can return `executionFailed: true` with bounded artifacts from
+a failed process. The engine retains those artifacts, skips all grading, and
+reports `failed` / `not_requested` / `not_assessed` with exit code `2`. A final
+message that happens to match a check cannot turn that failure into a pass.
+Claude retains its bounded event stream and, when present, up to 64 KiB of
+stderr as private host artifacts. Failure bodies do not appear in CLI diagnostics.

package/examples/basic/graded.json

@@ -0,0 +1,13 @@
+{
+  "id": "basic-graded",
+  "prompt": "Return ready.",
+  "fixture": { "files": { "README.md": "fixture\n" } },
+  "checks": [
+    {
+      "id": "ready",
+      "grader": "sevro.regex",
+      "configuration": { "pattern": "^ready$" }
+    }
+  ],
+  "requiredEvidence": []
+}

package/examples/basic/host.ts

@@ -0,0 +1,8 @@
+export default {
+  id: "sevro.host.example",
+  model: "deterministic-example",
+  effort: "none",
+  async run() {
+    return { finalMessage: "ready", complete: true };
+  },
+};

package/examples/basic/prompt-only.json

@@ -0,0 +1,7 @@
+{
+  "id": "basic-prompt-only",
+  "prompt": "Return ready.",
+  "fixture": { "files": { "README.md": "fixture\n" } },
+  "checks": [],
+  "requiredEvidence": []
+}

package/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "@bjoernrochel/sevro",
+  "version": "0.1.0-rc.1",
+  "license": "BUSL-1.1",
+  "description": "A standalone evaluation runner for agentic coding capabilities",
+  "type": "module",
+  "private": false,
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/BjRo/sevro.git"
+  },
+  "publishConfig": {
+    "registry": "https://registry.npmjs.org/",
+    "access": "public",
+    "tag": "next"
+  },
+  "packageManager": "bun@1.3.13",
+  "files": [
+    "README.md",
+    "docs/",
+    "examples/",
+    "schemas/",
+    "src/"
+  ],
+  "bin": {
+    "sevro": "./src/cli.ts"
+  },
+  "scripts": {
+    "cli:dev": "bun src/cli.ts",
+    "test": "bun test",
+    "test:package-install": "bun scripts/verify-package-install.ts",
+    "release:prepare": "bun scripts/prepare-release.ts",
+    "typecheck": "tsc --noEmit",
+    "format:check": "prettier --check README.md docs/*.md examples/**/*.json examples/**/*.ts schemas/*.json src/**/*.ts scripts/**/*.ts tests/**/*.ts package.json tsconfig.json .github/workflows/*.yml"
+  },
+  "dependencies": {
+    "ajv": "8.20.0",
+    "smol-toml": "1.8.0"
+  },
+  "devDependencies": {
+    "@types/bun": "^1.2.0",
+    "prettier": "^3.9.9",
+    "typescript": "^5.7.0"
+  }
+}

package/schemas/cli-result-v1.schema.json

@@ -0,0 +1,184 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "urn:sevro:schema:cli-result:v1",
+  "title": "Sevro CLI JSON result v1",
+  "type": "object",
+  "required": [
+    "format",
+    "runId",
+    "execution",
+    "grading",
+    "task",
+    "exitCode",
+    "evidencePath",
+    "cases"
+  ],
+  "properties": {
+    "format": { "const": "sevro.cli-result.v1" },
+    "runId": { "type": ["string", "null"] },
+    "execution": { "$ref": "#/$defs/execution" },
+    "grading": { "$ref": "#/$defs/grading" },
+    "task": { "$ref": "#/$defs/task" },
+    "exitCode": { "enum": [0, 1, 2, 3, 4, 64, 70, 130, 143] },
+    "evidencePath": { "type": ["string", "null"] },
+    "cases": { "type": "array", "items": { "$ref": "#/$defs/case" } },
+    "diagnostic": { "$ref": "#/$defs/diagnostic" }
+  },
+  "allOf": [{ "$ref": "#/$defs/stateConsistency" }],
+  "additionalProperties": false,
+  "$defs": {
+    "stateConsistency": {
+      "allOf": [
+        {
+          "if": {
+            "properties": {
+              "execution": {
+                "properties": {
+                  "status": { "enum": ["failed", "cancelled", "not_run"] }
+                }
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "task": {
+                "properties": { "verdict": { "not": { "const": "passed" } } }
+              }
+            }
+          }
+        },
+        {
+          "if": {
+            "properties": {
+              "grading": {
+                "properties": {
+                  "status": {
+                    "enum": ["error", "unavailable", "not_requested"]
+                  }
+                }
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "task": {
+                "properties": { "verdict": { "not": { "const": "passed" } } }
+              }
+            }
+          }
+        }
+      ]
+    },
+    "execution": {
+      "type": "object",
+      "required": ["status"],
+      "properties": {
+        "status": { "enum": ["completed", "failed", "cancelled", "not_run"] },
+        "errorCode": { "type": "string", "minLength": 1 }
+      },
+      "additionalProperties": false
+    },
+    "grading": {
+      "type": "object",
+      "required": ["status"],
+      "properties": {
+        "status": {
+          "enum": ["completed", "error", "unavailable", "not_requested"]
+        },
+        "errorCode": { "type": "string", "minLength": 1 }
+      },
+      "additionalProperties": false
+    },
+    "task": {
+      "type": "object",
+      "required": ["verdict"],
+      "properties": {
+        "verdict": { "enum": ["passed", "failed", "not_assessed"] }
+      },
+      "additionalProperties": false
+    },
+    "check": {
+      "type": "object",
+      "required": ["id", "grader", "status"],
+      "properties": {
+        "id": { "type": "string", "minLength": 1 },
+        "grader": { "type": "string", "minLength": 1 },
+        "status": { "enum": ["passed", "failed", "unavailable"] },
+        "detail": { "type": "string", "maxLength": 4096 },
+        "evidenceRefs": {
+          "type": "array",
+          "items": { "type": "string", "minLength": 1 },
+          "uniqueItems": true
+        }
+      },
+      "additionalProperties": false
+    },
+    "domainOutcome": {
+      "type": "object",
+      "required": ["id", "status", "evidenceRefs"],
+      "properties": {
+        "id": {
+          "type": "string",
+          "pattern": "^[a-z][a-z0-9]*(?:[.-][a-z0-9]+)+$",
+          "maxLength": 128
+        },
+        "status": { "enum": ["passed", "failed", "unavailable"] },
+        "detail": { "type": "string", "maxLength": 4096 },
+        "evidenceRefs": {
+          "type": "array",
+          "items": { "type": "string", "minLength": 1 },
+          "uniqueItems": true
+        },
+        "data": { "type": "object" }
+      },
+      "additionalProperties": false
+    },
+    "trial": {
+      "type": "object",
+      "required": [
+        "trial",
+        "execution",
+        "grading",
+        "task",
+        "checks",
+        "artifactPath"
+      ],
+      "properties": {
+        "trial": { "type": "integer", "minimum": 1 },
+        "execution": { "$ref": "#/$defs/execution" },
+        "grading": { "$ref": "#/$defs/grading" },
+        "task": { "$ref": "#/$defs/task" },
+        "checks": { "type": "array", "items": { "$ref": "#/$defs/check" } },
+        "domainOutcomes": {
+          "type": "array",
+          "items": { "$ref": "#/$defs/domainOutcome" }
+        },
+        "artifactPath": { "type": ["string", "null"] }
+      },
+      "allOf": [{ "$ref": "#/$defs/stateConsistency" }],
+      "additionalProperties": false
+    },
+    "case": {
+      "type": "object",
+      "required": ["caseId", "execution", "grading", "task", "trials"],
+      "properties": {
+        "caseId": { "type": "string", "minLength": 1 },
+        "execution": { "$ref": "#/$defs/execution" },
+        "grading": { "$ref": "#/$defs/grading" },
+        "task": { "$ref": "#/$defs/task" },
+        "trials": { "type": "array", "items": { "$ref": "#/$defs/trial" } }
+      },
+      "allOf": [{ "$ref": "#/$defs/stateConsistency" }],
+      "additionalProperties": false
+    },
+    "diagnostic": {
+      "type": "object",
+      "required": ["code", "message"],
+      "properties": {
+        "code": { "type": "string", "minLength": 1 },
+        "message": { "type": "string", "minLength": 1, "maxLength": 4096 }
+      },
+      "additionalProperties": false
+    }
+  }
+}