@dogfood-lab/verify 1.3.1 → 1.4.0

package/README.md

@@ -48,7 +48,8 @@ if (!result.ok) {
 
 | Validator | Purpose |
 |---|---|
-| `validators/schema.js` | JSON Schema check against `@dogfood-lab/schemas` |
+| `validators/schema.js` | JSON Schema check against `@dogfood-lab/schemas` (SHAPE gate) |
+| `validators/schema-version.js` | `schema_version` VALUE gate — refuses an incompatible MAJOR against `SUPPORTED_SCHEMA_VERSIONS` |
 | `validators/policy.js` | Per-repo policy compliance (prototype-pollution-safe deep merge) |
 | `validators/provenance.js` | GitHub Actions run-ID confirmation via API (with timeout guard) |
 | `validators/steps.js` | Step-by-step contract checks (gate accumulation, ordering) |
@@ -84,48 +85,67 @@ Provenance fields (`github_run_id`, `github_workflow_ref`) are required when `pr
 
 The verifier emits two prefix classes:
 
-**Submission-bad** (the submitter's payload failed a validator gate — the operator should fix the submission and resubmit):
+Discrimination happens by **class**, surfaced by `parseRejectionReason` (below). Every prefix maps to one of four classes: **submission-bad** (the submitter fixes the payload), **operational** (the verifier/tooling faulted), **ingest** (an ingest-side load fault), or **unknown** (unrecognized prefix).
+
+**Submission-bad** — `class: 'submission-bad'` (the submitter's payload failed a validator gate; fix the submission and resubmit):
 
 | Prefix | Source | Meaning |
 |---|---|---|
 | `schema:` | `validators/schema.js` | JSON Schema check on the submission/record envelope failed. The rest of the string carries the AJV path + message. |
-| `policy:` | `validators/policy.js` | Per-repo policy gate failed (forbidden tags, missing required fields, version-floor violation, etc.). |
+| `policy:` | `validators/policy.js` | Per-repo policy gate failed (forbidden tags, missing required fields, surface evidence/CI requirements, etc.). |
 | `steps[<id>]:` | `validators/steps.js` | Step-level contract check failed on a specific step id (gate accumulation, ordering, evidence shape). |
 | `provenance:` | `validators/provenance.js` | The GitHub run-id confirmation could not match the submitted commit/repo at the GitHub API. |
-| `scenario-load:` | `packages/ingest/load-context.js` | A scenario referenced by `scenario_results` could not be loaded from the source repo (typed-reason: `timeout` / `not_found` / `parse_error` / `invalid_id`). |
+| `repo:` | `index.js` cross-field guard | `submission.repo` does not match the owner/repo encoded in `source.run_url` (anti-forgery guard). Emitted as `repo:mismatch: …`. |
+| `submission-contains-verifier-field:` | `index.js` | The submission carried a verifier-owned field (`policy_version`, `verification`, or an object `overall_verdict`) it must not author. |
+| `CONTRACT_SCHEMA_TOO_NEW:` | `validators/schema-version.js` | The submission's `schema_version` declares a MAJOR **above** what this build supports (see `SUPPORTED_SCHEMA_VERSIONS` in `@dogfood-lab/schemas`). This build cannot understand a future contract — **the operator must upgrade testing-os**, but the routing class stays submission-bad (the payload as-shipped cannot be accepted by THIS build). |
+| `CONTRACT_SCHEMA_TOO_OLD:` | `validators/schema-version.js` | The submission's `schema_version` declares a MAJOR **below** the supported floor. **The submitter must re-emit** against the current contract. A patch/minor delta inside the supported major range is NOT rejected. |
 
-**Validator-crashed** (the validator itself threw an internal error — this is an operational fault, NOT submission-bad; the operator should investigate the verifier itself):
+**Operational** — `class: 'operational'` (the validator itself threw an internal error; investigate the verifier, do NOT bounce to the submitter):
 
 | Prefix | Source | Meaning |
 |---|---|---|
 | `VALIDATOR_FAULT_SCHEMA:` | `runValidator('schema', …)` catch | Internal exception inside the schema validator. The rest of the string carries the thrown `.message`. |
 | `VALIDATOR_FAULT_POLICY:` | `runValidator('policy', …)` catch | Internal exception inside the policy validator. |
 | `VALIDATOR_FAULT_STEPS:` | `runValidator('steps', …)` catch | Internal exception inside the steps validator. |
+| `VALIDATOR_FAULT_CONTRACT_SCHEMA_VERSION:` | `runValidator('contract_schema_version', …)` catch | The version gate was called with an unknown contract key (a programmer error at the call site, not a submission fault). |
+
+Any future `VALIDATOR_FAULT_<NEW>:` prefix is classified `operational` by family — `parseRejectionReason` matches the `VALIDATOR_FAULT_` head, so a new validator class needs no parser edit.
+
+**Ingest** — `class: 'ingest'` (an ingest-side load fault, not a verifier gate):
+
+| Prefix | Source | Meaning |
+|---|---|---|
+| `scenario-load:` | `packages/ingest/run.js` | A scenario referenced by `scenario_results` could not be loaded from the source repo (typed-reason: `timeout` / `not_found` / `parse_error` / `invalid_id`). |
 
 ### Operator hygiene
 
+Discriminate by **class**, not by hand-rolled `.startsWith()` chains. `parseRejectionReason(reason)` returns `{ class, prefix, detail }`:
+
 ```js
-// Discriminate by prefix
+import { parseRejectionReason } from '@dogfood-lab/verify';
+
 for (const r of result.rejection_reasons) {
-  if (r.startsWith('VALIDATOR_FAULT_')) {
-    // Operational incident — verifier-side. Page someone; do NOT route
-    // back to the submitter as a "fix your payload" message.
-    notifyOps(r);
-  } else if (r.startsWith('schema:') || r.startsWith('policy:') || r.startsWith('steps[')) {
-    // Submission-bad — surface to the submitter.
-    surfaceToSubmitter(r);
-  } else if (r.startsWith('provenance:')) {
-    // May be either class — GitHub API timeouts are operational; a real
-    // commit/repo mismatch is submission-bad. The string detail carries
-    // the discriminator.
-    triageProvenance(r);
-  } else if (r.startsWith('scenario-load:')) {
-    // Ingest-side: scenario fetch reason determines class. timeout is
-    // operational; not_found / parse_error / invalid_id are submission-bad.
-    triageScenarioLoad(r);
-  } else {
-    // Unknown prefix — log and surface as raw text.
-    log.warn('unknown rejection_reason prefix', r);
+  const { class: cls, prefix, detail } =
+    parseRejectionReason(r);
+  switch (cls) {
+    case 'operational':
+      // Verifier-side fault. Page ops; do NOT bounce
+      // back to the submitter as "fix your payload".
+      notifyOps(prefix, detail);
+      break;
+    case 'submission-bad':
+      // The payload failed a gate — surface to the
+      // submitter so they fix it and resubmit.
+      surfaceToSubmitter(prefix, detail);
+      break;
+    case 'ingest':
+      // Ingest-side scenario fetch. The typed reason in
+      // `detail` (timeout vs not_found/…) decides triage.
+      triageScenarioLoad(detail);
+      break;
+    default: // 'unknown'
+      // Unrecognized prefix — log + surface raw text.
+      log.warn('unknown rejection_reason', r);
   }
 }
 ```

package/index.js

@@ -9,7 +9,19 @@
 import { validateSubmissionSchema as _defaultValidateSubmissionSchema } from './validators/schema.js';
 import { validatePolicy as _defaultValidatePolicy } from './validators/policy.js';
 import { validateStepResults as _defaultValidateStepResults } from './validators/steps.js';
+import { validateSchemaVersion as _defaultValidateSchemaVersion } from './validators/schema-version.js';
 import { computeVerdict } from './validators/verdict.js';
+import { SUPPORTED_SCHEMA_VERSIONS } from '@dogfood-lab/schemas';
+
+// F1-CONTRACTS-003: re-export the rejection-reason classifier from the package
+// root so consumers `import { parseRejectionReason } from '@dogfood-lab/verify'`
+// instead of hand-rolling .startsWith() chains over the prefix taxonomy.
+export { parseRejectionReason } from './parse-rejection.js';
+
+// F1-CONTRACTS-001: the persisted record's `schema_version` is the SINGLE
+// source of truth from the contract package — not a hardcoded literal that
+// can drift from `SUPPORTED_SCHEMA_VERSIONS.record.current`.
+const RECORD_SCHEMA_VERSION = SUPPORTED_SCHEMA_VERSIONS.record.current;
 
 /**
  * D1B-003 (Stage C humanization): the SOLE catch wrapper for synchronous
@@ -75,7 +87,7 @@ export async function verify(submission, options) {
     // timing.finished_at). Mark _skipPersist so the ingest layer surfaces the
     // rejection without crashing the persist layer with `invalid repo format: undefined`.
     return {
-      schema_version: '1.0.0',
+      schema_version: RECORD_SCHEMA_VERSION,
       _skipPersist: true,
       verification: {
         status: 'rejected',
@@ -95,6 +107,7 @@ export async function verify(submission, options) {
   const validateSubmissionSchema = validatorOverrides.validateSubmissionSchema || _defaultValidateSubmissionSchema;
   const validateStepResults = validatorOverrides.validateStepResults || _defaultValidateStepResults;
   const validatePolicy = validatorOverrides.validatePolicy || _defaultValidatePolicy;
+  const validateSchemaVersion = validatorOverrides.validateSchemaVersion || _defaultValidateSchemaVersion;
   const now = new Date().toISOString();
   const reasons = [];
 
@@ -134,6 +147,26 @@ export async function verify(submission, options) {
     reasons.push(schemaRun.faultReason);
   }
 
+  // 1b. schema_version VALUE gate (F1-CONTRACTS-001)
+  // The schema check above gates `schema_version` by PATTERN only. This gate
+  // compares the declared MAJOR against `SUPPORTED_SCHEMA_VERSIONS` (the
+  // single source of truth in @dogfood-lab/schemas) and refuses an
+  // incompatible major instead of silently mis-validating a future contract
+  // against the live 1.x schema. It runs INDEPENDENT of `schemaResult.valid`:
+  // a future-major payload may also fail shape, but the version refusal is the
+  // operator-actionable signal and must land regardless. The validator emits a
+  // fully-prefixed `CONTRACT_SCHEMA_TOO_NEW:` / `CONTRACT_SCHEMA_TOO_OLD:`
+  // reason; an unknown-contract throw surfaces as
+  // `VALIDATOR_FAULT_CONTRACT_SCHEMA_VERSION` via the same runValidator seam.
+  const versionRun = runValidator('contract_schema_version', () => validateSchemaVersion(submission, 'recordSubmission'));
+  if (versionRun.ok) {
+    if (!versionRun.result.valid) {
+      reasons.push(...versionRun.result.errors);
+    }
+  } else {
+    reasons.push(versionRun.faultReason);
+  }
+
   // 2. Reject if submission includes verifier-owned fields
   const verifierFields = ['policy_version', 'verification'];
   for (const field of verifierFields) {
@@ -219,7 +252,7 @@ export async function verify(submission, options) {
 
   // 7. Assemble persisted record
   const persisted = {
-    schema_version: '1.0.0',
+    schema_version: RECORD_SCHEMA_VERSION,
     policy_version: policyVersion,
     run_id: submission.run_id,
     repo: submission.repo,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dogfood-lab/verify",
-  "version": "1.3.1",
+  "version": "1.4.0",
   "type": "module",
   "description": "Central verifier for testing-os. Validates submissions against schema and policy, produces persisted records.",
   "main": "index.js",

package/validators/policy.js

@@ -156,7 +156,15 @@ export function validatePolicy(submission, { globalPolicy, repoPolicy }) {
 
     if (ciReqs.coverage_min != null) {
       const coverageCheck = submission.ci_checks?.find(c => c.kind === 'coverage');
-      if (!coverageCheck) {
+      // ci_checks[].value is OPTIONAL in the submission schema (only id/kind/status
+      // are required). A value-less coverage check must be treated the SAME as a
+      // missing one: at the trust boundary the verifier rejects incomplete proof.
+      // Without this guard, `undefined < coverage_min` is false (and a coerced
+      // null/NaN is likewise not a real measurement), so the gate silently passed
+      // with no measured coverage.
+      const measured = coverageCheck && typeof coverageCheck.value === 'number'
+        && Number.isFinite(coverageCheck.value);
+      if (!coverageCheck || !measured) {
         errors.push(
           `surface[${surface}]: coverage_min is ${ciReqs.coverage_min}% but no coverage data provided`
         );

package/validators/schema-version.js

@@ -0,0 +1,111 @@
+/**
+ * schema_version VALUE gate — validators/schema-version.js
+ *
+ * F1-CONTRACTS-001 (Wave 2, HIGH). The contract JSON schemas gate
+ * `schema_version` by PATTERN only (`^\d+\.\d+\.\d+$`). Nothing compared the
+ * declared value against the set this build actually supports, so a
+ * submission declaring `schema_version: '2.0.0'` (or `99.0.0`) validated
+ * clean against the live 1.x schema whenever its shape happened to fit — a
+ * genuinely-incompatible future major was silently mis-validated instead of
+ * cleanly refused.
+ *
+ * This validator compares a payload's MAJOR against
+ * SUPPORTED_SCHEMA_VERSIONS (the single source of truth imported from
+ * `@dogfood-lab/schemas` — NOT a hand-copied literal) for the named contract
+ * and returns a TYPED rejection-reason string:
+ *
+ *   - major > maxMajor → `CONTRACT_SCHEMA_TOO_NEW: ...` (operator must upgrade
+ *     testing-os — this build does not understand the future contract).
+ *   - major < minMajor → `CONTRACT_SCHEMA_TOO_OLD: ...` (submitter must
+ *     re-emit against the current contract).
+ *   - in range         → no reason (PASS; a patch/minor delta inside the range
+ *     is accepted — do NOT reject on a non-major bump).
+ *   - malformed / absent version → no reason (the JSON-schema PATTERN gate
+ *     already owns shape; this validator only owns the VALUE comparison and
+ *     stays silent rather than double-reporting a shape fault).
+ *
+ * Unlike the `schema`/`policy`/`steps` validators (whose caller prepends a
+ * single fixed prefix), the TOO_NEW vs TOO_OLD discriminant is known HERE, at
+ * the branch that fired — so this validator emits the FULL prefixed reason
+ * string. `index.js` pushes it verbatim into `verification.rejection_reasons`
+ * via the SAME `runValidator` seam (a thrown unknown-contract fault still
+ * surfaces as `VALIDATOR_FAULT_CONTRACT_SCHEMA_VERSION`). The
+ * `CONTRACT_SCHEMA_*` prefixes join the documented prefix taxonomy (see
+ * verify/README.md → "Error shape").
+ *
+ * Return contract: `{ valid: boolean, errors: string[] }` — `errors` holds
+ * the already-prefixed `CONTRACT_SCHEMA_*: ...` reason(s).
+ */
+
+import { SUPPORTED_SCHEMA_VERSIONS } from '@dogfood-lab/schemas';
+
+/**
+ * Parse the MAJOR component of a semver-ish string. Returns null when the
+ * value is not a `\d+\.\d+\.\d+` triple — the shape gate owns that fault.
+ *
+ * @param {unknown} version
+ * @returns {number | null}
+ */
+function parseMajor(version) {
+  if (typeof version !== 'string') return null;
+  const m = version.match(/^(\d+)\.\d+\.\d+$/);
+  if (!m) return null;
+  return Number(m[1]);
+}
+
+/**
+ * Gate a payload's declared `schema_version` against the supported MAJOR
+ * range for a named contract.
+ *
+ * @param {object} payload - Payload carrying a `schema_version` field.
+ * @param {string} [contract='recordSubmission'] - SUPPORTED_SCHEMA_VERSIONS key.
+ * @returns {{ valid: boolean, errors: string[] }}
+ */
+export function validateSchemaVersion(payload, contract = 'recordSubmission') {
+  // Only gate payloads that actually declare a schema_version. A null/absent
+  // value is the shape gate's concern, not the value gate's.
+  if (!payload || typeof payload !== 'object') {
+    return { valid: true, errors: [] };
+  }
+  const declared = payload.schema_version;
+  if (declared === undefined || declared === null) {
+    return { valid: true, errors: [] };
+  }
+
+  const supported = SUPPORTED_SCHEMA_VERSIONS[contract];
+  if (!supported) {
+    // Unknown contract key is a programmer error in the call site, not a
+    // submission fault — throw so the runValidator seam surfaces it as a
+    // VALIDATOR_FAULT_* operational incident rather than a submission reason.
+    throw new Error(`validateSchemaVersion: unknown contract "${contract}"`);
+  }
+
+  const major = parseMajor(declared);
+  if (major === null) {
+    // Malformed shape — defer to the JSON-schema PATTERN gate; do not
+    // double-report.
+    return { valid: true, errors: [] };
+  }
+
+  if (major > supported.maxMajor) {
+    return {
+      valid: false,
+      errors: [
+        `CONTRACT_SCHEMA_TOO_NEW: ${contract} schema v${declared} but this build supports ` +
+          `v${supported.current} (major ${supported.minMajor}–${supported.maxMajor}) — upgrade testing-os`,
+      ],
+    };
+  }
+
+  if (major < supported.minMajor) {
+    return {
+      valid: false,
+      errors: [
+        `CONTRACT_SCHEMA_TOO_OLD: ${contract} schema v${declared} is below the supported floor ` +
+          `v${supported.current} (major ${supported.minMajor}–${supported.maxMajor}) — re-emit against the current contract`,
+      ],
+    };
+  }
+
+  return { valid: true, errors: [] };
+}

package/validators/steps.js

@@ -25,7 +25,11 @@ export function validateStepResults(scenarioResult) {
     return errors;
   }
 
-  const VALID_STATUSES = new Set(['pass', 'fail', 'blocked', 'skip']);
+  // Must match the step_results[].status enum in BOTH dogfood-record-submission.schema.json
+  // and dogfood-record.schema.json (["pass","fail","blocked","skip","partial"]). `partial`
+  // is a first-class, contract-blessed step status (verdict.js ranks it 2/3); omitting it
+  // here falsely rejected schema-valid submissions as "unknown status".
+  const VALID_STATUSES = new Set(['pass', 'fail', 'blocked', 'skip', 'partial']);
 
   for (let i = 0; i < step_results.length; i++) {
     const step = step_results[i];