@dogfood-lab/verify 1.9.0 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -16
- package/cli-lint.js +5 -5
- package/cli.js +16 -11
- package/index.js +110 -0
- package/package.json +1 -1
- package/parse-rejection.js +116 -28
- package/validators/f-2a5ddafa-provenance-retry-warn.test.js +226 -0
- package/validators/f-f50e779b-retry-warn-fallback-guard.test.js +153 -0
- package/validators/policy.js +22 -7
- package/validators/provenance-registry.test.js +35 -0
- package/validators/provenance.js +127 -8
- package/validators/repo-binding.js +17 -2
- package/validators/repo-binding.test.js +40 -0
- package/validators/steps.js +77 -0
- package/validators/verdict.js +9 -1
package/README.md
CHANGED
|
@@ -109,21 +109,23 @@ The verifier emits rejection-reason strings under stable prefixes, each mapping
|
|
|
109
109
|
|
|
110
110
|
Discrimination happens by **class**, surfaced by `parseRejectionReason` (below). Every prefix maps to one of four classes: **submission-bad** (the submitter fixes the payload), **operational** (the verifier/tooling faulted), **ingest** (an ingest-side load fault), or **unknown** (unrecognized prefix).
|
|
111
111
|
|
|
112
|
+
**Retryable** (F-f8952a50, wave 10) is a SEPARATE, narrower per-prefix flag, orthogonal to `class`: may a same-`run_id` resubmission whose ONLY prior rejection carries this prefix still reach an acceptance, once corrected? `packages/ingest/persist.js`'s duplicate guard (`isDuplicate` → `isRetryableRejection`) reads this flag — never `class` or `prefix` directly — to decide whether a stale `_rejected` record blocks a same-`run_id` retry that is now headed to acceptance. Every `operational` / `ingest` / `unknown` prefix is `retryable: false` (an ops fault or an unrecognized signal is never the submitter's to retry past). Within `submission-bad`, the taxonomy splits further — **shape/addressing** prefixes ("we could not even read/place/shape your submission") are retryable; **content-verdict** prefixes ("we read your submission and rendered a verdict against its own reported content") are not, so a submitter cannot launder a genuinely-bad run into an accepted one by resubmitting different self-reported content under the same `run_id`. The table below marks each `submission-bad` prefix's retryable value explicitly.
|
|
113
|
+
|
|
112
114
|
**Submission-bad** — `class: 'submission-bad'` (the submitter's payload failed a validator gate; fix the submission and resubmit):
|
|
113
115
|
|
|
114
|
-
| Prefix | Source | Meaning |
|
|
115
|
-
|
|
116
|
-
| `schema:` | `validators/schema.js` | JSON Schema check on the submission/record envelope failed. The rest of the string carries the AJV path + message. |
|
|
117
|
-
| `policy:` | `validators/policy.js` | Per-repo policy gate failed (forbidden tags, missing required fields, surface evidence/CI requirements, or a declarative `when`/`custom_rules` predicate matched — see the [policy DSL](https://dogfood-lab.github.io/testing-os/handbook/policy-dsl/)). |
|
|
118
|
-
| `policy-config:` | `validators/policy.js` | **VERIFY-F1.** A REPO custom-rule predicate hit an eval-time semantic fault the schema could not catch — an unknown leading field, a numeric operator against a non-number, or a depth/width/fan-out budget. The repo authored the bad
|
|
119
|
-
| `steps[<id>]:` | `validators/steps.js` | Step-level contract check failed on a specific step id (gate accumulation, ordering, evidence shape). |
|
|
120
|
-
| `provenance:` | `validators/provenance.js` | The run was genuinely **absent / not confirmable** — a 404 from the provider API, or the run head did not match the submitted commit/repo. The
|
|
121
|
-
| `repo:` | `index.js` cross-field guard | `submission.repo` does not match the owner/repo encoded in `source.run_url` (anti-forgery guard). Emitted as `repo:mismatch: …`. |
|
|
122
|
-
| `submission-contains-verifier-field:` | `index.js` | The submission carried a verifier-owned field (`policy_version`, `verification`, or an object `overall_verdict`) it must not author. |
|
|
123
|
-
| `
|
|
124
|
-
| `
|
|
125
|
-
|
|
126
|
-
**Operational** — `class: 'operational'` (the validator itself threw an internal error; investigate the verifier, do NOT bounce to the submitter):
|
|
116
|
+
| Prefix | Retryable | Source | Meaning |
|
|
117
|
+
|---|---|---|---|
|
|
118
|
+
| `schema:` | Yes — shape | `validators/schema.js` | JSON Schema check on the submission/record envelope failed. The rest of the string carries the AJV path + message. |
|
|
119
|
+
| `policy:` | **No — content verdict** | `validators/policy.js` | Per-repo policy gate failed (forbidden tags, missing required fields, surface evidence/CI requirements, or a declarative `when`/`custom_rules` predicate matched — see the [policy DSL](https://dogfood-lab.github.io/testing-os/handbook/policy-dsl/)). Judges the run's OWN reported content (tags, evidence, scenario results); consuming the run_id is the deliberate anti-gaming behavior. |
|
|
120
|
+
| `policy-config:` | Yes — shape | `validators/policy.js` | **VERIFY-F1.** A REPO custom-rule predicate hit an eval-time semantic fault the schema could not catch — an unknown leading field, a numeric operator against a non-number, or a depth/width/fan-out budget. The repo authored the bad RULE (config), not the run's content, so the fix belongs to the submitter and is retryable. (A malformed GLOBAL predicate is `VALIDATOR_FAULT_POLICY:` operational instead — see below.) |
|
|
121
|
+
| `steps[<id>]:` | Yes — shape | `validators/steps.js` | Step-level contract check failed on a specific step id (gate accumulation, ordering, evidence shape — a completeness/structure mismatch, not a verdict on whether the steps passed). |
|
|
122
|
+
| `provenance:` | **No — content verdict** | `validators/provenance.js` | The run was genuinely **absent / not confirmable** — a 404 from the provider API, or the run head did not match the submitted commit/repo. The provider could not confirm THIS specific run happened as claimed; resubmitting different self-reported content under the same run_id to "become confirmable" is exactly the laundering the anti-gaming doctrine blocks. (Operational provider faults — 429/5xx/401/403 — are NOT this class; see `provenance-fault:` below.) |
|
|
123
|
+
| `repo:` | Yes — shape | `index.js` cross-field guard | `submission.repo` does not match the owner/repo encoded in `source.run_url` (anti-forgery guard). Emitted as `repo:mismatch: …`. Pure identity/addressing — the run happened, only the repo/run_url pairing was mis-stated. |
|
|
124
|
+
| `submission-contains-verifier-field:` | Yes — shape | `index.js` | The submission carried a verifier-owned field (`policy_version`, `verification`, or an object `overall_verdict`) it must not author. |
|
|
125
|
+
| `CONTRACT_SCHEMA_TOO_OLD:` | Yes — shape | `validators/schema-version.js` | The submission's `schema_version` declares a MAJOR **below** the supported floor. **The submitter must re-emit** against the current contract. A patch/minor delta inside the supported major range is NOT rejected. (NOT symmetric with `CONTRACT_SCHEMA_TOO_NEW:` below — see that row for why.) |
|
|
126
|
+
| `unsafe-record-path:` | Yes — shape | `packages/ingest/run.js`, `writeRecord()` catch | The record passed schema validation but `computeRecordPath()`'s traversal guard (`isUnsafeSegment`, stricter than the schema's `repo` pattern — e.g. `../etc`) still refused to place it on disk. The submitter's own `repo` string is the problem; nothing is persisted (there is no safe path to write to). |
|
|
127
|
+
|
|
128
|
+
**Operational** — `class: 'operational'`, always `retryable: false` (the validator itself threw an internal error; investigate the verifier, do NOT bounce to the submitter):
|
|
127
129
|
|
|
128
130
|
| Prefix | Source | Meaning |
|
|
129
131
|
|---|---|---|
|
|
@@ -131,13 +133,14 @@ Discrimination happens by **class**, surfaced by `parseRejectionReason` (below).
|
|
|
131
133
|
| `VALIDATOR_FAULT_POLICY:` | `runValidator('policy', …)` catch | Internal exception inside the policy validator — including a **GLOBAL** declarative-rule predicate fault (VERIFY-F1): a broken `policies/global-policy.yaml` is an ops incident (the studio's own config), so its predicate fault throws here rather than bouncing to the submitter. The repo-authored counterpart is `policy-config:` submission-bad. |
|
|
132
134
|
| `VALIDATOR_FAULT_STEPS:` | `runValidator('steps', …)` catch | Internal exception inside the steps validator. |
|
|
133
135
|
| `VALIDATOR_FAULT_CONTRACT_SCHEMA_VERSION:` | `runValidator('contract_schema_version', …)` catch | The version gate was called with an unknown contract key (a programmer error at the call site, not a submission fault). |
|
|
136
|
+
| `CONTRACT_SCHEMA_TOO_NEW:` | `validators/schema-version.js` | **F-be0deacd (wave 20).** The submission's `schema_version` declares a MAJOR **above** what this build supports (see `SUPPORTED_SCHEMA_VERSIONS` in `@dogfood-lab/schemas`) — THIS BUILD is behind a schema major its own submitters have already adopted. No resubmission, corrected or not, can ever satisfy a `major > maxMajor` comparison until testing-os itself ships an upgrade — **the operator must upgrade testing-os**. Page ops; do NOT bounce it back to the submitter. Unlike every other row in this table, `validators/schema-version.js` RETURNS this as an ordinary rejection string rather than throwing, so (unlike the `VALIDATOR_FAULT_*`/`provenance-fault:`/`scenario-fetch-fault:` rows) it genuinely persists to `records/_rejected/` and is reachable by `packages/ingest/persist.js`'s `isRetryableRejection()` — **F-51780da9 (wave 22):** rather than trusting the frozen `retryable: false` this prefix's own classification carries, `isRetryableRejection()` re-derives retryability for this ONE prefix against the CURRENT build's `SUPPORTED_SCHEMA_VERSIONS` ceiling, so a stale TOO_NEW rejection unblocks once the operator upgrades testing-os past the declared major. Distinct from the submission-bad `CONTRACT_SCHEMA_TOO_OLD:` above — the two prefixes are asymmetric despite the shared emitter. |
|
|
134
137
|
| `submission-malformed:` | `index.js` null/non-object early-return | The submission itself was `null` or not an object — a malfunctioning **dispatcher** sent garbage, not a submitter who authored a bad-but-shaped payload. Page ops / inspect the dispatch pipeline; do NOT bounce it to a submitter. |
|
|
135
138
|
| `provenance-fault:` | `index.js` provenance catch | The provenance adapter THREW an operational error confirming the run — a provider **429 rate-limit, 5xx outage, or 401/403 token** fault (`validators/provenance.js` throws these on purpose for non-404 responses). The submitter's payload is fine; the verifier could not reach a verdict. Page ops / retry; do NOT bounce it to a submitter. Distinct from the submission-bad `provenance:` (genuine absence/404). |
|
|
136
139
|
| `scenario-fetch-fault:` | `packages/ingest/load-context.js` | The scenario fetcher THREW after exhausting its retry budget (**5xx/429 outage, transport reject**) or hit a **401/403 credential** fault loading a scenario definition. The submission may be perfectly good — the fetch infrastructure faulted. The ingest CLI lets this propagate (exit 2, nothing persisted); a true missing file is the ingest-class `scenario-load: … (reason: not_found)` instead. |
|
|
137
140
|
|
|
138
141
|
Any future `VALIDATOR_FAULT_<NEW>:` prefix is classified `operational` by family — `parseRejectionReason` matches the `VALIDATOR_FAULT_` head, so a new validator class needs no parser edit. The `submission-malformed:` prefix is matched literally (it is not part of the `VALIDATOR_FAULT_` family).
|
|
139
142
|
|
|
140
|
-
**Ingest** — `class: 'ingest'` (an ingest-side load fault, not a verifier gate):
|
|
143
|
+
**Ingest** — `class: 'ingest'`, always `retryable: false` (an ingest-side load fault, not a verifier gate):
|
|
141
144
|
|
|
142
145
|
| Prefix | Source | Meaning |
|
|
143
146
|
|---|---|---|
|
|
@@ -145,13 +148,13 @@ Any future `VALIDATOR_FAULT_<NEW>:` prefix is classified `operational` by family
|
|
|
145
148
|
|
|
146
149
|
### Operator hygiene
|
|
147
150
|
|
|
148
|
-
Discriminate by **class**, not by hand-rolled `.startsWith()` chains. `parseRejectionReason(reason)` returns `{ class, prefix, detail }`:
|
|
151
|
+
Discriminate by **class**, not by hand-rolled `.startsWith()` chains. `parseRejectionReason(reason)` returns `{ class, prefix, detail, retryable }`:
|
|
149
152
|
|
|
150
153
|
```js
|
|
151
154
|
import { parseRejectionReason } from '@dogfood-lab/verify';
|
|
152
155
|
|
|
153
156
|
for (const r of result.rejection_reasons) {
|
|
154
|
-
const { class: cls, prefix, detail } =
|
|
157
|
+
const { class: cls, prefix, detail, retryable } =
|
|
155
158
|
parseRejectionReason(r);
|
|
156
159
|
switch (cls) {
|
|
157
160
|
case 'operational':
|
package/cli-lint.js
CHANGED
|
@@ -40,11 +40,11 @@ class LintOperatorError extends Error {
|
|
|
40
40
|
|
|
41
41
|
const LINT_USAGE = `dogfood-verify lint — author-time static check for a policy or scenario file
|
|
42
42
|
|
|
43
|
-
|
|
43
|
+
Usage:
|
|
44
44
|
dogfood-verify lint <policy-file> [--json]
|
|
45
45
|
dogfood-verify lint --scenario <scenario-file> [--json]
|
|
46
46
|
|
|
47
|
-
|
|
47
|
+
What it checks — policy mode (default, no submission needed):
|
|
48
48
|
- structural validity against policy.schema.json
|
|
49
49
|
- every predicate's known leading field, combinator depth, and node budget
|
|
50
50
|
- an ADVISORY warning on the [] footgun (a negative op over a [] path fails open)
|
|
@@ -52,7 +52,7 @@ WHAT IT CHECKS — policy mode (default, no submission needed):
|
|
|
52
52
|
It CANNOT statically catch a type_mismatch or a fanout_budget overrun — those depend
|
|
53
53
|
on submission data. Run \`dogfood-verify --file <submission> --explain\` for that.
|
|
54
54
|
|
|
55
|
-
|
|
55
|
+
What it checks — --scenario mode (no submission needed):
|
|
56
56
|
- structural validity against scenario.schema.json
|
|
57
57
|
- every success_criteria.required_steps entry references a declared steps[].id
|
|
58
58
|
- step ids are unique
|
|
@@ -63,12 +63,12 @@ WHAT IT CHECKS — --scenario mode (no submission needed):
|
|
|
63
63
|
It CANNOT verify that a real submission's step_results satisfy required_steps, nor
|
|
64
64
|
that the receiver can fetch the file at the attested commit — run a real ingest.
|
|
65
65
|
|
|
66
|
-
|
|
66
|
+
Options:
|
|
67
67
|
--scenario Lint the file as a scenario definition (default: policy).
|
|
68
68
|
--json Machine-readable result for CI.
|
|
69
69
|
-h, --help Show this help.
|
|
70
70
|
|
|
71
|
-
|
|
71
|
+
Exit codes:
|
|
72
72
|
0 clean or warnings-only 1 errors found 2 operator error (bad flags / IO)`;
|
|
73
73
|
|
|
74
74
|
/**
|
package/cli.js
CHANGED
|
@@ -67,6 +67,21 @@ class OperatorError extends Error {
|
|
|
67
67
|
}
|
|
68
68
|
}
|
|
69
69
|
|
|
70
|
+
/**
|
|
71
|
+
* V2-CONTRACT-004: appended to EVERY --explain rendering (accepted and
|
|
72
|
+
* rejected alike) so a preview verdict can never be read as covering the
|
|
73
|
+
* required_steps gate, which only production ingest enforces (it fetches the
|
|
74
|
+
* scenario definitions this preview does not have).
|
|
75
|
+
*
|
|
76
|
+
* Declared ahead of USAGE and interpolated directly into its own "Not checked
|
|
77
|
+
* in preview" section below (F-e0c3fa21 — that section used to restate the
|
|
78
|
+
* header as its own redundant lowercase lead-in, worded slightly differently
|
|
79
|
+
* than this constant) so --help and --explain describe the identical gap
|
|
80
|
+
* from one shared string and can never drift apart again.
|
|
81
|
+
*/
|
|
82
|
+
const PREVIEW_GAP_NOTE =
|
|
83
|
+
'Not checked in preview: required_steps (needs scenario definitions — enforced only by real ingest).';
|
|
84
|
+
|
|
70
85
|
const USAGE = `verify — local dry-run / explain for dogfood submissions
|
|
71
86
|
|
|
72
87
|
Usage:
|
|
@@ -92,8 +107,7 @@ Provenance (default: stub):
|
|
|
92
107
|
|
|
93
108
|
-h, --help Show this help.
|
|
94
109
|
|
|
95
|
-
|
|
96
|
-
not checked in preview: required_steps (needs scenario definitions).
|
|
110
|
+
${PREVIEW_GAP_NOTE}
|
|
97
111
|
Scenario definitions live in the source repo; only a real ingest fetches
|
|
98
112
|
them and enforces success_criteria.required_steps.
|
|
99
113
|
|
|
@@ -362,15 +376,6 @@ function resolveProvenance(provenanceMode, submission) {
|
|
|
362
376
|
* the routing decision parseRejectionReason() makes; this maps it to operator
|
|
363
377
|
* language so the consumer knows WHOSE problem each reason is.
|
|
364
378
|
*/
|
|
365
|
-
/**
|
|
366
|
-
* V2-CONTRACT-004: appended to EVERY --explain rendering (accepted and
|
|
367
|
-
* rejected alike) so a preview verdict can never be read as covering the
|
|
368
|
-
* required_steps gate, which only production ingest enforces (it fetches the
|
|
369
|
-
* scenario definitions this preview does not have).
|
|
370
|
-
*/
|
|
371
|
-
const PREVIEW_GAP_NOTE =
|
|
372
|
-
'Not checked in preview: required_steps (needs scenario definitions — enforced only by real ingest).';
|
|
373
|
-
|
|
374
379
|
const CLASS_LABEL = {
|
|
375
380
|
'submission-bad': 'SUBMISSION — fix your payload and resubmit',
|
|
376
381
|
'operational': 'OPERATIONAL — verifier/tooling fault; page ops, do not bounce to submitter',
|
package/index.js
CHANGED
|
@@ -86,6 +86,39 @@ function runValidator(name, fn) {
|
|
|
86
86
|
}
|
|
87
87
|
}
|
|
88
88
|
|
|
89
|
+
/**
|
|
90
|
+
* F-4036ae25: does `submission` carry a repo / run_id / timing.finished_at
|
|
91
|
+
* shape that packages/ingest/persist.js's computeRecordPath() could place on
|
|
92
|
+
* disk? Mirrors computeRecordPath's OWN structural preconditions (segment
|
|
93
|
+
* count, run_id charset, a parseable finished_at) — deliberately NOT its
|
|
94
|
+
* stricter isUnsafeSegment traversal guard. That guard rejects strings (e.g.
|
|
95
|
+
* `../etc`) this submission schema's own repo pattern also accepts, so a
|
|
96
|
+
* submission can be schema-VALID and still fail it; that gap belongs to
|
|
97
|
+
* F-4acd28d8, handled downstream in writeRecord()/ingest() once the record
|
|
98
|
+
* already has a verdict. Here we only need "is this filable in principle" —
|
|
99
|
+
* a three-field structural question `verify()` can answer locally without
|
|
100
|
+
* importing `@dogfood-lab/ingest` (which already imports `@dogfood-lab/verify`;
|
|
101
|
+
* importing back would close a new cross-package cycle).
|
|
102
|
+
*
|
|
103
|
+
* @param {object} submission
|
|
104
|
+
* @returns {boolean}
|
|
105
|
+
*/
|
|
106
|
+
function hasFilablePathIdentity(submission) {
|
|
107
|
+
const repo = submission.repo;
|
|
108
|
+
if (typeof repo !== 'string') return false;
|
|
109
|
+
const segments = repo.split('/');
|
|
110
|
+
if (segments.length !== 2 || !segments[0] || !segments[1]) return false;
|
|
111
|
+
|
|
112
|
+
if (typeof submission.run_id !== 'string' || !/^[\w-]+$/.test(submission.run_id)) {
|
|
113
|
+
return false;
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
const finishedAt = submission.timing && typeof submission.timing === 'object'
|
|
117
|
+
? submission.timing.finished_at
|
|
118
|
+
: undefined;
|
|
119
|
+
return typeof finishedAt === 'string' && !isNaN(new Date(finishedAt).getTime());
|
|
120
|
+
}
|
|
121
|
+
|
|
89
122
|
/**
|
|
90
123
|
* Verify a dogfood submission and produce a persisted record.
|
|
91
124
|
*
|
|
@@ -328,8 +361,85 @@ export async function verify(submission, options) {
|
|
|
328
361
|
});
|
|
329
362
|
|
|
330
363
|
// 7. Assemble persisted record
|
|
364
|
+
//
|
|
365
|
+
// A submission whose repo/run_id/timing.finished_at cannot drive
|
|
366
|
+
// computeRecordPath() is marked `_skipPersist` — the SAME sentinel the
|
|
367
|
+
// null/non-object branch above sets, because it is the same claim: we
|
|
368
|
+
// cannot place this on disk, so there is nothing to file. The record is
|
|
369
|
+
// still assembled and returned (the submitter needs the reasons); it is
|
|
370
|
+
// simply never written.
|
|
371
|
+
//
|
|
372
|
+
// F-4036ae25: this used to fire for EVERY schema-invalid submission, not
|
|
373
|
+
// only this unfilable subset — broader than the rationale below justifies.
|
|
374
|
+
// A submission whose repo/run_id/timing.finished_at are perfectly fine but
|
|
375
|
+
// fails schema on an unrelated field (e.g. an unexpected top-level property)
|
|
376
|
+
// IS filable, and blanket-skipping it left the fleet-wide audit trail with a
|
|
377
|
+
// rejection nobody could see anywhere durable. hasFilablePathIdentity()
|
|
378
|
+
// re-checks the three path-deriving fields directly against `submission`
|
|
379
|
+
// (not against WHICH schema rule fired), so this stays correct regardless of
|
|
380
|
+
// which field the schema violation is actually on.
|
|
381
|
+
//
|
|
382
|
+
// Skipping is not merely an economy. isDuplicate checks the _rejected path,
|
|
383
|
+
// so persisting an UNFILABLE rejection isn't possible anyway (computeRecordPath
|
|
384
|
+
// would throw); persisting a FILABLE one, on the other hand, reopens the
|
|
385
|
+
// V2-CROSS-BO-001 / F-82429f90 run_id-poisoning pathology for the
|
|
386
|
+
// newly-persisted category — a corrected resubmission silently dropped as a
|
|
387
|
+
// duplicate (exit 0). packages/ingest/persist.js's isDuplicate() closes that
|
|
388
|
+
// gap: a prior `_rejected` record whose reasons are ALL retryable-class is
|
|
389
|
+
// treated as non-blocking for a same-run_id retry that goes on to be
|
|
390
|
+
// ACCEPTED, so the audit trail and the resubmission path both hold. (A
|
|
391
|
+
// schema-invalid submission that reaches writeRecord() may still fail
|
|
392
|
+
// dogfood-record.schema.json's own mirrored constraints on the SAME field —
|
|
393
|
+
// ingest() routes that RecordValidationError the same way _skipPersist
|
|
394
|
+
// routes here; see F-4acd28d8.) An UNCORRECTED retry — still rejected,
|
|
395
|
+
// same run_id and date — computes to the exact SAME `_rejected` path
|
|
396
|
+
// attempt 1 already occupies regardless of which violation it reports, so
|
|
397
|
+
// it stays an ordinary duplicate rather than reaching writeRecord()'s
|
|
398
|
+
// exclusive-create at all (F-0f9e4077: the retry carve-out only ever
|
|
399
|
+
// waives the collision for a record now headed to a DIFFERENT, accepted
|
|
400
|
+
// path).
|
|
401
|
+
//
|
|
402
|
+
// The line: the duplicate guard otherwise consumes a run_id when we
|
|
403
|
+
// rendered a VERDICT on a run's own reported content. A policy or
|
|
404
|
+
// provenance rejection is such a verdict — it persists, and PERMANENTLY
|
|
405
|
+
// consuming the run_id is the intended anti-gaming behavior (a submitter
|
|
406
|
+
// must not be able to launder a genuinely-bad run into an accepted one by
|
|
407
|
+
// resubmitting different self-reported content under the same run_id).
|
|
408
|
+
// "We could not read/place/shape your submission" (schema, repo,
|
|
409
|
+
// unsafe-record-path, ...) is not a content verdict — it persists too when
|
|
410
|
+
// filable, but stays reusable for a genuinely corrected resubmission.
|
|
411
|
+
//
|
|
412
|
+
// F-f8952a50 (wave 10): this split is now a PER-PREFIX flag —
|
|
413
|
+
// parse-rejection.js's `retryable` field, consulted by persist.js's
|
|
414
|
+
// isRetryableRejection() — not a blanket "schema: only" allowlist. The
|
|
415
|
+
// shape/addressing subset of `submission-bad` (schema:, repo:,
|
|
416
|
+
// unsafe-record-path:, steps[<id>]:, policy-config:,
|
|
417
|
+
// submission-contains-verifier-field:, CONTRACT_SCHEMA_TOO_OLD:) is
|
|
418
|
+
// `retryable: true`; the content-verdict subset (policy:, provenance:)
|
|
419
|
+
// stays `retryable: false`, exactly as this paragraph describes. See
|
|
420
|
+
// parse-rejection.js's file header for the full rationale and
|
|
421
|
+
// schema-invalid-skip-persist.test.js's "REGRESSION GUARD" /
|
|
422
|
+
// "persist-a-verdict doctrine is preserved" tests for the pinned proof that
|
|
423
|
+
// policy:/provenance: remain terminal.
|
|
424
|
+
//
|
|
425
|
+
// CONTRACT_SCHEMA_TOO_NEW: does NOT belong in the shape/addressing list
|
|
426
|
+
// above — corrected here, wave 22 (F-51780da9's sweep found this comment
|
|
427
|
+
// stale in the same way it found parse-rejection.test.js's F-be0deacd test
|
|
428
|
+
// comment stale). F-be0deacd (wave 20) moved it to class 'operational' with
|
|
429
|
+
// `retryable: false` permanently: only a testing-os upgrade satisfies
|
|
430
|
+
// `major > maxMajor`, never a payload correction. `retryable` is a static
|
|
431
|
+
// per-prefix flag with no visibility into a later upgrade, so
|
|
432
|
+
// persist.js's isRetryableRejection() carries one additional, narrow
|
|
433
|
+
// re-check for this prefix only — re-deriving whether the stored
|
|
434
|
+
// rejection's declared major is still above the CURRENT build's
|
|
435
|
+
// SUPPORTED_SCHEMA_VERSIONS ceiling, rather than trusting the frozen
|
|
436
|
+
// boolean — so a stale TOO_NEW rejection does not permanently poison a
|
|
437
|
+
// run_id once the operator upgrades past the declared major. See
|
|
438
|
+
// isRetryableRejection's own doc comment (packages/ingest/persist.js).
|
|
439
|
+
const canFilePath = schemaResult.valid || hasFilablePathIdentity(submission);
|
|
331
440
|
const persisted = {
|
|
332
441
|
schema_version: RECORD_SCHEMA_VERSION,
|
|
442
|
+
...(canFilePath ? {} : { _skipPersist: true }),
|
|
333
443
|
policy_version: policyVersion,
|
|
334
444
|
run_id: submission.run_id,
|
|
335
445
|
repo: submission.repo,
|
package/package.json
CHANGED
package/parse-rejection.js
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
* a fresh drift source the moment a prefix is added.
|
|
9
9
|
*
|
|
10
10
|
* `parseRejectionReason(reason)` is the single exported classifier. It maps a
|
|
11
|
-
* raw rejection string to `{ class, prefix, detail }`:
|
|
11
|
+
* raw rejection string to `{ class, prefix, detail, retryable }`:
|
|
12
12
|
*
|
|
13
13
|
* - class — the routing decision (who fixes this):
|
|
14
14
|
* 'submission-bad' → the submitter fixes the payload and resubmits
|
|
@@ -20,6 +20,36 @@
|
|
|
20
20
|
* `'steps[<id>]:'`), or `null` for the 'unknown' class.
|
|
21
21
|
* - detail — the human-readable remainder with the matched prefix stripped
|
|
22
22
|
* (for 'unknown', the whole string verbatim).
|
|
23
|
+
* - retryable — (F-f8952a50, wave 10) may a same-run_id resubmission whose
|
|
24
|
+
* ONLY prior rejection carries this prefix still reach an
|
|
25
|
+
* acceptance, once corrected? This is a NARROWER question than
|
|
26
|
+
* `class`. Every 'operational' / 'ingest' / 'unknown' reason is
|
|
27
|
+
* `retryable: false` (only the submitter's own payload earns a
|
|
28
|
+
* retry — an ops fault or an unrecognized signal never does).
|
|
29
|
+
* Within 'submission-bad', the taxonomy splits further:
|
|
30
|
+
* - retryable: true — the prefix means "we could not even
|
|
31
|
+
* read/place/shape your submission" (schema:, repo:,
|
|
32
|
+
* unsafe-record-path:, steps[<id>]:, policy-config:,
|
|
33
|
+
* submission-contains-verifier-field:,
|
|
34
|
+
* CONTRACT_SCHEMA_TOO_OLD:) — a correction changes nothing
|
|
35
|
+
* about what the run actually did, only how it was
|
|
36
|
+
* addressed/formatted/configured. (CONTRACT_SCHEMA_TOO_NEW:
|
|
37
|
+
* moved to class 'operational' / retryable: false in
|
|
38
|
+
* F-be0deacd, wave 20 — see the LITERAL_PREFIXES entry
|
|
39
|
+
* below for why it is NOT the submitter's problem.)
|
|
40
|
+
* - retryable: false — the prefix means "we read your
|
|
41
|
+
* submission and rendered a VERDICT against its content"
|
|
42
|
+
* (policy:, provenance:) — consuming the run_id is the
|
|
43
|
+
* INTENDED anti-gaming behavior: a submitter cannot launder
|
|
44
|
+
* a genuinely-failed run into an accepted one by resubmitting
|
|
45
|
+
* different self-reported content under the same run_id.
|
|
46
|
+
* Pinned end-to-end: schema-invalid-skip-persist.test.js's
|
|
47
|
+
* "REGRESSION GUARD: a resubmission after a persisted
|
|
48
|
+
* NON-schema rejection is STILL blocked" test (provenance:)
|
|
49
|
+
* and f-0f9e4077-retry-collision-duplicate-rejection.test.js's
|
|
50
|
+
* "F-f8952a50" describe block (policy: / provenance: cases).
|
|
51
|
+
* Consumers key off THIS field, never off `prefix` — see
|
|
52
|
+
* packages/ingest/persist.js's isRetryableRejection().
|
|
23
53
|
*
|
|
24
54
|
* The prefix vocabulary below is enumerated from the ACTUAL emitters — it is
|
|
25
55
|
* NOT invented:
|
|
@@ -32,9 +62,16 @@
|
|
|
32
62
|
* submission-contains-verifier-field:,
|
|
33
63
|
* submission-malformed:,
|
|
34
64
|
* VALIDATOR_FAULT_<NAME>:
|
|
35
|
-
* - validators/schema-version.js: CONTRACT_SCHEMA_TOO_NEW
|
|
36
|
-
*
|
|
37
|
-
*
|
|
65
|
+
* - validators/schema-version.js: CONTRACT_SCHEMA_TOO_NEW: (this BUILD is
|
|
66
|
+
* behind a schema major its submitters
|
|
67
|
+
* already adopted → operational, F-be0deacd
|
|
68
|
+
* wave 20), CONTRACT_SCHEMA_TOO_OLD: (the
|
|
69
|
+
* SUBMITTER is behind → submission-bad)
|
|
70
|
+
* - packages/ingest/run.js: scenario-load:,
|
|
71
|
+
* unsafe-record-path: (F-4acd28d8,
|
|
72
|
+
* computeRecordPath's traversal guard
|
|
73
|
+
* rejected a schema-valid repo →
|
|
74
|
+
* submission-bad)
|
|
38
75
|
* - packages/ingest/load-context.js: scenario-fetch-fault: (V2-CROSS-BO-001,
|
|
39
76
|
* scenario-fetch outage/credential fault
|
|
40
77
|
* → operational)
|
|
@@ -52,29 +89,42 @@
|
|
|
52
89
|
|
|
53
90
|
/**
|
|
54
91
|
* @typedef {object} ParsedRejection
|
|
55
|
-
* @property {RejectionClass} class
|
|
56
|
-
* @property {string | null} prefix
|
|
57
|
-
* @property {string} detail
|
|
92
|
+
* @property {RejectionClass} class Routing decision (who must act).
|
|
93
|
+
* @property {string | null} prefix Canonical prefix token, or null when unknown.
|
|
94
|
+
* @property {string} detail Human-readable remainder (prefix stripped).
|
|
95
|
+
* @property {boolean} retryable Whether a same-run_id resubmission whose
|
|
96
|
+
* ONLY prior rejection carries this prefix may still reach an acceptance.
|
|
97
|
+
* See the file header for the full submission-bad split (shape/addressing
|
|
98
|
+
* vs. rendered-verdict).
|
|
58
99
|
*/
|
|
59
100
|
|
|
60
101
|
/**
|
|
61
102
|
* Literal-prefix matchers, ordered most-specific-first. Each carries the
|
|
62
|
-
* canonical prefix token
|
|
63
|
-
*
|
|
64
|
-
*
|
|
65
|
-
*
|
|
103
|
+
* canonical prefix token, the routing class, and whether a resubmission past
|
|
104
|
+
* this rejection is retryable (F-f8952a50 — see file header). `match` is the
|
|
105
|
+
* exact string the reason must start with; `detail` is whatever follows it
|
|
106
|
+
* (trimmed of one leading space). `repo:` matches the `repo:mismatch: …`
|
|
107
|
+
* family — the canonical token reported is the stable `repo:` head.
|
|
66
108
|
*/
|
|
67
109
|
const LITERAL_PREFIXES = [
|
|
68
|
-
// submission-bad
|
|
69
|
-
{ match: 'schema:', prefix: 'schema:', class: 'submission-bad' },
|
|
110
|
+
// submission-bad, retryable — "we could not even read your submission".
|
|
111
|
+
{ match: 'schema:', prefix: 'schema:', class: 'submission-bad', retryable: true },
|
|
70
112
|
// VERIFY-F1: a malformed REPO custom-rule predicate (eval-time semantic fault the
|
|
71
113
|
// schema could not catch). submission-bad — the repo authored the bad policy YAML,
|
|
72
114
|
// so the fix belongs to the submitter, not studio ops. Ordered BEFORE `policy:`
|
|
73
115
|
// to keep the most-specific-first invariant (the tokens don't overlap — `policy:`
|
|
74
116
|
// ends at the colon, `policy-config:` continues with `-config:` — but the order is
|
|
75
117
|
// explicit). A malformed GLOBAL predicate is operational (VALIDATOR_FAULT_POLICY).
|
|
76
|
-
|
|
77
|
-
|
|
118
|
+
// retryable — the REPO's policy YAML was broken, not a judgment on this run's
|
|
119
|
+
// content; once the repo fixes policies/*.yaml, the identical submission can
|
|
120
|
+
// legitimately evaluate cleanly. Shape/config, not a verdict.
|
|
121
|
+
{ match: 'policy-config:', prefix: 'policy-config:', class: 'submission-bad', retryable: true },
|
|
122
|
+
// NOT retryable — a rendered verdict on the run's own reported content
|
|
123
|
+
// (forbidden tags, missing required evidence, a failed custom_rules
|
|
124
|
+
// predicate). Consuming the run_id here is the deliberate anti-gaming
|
|
125
|
+
// behavior: pinned by schema-invalid-skip-persist.test.js's "the
|
|
126
|
+
// persist-a-verdict doctrine is preserved" describe block.
|
|
127
|
+
{ match: 'policy:', prefix: 'policy:', class: 'submission-bad', retryable: false },
|
|
78
128
|
// operational — a provider-side provenance FAULT (429/5xx/401/403). The
|
|
79
129
|
// adapter THROWS these (verify-A-002); index.js catches the throw and emits
|
|
80
130
|
// this distinct `provenance-fault:` prefix so the incident pages ops. Ordered
|
|
@@ -82,29 +132,57 @@ const LITERAL_PREFIXES = [
|
|
|
82
132
|
// the longer, more-specific match first preserves the most-specific-first
|
|
83
133
|
// invariant the array is sorted by. The genuine not-confirmed case
|
|
84
134
|
// (`provenance: source run could not be confirmed`) stays submission-bad below.
|
|
85
|
-
{ match: 'provenance-fault:', prefix: 'provenance-fault:', class: 'operational' },
|
|
86
|
-
|
|
87
|
-
|
|
135
|
+
{ match: 'provenance-fault:', prefix: 'provenance-fault:', class: 'operational', retryable: false },
|
|
136
|
+
// submission-bad, NOT retryable — a rendered verdict: the provider could not
|
|
137
|
+
// confirm this specific run happened as claimed. Resubmitting different
|
|
138
|
+
// self-reported content under the same run_id to "become confirmable" is
|
|
139
|
+
// exactly the laundering the anti-gaming doctrine blocks. Pinned by
|
|
140
|
+
// schema-invalid-skip-persist.test.js's REGRESSION GUARD test.
|
|
141
|
+
{ match: 'provenance:', prefix: 'provenance:', class: 'submission-bad', retryable: false },
|
|
142
|
+
// submission-bad, retryable — pure identity/addressing: the run happened,
|
|
143
|
+
// only the repo/run_url pairing was mis-stated. Proven live (F-f8952a50): a
|
|
144
|
+
// repo:mismatch correction must reach verify() again, not vanish as a
|
|
145
|
+
// silent duplicate.
|
|
146
|
+
{ match: 'repo:', prefix: 'repo:', class: 'submission-bad', retryable: true },
|
|
88
147
|
{
|
|
89
148
|
match: 'submission-contains-verifier-field:',
|
|
90
149
|
prefix: 'submission-contains-verifier-field:',
|
|
91
150
|
class: 'submission-bad',
|
|
151
|
+
retryable: true,
|
|
92
152
|
},
|
|
93
|
-
|
|
94
|
-
|
|
153
|
+
// F-be0deacd (wave 20): TOO_NEW and TOO_OLD are NOT symmetric, despite the
|
|
154
|
+
// near-identical prefixes and shared emitter (validators/schema-version.js).
|
|
155
|
+
// TOO_OLD means the SUBMITTER is behind — re-emitting against the current
|
|
156
|
+
// contract fixes it, so it stays submission-bad/retryable. TOO_NEW means
|
|
157
|
+
// THIS BUILD is behind a schema major its own submitters have already
|
|
158
|
+
// adopted (schema-version.js's own JSDoc: "operator must upgrade
|
|
159
|
+
// testing-os") — no resubmission, corrected or not, can ever satisfy a
|
|
160
|
+
// `major > maxMajor` comparison until testing-os itself ships an upgrade.
|
|
161
|
+
// That is an ops action, not a submitter action, so TOO_NEW routes
|
|
162
|
+
// operational/not-retryable — the same bucket VALIDATOR_FAULT_* and
|
|
163
|
+
// submission-malformed: use for "page ops, don't bounce it back".
|
|
164
|
+
{ match: 'CONTRACT_SCHEMA_TOO_NEW:', prefix: 'CONTRACT_SCHEMA_TOO_NEW:', class: 'operational', retryable: false },
|
|
165
|
+
{ match: 'CONTRACT_SCHEMA_TOO_OLD:', prefix: 'CONTRACT_SCHEMA_TOO_OLD:', class: 'submission-bad', retryable: true },
|
|
166
|
+
// submission-bad — F-4acd28d8: the record passed schema validation but its
|
|
167
|
+
// OWN repo identifier is unsafe to file under (computeRecordPath's
|
|
168
|
+
// isUnsafeSegment traversal guard, stricter than the schema's repo
|
|
169
|
+
// pattern — see F-bbbe2e1f). The submitter's repo string is the problem,
|
|
170
|
+
// not the verifier/ingest tooling, so this stays submission-bad.
|
|
171
|
+
// retryable — addressing, not a verdict on the run's content.
|
|
172
|
+
{ match: 'unsafe-record-path:', prefix: 'unsafe-record-path:', class: 'submission-bad', retryable: true },
|
|
95
173
|
// operational — a null/non-object submission is a malfunctioning dispatcher
|
|
96
174
|
// (verify-B-003), not a submitter who sent a bad-but-shaped payload. Page ops;
|
|
97
175
|
// do NOT bounce it back to the submitter.
|
|
98
|
-
{ match: 'submission-malformed:', prefix: 'submission-malformed:', class: 'operational' },
|
|
176
|
+
{ match: 'submission-malformed:', prefix: 'submission-malformed:', class: 'operational', retryable: false },
|
|
99
177
|
// operational — the scenario fetcher THREW after exhausting its retry
|
|
100
178
|
// budget (5xx/429 outage, transport reject) or hit a credential fault
|
|
101
179
|
// (401/403). V2-CROSS-BO-001: packages/ingest/load-context.js throws this
|
|
102
180
|
// classified error instead of returning `not_found`, so an outage never
|
|
103
181
|
// rejects a good submission. Ordered before `scenario-load:` to keep the
|
|
104
182
|
// most-specific-first invariant explicit (the tokens do not overlap).
|
|
105
|
-
{ match: 'scenario-fetch-fault:', prefix: 'scenario-fetch-fault:', class: 'operational' },
|
|
183
|
+
{ match: 'scenario-fetch-fault:', prefix: 'scenario-fetch-fault:', class: 'operational', retryable: false },
|
|
106
184
|
// ingest
|
|
107
|
-
{ match: 'scenario-load:', prefix: 'scenario-load:', class: 'ingest' },
|
|
185
|
+
{ match: 'scenario-load:', prefix: 'scenario-load:', class: 'ingest', retryable: false },
|
|
108
186
|
];
|
|
109
187
|
|
|
110
188
|
/** Strip a matched literal prefix and one optional leading space from a reason. */
|
|
@@ -121,27 +199,33 @@ function stripLiteral(reason, match) {
|
|
|
121
199
|
*/
|
|
122
200
|
export function parseRejectionReason(reason) {
|
|
123
201
|
if (typeof reason !== 'string') {
|
|
124
|
-
return { class: 'unknown', prefix: null, detail: '' };
|
|
202
|
+
return { class: 'unknown', prefix: null, detail: '', retryable: false };
|
|
125
203
|
}
|
|
126
204
|
|
|
127
205
|
// 1. Operational faults — matched by FAMILY so future VALIDATOR_FAULT_<NEW>
|
|
128
206
|
// classes need no edit here. (runValidator emits `VALIDATOR_FAULT_<CLS>: …`.)
|
|
207
|
+
// Never retryable — an ops fault is not the submitter's to fix.
|
|
129
208
|
const faultMatch = reason.match(/^(VALIDATOR_FAULT_[A-Z0-9_]+):\s*/);
|
|
130
209
|
if (faultMatch) {
|
|
131
210
|
return {
|
|
132
211
|
class: 'operational',
|
|
133
212
|
prefix: `${faultMatch[1]}:`,
|
|
134
213
|
detail: reason.slice(faultMatch[0].length),
|
|
214
|
+
retryable: false,
|
|
135
215
|
};
|
|
136
216
|
}
|
|
137
217
|
|
|
138
218
|
// 2. steps[<id>]: — bracketed id, so matched by regex rather than a literal.
|
|
219
|
+
// retryable — a structural/completeness mismatch between reported step
|
|
220
|
+
// results and the scenario's declared required_steps, not a verdict on
|
|
221
|
+
// whether the run's steps passed.
|
|
139
222
|
const stepsMatch = reason.match(/^steps\[[^\]]*\]:\s*/);
|
|
140
223
|
if (stepsMatch) {
|
|
141
224
|
return {
|
|
142
225
|
class: 'submission-bad',
|
|
143
226
|
prefix: 'steps[<id>]:',
|
|
144
227
|
detail: reason.slice(stepsMatch[0].length),
|
|
228
|
+
retryable: true,
|
|
145
229
|
};
|
|
146
230
|
}
|
|
147
231
|
|
|
@@ -154,12 +238,16 @@ export function parseRejectionReason(reason) {
|
|
|
154
238
|
class: entry.class,
|
|
155
239
|
prefix: entry.prefix,
|
|
156
240
|
detail: stripLiteral(reason, entry.match),
|
|
241
|
+
retryable: entry.retryable,
|
|
157
242
|
};
|
|
158
243
|
}
|
|
159
244
|
}
|
|
160
245
|
|
|
161
|
-
// 4. Unrecognized — log + surface raw
|
|
162
|
-
//
|
|
163
|
-
//
|
|
164
|
-
|
|
246
|
+
// 4. Unrecognized — log + surface raw, and never retryable: an unrecognized
|
|
247
|
+
// prefix is not affirmatively known to be a shape/addressing mistake, so
|
|
248
|
+
// the same "fails closed" discipline the taxonomy applies elsewhere
|
|
249
|
+
// applies here too. (The null/non-object submission reason now carries
|
|
250
|
+
// the typed `submission-malformed:` prefix → 'operational', so it no
|
|
251
|
+
// longer falls through here; see verify-B-003.)
|
|
252
|
+
return { class: 'unknown', prefix: null, detail: reason, retryable: false };
|
|
165
253
|
}
|