eval-quality 0.1.0 → 0.2.0

package/schemas/eval-contract.schema.json

@@ -1069,13 +1069,23 @@
               }
             ],
             "description": "The temporal clause: the identifier of an earlier step, or `null`. That the named step carries no clause of its own is AD-39's one-level bound, which fires `nested-temporal-clause` at compile time and is deliberately not enforced here, so the code keeps a shape to fire on."
+          },
+          "cardinality": {
+            "type": "string",
+            "enum": [
+              "exactly-one",
+              "at-most-one",
+              "any"
+            ],
+            "description": "AD-39's declared selector cardinality (owed item 2): `exactly-one` or `at-most-one` when the step expects a single matching observation, `any` when several are legitimate. Several matches under `exactly-one`/`at-most-one` is the named ambiguity condition; a reference selection function returns it as data. Required rather than optional, which makes this a BREAKING `schemaVersion` bump under AD-11: adding an optional field is additive, and this field is not optional."
           }
         },
         "required": [
           "stepId",
           "operationId",
           "inputBinding",
-          "after"
+          "after",
+          "cardinality"
         ],
         "additionalProperties": false
       },
@@ -1148,11 +1158,71 @@
               "type": "null"
             }
           ]
+        },
+        "principals": {
+          "anyOf": [
+            {
+              "type": "object",
+              "propertyNames": {
+                "type": "string",
+                "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$",
+                "description": "A kebab-case slug. Excludes \"/\" and \"~\" so an identifier can be embedded in an interaction-rooted pointer without escaping."
+              },
+              "additionalProperties": {
+                "type": "object",
+                "properties": {
+                  "kind": {
+                    "type": "string",
+                    "minLength": 1
+                  }
+                },
+                "required": [
+                  "kind"
+                ],
+                "additionalProperties": false
+              }
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "description": "Owed item 3: the principals a `{ principal }` input binding may name. Caller-keyed by name, which makes a duplicate declaration unrepresentable, and valueless, since AD-19 forbids a credential value in a declaration and AD-18 keeps the name an opaque label. `null`, `{}`, and a populated map are three distinct answers, on `referenceSets`' own reasoning. A binding naming a key absent here still parses; it fires `undeclared-mandatory-input` at compile time in strict mode, because a cross-subtree constraint reaching from `interactionPlan` into this field cannot survive the export. Required, not optional, which makes this and `resources` below together the eval contract's `schemaVersion` 2 -> 3 BREAKING bump under AD-11, whose rule is that \"adding an optional field is a `schemaVersion` bump recorded in the field's own description; removing or retyping is breaking\". `null` is the legal spelling for a contract declaring none, but `TestData` is a `strictObject` and the key itself must be present, so no version-2 contract parses."
+        },
+        "resources": {
+          "anyOf": [
+            {
+              "type": "object",
+              "propertyNames": {
+                "type": "string",
+                "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$",
+                "description": "A kebab-case slug. Excludes \"/\" and \"~\" so an identifier can be embedded in an interaction-rooted pointer without escaping."
+              },
+              "additionalProperties": {
+                "type": "object",
+                "properties": {
+                  "kind": {
+                    "type": "string",
+                    "minLength": 1
+                  }
+                },
+                "required": [
+                  "kind"
+                ],
+                "additionalProperties": false
+              }
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "description": "Test-data resources, declared the same way as `principals` above. Any entry fires `scoped-reference-resolves-forbidden` at compile time exactly as a `scopedResources` entry does: a resource the contract points the evaluator at is a scoped reference wherever it is written down (AD-16). Required on the same terms as `principals`, and the other half of the eval contract's `schemaVersion` 2 -> 3 BREAKING bump under AD-11: `null` is the empty spelling and the key is mandatory."
         }
       },
       "required": [
         "setup",
-        "cleanup"
+        "cleanup",
+        "principals",
+        "resources"
       ],
       "additionalProperties": false
     },
@@ -2130,6 +2200,34 @@
                   "matcher"
                 ],
                 "additionalProperties": false
+              },
+              {
+                "type": "object",
+                "properties": {
+                  "captured": {
+                    "type": "string",
+                    "pattern": "^\\/interactions\\/[a-z0-9]+(?:-[a-z0-9]+)*\\/(?:(?:response-body|response-headers|stdout|stderr)(?:\\/(?:[^/~]|~[01])*)*|(?:response-status|exit-code)|call-inputs\\/(?:path|query|header|body)(?:\\/(?:[^/~]|~[01])*)*)$",
+                    "description": "AD-26 interaction-rooted pointer: \"/interactions/{stepId}/\" followed by one channel of the closed vocabulary. `call-inputs` takes one of the four transport channels as its next segment; `response-status` and `exit-code` take no tail. Syntax only: whether the step exists and whether the evidence is reachable are compile-time checks, not schema checks."
+                  }
+                },
+                "required": [
+                  "captured"
+                ],
+                "additionalProperties": false
+              },
+              {
+                "type": "object",
+                "properties": {
+                  "principal": {
+                    "type": "string",
+                    "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$",
+                    "description": "A kebab-case slug. Excludes \"/\" and \"~\" so an identifier can be embedded in an interaction-rooted pointer without escaping."
+                  }
+                },
+                "required": [
+                  "principal"
+                ],
+                "additionalProperties": false
               }
             ]
           }

package/schemas/evidence-artifact.schema.json

@@ -37,7 +37,7 @@
         "scoringVersion": {
           "type": "string",
           "pattern": "^sha256:[0-9a-f]{64}$",
-          "description": "AD-11: computed by the scorer over the five named inputs below and never caller-supplied."
+          "description": "AD-11: computed by the scorer over the six named inputs below and never caller-supplied."
         },
         "scoringVersionInputs": {
           "type": "object",
@@ -66,6 +66,11 @@
               "type": "string",
               "pattern": "^sha256:[0-9a-f]{64}$",
               "description": "AD-27 digest: \"sha256:\" plus 64 lowercase hexadecimal characters."
+            },
+            "mode": {
+              "type": "string",
+              "const": "production",
+              "description": "Owed item 4's identity clause: mode enters AD-11's scoring version, superseding AD-11's five-field sentence. Read from the sealed run record, never re-derived, so two runs scored under one mode never collide with a version computed for the other. Required, not optional, which makes this the evidence artifact's `schemaVersion` 1 -> 2 BREAKING bump under AD-11, whose rule is that \"adding an optional field is a `schemaVersion` bump recorded in the field's own description; removing or retyping is breaking\". A version-1 artifact carries five scoring-version inputs where this one carries six, so it does not parse, and no version computed before the bump compares with one computed after it."
             }
           },
           "required": [
@@ -73,7 +78,8 @@
             "corpusDigest",
             "fixtureDigest",
             "evaluatorConfigurationDigest",
-            "scoringPolicyDigest"
+            "scoringPolicyDigest",
+            "mode"
           ],
           "additionalProperties": false
         },
@@ -113,10 +119,11 @@
               "corpusDigest",
               "fixtureDigest",
               "evaluatorConfigurationDigest",
-              "scoringPolicyDigest"
+              "scoringPolicyDigest",
+              "mode"
             ]
           },
-          "description": "Which of the five scoring-version inputs were caller-attested rather than computed by this package. An enum over the five key names rather than free strings, because AD-32 requires the artifact to state *which* inputs were attested and an unconstrained string cannot be checked against anything. AD-11 names three of the five as caller-attested; the enum admits any subset so a stricter or looser integration stays representable."
+          "description": "Which of the six scoring-version inputs were caller-attested rather than computed by this package. An enum over the six key names rather than free strings, because AD-32 requires the artifact to state *which* inputs were attested and an unconstrained string cannot be checked against anything. AD-11 names three of the six as caller-attested; the enum admits any subset so a stricter or looser integration stays representable."
         },
         "trials": {
           "type": "object",
@@ -610,7 +617,7 @@
         "scoringVersion": {
           "type": "string",
           "pattern": "^sha256:[0-9a-f]{64}$",
-          "description": "AD-11: computed by the scorer over the five named inputs below and never caller-supplied."
+          "description": "AD-11: computed by the scorer over the six named inputs below and never caller-supplied."
         },
         "scoringVersionInputs": {
           "type": "object",
@@ -639,6 +646,11 @@
               "type": "string",
               "pattern": "^sha256:[0-9a-f]{64}$",
               "description": "AD-27 digest: \"sha256:\" plus 64 lowercase hexadecimal characters."
+            },
+            "mode": {
+              "type": "string",
+              "const": "contract-scoring",
+              "description": "Owed item 4's identity clause: mode enters AD-11's scoring version, superseding AD-11's five-field sentence. Read from the sealed run record, never re-derived, so two runs scored under one mode never collide with a version computed for the other. Required, not optional, which makes this the evidence artifact's `schemaVersion` 1 -> 2 BREAKING bump under AD-11, whose rule is that \"adding an optional field is a `schemaVersion` bump recorded in the field's own description; removing or retyping is breaking\". A version-1 artifact carries five scoring-version inputs where this one carries six, so it does not parse, and no version computed before the bump compares with one computed after it."
             }
           },
           "required": [
@@ -646,7 +658,8 @@
             "corpusDigest",
             "fixtureDigest",
             "evaluatorConfigurationDigest",
-            "scoringPolicyDigest"
+            "scoringPolicyDigest",
+            "mode"
           ],
           "additionalProperties": false
         },
@@ -686,10 +699,11 @@
               "corpusDigest",
               "fixtureDigest",
               "evaluatorConfigurationDigest",
-              "scoringPolicyDigest"
+              "scoringPolicyDigest",
+              "mode"
             ]
           },
-          "description": "Which of the five scoring-version inputs were caller-attested rather than computed by this package. An enum over the five key names rather than free strings, because AD-32 requires the artifact to state *which* inputs were attested and an unconstrained string cannot be checked against anything. AD-11 names three of the five as caller-attested; the enum admits any subset so a stricter or looser integration stays representable."
+          "description": "Which of the six scoring-version inputs were caller-attested rather than computed by this package. An enum over the six key names rather than free strings, because AD-32 requires the artifact to state *which* inputs were attested and an unconstrained string cannot be checked against anything. AD-11 names three of the six as caller-attested; the enum admits any subset so a stricter or looser integration stays representable."
         },
         "trials": {
           "type": "object",
@@ -1123,6 +1137,65 @@
         "contractVerdict": {
           "$ref": "#/$defs/Verdict"
         },
+        "uncitedFindingGaps": {
+          "type": "array",
+          "items": {
+            "type": "object",
+            "properties": {
+              "findingId": {
+                "type": "string",
+                "pattern": "^F-[0-9]{3,}$"
+              },
+              "observationIds": {
+                "minItems": 1,
+                "type": "array",
+                "items": {
+                  "type": "string",
+                  "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$",
+                  "description": "A kebab-case slug. Excludes \"/\" and \"~\" so an identifier can be embedded in an interaction-rooted pointer without escaping."
+                }
+              },
+              "quotedEvidence": {
+                "minItems": 1,
+                "type": "array",
+                "items": {
+                  "type": "object",
+                  "properties": {
+                    "quote": {
+                      "type": "string",
+                      "minLength": 1,
+                      "description": "The evaluator's verbatim quotation, per AD-23. Non-empty: an empty quotation quotes nothing, no AD-5 code names the condition, and under the admit-rule's second clause the schema is therefore the enforcement point. That this text appears in at least one of the finding's cited observations is NOT checked here; it is an AD-32 declared-versus-observed inconsistency that invalidates at ingest, and ADR-009 Decision 2 settles the precedence: \"cited identifiers govern the witness match; quotation audits it.\""
+                    },
+                    "channel": {
+                      "$ref": "#/$defs/EvidenceChannel"
+                    }
+                  },
+                  "required": [
+                    "quote",
+                    "channel"
+                  ],
+                  "additionalProperties": false
+                }
+              },
+              "severity": {
+                "type": "string",
+                "enum": [
+                  "low",
+                  "material",
+                  "critical"
+                ]
+              }
+            },
+            "required": [
+              "findingId",
+              "observationIds",
+              "quotedEvidence",
+              "severity"
+            ],
+            "additionalProperties": false
+          },
+          "description": "Owed item 5's rung record, contract-scoring only: one entry per uncited `defect` finding, per `core/score/outcome.ts`'s `uncitedDefectFindingGaps`. Required, not optional, which makes this the `schemaVersion` 2 -> 3 BREAKING bump under AD-11: an empty array is legal and is what a run with no uncited defect finding carries, but the key itself must be present. Never on the production branch, where the ladder signal alone is enough and this key would fail `strictObject`'s `unrecognized_keys` rule."
+        },
         "systemRecommendationRecorded": {
           "description": "The evaluator's recommendation about the SYSTEM, recorded and never promoted. On a knowingly defective probe a system-directed FAIL is an input rather than a signal about the contract.",
           "$ref": "#/$defs/EvaluatorRecommendation"
@@ -1158,6 +1231,7 @@
         "remediation",
         "mode",
         "contractVerdict",
+        "uncitedFindingGaps",
         "systemRecommendationRecorded",
         "systemRecommendationNote"
       ],
@@ -1165,7 +1239,7 @@
       "description": "Contract-scoring mode: the verdict is about the contract, and the evaluator's recommendation about the system is recorded beside it rather than promoted."
     }
   ],
-  "description": "The scored output, owned by `emit` per AD-24, with no prior art; `score` produces the outcome and verdict values it serializes. The two modes are separate branches because AD-21 requires that the production verdict and the contract verdict never share a field. This expresses that settled sentence and no more: Owed item 4's remaining half stays open, namely mode being fixed before ingest and entering AD-11's identity inputs rather than appearing first here; and the Sealed Run Record still carries no mode.",
+  "description": "The scored output, owned by `emit` per AD-24, with no prior art; `score` produces the outcome and verdict values it serializes. The two modes are separate branches because AD-21 requires that the production verdict and the contract verdict never share a field. Mode is no longer first stated here: the Sealed Run Record now carries a required `mode`, where AD-21's \"fixed before ingest\" puts it, so this artifact restates a mode the sealed record already fixed and is never the source. Owed item 4 is closed as of this schemaVersion: `mode` is `ScoringVersionInputs`'s sixth field, `core/score/ladder.ts` carries `ProductionAssessment`/`ContractAssessment` and their own total ladders, and `core/score/mode-agreement.ts` rejects a `production` record paired with a `contract-scoring` artifact in both directions as an AD-32 cross-artifact disagreement. That check is a hand-written function at the assembly boundary rather than a schema refinement, since no single schema sees both artifacts; `emit` is its one caller, immediately before returning the artifact it just built. Owed item 5 is closed as of this schemaVersion too: the contract-scoring branch carries a required `uncitedFindingGaps`, and both ladders in `core/score/ladder.ts` gain a shared `uncited-defect-finding` CONCERNS row over `AssessmentCommon.uncitedDefectFindings`, so an ingested defect finding citing no oracle now resolves at least CONCERNS instead of routing nowhere.",
   "$defs": {
     "CheckResolution": {
       "description": "One node of an oracle's `check` tree as it resolved. AD-4's resolution is three-valued rather than two, and the introduction vocabulary has exactly one member because AD-4 closes the introduction set at one: an empty collection. `null` where the resolution is not `insufficient-evidence`. A leaf carries an empty `children` array.",
@@ -1216,6 +1290,19 @@
       ],
       "description": "AD-21's four verdicts. Uppercase is one of the two named exceptions to the lowercase kebab-case enum convention. Exit codes are AD-21's and belong to the CLI rather than to this value: PASS, WAIVED, and CONCERNS exit zero and FAIL exits two."
     },
+    "EvidenceChannel": {
+      "type": "string",
+      "enum": [
+        "response-body",
+        "response-headers",
+        "response-status",
+        "call-inputs",
+        "stdout",
+        "stderr",
+        "exit-code"
+      ],
+      "description": "AD-26's closed evidence channel vocabulary. The same seven the interaction-rooted pointer addresses; a channel outside this set is a syntax error rather than an unreachable-evidence finding."
+    },
     "EvaluatorRecommendation": {
       "type": "string",
       "enum": [