@openpond/evals 0.4.1 → 0.5.0

package/CONTRACT.md

@@ -17,6 +17,8 @@ runtime processes.
 | `HarnessRunTrace` | `HarnessTrace` | Preserve ordered actions, observations, lifecycle events, terminal state, failure class, and trace hash. Learning-signal envelopes remain a training projection of the trace and receipt. |
 | `TaskAttemptResult` | `AttemptReceipt` | Preserve the old record for application persistence while adding a lossless receipt reference. Output becomes `outputHash`; trace and artifacts are separately hash-bound. |
 | `GradeResult` | `GraderEvidence[]` | Preserve component score, pass, reward eligibility, failure class, feedback, and visible/private evidence references. Aggregate UI results remain host projections. |
+| Human or model comparison of attempts | `PreferenceComparisonRelease` + `ComparisonAssignment` + `PreferenceReceipt` | Keep release policy, blinded candidate ordering, exact attempt/artifact references, reviewer identity, and ranking immutable. Human and model reviewers emit the same receipt; provider sessions, identity details, queue state, and artifact bytes remain host-owned. |
+| Comparative reward | `PreferenceAggregationReceipt` + standard `RewardComponentReceipt` | Aggregate only with the release's named quorum algorithm, calibrate automated reviewers against non-frozen human receipts, then project the named pairwise-win fraction into the existing per-attempt reward component boundary. |
 | managed-RL local receipt | `AttemptReceipt` | Submit canonical manifest/task/trace/artifact/grader identities; policy token responses and provider request IDs remain host-private trace data. |
 | resolved training bundle | `HarnessRelease` + host training binding | Agent snapshot, program, lifecycle, tool declarations, files, and grader interface belong to the Harness. Taskset environment/policy/graders and dataset/evidence, recipe, compute, engine, approval, and opaque leases remain explicit host-side bindings. |
 | completed Work or Development turn | `WorkEvidenceReceipt` | Project the authoritative terminal turn, immutable Agent snapshot when available, model/runtime identity, sanitized trace reference, exact output revisions, validation evidence, interventions, timing, usage, and explicit consent provenance. Keep the raw source and trace host-private. |
@@ -51,6 +53,18 @@ runtime processes.
   upper bounds.
 - Immutable content never contains secret values, opaque lease values, mutable
   provider resource IDs, database keys, UI state, or process handles.
+- Preference comparisons are an additive group-evaluation protocol, not a new
+  per-attempt grader kind and not a replacement for benchmark comparisons.
+  Releases contain two through four candidates, an ordered-tie-group result
+  policy, a frozen rubric artifact, presentation policy, aggregation policy,
+  reward projection, and calibration thresholds. Assignments bind every visible
+  artifact to an existing attempt and manifest under one Taskset, Harness,
+  Environment, Verifier, tool, and policy lineage.
+- A model preference receipt is reward-eligible only when a passed calibration
+  report binds the exact comparison release and immutable model-reviewer
+  release. Frozen-evaluation assignments cannot contribute calibration or
+  training evidence. Invalid or unrenderable candidates remain unscorable;
+  they are not silently converted into aesthetic losses.
 - The Work evidence schemas first ship as additive `0.2.x` package exports. The
   package version does not replace the `openpond.workEvidenceReceipt.v1`,
   `openpond.workProcessTrace.v1`, `openpond.workFeedbackReceipt.v1`, or

package/README.md

@@ -8,6 +8,11 @@ eligibility, and no-training/SFT/preference/RL qualification receipts. The packa
 does not re-export Harness APIs. Applications import the two packages directly,
 which keeps refinement and evaluation authority visibly separate.
 
+The `@openpond/evals/preferences` subpath adds a bounded, artifact-backed
+comparison protocol: two to four attempts can be ranked by a human or model
+reviewer, calibrated against held-out human receipts, then projected into the
+existing reward-component boundary without encoding a specific visual domain.
+
 ```ts
 import {
   AttemptReceiptSchema,
@@ -29,7 +34,7 @@ import {
 ```
 
 Subpath exports are available at `/harness`, `/tasksets`, `/benchmarks`, `/graders`, `/runs`,
-`/conformance`, `/evidence`, `/review`, and
+`/conformance`, `/evidence`, `/preferences`, `/review`, and
 `/model-improvement-qualification`. The package is an evaluation protocol library,
 not a hosted client. It does not execute OpenPond Desktop or Sandbox sessions,
 resolve credentials, or persist artifacts.

package/dist/artifact-verification.js

@@ -0,0 +1,172 @@
+import { ArtifactManifestEntrySchema, RewardComponentReceiptSchema, } from "./execution-contracts.js";
+import { createArtifactManifest } from "./execution-receipts.js";
+export function buildArtifactManifest(input) {
+    const byPath = new Map();
+    for (const artifact of input.collectedArtifacts) {
+        const matches = byPath.get(artifact.path) ?? [];
+        matches.push(artifact);
+        byPath.set(artifact.path, matches);
+    }
+    const consumed = new Set();
+    const entries = input.requiredOutputs.map((required) => {
+        const matches = byPath.get(required.path) ?? [];
+        if (matches.length === 0) {
+            return ArtifactManifestEntrySchema.parse({
+                requiredOutputPath: required.path,
+                collectedPath: null,
+                declaredMediaType: required.mediaType,
+                detectedMediaType: null,
+                artifact: null,
+                status: "missing",
+                parseStatus: "not_requested",
+                schemaStatus: "not_requested",
+                errorCode: "required_output_missing",
+                failureOwner: "policy",
+                evidenceRefs: [],
+                metadata: { maxBytes: required.maxBytes, schemaRef: required.schemaRef },
+            });
+        }
+        if (matches.length > 1) {
+            for (const match of matches)
+                consumed.add(match);
+            return ArtifactManifestEntrySchema.parse({
+                requiredOutputPath: required.path,
+                collectedPath: required.path,
+                declaredMediaType: required.mediaType,
+                detectedMediaType: null,
+                artifact: null,
+                status: "failed",
+                parseStatus: "not_requested",
+                schemaStatus: "not_requested",
+                errorCode: "artifact_collection_ambiguous",
+                failureOwner: "collector",
+                evidenceRefs: matches.flatMap((match) => match.evidenceRefs ?? []),
+                metadata: { duplicateCount: matches.length },
+            });
+        }
+        const [collected] = matches;
+        consumed.add(collected);
+        return manifestEntry(required, collected);
+    });
+    for (const collected of input.collectedArtifacts) {
+        if (consumed.has(collected))
+            continue;
+        entries.push(ArtifactManifestEntrySchema.parse({
+            requiredOutputPath: null,
+            collectedPath: collected.path,
+            declaredMediaType: null,
+            detectedMediaType: collected.detectedMediaType,
+            artifact: collected.artifact,
+            status: collected.status,
+            parseStatus: collected.parseStatus ?? "not_requested",
+            schemaStatus: collected.schemaStatus ?? "not_requested",
+            errorCode: collected.errorCode ?? null,
+            failureOwner: collected.failureOwner ?? null,
+            evidenceRefs: collected.evidenceRefs ?? [],
+            metadata: collected.metadata ?? {},
+        }));
+    }
+    return createArtifactManifest({
+        schemaVersion: "openpond.artifactManifest.v1",
+        id: input.id,
+        attemptRef: input.attemptRef,
+        entries,
+        createdAt: input.createdAt,
+        metadata: input.metadata ?? {},
+    });
+}
+export function verifyRequiredOutputs(input) {
+    return input.requiredOutputs.map((required) => {
+        const entry = input.manifest.entries.find((candidate) => candidate.requiredOutputPath === required.path);
+        if (!entry || entry.status === "missing") {
+            return requiredOutputComponent(required, entry ?? null, {
+                status: "scored",
+                score: 0,
+                passed: false,
+                rewardEligible: true,
+                failureOwner: "policy",
+                feedback: `Required output ${required.path} was not collected.`,
+            });
+        }
+        if (entry.status === "failed" && entry.failureOwner !== "policy") {
+            return requiredOutputComponent(required, entry, {
+                status: "unscorable",
+                score: null,
+                passed: false,
+                rewardEligible: false,
+                failureOwner: entry.failureOwner ?? "collector",
+                feedback: `Required output ${required.path} could not be collected reliably.`,
+            });
+        }
+        const failures = structuralFailures(required, entry);
+        return requiredOutputComponent(required, entry, {
+            status: "scored",
+            score: failures.length === 0 ? 1 : 0,
+            passed: failures.length === 0,
+            rewardEligible: true,
+            failureOwner: failures.length === 0 ? null : "policy",
+            feedback: failures.length === 0
+                ? `Required output ${required.path} passed structural verification.`
+                : failures.join(" "),
+        });
+    });
+}
+function manifestEntry(required, collected) {
+    return ArtifactManifestEntrySchema.parse({
+        requiredOutputPath: required.path,
+        collectedPath: collected.path,
+        declaredMediaType: required.mediaType,
+        detectedMediaType: collected.detectedMediaType,
+        artifact: collected.artifact,
+        status: collected.status,
+        parseStatus: collected.parseStatus ?? "not_requested",
+        schemaStatus: collected.schemaStatus ?? "not_requested",
+        errorCode: collected.errorCode ?? null,
+        failureOwner: collected.failureOwner ?? null,
+        evidenceRefs: collected.evidenceRefs ?? [],
+        metadata: {
+            ...collected.metadata,
+            maxBytes: required.maxBytes,
+            schemaRef: required.schemaRef,
+        },
+    });
+}
+function structuralFailures(required, entry) {
+    const failures = [];
+    if (!entry.artifact)
+        failures.push(`Required output ${required.path} has no immutable artifact reference.`);
+    if (entry.detectedMediaType !== required.mediaType) {
+        failures.push(`Required output ${required.path} has media type ${entry.detectedMediaType ?? "unknown"}; expected ${required.mediaType}.`);
+    }
+    if (required.maxBytes !== null && entry.artifact?.sizeBytes !== null && entry.artifact && entry.artifact.sizeBytes > required.maxBytes) {
+        failures.push(`Required output ${required.path} exceeds ${required.maxBytes} bytes.`);
+    }
+    if (entry.parseStatus === "failed")
+        failures.push(`Required output ${required.path} could not be parsed.`);
+    if (entry.schemaStatus === "failed")
+        failures.push(`Required output ${required.path} failed schema validation.`);
+    return failures;
+}
+function requiredOutputComponent(required, entry, result) {
+    const evidenceRefs = [
+        ...(entry?.evidenceRefs ?? []),
+        ...(entry?.artifact ? [entry.artifact] : []),
+    ];
+    return RewardComponentReceiptSchema.parse({
+        verifierId: `required-output:${required.path}`,
+        verifierVersion: "1",
+        status: result.status,
+        rawScore: result.score,
+        normalizedScore: result.score,
+        weight: 1,
+        passed: result.passed,
+        hardGate: true,
+        rewardEligible: result.rewardEligible,
+        rewardContribution: result.rewardEligible ? result.score : null,
+        failureOwner: result.failureOwner,
+        feedback: [result.feedback],
+        visibleEvidenceRefs: evidenceRefs,
+        privilegedEvidenceRefs: [],
+        metadata: { requiredOutputPath: required.path },
+    });
+}