@openpond/evals 0.4.2 → 0.6.0

package/CONTRACT.md

@@ -4,7 +4,8 @@
 model-driven refinement/review policy, improvements, traces, tools, model
 identities, and shared hashing.
 `@openpond/evals` owns Tasksets, graders, evaluation runs and receipts,
-execution orchestration, conformance fixtures, and Work-evidence eligibility.
+execution orchestration, conformance fixtures, Work-evidence eligibility, and
+portable Run telemetry/metric semantics.
 Evals depends on Harness for exact release identities but does not re-export
 Harness APIs; Harness never depends on Evals. Host applications own persistence, provider sessions, secret
 leases, connected-app authorization, model streaming, artifact bytes, and
@@ -17,6 +18,8 @@ runtime processes.
 | `HarnessRunTrace` | `HarnessTrace` | Preserve ordered actions, observations, lifecycle events, terminal state, failure class, and trace hash. Learning-signal envelopes remain a training projection of the trace and receipt. |
 | `TaskAttemptResult` | `AttemptReceipt` | Preserve the old record for application persistence while adding a lossless receipt reference. Output becomes `outputHash`; trace and artifacts are separately hash-bound. |
 | `GradeResult` | `GraderEvidence[]` | Preserve component score, pass, reward eligibility, failure class, feedback, and visible/private evidence references. Aggregate UI results remain host projections. |
+| Human or model comparison of attempts | `PreferenceComparisonRelease` + `ComparisonAssignment` + `PreferenceReceipt` | Keep release policy, blinded candidate ordering, exact attempt/artifact references, reviewer identity, and ranking immutable. Human and model reviewers emit the same receipt; provider sessions, identity details, queue state, and artifact bytes remain host-owned. |
+| Comparative reward | `PreferenceAggregationReceipt` + standard `RewardComponentReceipt` | Aggregate only with the release's named quorum algorithm, calibrate automated reviewers against non-frozen human receipts, then project the named pairwise-win fraction into the existing per-attempt reward component boundary. |
 | managed-RL local receipt | `AttemptReceipt` | Submit canonical manifest/task/trace/artifact/grader identities; policy token responses and provider request IDs remain host-private trace data. |
 | resolved training bundle | `HarnessRelease` + host training binding | Agent snapshot, program, lifecycle, tool declarations, files, and grader interface belong to the Harness. Taskset environment/policy/graders and dataset/evidence, recipe, compute, engine, approval, and opaque leases remain explicit host-side bindings. |
 | completed Work or Development turn | `WorkEvidenceReceipt` | Project the authoritative terminal turn, immutable Agent snapshot when available, model/runtime identity, sanitized trace reference, exact output revisions, validation evidence, interventions, timing, usage, and explicit consent provenance. Keep the raw source and trace host-private. |
@@ -24,6 +27,8 @@ runtime processes.
 | user feedback on Work output | `WorkFeedbackReceipt` | Append a new receipt bound to the evidence receipt and, when selected, the exact content-addressed output-revision descriptor. Corrections are separate artifacts and never mutate prior receipts. |
 | bounded cross-Work review | `HarnessEvaluationReviewReceipt` | Select only currently authorized immutable evidence, advance one watermark, group one stable claim, route to the smallest correct layer, and name the next authority without performing downstream effects. |
 | model-improvement qualification | `ModelImprovementQualificationReceipt` | Bind the originating review, exact Harness, Taskset, real baseline Evaluation, Model, Environment/tool/permission/policy hashes, Verifier, source policies, privacy, budget, and non-frozen signal. Weak or confounded evidence emits `no_training`; training and activation remain host effects. |
+| trainer/runtime telemetry | `RunTelemetryEvent` + `MetricObservation` | Emit compact ordered events and observations with exact Run lineage, bounded attributes/dimensions, source authority, visibility, and stable idempotency identity. Tenant identity, provider credentials, storage, retention, billing, and durable indexes remain host projections. |
+| local/hosted Run investigation export | `TelemetryExportBundle` | Export metric definitions, ordered evidence, bounded references, completeness, and a content hash. Apply visibility and redaction before crossing authority boundaries; never include raw privileged trace bytes. |
 
 ## Compatibility policy
 
@@ -46,11 +51,32 @@ runtime processes.
   immutable objects and rejects lifecycle, tool, grader-interface, or required
   Environment-tool drift before issuing the receipt.
 - The initial support target is Node.js ESM on Node 22.14 through Node 24.
+- Telemetry schema literals are shared across `@openpond/evals/telemetry` and
+  the `openpond-evals` Python distribution. Generated JSON Schemas and positive
+  and negative fixtures are the cross-language conformance authority.
+- Telemetry producer sequence is monotonic within a Run. Receivers deduplicate
+  exact retries, accept late delivery, and reject conflicting reuse of an
+  idempotency key or `(runId, sequence)` pair.
+- Core metrics reject unknown dimensions. Taskset- or Environment-specific
+  extensions require an explicit `MetricDefinition`; arbitrary metric names or
+  unbounded labels are not portable telemetry.
 - Portable paths are relative and at most 2,000 characters. Individual assets
   are at most 250 MB. Tasksets, traces, and evidence arrays have schema-level
   upper bounds.
 - Immutable content never contains secret values, opaque lease values, mutable
   provider resource IDs, database keys, UI state, or process handles.
+- Preference comparisons are an additive group-evaluation protocol, not a new
+  per-attempt grader kind and not a replacement for benchmark comparisons.
+  Releases contain two through four candidates, an ordered-tie-group result
+  policy, a frozen rubric artifact, presentation policy, aggregation policy,
+  reward projection, and calibration thresholds. Assignments bind every visible
+  artifact to an existing attempt and manifest under one Taskset, Harness,
+  Environment, Verifier, tool, and policy lineage.
+- A model preference receipt is reward-eligible only when a passed calibration
+  report binds the exact comparison release and immutable model-reviewer
+  release. Frozen-evaluation assignments cannot contribute calibration or
+  training evidence. Invalid or unrenderable candidates remain unscorable;
+  they are not silently converted into aesthetic losses.
 - The Work evidence schemas first ship as additive `0.2.x` package exports. The
   package version does not replace the `openpond.workEvidenceReceipt.v1`,
   `openpond.workProcessTrace.v1`, `openpond.workFeedbackReceipt.v1`, or

package/README.md

@@ -3,11 +3,17 @@
 Portable evaluation and benchmark contracts plus pure helpers for Tasksets,
 graders, run manifests, attempt and evaluation receipts, paired benchmark
 comparisons, execution adapters, conformance fixtures, Work-evidence
-eligibility, and no-training/SFT/preference/RL qualification receipts. The package depends on
+eligibility, no-training/SFT/preference/RL qualification receipts, and portable
+training telemetry. The package depends on
 [`@openpond/harness`](../harness/README.md) for exact Harness identities but
 does not re-export Harness APIs. Applications import the two packages directly,
 which keeps refinement and evaluation authority visibly separate.
 
+The `@openpond/evals/preferences` subpath adds a bounded, artifact-backed
+comparison protocol: two to four attempts can be ranked by a human or model
+reviewer, calibrated against held-out human receipts, then projected into the
+existing reward-component boundary without encoding a specific visual domain.
+
 ```ts
 import {
   AttemptReceiptSchema,
@@ -29,7 +35,7 @@ import {
 ```
 
 Subpath exports are available at `/harness`, `/tasksets`, `/benchmarks`, `/graders`, `/runs`,
-`/conformance`, `/evidence`, `/review`, and
+`/conformance`, `/evidence`, `/telemetry`, `/preferences`, `/review`, and
 `/model-improvement-qualification`. The package is an evaluation protocol library,
 not a hosted client. It does not execute OpenPond Desktop or Sandbox sessions,
 resolve credentials, or persist artifacts.
@@ -117,6 +123,29 @@ semantics. Infrastructure failures must remain reward-ineligible.
 Package semver and schema literals are independent. See [CONTRACT.md](./CONTRACT.md)
 for compatibility aliases, migration rules, size limits, and the field map.
 
+## Training telemetry
+
+The `@openpond/evals/telemetry` subpath defines the cross-runtime Run event,
+metric, cohort, evidence-completeness, and export-bundle protocol. It includes a
+bounded core metric catalog, deterministic builders, duplicate and late-delivery
+merge semantics, chart-ready aggregation, cohort filtering, and
+visibility-aware export helpers. The sibling `openpond-evals` Python
+distribution implements the producer-facing models, builders, and asynchronous
+buffered emitter used by GPU workers and external trainers; both languages
+validate the same fixtures and schema literals.
+
+Telemetry records what an admitted Run did; it does not configure the Taskset
+or execute training. Core observations are accepted only when their metric ID
+and bounded dimensions match the catalog. Custom metrics require an explicit
+versioned definition. Events carry immutable Run/Model/Harness/Taskset lineage,
+producer sequence, source authority, and evidence visibility.
+
+Portable export bundles contain definitions, events, observations, bounded
+evidence references, completeness state, and a content hash. Raw trace bytes,
+credentials, provider handles, tenant identity, billing policy, and hosted
+indexes remain host-owned. The package contains no trainer, optimizer,
+provisioner, persistence client, or diagnostic agent.
+
 ## Release preparation
 
 ```bash

package/conformance/telemetry/v1/invalid-batch.json

@@ -0,0 +1,17 @@
+{
+  "schemaVersion": "openpond.runTelemetryBatch.v1",
+  "events": [
+    {
+      "schemaVersion": "openpond.runTelemetryEvent.v1",
+      "eventId": "event-invalid",
+      "sequence": -1,
+      "occurredAt": "not-a-timestamp",
+      "source": "unknown_source",
+      "type": "optimizer_step_completed",
+      "visibility": "team_visible",
+      "lineage": {},
+      "attributes": {}
+    }
+  ],
+  "observations": []
+}

package/conformance/telemetry/v1/valid-batch.json

@@ -0,0 +1,58 @@
+{
+  "schemaVersion": "openpond.runTelemetryBatch.v1",
+  "events": [
+    {
+      "schemaVersion": "openpond.runTelemetryEvent.v1",
+      "eventId": "event-optimizer-step-1",
+      "sequence": 1,
+      "occurredAt": "2026-08-25T20:00:00.000Z",
+      "source": "optimizer",
+      "type": "optimizer_step_completed",
+      "visibility": "team_visible",
+      "lineage": {
+        "modelProjectId": "project-1",
+        "runId": "run-1",
+        "modelVersionId": "version-1",
+        "harnessReleaseHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+        "tasksetReleaseHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+        "environmentReleaseHash": null,
+        "checkpointId": null,
+        "step": 1,
+        "rolloutGroupId": "group-1",
+        "attemptId": null,
+        "scenarioId": "scenario-1"
+      },
+      "attributes": {
+        "learningRate": 0.00001,
+        "runner": "openpond_direct_grpo_v1"
+      }
+    }
+  ],
+  "observations": [
+    {
+      "schemaVersion": "openpond.metricObservation.v1",
+      "observationId": "observation-learning-rate-1",
+      "metricId": "optimizer.learning_rate",
+      "eventId": "event-optimizer-step-1",
+      "sequence": 2,
+      "observedAt": "2026-08-25T20:00:00.000Z",
+      "value": 0.00001,
+      "lineage": {
+        "modelProjectId": "project-1",
+        "runId": "run-1",
+        "modelVersionId": "version-1",
+        "harnessReleaseHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+        "tasksetReleaseHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+        "environmentReleaseHash": null,
+        "checkpointId": null,
+        "step": 1,
+        "rolloutGroupId": "group-1",
+        "attemptId": null,
+        "scenarioId": "scenario-1"
+      },
+      "dimensions": {
+        "split": "train"
+      }
+    }
+  ]
+}

package/dist/artifact-verification.js

@@ -0,0 +1,172 @@
+import { ArtifactManifestEntrySchema, RewardComponentReceiptSchema, } from "./execution-contracts.js";
+import { createArtifactManifest } from "./execution-receipts.js";
+export function buildArtifactManifest(input) {
+    const byPath = new Map();
+    for (const artifact of input.collectedArtifacts) {
+        const matches = byPath.get(artifact.path) ?? [];
+        matches.push(artifact);
+        byPath.set(artifact.path, matches);
+    }
+    const consumed = new Set();
+    const entries = input.requiredOutputs.map((required) => {
+        const matches = byPath.get(required.path) ?? [];
+        if (matches.length === 0) {
+            return ArtifactManifestEntrySchema.parse({
+                requiredOutputPath: required.path,
+                collectedPath: null,
+                declaredMediaType: required.mediaType,
+                detectedMediaType: null,
+                artifact: null,
+                status: "missing",
+                parseStatus: "not_requested",
+                schemaStatus: "not_requested",
+                errorCode: "required_output_missing",
+                failureOwner: "policy",
+                evidenceRefs: [],
+                metadata: { maxBytes: required.maxBytes, schemaRef: required.schemaRef },
+            });
+        }
+        if (matches.length > 1) {
+            for (const match of matches)
+                consumed.add(match);
+            return ArtifactManifestEntrySchema.parse({
+                requiredOutputPath: required.path,
+                collectedPath: required.path,
+                declaredMediaType: required.mediaType,
+                detectedMediaType: null,
+                artifact: null,
+                status: "failed",
+                parseStatus: "not_requested",
+                schemaStatus: "not_requested",
+                errorCode: "artifact_collection_ambiguous",
+                failureOwner: "collector",
+                evidenceRefs: matches.flatMap((match) => match.evidenceRefs ?? []),
+                metadata: { duplicateCount: matches.length },
+            });
+        }
+        const [collected] = matches;
+        consumed.add(collected);
+        return manifestEntry(required, collected);
+    });
+    for (const collected of input.collectedArtifacts) {
+        if (consumed.has(collected))
+            continue;
+        entries.push(ArtifactManifestEntrySchema.parse({
+            requiredOutputPath: null,
+            collectedPath: collected.path,
+            declaredMediaType: null,
+            detectedMediaType: collected.detectedMediaType,
+            artifact: collected.artifact,
+            status: collected.status,
+            parseStatus: collected.parseStatus ?? "not_requested",
+            schemaStatus: collected.schemaStatus ?? "not_requested",
+            errorCode: collected.errorCode ?? null,
+            failureOwner: collected.failureOwner ?? null,
+            evidenceRefs: collected.evidenceRefs ?? [],
+            metadata: collected.metadata ?? {},
+        }));
+    }
+    return createArtifactManifest({
+        schemaVersion: "openpond.artifactManifest.v1",
+        id: input.id,
+        attemptRef: input.attemptRef,
+        entries,
+        createdAt: input.createdAt,
+        metadata: input.metadata ?? {},
+    });
+}
+export function verifyRequiredOutputs(input) {
+    return input.requiredOutputs.map((required) => {
+        const entry = input.manifest.entries.find((candidate) => candidate.requiredOutputPath === required.path);
+        if (!entry || entry.status === "missing") {
+            return requiredOutputComponent(required, entry ?? null, {
+                status: "scored",
+                score: 0,
+                passed: false,
+                rewardEligible: true,
+                failureOwner: "policy",
+                feedback: `Required output ${required.path} was not collected.`,
+            });
+        }
+        if (entry.status === "failed" && entry.failureOwner !== "policy") {
+            return requiredOutputComponent(required, entry, {
+                status: "unscorable",
+                score: null,
+                passed: false,
+                rewardEligible: false,
+                failureOwner: entry.failureOwner ?? "collector",
+                feedback: `Required output ${required.path} could not be collected reliably.`,
+            });
+        }
+        const failures = structuralFailures(required, entry);
+        return requiredOutputComponent(required, entry, {
+            status: "scored",
+            score: failures.length === 0 ? 1 : 0,
+            passed: failures.length === 0,
+            rewardEligible: true,
+            failureOwner: failures.length === 0 ? null : "policy",
+            feedback: failures.length === 0
+                ? `Required output ${required.path} passed structural verification.`
+                : failures.join(" "),
+        });
+    });
+}
+function manifestEntry(required, collected) {
+    return ArtifactManifestEntrySchema.parse({
+        requiredOutputPath: required.path,
+        collectedPath: collected.path,
+        declaredMediaType: required.mediaType,
+        detectedMediaType: collected.detectedMediaType,
+        artifact: collected.artifact,
+        status: collected.status,
+        parseStatus: collected.parseStatus ?? "not_requested",
+        schemaStatus: collected.schemaStatus ?? "not_requested",
+        errorCode: collected.errorCode ?? null,
+        failureOwner: collected.failureOwner ?? null,
+        evidenceRefs: collected.evidenceRefs ?? [],
+        metadata: {
+            ...collected.metadata,
+            maxBytes: required.maxBytes,
+            schemaRef: required.schemaRef,
+        },
+    });
+}
+function structuralFailures(required, entry) {
+    const failures = [];
+    if (!entry.artifact)
+        failures.push(`Required output ${required.path} has no immutable artifact reference.`);
+    if (entry.detectedMediaType !== required.mediaType) {
+        failures.push(`Required output ${required.path} has media type ${entry.detectedMediaType ?? "unknown"}; expected ${required.mediaType}.`);
+    }
+    if (required.maxBytes !== null && entry.artifact?.sizeBytes !== null && entry.artifact && entry.artifact.sizeBytes > required.maxBytes) {
+        failures.push(`Required output ${required.path} exceeds ${required.maxBytes} bytes.`);
+    }
+    if (entry.parseStatus === "failed")
+        failures.push(`Required output ${required.path} could not be parsed.`);
+    if (entry.schemaStatus === "failed")
+        failures.push(`Required output ${required.path} failed schema validation.`);
+    return failures;
+}
+function requiredOutputComponent(required, entry, result) {
+    const evidenceRefs = [
+        ...(entry?.evidenceRefs ?? []),
+        ...(entry?.artifact ? [entry.artifact] : []),
+    ];
+    return RewardComponentReceiptSchema.parse({
+        verifierId: `required-output:${required.path}`,
+        verifierVersion: "1",
+        status: result.status,
+        rawScore: result.score,
+        normalizedScore: result.score,
+        weight: 1,
+        passed: result.passed,
+        hardGate: true,
+        rewardEligible: result.rewardEligible,
+        rewardContribution: result.rewardEligible ? result.score : null,
+        failureOwner: result.failureOwner,
+        feedback: [result.feedback],
+        visibleEvidenceRefs: evidenceRefs,
+        privilegedEvidenceRefs: [],
+        metadata: { requiredOutputPath: required.path },
+    });
+}