@openpond/evals 0.5.0 → 0.6.0

package/CONTRACT.md

@@ -4,7 +4,8 @@
 model-driven refinement/review policy, improvements, traces, tools, model
 identities, and shared hashing.
 `@openpond/evals` owns Tasksets, graders, evaluation runs and receipts,
-execution orchestration, conformance fixtures, and Work-evidence eligibility.
+execution orchestration, conformance fixtures, Work-evidence eligibility, and
+portable Run telemetry/metric semantics.
 Evals depends on Harness for exact release identities but does not re-export
 Harness APIs; Harness never depends on Evals. Host applications own persistence, provider sessions, secret
 leases, connected-app authorization, model streaming, artifact bytes, and
@@ -26,6 +27,8 @@ runtime processes.
 | user feedback on Work output | `WorkFeedbackReceipt` | Append a new receipt bound to the evidence receipt and, when selected, the exact content-addressed output-revision descriptor. Corrections are separate artifacts and never mutate prior receipts. |
 | bounded cross-Work review | `HarnessEvaluationReviewReceipt` | Select only currently authorized immutable evidence, advance one watermark, group one stable claim, route to the smallest correct layer, and name the next authority without performing downstream effects. |
 | model-improvement qualification | `ModelImprovementQualificationReceipt` | Bind the originating review, exact Harness, Taskset, real baseline Evaluation, Model, Environment/tool/permission/policy hashes, Verifier, source policies, privacy, budget, and non-frozen signal. Weak or confounded evidence emits `no_training`; training and activation remain host effects. |
+| trainer/runtime telemetry | `RunTelemetryEvent` + `MetricObservation` | Emit compact ordered events and observations with exact Run lineage, bounded attributes/dimensions, source authority, visibility, and stable idempotency identity. Tenant identity, provider credentials, storage, retention, billing, and durable indexes remain host projections. |
+| local/hosted Run investigation export | `TelemetryExportBundle` | Export metric definitions, ordered evidence, bounded references, completeness, and a content hash. Apply visibility and redaction before crossing authority boundaries; never include raw privileged trace bytes. |
 
 ## Compatibility policy
 
@@ -48,6 +51,15 @@ runtime processes.
   immutable objects and rejects lifecycle, tool, grader-interface, or required
   Environment-tool drift before issuing the receipt.
 - The initial support target is Node.js ESM on Node 22.14 through Node 24.
+- Telemetry schema literals are shared across `@openpond/evals/telemetry` and
+  the `openpond-evals` Python distribution. Generated JSON Schemas and positive
+  and negative fixtures are the cross-language conformance authority.
+- Telemetry producer sequence is monotonic within a Run. Receivers deduplicate
+  exact retries, accept late delivery, and reject conflicting reuse of an
+  idempotency key or `(runId, sequence)` pair.
+- Core metrics reject unknown dimensions. Taskset- or Environment-specific
+  extensions require an explicit `MetricDefinition`; arbitrary metric names or
+  unbounded labels are not portable telemetry.
 - Portable paths are relative and at most 2,000 characters. Individual assets
   are at most 250 MB. Tasksets, traces, and evidence arrays have schema-level
   upper bounds.

package/README.md

@@ -3,7 +3,8 @@
 Portable evaluation and benchmark contracts plus pure helpers for Tasksets,
 graders, run manifests, attempt and evaluation receipts, paired benchmark
 comparisons, execution adapters, conformance fixtures, Work-evidence
-eligibility, and no-training/SFT/preference/RL qualification receipts. The package depends on
+eligibility, no-training/SFT/preference/RL qualification receipts, and portable
+training telemetry. The package depends on
 [`@openpond/harness`](../harness/README.md) for exact Harness identities but
 does not re-export Harness APIs. Applications import the two packages directly,
 which keeps refinement and evaluation authority visibly separate.
@@ -34,7 +35,7 @@ import {
 ```
 
 Subpath exports are available at `/harness`, `/tasksets`, `/benchmarks`, `/graders`, `/runs`,
-`/conformance`, `/evidence`, `/preferences`, `/review`, and
+`/conformance`, `/evidence`, `/telemetry`, `/preferences`, `/review`, and
 `/model-improvement-qualification`. The package is an evaluation protocol library,
 not a hosted client. It does not execute OpenPond Desktop or Sandbox sessions,
 resolve credentials, or persist artifacts.
@@ -122,6 +123,29 @@ semantics. Infrastructure failures must remain reward-ineligible.
 Package semver and schema literals are independent. See [CONTRACT.md](./CONTRACT.md)
 for compatibility aliases, migration rules, size limits, and the field map.
 
+## Training telemetry
+
+The `@openpond/evals/telemetry` subpath defines the cross-runtime Run event,
+metric, cohort, evidence-completeness, and export-bundle protocol. It includes a
+bounded core metric catalog, deterministic builders, duplicate and late-delivery
+merge semantics, chart-ready aggregation, cohort filtering, and
+visibility-aware export helpers. The sibling `openpond-evals` Python
+distribution implements the producer-facing models, builders, and asynchronous
+buffered emitter used by GPU workers and external trainers; both languages
+validate the same fixtures and schema literals.
+
+Telemetry records what an admitted Run did; it does not configure the Taskset
+or execute training. Core observations are accepted only when their metric ID
+and bounded dimensions match the catalog. Custom metrics require an explicit
+versioned definition. Events carry immutable Run/Model/Harness/Taskset lineage,
+producer sequence, source authority, and evidence visibility.
+
+Portable export bundles contain definitions, events, observations, bounded
+evidence references, completeness state, and a content hash. Raw trace bytes,
+credentials, provider handles, tenant identity, billing policy, and hosted
+indexes remain host-owned. The package contains no trainer, optimizer,
+provisioner, persistence client, or diagnostic agent.
+
 ## Release preparation
 
 ```bash

package/conformance/telemetry/v1/invalid-batch.json

@@ -0,0 +1,17 @@
+{
+  "schemaVersion": "openpond.runTelemetryBatch.v1",
+  "events": [
+    {
+      "schemaVersion": "openpond.runTelemetryEvent.v1",
+      "eventId": "event-invalid",
+      "sequence": -1,
+      "occurredAt": "not-a-timestamp",
+      "source": "unknown_source",
+      "type": "optimizer_step_completed",
+      "visibility": "team_visible",
+      "lineage": {},
+      "attributes": {}
+    }
+  ],
+  "observations": []
+}

package/conformance/telemetry/v1/valid-batch.json

@@ -0,0 +1,58 @@
+{
+  "schemaVersion": "openpond.runTelemetryBatch.v1",
+  "events": [
+    {
+      "schemaVersion": "openpond.runTelemetryEvent.v1",
+      "eventId": "event-optimizer-step-1",
+      "sequence": 1,
+      "occurredAt": "2026-08-25T20:00:00.000Z",
+      "source": "optimizer",
+      "type": "optimizer_step_completed",
+      "visibility": "team_visible",
+      "lineage": {
+        "modelProjectId": "project-1",
+        "runId": "run-1",
+        "modelVersionId": "version-1",
+        "harnessReleaseHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+        "tasksetReleaseHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+        "environmentReleaseHash": null,
+        "checkpointId": null,
+        "step": 1,
+        "rolloutGroupId": "group-1",
+        "attemptId": null,
+        "scenarioId": "scenario-1"
+      },
+      "attributes": {
+        "learningRate": 0.00001,
+        "runner": "openpond_direct_grpo_v1"
+      }
+    }
+  ],
+  "observations": [
+    {
+      "schemaVersion": "openpond.metricObservation.v1",
+      "observationId": "observation-learning-rate-1",
+      "metricId": "optimizer.learning_rate",
+      "eventId": "event-optimizer-step-1",
+      "sequence": 2,
+      "observedAt": "2026-08-25T20:00:00.000Z",
+      "value": 0.00001,
+      "lineage": {
+        "modelProjectId": "project-1",
+        "runId": "run-1",
+        "modelVersionId": "version-1",
+        "harnessReleaseHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+        "tasksetReleaseHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+        "environmentReleaseHash": null,
+        "checkpointId": null,
+        "step": 1,
+        "rolloutGroupId": "group-1",
+        "attemptId": null,
+        "scenarioId": "scenario-1"
+      },
+      "dimensions": {
+        "split": "train"
+      }
+    }
+  ]
+}

package/dist/index.js

@@ -6,9 +6,14 @@ export * from "./execution-contracts.js";
 export * from "./execution-receipts.js";
 export * from "./graders.js";
 export * from "./harness.js";
+export * from "./learned-preference.js";
 export * from "./runs.js";
 export * from "./model-improvement-qualification.js";
 export * from "./preferences.js";
 export * from "./review-conformance.js";
 export * from "./rollouts.js";
+export * from "./telemetry.js";
+export * from "./telemetry-catalog.js";
+export * from "./telemetry-analysis.js";
+export * from "./telemetry-bundle.js";
 export * from "./tasksets.js";

package/dist/learned-preference.js

@@ -0,0 +1,334 @@
+import { z } from "zod";
+import { ImmutableReleaseRefSchema, MetadataSchema, ReleaseHashSchema, ReleaseIdSchema, ReleaseTimestampSchema, contentHash, } from "@openpond/harness";
+import { verifyComparisonAssignment, verifyPreferenceReceipt, } from "./preferences.js";
+const PreferenceDatasetPartitionSchema = z.enum([
+    "reward_train",
+    "reward_validation",
+    "reward_qualification",
+]);
+export const PreferenceEvidenceAuthoritySchema = z.enum([
+    "human",
+    "synthetic_fixture",
+]);
+export const RewardModelQualificationKindSchema = z.enum([
+    "synthetic_smoke",
+    "human_heldout",
+]);
+const PreferenceDatasetGroupSchema = z.object({
+    id: ReleaseIdSchema,
+    assignmentRef: ImmutableReleaseRefSchema,
+    scenarioRef: ImmutableReleaseRefSchema,
+    scenarioSplit: z.enum(["train", "validation"]),
+    preferenceResultRef: ImmutableReleaseRefSchema,
+    receiptRefs: z.array(ImmutableReleaseRefSchema).min(1).max(100),
+    attemptRefs: z.array(ImmutableReleaseRefSchema).min(2).max(4),
+    artifactManifestRefs: z.array(ImmutableReleaseRefSchema).min(2).max(4),
+    orderedBuckets: z.array(z.array(ReleaseIdSchema).min(1).max(4)).max(4),
+    rejectAll: z.boolean(),
+    partition: PreferenceDatasetPartitionSchema,
+    metadata: MetadataSchema,
+}).strict().superRefine((group, context) => {
+    const attemptIds = group.attemptRefs.map((attempt) => attempt.id);
+    if (new Set(attemptIds).size !== attemptIds.length) {
+        context.addIssue({
+            code: "custom",
+            path: ["attemptRefs"],
+            message: "Preference dataset group Attempt refs must be unique.",
+        });
+    }
+    if (group.artifactManifestRefs.length !== group.attemptRefs.length) {
+        context.addIssue({
+            code: "custom",
+            path: ["artifactManifestRefs"],
+            message: "Every preference dataset Attempt requires one Artifact Manifest ref.",
+        });
+    }
+    const bucketIds = group.orderedBuckets.flat();
+    if (group.rejectAll && bucketIds.length > 0) {
+        context.addIssue({
+            code: "custom",
+            path: ["orderedBuckets"],
+            message: "A reject-all preference dataset group cannot contain ordered buckets.",
+        });
+    }
+    if (!group.rejectAll
+        && (bucketIds.length !== attemptIds.length
+            || new Set(bucketIds).size !== bucketIds.length
+            || bucketIds.some((attemptId) => !attemptIds.includes(attemptId)))) {
+        context.addIssue({
+            code: "custom",
+            path: ["orderedBuckets"],
+            message: "Ordered buckets must contain every group Attempt exactly once.",
+        });
+    }
+});
+const PreferenceDatasetPairSchema = z.object({
+    groupId: ReleaseIdSchema,
+    preferredAttemptRef: ImmutableReleaseRefSchema,
+    dispreferredAttemptRef: ImmutableReleaseRefSchema,
+    relation: z.enum(["preferred", "tie"]),
+}).strict().superRefine((pair, context) => {
+    if (pair.preferredAttemptRef.id === pair.dispreferredAttemptRef.id
+        && pair.preferredAttemptRef.contentHash === pair.dispreferredAttemptRef.contentHash) {
+        context.addIssue({
+            code: "custom",
+            message: "A derived preference pair must reference two different Attempts.",
+        });
+    }
+});
+export const PreferenceDatasetReleaseContentSchema = z.object({
+    schemaVersion: z.literal("openpond.preferenceDatasetRelease.v1"),
+    id: ReleaseIdSchema,
+    revision: z.number().int().positive(),
+    tasksetRelease: ImmutableReleaseRefSchema,
+    comparisonRelease: ImmutableReleaseRefSchema,
+    authority: PreferenceEvidenceAuthoritySchema,
+    qualificationEligibility: z.enum(["smoke_only", "human_heldout"]),
+    fixtureRelease: ImmutableReleaseRefSchema.nullable(),
+    groups: z.array(PreferenceDatasetGroupSchema).min(1).max(100_000),
+    derivedPairs: z.array(PreferenceDatasetPairSchema).max(1_000_000),
+    createdAt: ReleaseTimestampSchema,
+    metadata: MetadataSchema,
+}).strict().superRefine((release, context) => {
+    if (release.authority === "synthetic_fixture"
+        && (release.qualificationEligibility !== "smoke_only"
+            || release.fixtureRelease === null)) {
+        context.addIssue({
+            code: "custom",
+            path: ["qualificationEligibility"],
+            message: "Synthetic preference datasets require a fixture release and are smoke-only.",
+        });
+    }
+    if (release.authority === "human"
+        && (release.qualificationEligibility !== "human_heldout"
+            || release.fixtureRelease !== null)) {
+        context.addIssue({
+            code: "custom",
+            path: ["authority"],
+            message: "Human preference datasets must be human-heldout eligible and cannot bind a fixture release.",
+        });
+    }
+    const groupIds = release.groups.map((group) => group.id);
+    if (new Set(groupIds).size !== groupIds.length) {
+        context.addIssue({
+            code: "custom",
+            path: ["groups"],
+            message: "Preference dataset group IDs must be unique.",
+        });
+    }
+    const groups = new Map(release.groups.map((group) => [group.id, group]));
+    for (const [index, pair] of release.derivedPairs.entries()) {
+        const group = groups.get(pair.groupId);
+        if (!group) {
+            context.addIssue({
+                code: "custom",
+                path: ["derivedPairs", index, "groupId"],
+                message: "Derived preference pairs must reference a group in the same release.",
+            });
+            continue;
+        }
+        const attemptIds = new Set(group.attemptRefs.map((attempt) => attempt.id));
+        if (!attemptIds.has(pair.preferredAttemptRef.id)
+            || !attemptIds.has(pair.dispreferredAttemptRef.id)) {
+            context.addIssue({
+                code: "custom",
+                path: ["derivedPairs", index],
+                message: "Derived preference pairs must reference Attempts in their source group.",
+            });
+        }
+    }
+});
+export const PreferenceDatasetReleaseSchema = PreferenceDatasetReleaseContentSchema
+    .extend({ contentHash: ReleaseHashSchema })
+    .strict();
+const RewardModelQualificationMetricsSchema = z.object({
+    sampleCount: z.number().int().positive(),
+    finiteScoreRate: z.number().min(0).max(1),
+    scoreVariance: z.number().nonnegative(),
+    checkpointReloadPassed: z.boolean(),
+    processorCompatibilityPassed: z.boolean(),
+    invalidAttemptExclusionPassed: z.boolean(),
+    orderedPairAccuracy: z.number().min(0).max(1).nullable(),
+    bucketAccuracy: z.number().min(0).max(1).nullable(),
+    tieAgreement: z.number().min(0).max(1).nullable(),
+}).strict();
+export const RewardModelQualificationReportContentSchema = z.object({
+    schemaVersion: z.literal("openpond.rewardModelQualificationReport.v1"),
+    id: ReleaseIdSchema,
+    kind: RewardModelQualificationKindSchema,
+    rewardModelVersion: ImmutableReleaseRefSchema,
+    preferenceDatasetRelease: ImmutableReleaseRefSchema,
+    tasksetRelease: ImmutableReleaseRefSchema,
+    processorRelease: ImmutableReleaseRefSchema,
+    metrics: RewardModelQualificationMetricsSchema,
+    passed: z.boolean(),
+    productionRewardEligible: z.boolean(),
+    createdAt: ReleaseTimestampSchema,
+    metadata: MetadataSchema,
+}).strict().superRefine((report, context) => {
+    if (report.kind === "synthetic_smoke" && report.productionRewardEligible) {
+        context.addIssue({
+            code: "custom",
+            path: ["productionRewardEligible"],
+            message: "Synthetic-smoke Reward Model qualification can never grant production reward eligibility.",
+        });
+    }
+    if (report.kind === "human_heldout"
+        && report.productionRewardEligible !== report.passed) {
+        context.addIssue({
+            code: "custom",
+            path: ["productionRewardEligible"],
+            message: "Human-heldout reward eligibility must match the frozen qualification outcome.",
+        });
+    }
+});
+export const RewardModelQualificationReportSchema = RewardModelQualificationReportContentSchema
+    .extend({ contentHash: ReleaseHashSchema })
+    .strict();
+export function createPreferenceDatasetRelease(input) {
+    const content = PreferenceDatasetReleaseContentSchema.parse(input);
+    return PreferenceDatasetReleaseSchema.parse({
+        ...content,
+        contentHash: contentHash(content),
+    });
+}
+export function materializePreferenceDatasetRelease(input) {
+    if (input.groups.length === 0) {
+        throw new Error("Preference dataset materialization requires at least one reviewed group.");
+    }
+    const tasksetRef = releaseRef(input.tasksetRelease);
+    const comparisonRef = releaseRef(input.comparisonRelease);
+    const fixtureRefs = new Map();
+    const groups = input.groups.map(({ assignment, receipt, partition }) => {
+        if (!verifyComparisonAssignment(assignment)) {
+            throw new Error("Preference dataset assignment failed immutable verification.");
+        }
+        if (!verifyPreferenceReceipt(receipt)) {
+            throw new Error("Preference dataset receipt failed immutable verification.");
+        }
+        if (!sameReleaseRef(assignment.lineage.tasksetRelease, tasksetRef)) {
+            throw new Error("Preference dataset assignment belongs to a different Taskset release.");
+        }
+        if (!sameReleaseRef(assignment.comparisonRelease, comparisonRef)) {
+            throw new Error("Preference dataset assignment belongs to a different comparison release.");
+        }
+        if (!sameReleaseRef(receipt.assignmentRef, releaseRef(assignment))) {
+            throw new Error("Preference dataset receipt belongs to a different assignment.");
+        }
+        if (input.authority === "synthetic_fixture") {
+            if (receipt.reviewer.kind !== "fixture") {
+                throw new Error("Synthetic preference datasets accept only fixture receipts.");
+            }
+            fixtureRefs.set(receipt.reviewer.fixtureRelease.contentHash, receipt.reviewer.fixtureRelease);
+        }
+        else if (receipt.reviewer.kind !== "human") {
+            throw new Error("Human preference datasets accept only human receipts.");
+        }
+        const scenario = input.tasksetRelease.tasks.find((task) => task.id === assignment.taskRef.id);
+        if (!scenario || (scenario.split !== "train" && scenario.split !== "validation")) {
+            throw new Error("Preference model datasets require train or validation scenarios.");
+        }
+        if (partition === "reward_train" && scenario.split !== "train") {
+            throw new Error("Reward training groups must use train scenarios.");
+        }
+        if (partition !== "reward_train" && scenario.split !== "validation") {
+            throw new Error("Reward validation and qualification groups must use validation scenarios.");
+        }
+        return {
+            id: `preference-group:${assignment.id}`,
+            assignmentRef: releaseRef(assignment),
+            scenarioRef: assignment.taskRef,
+            scenarioSplit: scenario.split,
+            preferenceResultRef: releaseRef(receipt),
+            receiptRefs: [releaseRef(receipt)],
+            attemptRefs: assignment.candidates.map((candidate) => candidate.attemptRef),
+            artifactManifestRefs: assignment.candidates.map((candidate) => candidate.artifactManifestRef),
+            orderedBuckets: receipt.order,
+            rejectAll: receipt.rejectAll,
+            partition,
+            metadata: { purpose: assignment.purpose, ...assignment.metadata },
+        };
+    });
+    if (input.authority === "synthetic_fixture" && fixtureRefs.size !== 1) {
+        throw new Error("Synthetic preference dataset groups must share one immutable fixture release.");
+    }
+    const fixtureRelease = input.authority === "synthetic_fixture"
+        ? [...fixtureRefs.values()][0]
+        : null;
+    return createPreferenceDatasetRelease({
+        schemaVersion: "openpond.preferenceDatasetRelease.v1",
+        id: input.id,
+        revision: input.revision,
+        tasksetRelease: tasksetRef,
+        comparisonRelease: comparisonRef,
+        authority: input.authority,
+        qualificationEligibility: input.authority === "synthetic_fixture" ? "smoke_only" : "human_heldout",
+        fixtureRelease,
+        groups,
+        derivedPairs: groups.flatMap(derivePairs),
+        createdAt: input.createdAt,
+        metadata: input.metadata ?? {},
+    });
+}
+export function verifyPreferenceDatasetRelease(value) {
+    return verifyHashed(value, PreferenceDatasetReleaseContentSchema, PreferenceDatasetReleaseSchema);
+}
+export function createRewardModelQualificationReport(input) {
+    const content = RewardModelQualificationReportContentSchema.parse(input);
+    return RewardModelQualificationReportSchema.parse({
+        ...content,
+        contentHash: contentHash(content),
+    });
+}
+export function verifyRewardModelQualificationReport(value) {
+    return verifyHashed(value, RewardModelQualificationReportContentSchema, RewardModelQualificationReportSchema);
+}
+function verifyHashed(value, contentSchema, fullSchema) {
+    const parsed = fullSchema.safeParse(value);
+    if (!parsed.success || !parsed.data)
+        return false;
+    const { contentHash: actual, ...content } = parsed.data;
+    try {
+        return contentHash(contentSchema.parse(content)) === actual;
+    }
+    catch {
+        return false;
+    }
+}
+function derivePairs(group) {
+    if (group.rejectAll)
+        return [];
+    const attemptById = new Map(group.attemptRefs.map((attempt) => [attempt.id, attempt]));
+    const pairs = [];
+    for (const [bucketIndex, bucket] of group.orderedBuckets.entries()) {
+        for (let left = 0; left < bucket.length; left += 1) {
+            for (let right = left + 1; right < bucket.length; right += 1) {
+                pairs.push({
+                    groupId: group.id,
+                    preferredAttemptRef: attemptById.get(bucket[left]),
+                    dispreferredAttemptRef: attemptById.get(bucket[right]),
+                    relation: "tie",
+                });
+            }
+        }
+        for (const lowerBucket of group.orderedBuckets.slice(bucketIndex + 1)) {
+            for (const preferredId of bucket) {
+                for (const dispreferredId of lowerBucket) {
+                    pairs.push({
+                        groupId: group.id,
+                        preferredAttemptRef: attemptById.get(preferredId),
+                        dispreferredAttemptRef: attemptById.get(dispreferredId),
+                        relation: "preferred",
+                    });
+                }
+            }
+        }
+    }
+    return pairs;
+}
+function releaseRef(value) {
+    return { id: value.id, contentHash: value.contentHash };
+}
+function sameReleaseRef(left, right) {
+    return left.id === right.id && left.contentHash === right.contentHash;
+}

package/dist/preferences.js

@@ -13,6 +13,11 @@ export const PreferenceReviewerKindSchema = z.enum([
     "human",
     "model",
     "deterministic",
+    "fixture",
+]);
+export const PreferenceRewardScopeSchema = z.enum([
+    "production",
+    "synthetic_smoke",
 ]);
 export const PreferenceRendererSchema = z.enum([
     "markdown",
@@ -111,7 +116,7 @@ const ComparisonCandidateSchema = z.object({
     attemptRef: ImmutableReleaseRefSchema,
     runManifestRef: ImmutableReleaseRefSchema,
     artifactManifestRef: ImmutableReleaseRefSchema,
-    visibleArtifactIds: z.array(ReleaseIdSchema).min(1).max(100_000),
+    visibleArtifactIds: z.array(ReleaseIdSchema).max(100_000),
 }).strict().superRefine((candidate, context) => {
     if (new Set(candidate.visibleArtifactIds).size !== candidate.visibleArtifactIds.length) {
         context.addIssue({
@@ -158,14 +163,24 @@ export const ComparisonAssignmentSchema = ComparisonAssignmentBaseSchema
     }
 });
 const PreferenceOrderSchema = z.array(z.array(ReleaseIdSchema).min(1).max(4)).max(4);
+const PreferenceReviewerSchema = z.discriminatedUnion("kind", [
+    z.object({
+        kind: z.enum(["human", "model", "deterministic"]),
+        releaseRef: ImmutableReleaseRefSchema,
+    }).strict(),
+    z.object({
+        kind: z.literal("fixture"),
+        releaseRef: ImmutableReleaseRefSchema,
+        fixtureRelease: ImmutableReleaseRefSchema,
+        labelSource: z.literal("synthetic"),
+        qualificationEligibility: z.literal("smoke_only"),
+    }).strict(),
+]);
 const PreferenceReceiptBaseSchema = z.object({
     schemaVersion: z.literal("openpond.preferenceReceipt.v1"),
     id: ReleaseIdSchema,
     assignmentRef: ImmutableReleaseRefSchema,
-    reviewer: z.object({
-        kind: PreferenceReviewerKindSchema,
-        releaseRef: ImmutableReleaseRefSchema,
-    }).strict(),
+    reviewer: PreferenceReviewerSchema,
     order: PreferenceOrderSchema,
     rejectAll: z.boolean(),
     criterionScores: z.record(ReleaseIdSchema, z.record(ReleaseIdSchema, z.number().finite().min(0).max(1))),
@@ -244,7 +259,8 @@ export function createComparisonAssignment(input) {
     if (input.candidates.length !== release.candidateCount) {
         throw new Error("Comparison assignment candidate count does not match the immutable comparison release.");
     }
-    const normalized = input.candidates.map((candidate) => validateComparisonCandidate(candidate, taskset));
+    const requiresVisibleArtifact = release.presentation.parts.some((part) => part.source === "artifact");
+    const normalized = input.candidates.map((candidate) => validateComparisonCandidate(candidate, taskset, requiresVisibleArtifact));
     const first = normalized[0];
     for (const candidate of normalized.slice(1)) {
         if (candidate.attempt.taskId !== first.attempt.taskId) {
@@ -331,6 +347,45 @@ export function createPreferenceReceipt(input) {
     validateCriterionScores(content.criterionScores, assignment, release);
     return PreferenceReceiptSchema.parse({ ...content, contentHash: contentHash(content) });
 }
+export function createSyntheticFixturePreferenceReceipt(input) {
+    const candidateIds = input.assignment.candidates.map((candidate) => candidate.attemptRef.id);
+    const ratingIds = Object.keys(input.ratings);
+    if (ratingIds.length !== candidateIds.length
+        || candidateIds.some((candidateId) => !input.ratings[candidateId])
+        || ratingIds.some((candidateId) => !candidateIds.includes(candidateId))) {
+        throw new Error("Synthetic fixture ratings must label every assignment candidate exactly once.");
+    }
+    const orderedRatings = ["love", "like", "reject"];
+    const rejectAll = candidateIds.every((candidateId) => input.ratings[candidateId] === "reject");
+    const order = rejectAll ? [] : orderedRatings
+        .map((rating) => candidateIds.filter((candidateId) => input.ratings[candidateId] === rating))
+        .filter((bucket) => bucket.length > 0);
+    const score = { love: 1, like: 0.5, reject: 0 };
+    return createPreferenceReceipt({
+        id: input.id,
+        assignment: input.assignment,
+        comparisonRelease: input.comparisonRelease,
+        reviewer: {
+            kind: "fixture",
+            releaseRef: input.labelerRelease,
+            fixtureRelease: input.fixtureRelease,
+            labelSource: "synthetic",
+            qualificationEligibility: "smoke_only",
+        },
+        order,
+        rejectAll,
+        criterionScores: Object.fromEntries(candidateIds.map((candidateId) => [
+            candidateId,
+            Object.fromEntries(input.comparisonRelease.criteria.map((criterion) => [
+                criterion.id,
+                score[input.ratings[candidateId]],
+            ])),
+        ])),
+        startedAt: input.startedAt,
+        completedAt: input.completedAt,
+        metadata: input.metadata ?? {},
+    });
+}
 export function verifyPreferenceReceipt(value) {
     return verifyHashed(value, PreferenceReceiptContentSchema, PreferenceReceiptSchema);
 }
@@ -436,12 +491,16 @@ export function createPreferenceRewardComponents(input) {
         ? input.result
         : null;
     const automatedReceipt = modelReceipt?.reviewer.kind === "model";
-    const calibrated = !automatedReceipt || isCalibratedAutomatedReviewer(modelReceipt, release, input.calibrationReport ?? null);
+    const fixtureReceipt = modelReceipt?.reviewer.kind === "fixture";
+    const rewardScope = PreferenceRewardScopeSchema.parse(input.rewardScope ?? "production");
+    const admittedReviewer = fixtureReceipt
+        ? rewardScope === "synthetic_smoke"
+        : !automatedReceipt || isCalibratedAutomatedReviewer(modelReceipt, release, input.calibrationReport ?? null);
     return Object.fromEntries(assignment.candidates.map((candidate) => {
         const candidateEligibility = eligible.get(candidate.attemptRef.id);
         const canScore = candidateEligibility?.eligible ?? true;
         const score = scores[candidate.attemptRef.id];
-        const component = canScore && calibrated
+        const component = canScore && admittedReviewer
             ? RewardComponentReceiptSchema.parse({
                 verifierId: release.rewardProjection.verifierId,
                 verifierVersion: release.rewardProjection.verifierVersion,
@@ -475,7 +534,11 @@ export function createPreferenceRewardComponents(input) {
                 rewardEligible: false,
                 rewardContribution: null,
                 failureOwner: candidateEligibility?.failureOwner ?? "verifier",
-                feedback: [canScore ? "Automated preference reviewer is uncalibrated or inconsistent." : "Candidate is not eligible for comparative preference scoring."],
+                feedback: [canScore
+                        ? fixtureReceipt
+                            ? "Synthetic fixture preference evidence is admitted only to synthetic-smoke reward scope."
+                            : "Automated preference reviewer is uncalibrated or inconsistent."
+                        : "Candidate is not eligible for comparative preference scoring."],
                 visibleEvidenceRefs: [],
                 privilegedEvidenceRefs: [],
                 metadata: {
@@ -487,7 +550,7 @@ export function createPreferenceRewardComponents(input) {
         return [candidate.attemptRef.id, component];
     }));
 }
-function validateComparisonCandidate(input, taskset) {
+function validateComparisonCandidate(input, taskset, requiresVisibleArtifact) {
     const attempt = AttemptReceiptSchema.parse(input.attempt);
     if (!verifyAttemptReceipt(attempt))
         throw new Error("Comparison candidate Attempt Receipt has an invalid content hash.");
@@ -514,7 +577,10 @@ function validateComparisonCandidate(input, taskset) {
     const available = new Set(artifactManifest.entries
         .filter((entry) => entry.status === "collected" && entry.artifact !== null)
         .map((entry) => entry.artifact.id));
-    if (!visibleArtifactIds.length || visibleArtifactIds.some((id) => !available.has(id))) {
+    if (requiresVisibleArtifact && !visibleArtifactIds.length) {
+        throw new Error("Comparison presentation requires each candidate to expose a reviewable artifact.");
+    }
+    if (visibleArtifactIds.some((id) => !available.has(id))) {
         throw new Error("Comparison candidate exposes an artifact that is missing, uncollected, or unreviewable.");
     }
     if (visible.size !== visibleArtifactIds.length)

package/dist/telemetry/index.js

@@ -0,0 +1,4 @@
+export * from "../telemetry.js";
+export * from "../telemetry-catalog.js";
+export * from "../telemetry-analysis.js";
+export * from "../telemetry-bundle.js";