@openpond/evals 0.5.0 → 0.7.0

package/CONTRACT.md

@@ -4,7 +4,8 @@
 model-driven refinement/review policy, improvements, traces, tools, model
 identities, and shared hashing.
 `@openpond/evals` owns Tasksets, graders, evaluation runs and receipts,
-execution orchestration, conformance fixtures, and Work-evidence eligibility.
+execution orchestration, conformance fixtures, Work-evidence eligibility, and
+portable Run telemetry/metric semantics.
 Evals depends on Harness for exact release identities but does not re-export
 Harness APIs; Harness never depends on Evals. Host applications own persistence, provider sessions, secret
 leases, connected-app authorization, model streaming, artifact bytes, and
@@ -26,6 +27,8 @@ runtime processes.
 | user feedback on Work output | `WorkFeedbackReceipt` | Append a new receipt bound to the evidence receipt and, when selected, the exact content-addressed output-revision descriptor. Corrections are separate artifacts and never mutate prior receipts. |
 | bounded cross-Work review | `HarnessEvaluationReviewReceipt` | Select only currently authorized immutable evidence, advance one watermark, group one stable claim, route to the smallest correct layer, and name the next authority without performing downstream effects. |
 | model-improvement qualification | `ModelImprovementQualificationReceipt` | Bind the originating review, exact Harness, Taskset, real baseline Evaluation, Model, Environment/tool/permission/policy hashes, Verifier, source policies, privacy, budget, and non-frozen signal. Weak or confounded evidence emits `no_training`; training and activation remain host effects. |
+| trainer/runtime telemetry | `RunTelemetryEvent` + `MetricObservation` | Emit compact ordered events and observations with exact Run lineage, bounded attributes/dimensions, source authority, visibility, and stable idempotency identity. Tenant identity, provider credentials, storage, retention, billing, and durable indexes remain host projections. |
+| local/hosted Run investigation export | `TelemetryExportBundle` | Export metric definitions, ordered evidence, bounded references, completeness, and a content hash. Apply visibility and redaction before crossing authority boundaries; never include raw privileged trace bytes. |
 
 ## Compatibility policy
 
@@ -48,6 +51,15 @@ runtime processes.
   immutable objects and rejects lifecycle, tool, grader-interface, or required
   Environment-tool drift before issuing the receipt.
 - The initial support target is Node.js ESM on Node 22.14 through Node 24.
+- Telemetry schema literals are shared across `@openpond/evals/telemetry` and
+  the `openpond-evals` Python distribution. Generated JSON Schemas and positive
+  and negative fixtures are the cross-language conformance authority.
+- Telemetry producer sequence is monotonic within a Run. Receivers deduplicate
+  exact retries, accept late delivery, and reject conflicting reuse of an
+  idempotency key or `(runId, sequence)` pair.
+- Core metrics reject unknown dimensions. Taskset- or Environment-specific
+  extensions require an explicit `MetricDefinition`; arbitrary metric names or
+  unbounded labels are not portable telemetry.
 - Portable paths are relative and at most 2,000 characters. Individual assets
   are at most 250 MB. Tasksets, traces, and evidence arrays have schema-level
   upper bounds.

package/LEARNING.md

@@ -0,0 +1,106 @@
+# Task evidence and reusable Rewards
+
+`@openpond/evals/learning` owns the portable data and domain rules for task
+definitions, sources, submissions, feedback, admission, immutable task batches,
+learning policies and grading jobs. `@openpond/evals/rewards` owns immutable
+Reward releases and exact versioned bindings. The package does not contain a
+database, credentials, cloud client or execution infrastructure.
+
+Use `openpond-sdk/learning` to submit the same commands to an authenticated local
+or hosted execution owner. Plain HTTP producers use `POST /v1/learning/commands`
+with `{ "scope": "<profile-or-team>", "command": { ... } }` and a Bearer token.
+Reads use `POST /v1/learning/read`. Scope is a requested ownership boundary; the
+host must authorize it from the credential and assign the actor itself.
+
+## Data flow
+
+1. Publish a Reward, then a binding, task definition and source. Use
+   `publish_resources` to publish dependent resources atomically. Pin an existing
+   release by `{ id, revision, contentHash }`; edits create the next revision with
+   an expected current revision.
+2. Submit task examples under a stable source/example/attempt identity. Preserve
+   the producer idempotency key on retries. Reusing a key with different content
+   fails with a conflict. Invalid task-specific evidence remains reviewable;
+   accepting the submission does not approve it for training.
+3. Queue durable grading of the observed output or a proposed supervised target.
+   Hosts execute the binding through `TaskGradeExecutor`; the built-in executor
+   supports portable deterministic checks and reports other implementations as
+   unavailable. Publication and schema validity are not execution receipts.
+4. Record a reviewer decision referencing completed grader-run IDs. The service
+   checks the exact task, evidence, binding and output hashes. A failed observed
+   response may reveal a valid task. Supervised learning additionally requires a
+   separately approved, schema-valid target whose required checks passed.
+5. Seal reviewed evidence into a batch. The service reserves both task-family and
+   exact-input identities across splits in the same transaction. Unresolved
+   families, stale evidence, stale decisions and held-out/training overlap fail.
+   `compileTaskBatch` creates the existing `openpond.tasksetRelease.v2` package.
+
+Observed output, verifier expected output, private evaluator context and an
+approved supervised target have separate fields. The public policy view excludes
+expected output and private context. Compiled packages retain the approved target
+in typed learning metadata for a training adapter; they do not replace verifier
+ground truth with a demonstration.
+
+Feedback addresses source/example/attempt identity and may arrive before the
+example. Input, ground-truth and family corrections create new evidence revisions.
+Existing batches retain their exact historical evidence. Proposed-target feedback
+must be graded and reviewed before becoming a supervised target.
+
+## JSON and schema profile
+
+Generated HTTP/producer schemas are in `schemas/learning/v1`. They describe the
+structural contract. Use the domain service for authorization, revision/hash
+identity, admission and transactional constraints; JSON Schema alone cannot prove
+those relationships. Input/output schemas use object envelopes and JSON Schema
+2020-12 through Ajv, with a bounded portable profile:
+
+- Local `#/$defs/...` references; no remote loading or recursive schemas.
+- No regex, formats, custom keywords or runtime code inside schema validation.
+- At most 32 KiB of schema, bounded expansion/alternatives and validation caches.
+- Plain JSON values, finite numbers, no accessors/serializers/cycles, bounded depth,
+  node count and UTF-8 bytes. Host request limits apply in addition to field limits.
+
+This profile follows [Ajv's security guidance](https://ajv.js.org/security.html)
+for untrusted schemas and deliberately rejects unsupported capabilities.
+
+## Host responsibilities
+
+Authored verifier code and rubrics use immutable `asset` resources. Publish source
+and its Reward with `publish_resources`; the service checks UTF-8 size, SHA-256,
+full reference identity, scope and evaluator visibility in the same transaction.
+Editing source creates a new asset identity. Existing Rewards retain their bytes.
+
+`@openpond/evals/javascript-verifier/node` executes ESM verifier source in a
+worker containing a fresh QuickJS WebAssembly interpreter. The verifier receives
+JSON data and returns `{ score, passed, feedback, evidenceRefs? }`. It has no host
+functions or import loader. Memory, stack, source/result sizes and execution time
+are bounded; cancellation settles only after the worker has terminated. The
+interpreter and worker are embedded in the package, so a clean install needs no
+separate WASM download or native compiler. Browser/worker hosts can use the
+portable `@openpond/evals/javascript-verifier` entry point directly.
+
+This boundary follows the [QuickJS runtime isolation and resource APIs](https://github.com/justjake/quickjs-emscripten).
+Node's [VM documentation](https://nodejs.org/api/vm.html) explicitly excludes
+untrusted-code isolation; the local Taskset adapter now uses the same public
+interpreter worker as task-evidence grading.
+
+Implement `LearningRepository.transaction` with serialized writes and atomic
+rollback. Persist immutable historical revisions, exact operation receipts and
+unique split reservations. Never share a transaction object after it closes.
+List results are cursor-paginated at no more than 100 resources.
+
+Authorize every operation. Source credentials can submit only to their assigned
+source; human admission/correction/sealing require a reviewer. Derive identities
+from authentication, never from a producer-supplied actor field.
+
+Grading workers use leases, stable job identity, bounded timeout/spend and terminal
+receipts. Execution adapters must deduplicate submission and confirm cleanup.
+Cancellation remains `cancelling` until the execution owner confirms termination;
+a UI cancellation request is not proof that compute stopped. Restart recovery
+must use the same durable jobs. Unexpected host failures remain server errors;
+`LearningDomainError` exposes rejected requests as typed 4xx failures.
+
+Policy and iteration contracts distinguish training parent, optional teacher and
+upstream trigger. The contracts themselves do not launch training, schedule jobs,
+qualify a model, accept a candidate or promote a serving endpoint. Hosts must
+implement those transitions and record actual execution/evaluation receipts.

package/README.md

@@ -1,9 +1,12 @@
 # `@openpond/evals`
 
+See [task evidence and learning](./LEARNING.md) for reusable Rewards, task-format publication, SDK/HTTP intake, grading, review and approved batches.
+
 Portable evaluation and benchmark contracts plus pure helpers for Tasksets,
 graders, run manifests, attempt and evaluation receipts, paired benchmark
 comparisons, execution adapters, conformance fixtures, Work-evidence
-eligibility, and no-training/SFT/preference/RL qualification receipts. The package depends on
+eligibility, no-training/SFT/preference/RL qualification receipts, and portable
+training telemetry. The package depends on
 [`@openpond/harness`](../harness/README.md) for exact Harness identities but
 does not re-export Harness APIs. Applications import the two packages directly,
 which keeps refinement and evaluation authority visibly separate.
@@ -34,7 +37,7 @@ import {
 ```
 
 Subpath exports are available at `/harness`, `/tasksets`, `/benchmarks`, `/graders`, `/runs`,
-`/conformance`, `/evidence`, `/preferences`, `/review`, and
+`/conformance`, `/evidence`, `/telemetry`, `/preferences`, `/review`, and
 `/model-improvement-qualification`. The package is an evaluation protocol library,
 not a hosted client. It does not execute OpenPond Desktop or Sandbox sessions,
 resolve credentials, or persist artifacts.
@@ -122,6 +125,29 @@ semantics. Infrastructure failures must remain reward-ineligible.
 Package semver and schema literals are independent. See [CONTRACT.md](./CONTRACT.md)
 for compatibility aliases, migration rules, size limits, and the field map.
 
+## Training telemetry
+
+The `@openpond/evals/telemetry` subpath defines the cross-runtime Run event,
+metric, cohort, evidence-completeness, and export-bundle protocol. It includes a
+bounded core metric catalog, deterministic builders, duplicate and late-delivery
+merge semantics, chart-ready aggregation, cohort filtering, and
+visibility-aware export helpers. The sibling `openpond-evals` Python
+distribution implements the producer-facing models, builders, and asynchronous
+buffered emitter used by GPU workers and external trainers; both languages
+validate the same fixtures and schema literals.
+
+Telemetry records what an admitted Run did; it does not configure the Taskset
+or execute training. Core observations are accepted only when their metric ID
+and bounded dimensions match the catalog. Custom metrics require an explicit
+versioned definition. Events carry immutable Run/Model/Harness/Taskset lineage,
+producer sequence, source authority, and evidence visibility.
+
+Portable export bundles contain definitions, events, observations, bounded
+evidence references, completeness state, and a content hash. Raw trace bytes,
+credentials, provider handles, tenant identity, billing policy, and hosted
+indexes remain host-owned. The package contains no trainer, optimizer,
+provisioner, persistence client, or diagnostic agent.
+
 ## Release preparation
 
 ```bash

package/conformance/telemetry/v1/invalid-batch.json

@@ -0,0 +1,17 @@
+{
+  "schemaVersion": "openpond.runTelemetryBatch.v1",
+  "events": [
+    {
+      "schemaVersion": "openpond.runTelemetryEvent.v1",
+      "eventId": "event-invalid",
+      "sequence": -1,
+      "occurredAt": "not-a-timestamp",
+      "source": "unknown_source",
+      "type": "optimizer_step_completed",
+      "visibility": "team_visible",
+      "lineage": {},
+      "attributes": {}
+    }
+  ],
+  "observations": []
+}

package/conformance/telemetry/v1/valid-batch.json

@@ -0,0 +1,58 @@
+{
+  "schemaVersion": "openpond.runTelemetryBatch.v1",
+  "events": [
+    {
+      "schemaVersion": "openpond.runTelemetryEvent.v1",
+      "eventId": "event-optimizer-step-1",
+      "sequence": 1,
+      "occurredAt": "2026-08-25T20:00:00.000Z",
+      "source": "optimizer",
+      "type": "optimizer_step_completed",
+      "visibility": "team_visible",
+      "lineage": {
+        "modelProjectId": "project-1",
+        "runId": "run-1",
+        "modelVersionId": "version-1",
+        "harnessReleaseHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+        "tasksetReleaseHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+        "environmentReleaseHash": null,
+        "checkpointId": null,
+        "step": 1,
+        "rolloutGroupId": "group-1",
+        "attemptId": null,
+        "scenarioId": "scenario-1"
+      },
+      "attributes": {
+        "learningRate": 0.00001,
+        "runner": "openpond_direct_grpo_v1"
+      }
+    }
+  ],
+  "observations": [
+    {
+      "schemaVersion": "openpond.metricObservation.v1",
+      "observationId": "observation-learning-rate-1",
+      "metricId": "optimizer.learning_rate",
+      "eventId": "event-optimizer-step-1",
+      "sequence": 2,
+      "observedAt": "2026-08-25T20:00:00.000Z",
+      "value": 0.00001,
+      "lineage": {
+        "modelProjectId": "project-1",
+        "runId": "run-1",
+        "modelVersionId": "version-1",
+        "harnessReleaseHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+        "tasksetReleaseHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+        "environmentReleaseHash": null,
+        "checkpointId": null,
+        "step": 1,
+        "rolloutGroupId": "group-1",
+        "attemptId": null,
+        "scenarioId": "scenario-1"
+      },
+      "dimensions": {
+        "split": "train"
+      }
+    }
+  ]
+}

package/dist/graders.js

@@ -1,5 +1,6 @@
 import { z } from "zod";
 import { FailureClassSchema, ReleaseHashSchema, ReleaseIdSchema, contentHash } from "@openpond/harness";
+import { validateTaskSchema, validateTaskValue } from "./task-schema.js";
 export const GraderEvidenceContentSchema = z.object({
     schemaVersion: z.literal("openpond.graderEvidence.v1"),
     graderId: ReleaseIdSchema,
@@ -48,22 +49,42 @@ function gradeDeterministic(grader, task, attempt) {
         const outputField = string(config.outputField) ?? "text";
         const actual = string(attempt.output[outputField]);
         const expected = string(config.expectedValue) ?? string(task.expectedOutput?.[string(config.expectedField) ?? "text"]);
+        if (expected === null || !normalize(expected))
+            return unavailable("Content grading requires a nonempty expected value.");
         passed = actual !== null && expected !== null && normalize(actual) === normalize(expected);
     }
     else if (grader.kind === "schema") {
-        passed = strings(config.requiredKeys).every((key) => Object.hasOwn(attempt.output, key));
+        if (config.jsonSchema !== undefined) {
+            const schema = validateTaskSchema(config.jsonSchema);
+            if (!schema.valid)
+                return unavailable(schema.issues[0].message);
+            passed = validateTaskValue(config.jsonSchema, attempt.output).valid;
+        }
+        else {
+            const required = strings(config.requiredKeys);
+            if (!required.length)
+                return unavailable("Schema grading requires a JSON Schema or explicit required keys.");
+            passed = required.every((key) => Object.hasOwn(attempt.output, key));
+        }
     }
     else if (grader.kind === "artifact") {
         const contains = string(config.refIncludes) ?? "";
+        if (!contains)
+            return unavailable("Artifact grading requires a declared artifact reference.");
         passed = attempt.artifactRefs.some((ref) => ref.includes(contains));
     }
     else if (grader.kind === "runtime_event") {
-        passed = strings(config.requiredEvents).every((required) => attempt.runtimeEventRefs.some((ref) => ref.includes(required)));
+        const requiredEvents = strings(config.requiredEvents);
+        if (!requiredEvents.length)
+            return unavailable("Runtime event grading requires declared events.");
+        passed = requiredEvents.every((required) => attempt.runtimeEventRefs.some((ref) => ref.includes(required)));
     }
     else {
         const fields = strings(config.fields);
         const compared = fields.length ? fields : Object.keys(task.expectedOutput ?? {});
-        passed = compared.every((field) => Object.is(attempt.output[field], task.expectedOutput?.[field]));
+        if (!compared.length || compared.some((field) => !task.expectedOutput || !Object.hasOwn(task.expectedOutput, field)))
+            return unavailable("State grading requires expected values for every compared field.");
+        passed = compared.every((field) => Object.hasOwn(attempt.output, field) && contentHash(attempt.output[field]) === contentHash(task.expectedOutput[field]));
     }
     return {
         score: passed ? 1 : 0,
@@ -81,7 +102,7 @@ function evidence(grader, result) {
         graderId: grader.id,
         graderVersion: grader.version,
         ...result,
-        rewardEligible: grader.rewardEligible && result.score !== null,
+        rewardEligible: grader.rewardEligible && result.rewardEligible && result.score !== null,
     });
     return GraderEvidenceSchema.parse({ ...content, contentHash: contentHash(content) });
 }
@@ -89,5 +110,5 @@ function unavailable(message) {
     return { score: null, passed: false, rewardEligible: false, failureClass: "grader_failure", feedback: [message], visibleEvidenceRefs: [], privilegedEvidenceRefs: [] };
 }
 function string(value) { return typeof value === "string" ? value : null; }
-function strings(value) { return Array.isArray(value) ? value.filter((item) => typeof item === "string") : []; }
+function strings(value) { return Array.isArray(value) ? value.filter((item) => typeof item === "string" && item.trim().length > 0) : []; }
 function normalize(value) { return value.normalize("NFKC").trim().replace(/[,，]/g, "").replace(/[.\s]+$/g, "").replace(/\s+/g, " "); }

package/dist/index.js

@@ -6,9 +6,17 @@ export * from "./execution-contracts.js";
 export * from "./execution-receipts.js";
 export * from "./graders.js";
 export * from "./harness.js";
+export * from "./learned-preference.js";
 export * from "./runs.js";
 export * from "./model-improvement-qualification.js";
 export * from "./preferences.js";
 export * from "./review-conformance.js";
 export * from "./rollouts.js";
+export * from "./telemetry.js";
+export * from "./telemetry-catalog.js";
+export * from "./telemetry-analysis.js";
+export * from "./telemetry-bundle.js";
 export * from "./tasksets.js";
+export * from "./task-schema.js";
+export * from "./rewards.js";
+export * from "./learning/index.js";

package/dist/javascript-verifier-contract.js

@@ -0,0 +1,7 @@
+import { z } from "zod";
+export const JavaScriptVerifierResultSchema = z.object({
+    score: z.number().finite().min(0).max(1),
+    passed: z.boolean(),
+    feedback: z.string().max(20_000),
+    evidenceRefs: z.array(z.string().max(240)).max(1_000).default([]),
+}).strict();

package/dist/javascript-verifier-node.js

@@ -0,0 +1,56 @@
+import { Worker } from "node:worker_threads";
+import { JavaScriptVerifierResultSchema } from "./javascript-verifier-contract.js";
+import { javascriptVerifierWorkerSource } from "./javascript-verifier-worker-source.js";
+import { assertBoundedTaskJson } from "./task-schema.js";
+/** Resolves/rejects only after the execution owner has stopped the worker. */
+export async function executeJavaScriptVerifierInWorker(input) {
+    input.signal?.throwIfAborted();
+    if (!Number.isInteger(input.timeoutMs) || input.timeoutMs < 1 || input.timeoutMs > 300_000)
+        throw new Error("verifier_timeout_invalid");
+    assertBoundedTaskJson(input.value, 4_194_304);
+    if (new TextEncoder().encode(input.source).byteLength > 524_288)
+        throw new Error("verifier_source_too_large");
+    const { signal, ...workerData } = input;
+    return new Promise((resolve, reject) => {
+        const worker = new Worker(javascriptVerifierWorkerSource, {
+            eval: true, workerData, execArgv: [], env: {},
+            resourceLimits: { maxOldGenerationSizeMb: 64, maxYoungGenerationSizeMb: 16, stackSizeMb: 4 },
+        });
+        let settling = false;
+        const finish = (outcome) => {
+            if (settling)
+                return;
+            settling = true;
+            clearTimeout(timer);
+            signal?.removeEventListener("abort", cancel);
+            void worker.terminate().then(() => {
+                if ("error" in outcome)
+                    reject(outcome.error);
+                else
+                    resolve(outcome.result);
+            }, reject);
+        };
+        const cancel = () => finish({ error: signal?.reason instanceof Error ? signal.reason : new Error("verifier_cancelled") });
+        const timer = setTimeout(() => finish({ error: new Error("verifier_timeout") }), input.timeoutMs);
+        signal?.addEventListener("abort", cancel, { once: true });
+        worker.once("error", (error) => finish({ error }));
+        worker.once("exit", () => { if (!settling)
+            finish({ error: new Error("verifier_worker_exited_without_result") }); });
+        worker.once("message", (message) => {
+            try {
+                if (!message || typeof message !== "object" || !("ok" in message))
+                    throw new Error("verifier_worker_invalid_response");
+                if (message.ok !== true)
+                    throw new Error("error" in message && typeof message.error === "string" ? message.error : "verifier_execution_failed");
+                if (!("result" in message))
+                    throw new Error("verifier_worker_invalid_response");
+                finish({ result: JavaScriptVerifierResultSchema.parse(message.result) });
+            }
+            catch (error) {
+                finish({ error: error instanceof Error ? error : new Error("verifier_worker_invalid_response") });
+            }
+        });
+        if (signal?.aborted)
+            cancel();
+    });
+}