@openpond/evals 0.6.0 → 0.7.0

package/LEARNING.md

@@ -0,0 +1,106 @@
+# Task evidence and reusable Rewards
+
+`@openpond/evals/learning` owns the portable data and domain rules for task
+definitions, sources, submissions, feedback, admission, immutable task batches,
+learning policies and grading jobs. `@openpond/evals/rewards` owns immutable
+Reward releases and exact versioned bindings. The package does not contain a
+database, credentials, cloud client or execution infrastructure.
+
+Use `openpond-sdk/learning` to submit the same commands to an authenticated local
+or hosted execution owner. Plain HTTP producers use `POST /v1/learning/commands`
+with `{ "scope": "<profile-or-team>", "command": { ... } }` and a Bearer token.
+Reads use `POST /v1/learning/read`. Scope is a requested ownership boundary; the
+host must authorize it from the credential and assign the actor itself.
+
+## Data flow
+
+1. Publish a Reward, then a binding, task definition and source. Use
+   `publish_resources` to publish dependent resources atomically. Pin an existing
+   release by `{ id, revision, contentHash }`; edits create the next revision with
+   an expected current revision.
+2. Submit task examples under a stable source/example/attempt identity. Preserve
+   the producer idempotency key on retries. Reusing a key with different content
+   fails with a conflict. Invalid task-specific evidence remains reviewable;
+   accepting the submission does not approve it for training.
+3. Queue durable grading of the observed output or a proposed supervised target.
+   Hosts execute the binding through `TaskGradeExecutor`; the built-in executor
+   supports portable deterministic checks and reports other implementations as
+   unavailable. Publication and schema validity are not execution receipts.
+4. Record a reviewer decision referencing completed grader-run IDs. The service
+   checks the exact task, evidence, binding and output hashes. A failed observed
+   response may reveal a valid task. Supervised learning additionally requires a
+   separately approved, schema-valid target whose required checks passed.
+5. Seal reviewed evidence into a batch. The service reserves both task-family and
+   exact-input identities across splits in the same transaction. Unresolved
+   families, stale evidence, stale decisions and held-out/training overlap fail.
+   `compileTaskBatch` creates the existing `openpond.tasksetRelease.v2` package.
+
+Observed output, verifier expected output, private evaluator context and an
+approved supervised target have separate fields. The public policy view excludes
+expected output and private context. Compiled packages retain the approved target
+in typed learning metadata for a training adapter; they do not replace verifier
+ground truth with a demonstration.
+
+Feedback addresses source/example/attempt identity and may arrive before the
+example. Input, ground-truth and family corrections create new evidence revisions.
+Existing batches retain their exact historical evidence. Proposed-target feedback
+must be graded and reviewed before becoming a supervised target.
+
+## JSON and schema profile
+
+Generated HTTP/producer schemas are in `schemas/learning/v1`. They describe the
+structural contract. Use the domain service for authorization, revision/hash
+identity, admission and transactional constraints; JSON Schema alone cannot prove
+those relationships. Input/output schemas use object envelopes and JSON Schema
+2020-12 through Ajv, with a bounded portable profile:
+
+- Local `#/$defs/...` references; no remote loading or recursive schemas.
+- No regex, formats, custom keywords or runtime code inside schema validation.
+- At most 32 KiB of schema, bounded expansion/alternatives and validation caches.
+- Plain JSON values, finite numbers, no accessors/serializers/cycles, bounded depth,
+  node count and UTF-8 bytes. Host request limits apply in addition to field limits.
+
+This profile follows [Ajv's security guidance](https://ajv.js.org/security.html)
+for untrusted schemas and deliberately rejects unsupported capabilities.
+
+## Host responsibilities
+
+Authored verifier code and rubrics use immutable `asset` resources. Publish source
+and its Reward with `publish_resources`; the service checks UTF-8 size, SHA-256,
+full reference identity, scope and evaluator visibility in the same transaction.
+Editing source creates a new asset identity. Existing Rewards retain their bytes.
+
+`@openpond/evals/javascript-verifier/node` executes ESM verifier source in a
+worker containing a fresh QuickJS WebAssembly interpreter. The verifier receives
+JSON data and returns `{ score, passed, feedback, evidenceRefs? }`. It has no host
+functions or import loader. Memory, stack, source/result sizes and execution time
+are bounded; cancellation settles only after the worker has terminated. The
+interpreter and worker are embedded in the package, so a clean install needs no
+separate WASM download or native compiler. Browser/worker hosts can use the
+portable `@openpond/evals/javascript-verifier` entry point directly.
+
+This boundary follows the [QuickJS runtime isolation and resource APIs](https://github.com/justjake/quickjs-emscripten).
+Node's [VM documentation](https://nodejs.org/api/vm.html) explicitly excludes
+untrusted-code isolation; the local Taskset adapter now uses the same public
+interpreter worker as task-evidence grading.
+
+Implement `LearningRepository.transaction` with serialized writes and atomic
+rollback. Persist immutable historical revisions, exact operation receipts and
+unique split reservations. Never share a transaction object after it closes.
+List results are cursor-paginated at no more than 100 resources.
+
+Authorize every operation. Source credentials can submit only to their assigned
+source; human admission/correction/sealing require a reviewer. Derive identities
+from authentication, never from a producer-supplied actor field.
+
+Grading workers use leases, stable job identity, bounded timeout/spend and terminal
+receipts. Execution adapters must deduplicate submission and confirm cleanup.
+Cancellation remains `cancelling` until the execution owner confirms termination;
+a UI cancellation request is not proof that compute stopped. Restart recovery
+must use the same durable jobs. Unexpected host failures remain server errors;
+`LearningDomainError` exposes rejected requests as typed 4xx failures.
+
+Policy and iteration contracts distinguish training parent, optional teacher and
+upstream trigger. The contracts themselves do not launch training, schedule jobs,
+qualify a model, accept a candidate or promote a serving endpoint. Hosts must
+implement those transitions and record actual execution/evaluation receipts.

package/README.md

@@ -1,5 +1,7 @@
 # `@openpond/evals`
 
+See [task evidence and learning](./LEARNING.md) for reusable Rewards, task-format publication, SDK/HTTP intake, grading, review and approved batches.
+
 Portable evaluation and benchmark contracts plus pure helpers for Tasksets,
 graders, run manifests, attempt and evaluation receipts, paired benchmark
 comparisons, execution adapters, conformance fixtures, Work-evidence

package/dist/graders.js

@@ -1,5 +1,6 @@
 import { z } from "zod";
 import { FailureClassSchema, ReleaseHashSchema, ReleaseIdSchema, contentHash } from "@openpond/harness";
+import { validateTaskSchema, validateTaskValue } from "./task-schema.js";
 export const GraderEvidenceContentSchema = z.object({
     schemaVersion: z.literal("openpond.graderEvidence.v1"),
     graderId: ReleaseIdSchema,
@@ -48,22 +49,42 @@ function gradeDeterministic(grader, task, attempt) {
         const outputField = string(config.outputField) ?? "text";
         const actual = string(attempt.output[outputField]);
         const expected = string(config.expectedValue) ?? string(task.expectedOutput?.[string(config.expectedField) ?? "text"]);
+        if (expected === null || !normalize(expected))
+            return unavailable("Content grading requires a nonempty expected value.");
         passed = actual !== null && expected !== null && normalize(actual) === normalize(expected);
     }
     else if (grader.kind === "schema") {
-        passed = strings(config.requiredKeys).every((key) => Object.hasOwn(attempt.output, key));
+        if (config.jsonSchema !== undefined) {
+            const schema = validateTaskSchema(config.jsonSchema);
+            if (!schema.valid)
+                return unavailable(schema.issues[0].message);
+            passed = validateTaskValue(config.jsonSchema, attempt.output).valid;
+        }
+        else {
+            const required = strings(config.requiredKeys);
+            if (!required.length)
+                return unavailable("Schema grading requires a JSON Schema or explicit required keys.");
+            passed = required.every((key) => Object.hasOwn(attempt.output, key));
+        }
     }
     else if (grader.kind === "artifact") {
         const contains = string(config.refIncludes) ?? "";
+        if (!contains)
+            return unavailable("Artifact grading requires a declared artifact reference.");
         passed = attempt.artifactRefs.some((ref) => ref.includes(contains));
     }
     else if (grader.kind === "runtime_event") {
-        passed = strings(config.requiredEvents).every((required) => attempt.runtimeEventRefs.some((ref) => ref.includes(required)));
+        const requiredEvents = strings(config.requiredEvents);
+        if (!requiredEvents.length)
+            return unavailable("Runtime event grading requires declared events.");
+        passed = requiredEvents.every((required) => attempt.runtimeEventRefs.some((ref) => ref.includes(required)));
     }
     else {
         const fields = strings(config.fields);
         const compared = fields.length ? fields : Object.keys(task.expectedOutput ?? {});
-        passed = compared.every((field) => Object.is(attempt.output[field], task.expectedOutput?.[field]));
+        if (!compared.length || compared.some((field) => !task.expectedOutput || !Object.hasOwn(task.expectedOutput, field)))
+            return unavailable("State grading requires expected values for every compared field.");
+        passed = compared.every((field) => Object.hasOwn(attempt.output, field) && contentHash(attempt.output[field]) === contentHash(task.expectedOutput[field]));
     }
     return {
         score: passed ? 1 : 0,
@@ -81,7 +102,7 @@ function evidence(grader, result) {
         graderId: grader.id,
         graderVersion: grader.version,
         ...result,
-        rewardEligible: grader.rewardEligible && result.score !== null,
+        rewardEligible: grader.rewardEligible && result.rewardEligible && result.score !== null,
     });
     return GraderEvidenceSchema.parse({ ...content, contentHash: contentHash(content) });
 }
@@ -89,5 +110,5 @@ function unavailable(message) {
     return { score: null, passed: false, rewardEligible: false, failureClass: "grader_failure", feedback: [message], visibleEvidenceRefs: [], privilegedEvidenceRefs: [] };
 }
 function string(value) { return typeof value === "string" ? value : null; }
-function strings(value) { return Array.isArray(value) ? value.filter((item) => typeof item === "string") : []; }
+function strings(value) { return Array.isArray(value) ? value.filter((item) => typeof item === "string" && item.trim().length > 0) : []; }
 function normalize(value) { return value.normalize("NFKC").trim().replace(/[,，]/g, "").replace(/[.\s]+$/g, "").replace(/\s+/g, " "); }

package/dist/index.js

@@ -17,3 +17,6 @@ export * from "./telemetry-catalog.js";
 export * from "./telemetry-analysis.js";
 export * from "./telemetry-bundle.js";
 export * from "./tasksets.js";
+export * from "./task-schema.js";
+export * from "./rewards.js";
+export * from "./learning/index.js";

package/dist/javascript-verifier-contract.js

@@ -0,0 +1,7 @@
+import { z } from "zod";
+export const JavaScriptVerifierResultSchema = z.object({
+    score: z.number().finite().min(0).max(1),
+    passed: z.boolean(),
+    feedback: z.string().max(20_000),
+    evidenceRefs: z.array(z.string().max(240)).max(1_000).default([]),
+}).strict();

package/dist/javascript-verifier-node.js

@@ -0,0 +1,56 @@
+import { Worker } from "node:worker_threads";
+import { JavaScriptVerifierResultSchema } from "./javascript-verifier-contract.js";
+import { javascriptVerifierWorkerSource } from "./javascript-verifier-worker-source.js";
+import { assertBoundedTaskJson } from "./task-schema.js";
+/** Resolves/rejects only after the execution owner has stopped the worker. */
+export async function executeJavaScriptVerifierInWorker(input) {
+    input.signal?.throwIfAborted();
+    if (!Number.isInteger(input.timeoutMs) || input.timeoutMs < 1 || input.timeoutMs > 300_000)
+        throw new Error("verifier_timeout_invalid");
+    assertBoundedTaskJson(input.value, 4_194_304);
+    if (new TextEncoder().encode(input.source).byteLength > 524_288)
+        throw new Error("verifier_source_too_large");
+    const { signal, ...workerData } = input;
+    return new Promise((resolve, reject) => {
+        const worker = new Worker(javascriptVerifierWorkerSource, {
+            eval: true, workerData, execArgv: [], env: {},
+            resourceLimits: { maxOldGenerationSizeMb: 64, maxYoungGenerationSizeMb: 16, stackSizeMb: 4 },
+        });
+        let settling = false;
+        const finish = (outcome) => {
+            if (settling)
+                return;
+            settling = true;
+            clearTimeout(timer);
+            signal?.removeEventListener("abort", cancel);
+            void worker.terminate().then(() => {
+                if ("error" in outcome)
+                    reject(outcome.error);
+                else
+                    resolve(outcome.result);
+            }, reject);
+        };
+        const cancel = () => finish({ error: signal?.reason instanceof Error ? signal.reason : new Error("verifier_cancelled") });
+        const timer = setTimeout(() => finish({ error: new Error("verifier_timeout") }), input.timeoutMs);
+        signal?.addEventListener("abort", cancel, { once: true });
+        worker.once("error", (error) => finish({ error }));
+        worker.once("exit", () => { if (!settling)
+            finish({ error: new Error("verifier_worker_exited_without_result") }); });
+        worker.once("message", (message) => {
+            try {
+                if (!message || typeof message !== "object" || !("ok" in message))
+                    throw new Error("verifier_worker_invalid_response");
+                if (message.ok !== true)
+                    throw new Error("error" in message && typeof message.error === "string" ? message.error : "verifier_execution_failed");
+                if (!("result" in message))
+                    throw new Error("verifier_worker_invalid_response");
+                finish({ result: JavaScriptVerifierResultSchema.parse(message.result) });
+            }
+            catch (error) {
+                finish({ error: error instanceof Error ? error : new Error("verifier_worker_invalid_response") });
+            }
+        });
+        if (signal?.aborted)
+            cancel();
+    });
+}