@openpond/evals 0.7.5 → 0.8.0

package/JAVASCRIPT_ENVIRONMENTS.md

@@ -63,3 +63,50 @@ is cancelled, waits for worker termination, and fails explicitly when that bound
 operation deadline is exceeded.
 
 The policy adapter must honor its supplied abort signal and settle after cancelling its own request. `environmentCleanupComplete` describes environment cleanup only; it is not a provider billing or infrastructure-cleanup receipt. Failed creation, collection, cancellation and cleanup remain explicit states, without fabricated scores.
+
+## Submitted SQL and JavaScript
+
+An optional `executionServices` array on the immutable environment definition binds
+an owner-controlled service to a named `step` tool or to `collect`. Omission preserves
+existing definition hashes. Each service declares an identity, deadline and explicit
+value references: `{ scope: "input" | "initialState" | "state" | "arguments", path: [...] }`.
+References traverse own JSON properties only. The controller receives service results
+as `services[id]`; candidates cannot supply that field.
+
+A `sqlite.v1` binding declares `sql`, `snapshot`, `maxRows` and `maxResultBytes`.
+Snapshots contain named tables, typed columns (`INTEGER`, `REAL`, `TEXT`, `BLOB`) and
+array rows. SQLite executes exactly one statement in a fresh in-memory database.
+The Node adapter enforces read-only authorization, a deterministic function allowlist,
+a 32 MiB WASM memory maximum, value/SQL/result limits and a parent-owned deadline.
+Writes, attachment, pragma changes, extensions and nondeterministic functions are
+rejected. Results retain ordered column names and array rows, including duplicates.
+SQL null is JSON null; exact large integers use `{ integer: "9223372036854775807" }`
+and blobs use `{ blobBase64: "AP8=" }`. JSON numbers retain their ordinary IEEE-754
+semantics. Use tagged integers when exact 64-bit storage is required.
+
+A `javascript.v1` binding declares `source`, `cases`, `exportName`, `maxCases` and
+`maxResultBytes`. Cases are owner-held objects with an `input` property. Each case
+runs in a fresh QuickJS interpreter, and only its `input` crosses into the candidate.
+Other properties, including expected results, remain with the owner. Put hidden cases
+and expectations in private `initialState`, not task `input`: task input is part of
+the policy transcript. The authored controller grades returned case values against
+its own expectations. No host functions, module loader, filesystem or network are
+exposed to candidate JavaScript.
+
+Service outcomes are either completed results or explicit rejected submissions.
+Syntax, authorization, output and engine-memory failures cannot manufacture a passing
+answer. A service timeout produces a rejected outcome only after child termination;
+owner cancellation and infrastructure setup failures still fail the operation.
+Service preparation and the authored controller share the operation's overall budget
+and its tracked promise. Collection remains unable to mutate session state, and
+`destroy()` aborts and awaits all active work before running authored cleanup.
+
+`executeJavaScriptEnvironmentInProcess` and `executeJavaScriptEnvironmentInWorker`
+provide these services on Node hosts; service children always use Node processes.
+The portable in-process interpreter supports candidate JavaScript and rejects a SQL
+binding without a Node SQL executor. The standalone Node API is
+`executeSqlInProcess({ request, timeoutMs, signal })` from
+`@openpond/evals/sql-execution/node`; request/result schemas are available from
+`@openpond/evals/sql-execution`. The SQLite WASM bytes are embedded in the trusted
+child program, so execution survives installation and host bundling without resolving
+an application-relative WASM file or loading submitted code as host JavaScript.

package/LEARNING.md

@@ -79,6 +79,36 @@ for untrusted schemas and deliberately rejects unsupported capabilities.
 
 ## Host responsibilities
 
+### Reward fixture checks
+
+Reward drafts may contain up to 50 named fixtures. JSON and numeric editor fields
+remain strings while a draft is unfinished. `compileRewardAuthoring` is the shared
+source/rubric compiler; `compileRewardFixtures` validates executable fixture input.
+Published fixtures are an immutable evaluator-private `fixtureSetRef` asset included
+in the Reward's `assets`, so editing the next release restores the same examples.
+
+`queue_reward_check` takes an exact current Reward draft reference, timeout and
+explicit spend ceiling. It stores a `reward_check` resource without publishing a
+Reward, Task format, source or task evidence. The job binds the compiled Reward and
+fixture hashes. Its worker resolves historical draft bytes even if another editor
+saves or publishes a newer revision. Retry uses the same operation receipt; a
+different request under that operation ID conflicts.
+
+Hosts run `createRewardCheckWorker` with an execution adapter that records its
+runtime/package identity and settles only after cleanup. The isolated adapter runs
+portable deterministic checks and authored JavaScript with the supplied interpreter.
+Human review remains pending; missing judge/learned-model adapters remain unavailable.
+No fixture result implicitly calibrates a judge or qualifies a training runtime.
+Per-fixture results distinguish a scored rejection, unavailable grading and grader
+failure, and record whether each outcome matched its authored expectation.
+
+Checks are indexed by Reward ID, retain their original draft reference and remain
+readable after publication. Editing source or fixtures must mark prior checks as
+historical. `cancel_reward_check` requests cancellation with revision concurrency;
+the worker publishes terminal cancellation only after execution-owner cleanup.
+
+### Source and execution ownership
+
 Authored verifier code and rubrics use immutable `asset` resources. Publish source
 and its Reward with `publish_resources`; the service checks UTF-8 size, SHA-256,
 full reference identity, scope and evaluator visibility in the same transaction.

package/dist/environment-execution-services-contract.js

@@ -0,0 +1,27 @@
+import { z } from "zod";
+import { SqlExecutionRequestSchema } from "./sql-execution-contract.js";
+/** Explicit owner-side paths; resolution never evaluates code or traverses prototypes. */
+export const EnvironmentValueReferenceSchema = z.object({
+    scope: z.enum(["input", "initialState", "state", "arguments"]),
+    path: z.array(z.string().min(1).max(256)).max(16),
+}).strict();
+const common = {
+    id: z.string().min(1).max(128),
+    operation: z.enum(["step", "collect"]),
+    toolName: z.string().min(1).max(500).optional(),
+    timeoutMs: z.number().int().min(1).max(20_000),
+};
+export const EnvironmentExecutionServiceSchema = z.discriminatedUnion("kind", [
+    z.object({ ...common, kind: z.literal("sqlite.v1"), sql: EnvironmentValueReferenceSchema, snapshot: EnvironmentValueReferenceSchema,
+        maxRows: SqlExecutionRequestSchema.shape.maxRows, maxResultBytes: SqlExecutionRequestSchema.shape.maxResultBytes }).strict(),
+    z.object({ ...common, kind: z.literal("javascript.v1"), source: EnvironmentValueReferenceSchema, cases: EnvironmentValueReferenceSchema,
+        exportName: z.string().regex(/^[A-Za-z_$][A-Za-z0-9_$]*$/).max(128), maxCases: z.number().int().min(1).max(32), maxResultBytes: z.number().int().min(256).max(262_144) }).strict(),
+]).superRefine((service, context) => {
+    if ((service.operation === "step") !== (service.toolName !== undefined))
+        context.addIssue({ code: "custom", path: ["toolName"], message: "Only step services require a declared tool name." });
+    if (service.operation === "collect") {
+        const references = service.kind === "sqlite.v1" ? [service.sql, service.snapshot] : [service.source, service.cases];
+        if (references.some(reference => reference.scope === "arguments"))
+            context.addIssue({ code: "custom", message: "Collection has no tool arguments." });
+    }
+});

package/dist/environment-execution-services.js

@@ -0,0 +1,100 @@
+import { EnvironmentExecutionServiceSchema } from "./environment-execution-services-contract.js";
+import { JavaScriptIsolateExecutionError } from "./javascript-isolate.js";
+import { assertBoundedTaskJson } from "./task-schema.js";
+import { assertSqlExecutionRequest } from "./sql-execution-contract.js";
+function resolveValue(value, reference) {
+    let resolved = reference.scope === "arguments" ? value.action?.arguments : value[reference.scope];
+    for (const key of reference.path) {
+        if (!resolved || typeof resolved !== "object" || !Object.hasOwn(resolved, key))
+            throw new Error("environment_service_reference_missing");
+        resolved = resolved[key];
+    }
+    return resolved;
+}
+/** The single returned promise owns service work, controller execution and cancellation. */
+export async function executeEnvironmentWithServices(input, runners) {
+    if (!input.executionServices?.length)
+        return runners.controller(input);
+    if (!Number.isSafeInteger(input.timeoutMs) || input.timeoutMs < 1 || input.timeoutMs > 30_000)
+        throw new Error("environment_timeout_invalid");
+    assertBoundedTaskJson(input.value, 4_194_304);
+    const services = input.executionServices.map(service => EnvironmentExecutionServiceSchema.parse(service));
+    if (services.length > 16)
+        throw new Error("environment_service_limit");
+    if (new Set(services.map(service => service.id)).size !== services.length)
+        throw new Error("environment_service_identity_collision");
+    const started = Date.now();
+    const controller = new AbortController();
+    const signal = input.signal ? AbortSignal.any([input.signal, controller.signal]) : controller.signal;
+    const timer = setTimeout(() => controller.abort(new Error("environment_timeout")), input.timeoutMs);
+    try {
+        const results = Object.create(null);
+        for (const service of services) {
+            signal.throwIfAborted();
+            if (service.operation !== input.operation || (service.operation === "step" && service.toolName !== input.value.action?.name))
+                throw new Error("environment_service_operation_mismatch");
+            const budget = Math.min(service.timeoutMs, input.timeoutMs - (Date.now() - started));
+            if (budget < 1)
+                throw new Error("environment_timeout");
+            const deadline = new AbortController();
+            const serviceSignal = AbortSignal.any([signal, deadline.signal]);
+            const serviceTimer = setTimeout(() => deadline.abort(new Error("environment_service_timeout")), budget);
+            const serviceStarted = Date.now();
+            try {
+                if (service.kind === "sqlite.v1") {
+                    if (!runners.sql)
+                        throw new Error("environment_sql_executor_required");
+                    const sql = resolveValue(input.value, service.sql);
+                    if (typeof sql !== "string" || !sql.length || new TextEncoder().encode(sql).byteLength > 65_536 || sql.includes("\0")) {
+                        results[service.id] = { status: "rejected", code: "invalid_submission" };
+                        continue;
+                    }
+                    const request = assertSqlExecutionRequest({ schemaVersion: "openpond.sqlExecution.v1", snapshot: resolveValue(input.value, service.snapshot), sql, maxRows: service.maxRows, maxResultBytes: service.maxResultBytes });
+                    results[service.id] = await runners.sql({ request, timeoutMs: input.timeoutMs, signal: serviceSignal });
+                }
+                else {
+                    const source = resolveValue(input.value, service.source), cases = resolveValue(input.value, service.cases);
+                    if (typeof source !== "string" || new TextEncoder().encode(source).byteLength > 524_288) {
+                        results[service.id] = { status: "rejected", code: "invalid_submission" };
+                        continue;
+                    }
+                    if (!Array.isArray(cases) || !cases.length || cases.length > service.maxCases || cases.some(item => !item || typeof item !== "object" || !Object.hasOwn(item, "input")))
+                        throw new Error("environment_service_cases_invalid");
+                    const values = [];
+                    for (const item of cases) {
+                        const remaining = budget - (Date.now() - serviceStarted);
+                        if (remaining < 1) {
+                            deadline.abort(new Error("environment_service_timeout"));
+                            serviceSignal.throwIfAborted();
+                        }
+                        // Hidden expected results and all other case fields remain with the owner.
+                        values.push(await runners.candidate({ source, exportName: service.exportName, value: structuredClone(item.input), timeoutMs: Math.max(1, remaining), maxResultBytes: service.maxResultBytes, deterministic: true, errorPrefix: "candidate", signal: serviceSignal }));
+                        if (new TextEncoder().encode(JSON.stringify({ status: "completed", cases: values })).byteLength > service.maxResultBytes)
+                            throw new JavaScriptIsolateExecutionError("candidate_result_too_large");
+                    }
+                    results[service.id] = { status: "completed", cases: values };
+                }
+            }
+            catch (error) {
+                signal.throwIfAborted();
+                if (deadline.signal.aborted)
+                    results[service.id] = { status: "rejected", code: "timeout" };
+                else if (error instanceof JavaScriptIsolateExecutionError)
+                    results[service.id] = { status: "rejected", code: "execution_failed" };
+                else
+                    throw error;
+            }
+            finally {
+                clearTimeout(serviceTimer);
+            }
+        }
+        signal.throwIfAborted();
+        const remaining = input.timeoutMs - (Date.now() - started);
+        if (remaining < 1)
+            throw new Error("environment_timeout");
+        return await runners.controller({ ...input, executionServices: undefined, value: { ...input.value, services: results }, timeoutMs: remaining, signal });
+    }
+    finally {
+        clearTimeout(timer);
+    }
+}

package/dist/javascript-environment-contract.js

@@ -1,6 +1,8 @@
 import { z } from "zod";
 import { ImmutableAssetRefSchema, ToolDeclarationSchema, contentHash } from "@openpond/harness";
 import { assertBoundedTaskJson, validateTaskSchema } from "./task-schema-validation.js";
+import { EnvironmentExecutionServiceSchema } from "./environment-execution-services-contract.js";
+export * from "./environment-execution-services-contract.js";
 export const JavaScriptEnvironmentDefinitionContentSchema = z.object({
     schemaVersion: z.literal("openpond.javascriptEnvironment.v1"),
     id: z.string().trim().min(1).max(500),
@@ -11,6 +13,7 @@ export const JavaScriptEnvironmentDefinitionContentSchema = z.object({
     maxStateBytes: z.number().int().min(1).max(1_048_576),
     maxObservationBytes: z.number().int().min(1).max(262_144),
     operationTimeoutMs: z.number().int().min(1).max(30_000),
+    executionServices: z.array(EnvironmentExecutionServiceSchema).min(1).max(16).optional(),
 }).strict().superRefine((value, context) => {
     if (new Set(value.tools.map(tool => tool.name)).size !== value.tools.length)
         context.addIssue({ code: "custom", path: ["tools"], message: "Environment tool names must be unique." });
@@ -20,6 +23,16 @@ export const JavaScriptEnvironmentDefinitionContentSchema = z.object({
         if (tool.inputSchemaHash !== contentHash(tool.inputSchema))
             context.addIssue({ code: "custom", path: ["tools", index, "inputSchemaHash"], message: "Tool schema identity differs from its declared schema." });
     });
+    const services = value.executionServices ?? [];
+    if (new Set(services.map(service => service.id)).size !== services.length)
+        context.addIssue({ code: "custom", path: ["executionServices"], message: "Execution service identities must be unique." });
+    services.forEach((service, index) => {
+        const tool = value.tools.find(tool => tool.name === service.toolName);
+        if (service.operation === "step" && !tool)
+            context.addIssue({ code: "custom", path: ["executionServices", index, "toolName"], message: "Execution services must target a declared tool." });
+        if (service.timeoutMs >= Math.min(value.operationTimeoutMs, tool?.timeoutMs ?? value.operationTimeoutMs))
+            context.addIssue({ code: "custom", path: ["executionServices", index, "timeoutMs"], message: "Service budgets must leave time for the authored controller." });
+    });
 });
 export const JavaScriptEnvironmentDefinitionSchema = JavaScriptEnvironmentDefinitionContentSchema.safeExtend({ contentHash: z.string().regex(/^[a-f0-9]{64}$/) });
 export const JavaScriptEnvironmentOperationSchema = z.enum(["create", "reset", "step", "collect", "destroy"]);

package/dist/javascript-environment-node.js

@@ -1,13 +1,15 @@
 import { JavaScriptEnvironmentOperationSchema, JavaScriptEnvironmentResultSchema } from "./javascript-environment-contract.js";
 import { executeJavaScriptIsolateInWorker } from "./javascript-isolate-node.js";
 import { executeJavaScriptIsolateInProcess } from "./javascript-isolate-process.js";
+import { executeEnvironmentWithServices } from "./environment-execution-services.js";
+import { executeSqlInProcess } from "./sql-execution-node.js";
 /** The execution owner awaits termination before recording a step or cleanup. */
 export async function executeJavaScriptEnvironmentInWorker(input) {
     const operation = JavaScriptEnvironmentOperationSchema.parse(input.operation);
-    return JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolateInWorker({ ...input, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" }));
+    return executeEnvironmentWithServices(input, { sql: executeSqlInProcess, candidate: executeJavaScriptIsolateInProcess, controller: async (prepared) => JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolateInWorker({ ...prepared, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" })) });
 }
 /** Uses a Node child process independently of the execution owner's runtime. */
 export async function executeJavaScriptEnvironmentInProcess(input) {
     const operation = JavaScriptEnvironmentOperationSchema.parse(input.operation);
-    return JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolateInProcess({ ...input, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" }));
+    return executeEnvironmentWithServices(input, { sql: executeSqlInProcess, candidate: executeJavaScriptIsolateInProcess, controller: async (prepared) => JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolateInProcess({ ...prepared, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" })) });
 }

package/dist/javascript-environment.js

@@ -2,12 +2,13 @@ import { contentHash } from "@openpond/harness";
 import { verifyLearningTextAsset } from "./learning/assets.js";
 import { assertBoundedTaskJson, validateTaskValue } from "./task-schema.js";
 import { executeJavaScriptIsolate } from "./javascript-isolate.js";
+import { executeEnvironmentWithServices } from "./environment-execution-services.js";
 import { assertJavaScriptEnvironmentDefinition, JavaScriptEnvironmentOperationSchema, JavaScriptEnvironmentResultSchema } from "./javascript-environment-contract.js";
 export * from "./javascript-environment-contract.js";
 /** Pure, bounded module execution. Server hosts should provide the worker runner. */
 export async function executeJavaScriptEnvironment(input) {
     const operation = JavaScriptEnvironmentOperationSchema.parse(input.operation);
-    return JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolate({ ...input, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" }));
+    return executeEnvironmentWithServices(input, { candidate: executeJavaScriptIsolate, controller: async (prepared) => JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolate({ ...prepared, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" })) });
 }
 export class JavaScriptEnvironmentActionError extends Error {
     code;
@@ -59,7 +60,8 @@ export async function createJavaScriptEnvironmentSession(input) {
                 throw new JavaScriptEnvironmentActionError("unknown_tool");
             if (action && tool && !validateTaskValue(tool.inputSchema, action.arguments).valid)
                 throw new JavaScriptEnvironmentActionError("invalid_arguments");
-            const promise = execute({ source, operation, value: { input: taskInput, initialState, seed, state: structuredClone(state), sequence: steps, action: action ? structuredClone(action) : null }, timeoutMs: Math.min(definition.operationTimeoutMs, tool?.timeoutMs ?? definition.operationTimeoutMs), signal });
+            const executionServices = definition.executionServices?.filter(service => service.operation === operation && (operation !== "step" || service.toolName === action?.name));
+            const promise = execute({ source, operation, value: { input: taskInput, initialState, seed, state: structuredClone(state), sequence: steps, action: action ? structuredClone(action) : null }, timeoutMs: Math.min(definition.operationTimeoutMs, tool?.timeoutMs ?? definition.operationTimeoutMs), signal, ...(executionServices?.length ? { executionServices } : {}) });
             active = promise;
             const result = JavaScriptEnvironmentResultSchema.parse(await promise);
             signal.throwIfAborted();

package/dist/javascript-isolate-process-entry.js

@@ -1,8 +1,8 @@
-import { executeJavaScriptIsolate } from "./javascript-isolate.js";
+import { executeJavaScriptIsolate, JavaScriptIsolateExecutionError } from "./javascript-isolate.js";
 // Only this bundled, trusted program runs in Node. Authored source is interpreter data.
 const owner = globalThis;
 const input = owner.__openpondIsolateInput;
 delete owner.__openpondIsolateInput;
 if (!input)
     throw new Error("JavaScript process requires its execution owner.");
-void executeJavaScriptIsolate(input).then((result) => process.stdout.write(JSON.stringify({ ok: true, result })), (error) => process.stdout.write(JSON.stringify({ ok: false, error: (error instanceof Error ? error.message : "JavaScript execution failed.").slice(0, 8_192) })));
+void executeJavaScriptIsolate(input).then((result) => process.stdout.write(JSON.stringify({ ok: true, result })), (error) => process.stdout.write(JSON.stringify({ ok: false, executionError: error instanceof JavaScriptIsolateExecutionError, error: (error instanceof Error ? error.message : "JavaScript execution failed.").slice(0, 8_192) })));