@openpond/evals 0.7.4 → 0.7.6

package/JAVASCRIPT_ENVIRONMENTS.md

@@ -3,7 +3,7 @@
 These entrypoints execute authored tool modules over owner-held JSON state:
 
 - `@openpond/evals/javascript-environment` validates immutable definitions and creates an episode with `step`, `collect`, `snapshot` and `destroy`.
-- `@openpond/evals/javascript-environment/node` executes each operation in a terminable worker around the isolated interpreter.
+- `@openpond/evals/javascript-environment/node` exposes worker and Node subprocess owners around the isolated interpreter.
 - `@openpond/evals/javascript-environment/attempt` runs a policy adapter through the declared tools and returns its messages, output and independently recorded state.
 
 The interpreter exposes no host functions, filesystem, network, timers or module loader. Environment code receives an explicit seed and supplied inputs; ambient `Date` and `Math.random` are unavailable. A module cannot call a production connector or launch a process. Tools that require those capabilities need a separate host adapter and its permissions.
@@ -53,7 +53,9 @@ Hosts persist the returned attempt under their task/model/release identity and g
 
 ## Lifecycle ownership
 
-Server hosts should pass `executeJavaScriptEnvironmentInWorker` as the session or attempt runner. Each worker runs a fresh interpreter. A promise settles only after the worker is terminated, including timeout and cancellation. The session rejects overlapping operations and waits for an active operation to stop before cleanup. Call `destroy` in `finally` when using the session API directly; the attempt runner does this automatically.
+Server hosts that require process isolation should pass `executeJavaScriptEnvironmentInProcess` as the session or attempt runner. It requires Node 22.14–24 on `PATH` and gives each operation a fresh Node child and QuickJS interpreter. The child inherits only `PATH`, uses a bounded heap and response stream, and is killed on timeout or cancellation. Authored source travels as JSON interpreter data. A promise settles only after the child exits. This keeps interpreter allocation outside the execution owner's worker-thread lifecycle. `executeJavaScriptVerifierInProcess` provides the same ownership for graders from `@openpond/evals/javascript-verifier/node`.
+
+Node hosts can also use `executeJavaScriptEnvironmentInWorker`. Each worker runs a fresh interpreter and settles only after termination. Both owners include startup in the declared deadline; neither retries failed operations. The session rejects overlapping operations and waits for active execution to stop before cleanup. Call `destroy` in `finally` when using the session API directly; the attempt runner does this automatically.
 
 Cleanup receives the definition's `operationTimeoutMs`, including interpreter and
 worker startup, with no shorter implicit deadline. It still runs after the episode
@@ -61,3 +63,50 @@ is cancelled, waits for worker termination, and fails explicitly when that bound
 operation deadline is exceeded.
 
 The policy adapter must honor its supplied abort signal and settle after cancelling its own request. `environmentCleanupComplete` describes environment cleanup only; it is not a provider billing or infrastructure-cleanup receipt. Failed creation, collection, cancellation and cleanup remain explicit states, without fabricated scores.
+
+## Submitted SQL and JavaScript
+
+An optional `executionServices` array on the immutable environment definition binds
+an owner-controlled service to a named `step` tool or to `collect`. Omission preserves
+existing definition hashes. Each service declares an identity, deadline and explicit
+value references: `{ scope: "input" | "initialState" | "state" | "arguments", path: [...] }`.
+References traverse own JSON properties only. The controller receives service results
+as `services[id]`; candidates cannot supply that field.
+
+A `sqlite.v1` binding declares `sql`, `snapshot`, `maxRows` and `maxResultBytes`.
+Snapshots contain named tables, typed columns (`INTEGER`, `REAL`, `TEXT`, `BLOB`) and
+array rows. SQLite executes exactly one statement in a fresh in-memory database.
+The Node adapter enforces read-only authorization, a deterministic function allowlist,
+a 32 MiB WASM memory maximum, value/SQL/result limits and a parent-owned deadline.
+Writes, attachment, pragma changes, extensions and nondeterministic functions are
+rejected. Results retain ordered column names and array rows, including duplicates.
+SQL null is JSON null; exact large integers use `{ integer: "9223372036854775807" }`
+and blobs use `{ blobBase64: "AP8=" }`. JSON numbers retain their ordinary IEEE-754
+semantics. Use tagged integers when exact 64-bit storage is required.
+
+A `javascript.v1` binding declares `source`, `cases`, `exportName`, `maxCases` and
+`maxResultBytes`. Cases are owner-held objects with an `input` property. Each case
+runs in a fresh QuickJS interpreter, and only its `input` crosses into the candidate.
+Other properties, including expected results, remain with the owner. Put hidden cases
+and expectations in private `initialState`, not task `input`: task input is part of
+the policy transcript. The authored controller grades returned case values against
+its own expectations. No host functions, module loader, filesystem or network are
+exposed to candidate JavaScript.
+
+Service outcomes are either completed results or explicit rejected submissions.
+Syntax, authorization, output and engine-memory failures cannot manufacture a passing
+answer. A service timeout produces a rejected outcome only after child termination;
+owner cancellation and infrastructure setup failures still fail the operation.
+Service preparation and the authored controller share the operation's overall budget
+and its tracked promise. Collection remains unable to mutate session state, and
+`destroy()` aborts and awaits all active work before running authored cleanup.
+
+`executeJavaScriptEnvironmentInProcess` and `executeJavaScriptEnvironmentInWorker`
+provide these services on Node hosts; service children always use Node processes.
+The portable in-process interpreter supports candidate JavaScript and rejects a SQL
+binding without a Node SQL executor. The standalone Node API is
+`executeSqlInProcess({ request, timeoutMs, signal })` from
+`@openpond/evals/sql-execution/node`; request/result schemas are available from
+`@openpond/evals/sql-execution`. The SQLite WASM bytes are embedded in the trusted
+child program, so execution survives installation and host bundling without resolving
+an application-relative WASM file or loading submitted code as host JavaScript.

package/dist/environment-execution-services-contract.js

@@ -0,0 +1,27 @@
+import { z } from "zod";
+import { SqlExecutionRequestSchema } from "./sql-execution-contract.js";
+/** Explicit owner-side paths; resolution never evaluates code or traverses prototypes. */
+export const EnvironmentValueReferenceSchema = z.object({
+    scope: z.enum(["input", "initialState", "state", "arguments"]),
+    path: z.array(z.string().min(1).max(256)).max(16),
+}).strict();
+const common = {
+    id: z.string().min(1).max(128),
+    operation: z.enum(["step", "collect"]),
+    toolName: z.string().min(1).max(500).optional(),
+    timeoutMs: z.number().int().min(1).max(20_000),
+};
+export const EnvironmentExecutionServiceSchema = z.discriminatedUnion("kind", [
+    z.object({ ...common, kind: z.literal("sqlite.v1"), sql: EnvironmentValueReferenceSchema, snapshot: EnvironmentValueReferenceSchema,
+        maxRows: SqlExecutionRequestSchema.shape.maxRows, maxResultBytes: SqlExecutionRequestSchema.shape.maxResultBytes }).strict(),
+    z.object({ ...common, kind: z.literal("javascript.v1"), source: EnvironmentValueReferenceSchema, cases: EnvironmentValueReferenceSchema,
+        exportName: z.string().regex(/^[A-Za-z_$][A-Za-z0-9_$]*$/).max(128), maxCases: z.number().int().min(1).max(32), maxResultBytes: z.number().int().min(256).max(262_144) }).strict(),
+]).superRefine((service, context) => {
+    if ((service.operation === "step") !== (service.toolName !== undefined))
+        context.addIssue({ code: "custom", path: ["toolName"], message: "Only step services require a declared tool name." });
+    if (service.operation === "collect") {
+        const references = service.kind === "sqlite.v1" ? [service.sql, service.snapshot] : [service.source, service.cases];
+        if (references.some(reference => reference.scope === "arguments"))
+            context.addIssue({ code: "custom", message: "Collection has no tool arguments." });
+    }
+});

package/dist/environment-execution-services.js

@@ -0,0 +1,100 @@
+import { EnvironmentExecutionServiceSchema } from "./environment-execution-services-contract.js";
+import { JavaScriptIsolateExecutionError } from "./javascript-isolate.js";
+import { assertBoundedTaskJson } from "./task-schema.js";
+import { assertSqlExecutionRequest } from "./sql-execution-contract.js";
+function resolveValue(value, reference) {
+    let resolved = reference.scope === "arguments" ? value.action?.arguments : value[reference.scope];
+    for (const key of reference.path) {
+        if (!resolved || typeof resolved !== "object" || !Object.hasOwn(resolved, key))
+            throw new Error("environment_service_reference_missing");
+        resolved = resolved[key];
+    }
+    return resolved;
+}
+/** The single returned promise owns service work, controller execution and cancellation. */
+export async function executeEnvironmentWithServices(input, runners) {
+    if (!input.executionServices?.length)
+        return runners.controller(input);
+    if (!Number.isSafeInteger(input.timeoutMs) || input.timeoutMs < 1 || input.timeoutMs > 30_000)
+        throw new Error("environment_timeout_invalid");
+    assertBoundedTaskJson(input.value, 4_194_304);
+    const services = input.executionServices.map(service => EnvironmentExecutionServiceSchema.parse(service));
+    if (services.length > 16)
+        throw new Error("environment_service_limit");
+    if (new Set(services.map(service => service.id)).size !== services.length)
+        throw new Error("environment_service_identity_collision");
+    const started = Date.now();
+    const controller = new AbortController();
+    const signal = input.signal ? AbortSignal.any([input.signal, controller.signal]) : controller.signal;
+    const timer = setTimeout(() => controller.abort(new Error("environment_timeout")), input.timeoutMs);
+    try {
+        const results = Object.create(null);
+        for (const service of services) {
+            signal.throwIfAborted();
+            if (service.operation !== input.operation || (service.operation === "step" && service.toolName !== input.value.action?.name))
+                throw new Error("environment_service_operation_mismatch");
+            const budget = Math.min(service.timeoutMs, input.timeoutMs - (Date.now() - started));
+            if (budget < 1)
+                throw new Error("environment_timeout");
+            const deadline = new AbortController();
+            const serviceSignal = AbortSignal.any([signal, deadline.signal]);
+            const serviceTimer = setTimeout(() => deadline.abort(new Error("environment_service_timeout")), budget);
+            const serviceStarted = Date.now();
+            try {
+                if (service.kind === "sqlite.v1") {
+                    if (!runners.sql)
+                        throw new Error("environment_sql_executor_required");
+                    const sql = resolveValue(input.value, service.sql);
+                    if (typeof sql !== "string" || !sql.length || new TextEncoder().encode(sql).byteLength > 65_536 || sql.includes("\0")) {
+                        results[service.id] = { status: "rejected", code: "invalid_submission" };
+                        continue;
+                    }
+                    const request = assertSqlExecutionRequest({ schemaVersion: "openpond.sqlExecution.v1", snapshot: resolveValue(input.value, service.snapshot), sql, maxRows: service.maxRows, maxResultBytes: service.maxResultBytes });
+                    results[service.id] = await runners.sql({ request, timeoutMs: input.timeoutMs, signal: serviceSignal });
+                }
+                else {
+                    const source = resolveValue(input.value, service.source), cases = resolveValue(input.value, service.cases);
+                    if (typeof source !== "string" || new TextEncoder().encode(source).byteLength > 524_288) {
+                        results[service.id] = { status: "rejected", code: "invalid_submission" };
+                        continue;
+                    }
+                    if (!Array.isArray(cases) || !cases.length || cases.length > service.maxCases || cases.some(item => !item || typeof item !== "object" || !Object.hasOwn(item, "input")))
+                        throw new Error("environment_service_cases_invalid");
+                    const values = [];
+                    for (const item of cases) {
+                        const remaining = budget - (Date.now() - serviceStarted);
+                        if (remaining < 1) {
+                            deadline.abort(new Error("environment_service_timeout"));
+                            serviceSignal.throwIfAborted();
+                        }
+                        // Hidden expected results and all other case fields remain with the owner.
+                        values.push(await runners.candidate({ source, exportName: service.exportName, value: structuredClone(item.input), timeoutMs: Math.max(1, remaining), maxResultBytes: service.maxResultBytes, deterministic: true, errorPrefix: "candidate", signal: serviceSignal }));
+                        if (new TextEncoder().encode(JSON.stringify({ status: "completed", cases: values })).byteLength > service.maxResultBytes)
+                            throw new JavaScriptIsolateExecutionError("candidate_result_too_large");
+                    }
+                    results[service.id] = { status: "completed", cases: values };
+                }
+            }
+            catch (error) {
+                signal.throwIfAborted();
+                if (deadline.signal.aborted)
+                    results[service.id] = { status: "rejected", code: "timeout" };
+                else if (error instanceof JavaScriptIsolateExecutionError)
+                    results[service.id] = { status: "rejected", code: "execution_failed" };
+                else
+                    throw error;
+            }
+            finally {
+                clearTimeout(serviceTimer);
+            }
+        }
+        signal.throwIfAborted();
+        const remaining = input.timeoutMs - (Date.now() - started);
+        if (remaining < 1)
+            throw new Error("environment_timeout");
+        return await runners.controller({ ...input, executionServices: undefined, value: { ...input.value, services: results }, timeoutMs: remaining, signal });
+    }
+    finally {
+        clearTimeout(timer);
+    }
+}

package/dist/javascript-environment-contract.js

@@ -1,6 +1,8 @@
 import { z } from "zod";
 import { ImmutableAssetRefSchema, ToolDeclarationSchema, contentHash } from "@openpond/harness";
 import { assertBoundedTaskJson, validateTaskSchema } from "./task-schema-validation.js";
+import { EnvironmentExecutionServiceSchema } from "./environment-execution-services-contract.js";
+export * from "./environment-execution-services-contract.js";
 export const JavaScriptEnvironmentDefinitionContentSchema = z.object({
     schemaVersion: z.literal("openpond.javascriptEnvironment.v1"),
     id: z.string().trim().min(1).max(500),
@@ -11,6 +13,7 @@ export const JavaScriptEnvironmentDefinitionContentSchema = z.object({
     maxStateBytes: z.number().int().min(1).max(1_048_576),
     maxObservationBytes: z.number().int().min(1).max(262_144),
     operationTimeoutMs: z.number().int().min(1).max(30_000),
+    executionServices: z.array(EnvironmentExecutionServiceSchema).min(1).max(16).optional(),
 }).strict().superRefine((value, context) => {
     if (new Set(value.tools.map(tool => tool.name)).size !== value.tools.length)
         context.addIssue({ code: "custom", path: ["tools"], message: "Environment tool names must be unique." });
@@ -20,6 +23,16 @@ export const JavaScriptEnvironmentDefinitionContentSchema = z.object({
         if (tool.inputSchemaHash !== contentHash(tool.inputSchema))
             context.addIssue({ code: "custom", path: ["tools", index, "inputSchemaHash"], message: "Tool schema identity differs from its declared schema." });
     });
+    const services = value.executionServices ?? [];
+    if (new Set(services.map(service => service.id)).size !== services.length)
+        context.addIssue({ code: "custom", path: ["executionServices"], message: "Execution service identities must be unique." });
+    services.forEach((service, index) => {
+        const tool = value.tools.find(tool => tool.name === service.toolName);
+        if (service.operation === "step" && !tool)
+            context.addIssue({ code: "custom", path: ["executionServices", index, "toolName"], message: "Execution services must target a declared tool." });
+        if (service.timeoutMs >= Math.min(value.operationTimeoutMs, tool?.timeoutMs ?? value.operationTimeoutMs))
+            context.addIssue({ code: "custom", path: ["executionServices", index, "timeoutMs"], message: "Service budgets must leave time for the authored controller." });
+    });
 });
 export const JavaScriptEnvironmentDefinitionSchema = JavaScriptEnvironmentDefinitionContentSchema.safeExtend({ contentHash: z.string().regex(/^[a-f0-9]{64}$/) });
 export const JavaScriptEnvironmentOperationSchema = z.enum(["create", "reset", "step", "collect", "destroy"]);

package/dist/javascript-environment-node.js

@@ -1,7 +1,15 @@
 import { JavaScriptEnvironmentOperationSchema, JavaScriptEnvironmentResultSchema } from "./javascript-environment-contract.js";
 import { executeJavaScriptIsolateInWorker } from "./javascript-isolate-node.js";
+import { executeJavaScriptIsolateInProcess } from "./javascript-isolate-process.js";
+import { executeEnvironmentWithServices } from "./environment-execution-services.js";
+import { executeSqlInProcess } from "./sql-execution-node.js";
 /** The execution owner awaits termination before recording a step or cleanup. */
 export async function executeJavaScriptEnvironmentInWorker(input) {
     const operation = JavaScriptEnvironmentOperationSchema.parse(input.operation);
-    return JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolateInWorker({ ...input, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" }));
+    return executeEnvironmentWithServices(input, { sql: executeSqlInProcess, candidate: executeJavaScriptIsolateInProcess, controller: async (prepared) => JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolateInWorker({ ...prepared, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" })) });
+}
+/** Uses a Node child process independently of the execution owner's runtime. */
+export async function executeJavaScriptEnvironmentInProcess(input) {
+    const operation = JavaScriptEnvironmentOperationSchema.parse(input.operation);
+    return executeEnvironmentWithServices(input, { sql: executeSqlInProcess, candidate: executeJavaScriptIsolateInProcess, controller: async (prepared) => JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolateInProcess({ ...prepared, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" })) });
 }

package/dist/javascript-environment.js

@@ -2,12 +2,13 @@ import { contentHash } from "@openpond/harness";
 import { verifyLearningTextAsset } from "./learning/assets.js";
 import { assertBoundedTaskJson, validateTaskValue } from "./task-schema.js";
 import { executeJavaScriptIsolate } from "./javascript-isolate.js";
+import { executeEnvironmentWithServices } from "./environment-execution-services.js";
 import { assertJavaScriptEnvironmentDefinition, JavaScriptEnvironmentOperationSchema, JavaScriptEnvironmentResultSchema } from "./javascript-environment-contract.js";
 export * from "./javascript-environment-contract.js";
 /** Pure, bounded module execution. Server hosts should provide the worker runner. */
 export async function executeJavaScriptEnvironment(input) {
     const operation = JavaScriptEnvironmentOperationSchema.parse(input.operation);
-    return JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolate({ ...input, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" }));
+    return executeEnvironmentWithServices(input, { candidate: executeJavaScriptIsolate, controller: async (prepared) => JavaScriptEnvironmentResultSchema.parse(await executeJavaScriptIsolate({ ...prepared, exportName: operation, maxResultBytes: 1_572_864, deterministic: true, errorPrefix: "environment" })) });
 }
 export class JavaScriptEnvironmentActionError extends Error {
     code;
@@ -59,7 +60,8 @@ export async function createJavaScriptEnvironmentSession(input) {
                 throw new JavaScriptEnvironmentActionError("unknown_tool");
             if (action && tool && !validateTaskValue(tool.inputSchema, action.arguments).valid)
                 throw new JavaScriptEnvironmentActionError("invalid_arguments");
-            const promise = execute({ source, operation, value: { input: taskInput, initialState, seed, state: structuredClone(state), sequence: steps, action: action ? structuredClone(action) : null }, timeoutMs: Math.min(definition.operationTimeoutMs, tool?.timeoutMs ?? definition.operationTimeoutMs), signal });
+            const executionServices = definition.executionServices?.filter(service => service.operation === operation && (operation !== "step" || service.toolName === action?.name));
+            const promise = execute({ source, operation, value: { input: taskInput, initialState, seed, state: structuredClone(state), sequence: steps, action: action ? structuredClone(action) : null }, timeoutMs: Math.min(definition.operationTimeoutMs, tool?.timeoutMs ?? definition.operationTimeoutMs), signal, ...(executionServices?.length ? { executionServices } : {}) });
             active = promise;
             const result = JavaScriptEnvironmentResultSchema.parse(await promise);
             signal.throwIfAborted();

package/dist/javascript-isolate-process-entry.js

@@ -0,0 +1,8 @@
+import { executeJavaScriptIsolate, JavaScriptIsolateExecutionError } from "./javascript-isolate.js";
+// Only this bundled, trusted program runs in Node. Authored source is interpreter data.
+const owner = globalThis;
+const input = owner.__openpondIsolateInput;
+delete owner.__openpondIsolateInput;
+if (!input)
+    throw new Error("JavaScript process requires its execution owner.");
+void executeJavaScriptIsolate(input).then((result) => process.stdout.write(JSON.stringify({ ok: true, result })), (error) => process.stdout.write(JSON.stringify({ ok: false, executionError: error instanceof JavaScriptIsolateExecutionError, error: (error instanceof Error ? error.message : "JavaScript execution failed.").slice(0, 8_192) })));