@openpond/evals 0.1.0

package/CONTRACT.md

@@ -0,0 +1,39 @@
+# Portable contract map
+
+`@openpond/evals` owns the portable wire formats and pure protocol logic. Host
+applications own persistence, provider sessions, secret leases, connected-app
+authorization, model streaming, artifact bytes, and runtime processes.
+
+| Existing object | Portable object | Migration rule |
+| --- | --- | --- |
+| `Taskset` | `TasksetRelease` | Project only the released tasks, policy, environment, tools, capabilities, graders, and immutable Harness binding. Authoring state and UI readiness remain host state. |
+| `HarnessRunManifest` (`openpond.harnessRunManifest.v1`) | `RunManifest` (`openpond.runManifest.v1`) | Treat the old object as a legacy training projection. Normalize its release/model/runtime identities into one new manifest; recipe, compute, engine, secret leases, and approval records remain host bindings referenced by hashes. |
+| `HarnessRunTrace` | `HarnessTrace` | Preserve ordered actions, observations, lifecycle events, terminal state, failure class, and trace hash. Learning-signal envelopes remain a training projection of the trace and receipt. |
+| `TaskAttemptResult` | `AttemptReceipt` | Preserve the old record for application persistence while adding a lossless receipt reference. Output becomes `outputHash`; trace and artifacts are separately hash-bound. |
+| `GradeResult` | `GraderEvidence[]` | Preserve component score, pass, reward eligibility, failure class, feedback, and visible/private evidence references. Aggregate UI results remain host projections. |
+| managed-RL local receipt | `AttemptReceipt` | Submit canonical manifest/task/trace/artifact/grader identities; policy token responses and provider request IDs remain host-private trace data. |
+| resolved training bundle | `HarnessRelease` + host training binding | Environment, tools, program, policy, files, and grader interface belong to the Harness. Dataset/evidence, recipe, compute, engine, approval, and opaque leases remain explicit host-side bindings. |
+
+## Compatibility policy
+
+- Schema versions are literal and independent from package semver.
+- `openpond.harnessRunManifest.v1` and `openpond.taskAttempt.v1` remain accepted
+  host persistence formats during migration, but are not emitted as new public
+  identities by this package.
+- Compatible package releases may add optional helpers and exports. Changing a
+  required field, identity hash, or privacy boundary requires a new schema
+  literal and an explicit normalizer.
+- The initial support target is Node.js ESM on Node 22.14 through Node 24.
+- Portable paths are relative and at most 2,000 characters. Individual assets
+  are at most 250 MB. Tasksets, traces, and evidence arrays have schema-level
+  upper bounds.
+- Immutable content never contains secret values, opaque lease values, mutable
+  provider resource IDs, database keys, UI state, or process handles.
+
+## Runtime interfaces
+
+`HarnessRuntime` owns create/reset/step/collect/destroy environment state.
+`HarnessExecutor` owns the existing model or Agent loop and emits one receipt
+plus a hash-bound trace. `EvaluationRunner` selects tasks, seeds, repetitions,
+and models, invokes the executor, runs graders in their declared boundary, and
+aggregates host results. Runtime adapters stay with Desktop or Sandbox.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 OpenPond
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,60 @@
+# `@openpond/evals`
+
+Portable OpenPond contracts and reference helpers for immutable Agent snapshots,
+Harness releases, Taskset releases, run manifests, attempt receipts, deterministic
+graders, and conformance fixtures.
+
+```ts
+import {
+  AttemptReceiptSchema,
+  HarnessReleaseSchema,
+  TasksetReleaseSchema,
+  validateTasksetRelease,
+  verifyAttemptReceipt,
+} from "@openpond/evals";
+```
+
+Subpath exports are available at `/harness`, `/tasksets`, `/graders`, `/runs`,
+and `/conformance`. The package is a protocol library, not a hosted client. It
+does not execute OpenPond Desktop or Sandbox sessions, resolve credentials, or
+persist artifacts.
+
+## Privacy boundary
+
+Release content may include approved policy-visible assets and immutable hashes
+for verifier/host-private assets. Never serialize secrets, credential leases,
+provider resource handles, database identifiers, unrelated conversations, or
+mutable local UI state. Private graders receive only their declared evidence and
+must execute outside the policy-controlled Agent environment when tamper
+resistance matters.
+
+## Runtime adapters
+
+Implement `HarnessRuntime` for environment state and `HarnessExecutor` around
+the host's existing model/Agent loop. Do not reimplement prompting, tool dispatch,
+session persistence, cost accounting, cancellation, or cleanup in this package.
+A Prime-style environment maps reset/step/observation semantics to the same
+interfaces and keeps provider allocation, authentication, and cleanup in its host.
+
+## Conformance
+
+Run both `genericToolConformance` and `marketingPortfolioConformance` through the
+same adapter. The marketing fixture is named test data, not a runtime switch.
+Compare manifest, terminal/failure, output, trace, artifact, and grader receipt
+semantics. Infrastructure failures must remain reward-ineligible.
+
+## Schema lifecycle
+
+Package semver and schema literals are independent. See [CONTRACT.md](./CONTRACT.md)
+for compatibility aliases, migration rules, size limits, and the field map.
+
+## Release preparation
+
+```bash
+pnpm --dir packages/evals run check
+pnpm run release:evals:patch -- --dry-run
+```
+
+The first npm publication must be bootstrapped manually before trusted publishing
+can take over. Publication uses package-specific `evals-vX.Y.Z` tags and npm
+provenance; see the release workflow for the exact one-time setup.

package/RELEASING.md

@@ -0,0 +1,49 @@
+# Releasing `@openpond/evals`
+
+The package version is independent from OpenPond application and schema
+versions. Version `0.1.0` supports these initial schema literals:
+
+- `openpond.agentSnapshot.v1`
+- `openpond.harnessRelease.v1`
+- `openpond.tasksetRelease.v1`
+- `openpond.runManifest.v1`
+- `openpond.attemptReceipt.v1`
+- `openpond.harnessTrace.v1`
+- `openpond.graderEvidence.v1`
+- `openpond.evaluationResult.v1`
+
+## One-time npm bootstrap
+
+Trusted publishing can only be configured after the scoped package exists. Once
+this feature is merged, use a clean current `master` checkout:
+
+```bash
+pnpm evals:check
+npm login
+npm publish ./packages/evals --access public --ignore-scripts --provenance=false
+npm install --global npm@^11.15.0
+npm trust github @openpond/evals \
+  --repo openpond/openpond \
+  --file release-evals.yml \
+  --environment npm-production \
+  --allow-publish
+npm trust list @openpond/evals
+gh workflow run release-evals.yml --ref master
+```
+
+The last command is idempotent recovery: because `0.1.0` is already on npm, the
+workflow skips publication and creates the missing `evals-v0.1.0` GitHub release.
+
+## Later releases
+
+From a clean current `master` checkout:
+
+```bash
+pnpm release:evals:patch
+# or release:evals:minor / release:evals:major
+```
+
+The release helper creates an Evals-only release PR. Merging it triggers trusted
+publishing with provenance, registry verification, and a package-specific tag.
+Record the registry integrity/provenance evidence and merge commit in the
+release notes after publication.

package/dist/common.js

@@ -0,0 +1,67 @@
+import { z } from "zod";
+import { sha256Hex } from "./sha256.js";
+export const MAX_PORTABLE_PATH_BYTES = 2_000;
+export const MAX_PORTABLE_ASSET_BYTES = 250_000_000;
+export const ReleaseIdSchema = z.string().trim().min(1).max(240);
+export const ReleaseHashSchema = z.string().regex(/^[a-f0-9]{64}$/);
+export const ReleaseTimestampSchema = z.string().datetime({ offset: true });
+export const MetadataSchema = z.record(z.string(), z.unknown()).default({});
+export const ImmutableReleaseRefSchema = z.object({
+    id: ReleaseIdSchema,
+    contentHash: ReleaseHashSchema,
+}).strict();
+export const ImmutableAssetRefSchema = z.object({
+    id: ReleaseIdSchema,
+    path: z.string().trim().min(1).max(MAX_PORTABLE_PATH_BYTES).refine(safeRelativePath),
+    contentHash: ReleaseHashSchema,
+    sizeBytes: z.number().int().nonnegative().max(MAX_PORTABLE_ASSET_BYTES),
+    mediaType: z.string().trim().min(1).max(200),
+    visibility: z.enum(["policy", "verifier", "host_private"]),
+}).strict();
+export const ImmutableArtifactRefSchema = z.object({
+    id: ReleaseIdSchema,
+    contentHash: ReleaseHashSchema,
+    mediaType: z.string().trim().min(1).max(200).nullable().default(null),
+    sizeBytes: z.number().int().nonnegative().max(MAX_PORTABLE_ASSET_BYTES).nullable().default(null),
+}).strict();
+export const FailureClassSchema = z.enum([
+    "policy_failure",
+    "grader_failure",
+    "environment_failure",
+    "infrastructure_failure",
+    "timeout",
+    "cancelled",
+]);
+export function canonicalJson(value) {
+    return `${JSON.stringify(sortValue(value), null, 2)}\n`;
+}
+export function sha256(value) {
+    return sha256Hex(value);
+}
+export function contentHash(value) {
+    return sha256(canonicalJson(value));
+}
+export function withContentHash(value) {
+    return { ...value, contentHash: contentHash(value) };
+}
+export function assertContentHash(value, label) {
+    const { contentHash: actual, ...hashable } = value;
+    const expected = contentHash(hashable);
+    if (actual !== expected)
+        throw new Error(`${label} contentHash is ${actual}; expected ${expected}.`);
+}
+function safeRelativePath(value) {
+    const normalized = value.replaceAll("\\", "/");
+    if (!normalized || normalized.startsWith("/") || normalized.includes("\0"))
+        return false;
+    return !normalized.split("/").some((part) => !part || part === "." || part === "..");
+}
+function sortValue(value) {
+    if (Array.isArray(value))
+        return value.map(sortValue);
+    if (!value || typeof value !== "object")
+        return value;
+    return Object.fromEntries(Object.entries(value)
+        .sort(([left], [right]) => left.localeCompare(right))
+        .map(([key, child]) => [key, sortValue(child)]));
+}

package/dist/conformance.js

@@ -0,0 +1,66 @@
+import { contentHash, withContentHash } from "./common.js";
+import { createAgentSnapshot, createHarnessRelease } from "./harness.js";
+import { createRunManifest } from "./runs.js";
+import { TasksetReleaseSchema } from "./tasksets.js";
+const EMPTY_HASH = contentHash("");
+const dependencyLock = { id: "fixture-lock", path: "package-lock.json", contentHash: EMPTY_HASH, sizeBytes: 0, mediaType: "application/json", visibility: "policy" };
+const program = { id: "fixture-program", path: "harness.mjs", contentHash: EMPTY_HASH, sizeBytes: 0, mediaType: "text/javascript", visibility: "policy" };
+export const genericToolConformance = fixture("generic-tool-v1", [
+    { name: "lookup", description: "Look up a deterministic value.", inputSchema: { type: "object" }, inputSchemaHash: contentHash({ type: "object" }), sideEffect: "read", timeoutMs: 1_000 },
+]);
+export const marketingPortfolioConformance = fixture("marketing-portfolio-v1", [
+    { name: "get_portfolio_snapshot", description: "Read the fixture portfolio.", inputSchema: { type: "object" }, inputSchemaHash: contentHash({ type: "object" }), sideEffect: "read", timeoutMs: 1_000 },
+    { name: "submit_budget_decision", description: "Submit a fixture decision.", inputSchema: { type: "object" }, inputSchemaHash: contentHash({ type: "object" }), sideEffect: "write", timeoutMs: 1_000 },
+]);
+function fixture(id, tools) {
+    const snapshot = createAgentSnapshot({
+        schemaVersion: "openpond.agentSnapshot.v1",
+        id: `${id}-agent`,
+        profileRelease: null,
+        instructions: [], skills: [], agents: [], toolDeclarations: tools,
+        capabilityRequirements: [], dependencyLock,
+        portability: { portable: true, blockers: [], localOnlyAssetRefs: [], hostPrivateAssetRefs: [] },
+        metadata: { conformanceFixture: id },
+    });
+    const harness = createHarnessRelease({
+        schemaVersion: "openpond.harnessRelease.v1",
+        id: `${id}-harness`,
+        agentSnapshot: { id: snapshot.id, contentHash: snapshot.contentHash },
+        program,
+        environment: { protocolVersion: "openpond.environment.v1", kind: "agent", entrypoint: id, stateful: true, deterministicSeeds: true, lifecycle: ["create", "reset", "step", "collect", "destroy"], networkPolicy: "none", defaultTimeoutMs: 5_000 },
+        tools,
+        lifecycle: { create: true, reset: true, step: true, collect: true, destroy: true, resetScope: "attempt" },
+        graderInterface: { visibleEvidence: ["output"], privilegedEvidence: ["expected"], privateVerifierIsolation: true },
+        policy: { policyVisibleFields: ["input"], privilegedFields: ["expectedOutput"], hiddenGraderRefs: [], connectedAppScopes: [] },
+        files: [], metadata: { conformanceFixture: id },
+    });
+    const tasksetContent = {
+        schemaVersion: "openpond.tasksetRelease.v1",
+        id: `${id}-taskset`, revision: 1,
+        harnessRelease: { id: harness.id, contentHash: harness.contentHash },
+        policy: harness.policy,
+        environment: harness.environment,
+        tools,
+        capabilities: [],
+        tasks: [
+            { id: `${id}-train`, clusterKey: `${id}-train`, split: "train", input: { prompt: "Use the declared tool." }, expectedOutput: { text: "done" }, policyVisibleContext: {}, privilegedContextRef: `${id}-expected`, artifactRefs: [], tags: ["conformance"] },
+            { id: `${id}-frozen`, clusterKey: `${id}-frozen`, split: "frozen_eval", input: { prompt: "Repeat the protocol." }, expectedOutput: { text: "done" }, policyVisibleContext: {}, privilegedContextRef: `${id}-expected-frozen`, artifactRefs: [], tags: ["conformance"] },
+        ],
+        graders: [{ id: `${id}-exact`, version: "1", kind: "content", weight: 1, hardGate: true, rewardEligible: true, privileged: true, config: { outputField: "text", expectedField: "text" } }],
+        metadata: { conformanceFixture: id },
+    };
+    const taskset = TasksetReleaseSchema.parse(withContentHash(tasksetContent));
+    const manifest = createRunManifest({
+        schemaVersion: "openpond.runManifest.v1",
+        id: `${id}-run`,
+        harnessRelease: { id: harness.id, contentHash: harness.contentHash },
+        tasksetRelease: { id: taskset.id, contentHash: taskset.contentHash },
+        model: { provider: "fixture", model: "scripted", revision: "1", artifactHash: null, tokenizerRevision: null, chatTemplateHash: null },
+        runtimeTarget: { adapterId: "fixture-runtime", placement: "local", runtimeVersion: "1", capabilityReceipt: contentHash(tools.map((tool) => tool.name)) },
+        limits: { maxTurns: 8, timeoutMs: 5_000, maxOutputBytes: 1_000_000, maximumSpendUsd: 0 },
+        approval: null,
+        createdAt: "2026-08-03T00:00:00.000Z",
+        metadata: { conformanceFixture: id },
+    });
+    return { snapshot, harness, taskset, manifest };
+}

package/dist/graders.js

@@ -0,0 +1,87 @@
+import { z } from "zod";
+import { FailureClassSchema, ReleaseHashSchema, ReleaseIdSchema, contentHash } from "./common.js";
+export const GraderEvidenceContentSchema = z.object({
+    schemaVersion: z.literal("openpond.graderEvidence.v1"),
+    graderId: ReleaseIdSchema,
+    graderVersion: z.string().trim().min(1).max(100),
+    score: z.number().min(0).max(1).nullable(),
+    passed: z.boolean(),
+    rewardEligible: z.boolean(),
+    failureClass: FailureClassSchema.nullable(),
+    feedback: z.array(z.string().max(20_000)).max(1_000),
+    visibleEvidenceRefs: z.array(ReleaseIdSchema).max(10_000),
+    privilegedEvidenceRefs: z.array(ReleaseIdSchema).max(10_000),
+}).strict();
+export const GraderEvidenceSchema = GraderEvidenceContentSchema.extend({ contentHash: ReleaseHashSchema }).strict();
+export async function gradeEvidence(input) {
+    if (input.evidence.infrastructureError) {
+        return input.graders.map((grader) => evidence(grader, {
+            score: null,
+            passed: false,
+            rewardEligible: false,
+            failureClass: "infrastructure_failure",
+            feedback: [input.evidence.infrastructureError],
+            visibleEvidenceRefs: [],
+            privilegedEvidenceRefs: [],
+        }));
+    }
+    return Promise.all(input.graders.map(async (grader) => {
+        if (grader.kind === "model_judge") {
+            if (!input.modelJudge || grader.calibrationStatus !== "passed")
+                return evidence(grader, unavailable("Model judge is unavailable or uncalibrated."));
+            return evidence(grader, await input.modelJudge({ grader, task: input.task, evidence: input.evidence }));
+        }
+        if (grader.kind === "custom_verifier") {
+            if (!input.customVerifier)
+                return evidence(grader, unavailable("Custom verifier is unavailable."));
+            return evidence(grader, await input.customVerifier({ grader, task: input.task, evidence: input.evidence }));
+        }
+        if (grader.kind === "human")
+            return evidence(grader, unavailable("Human review is pending."));
+        return evidence(grader, gradeDeterministic(grader, input.task, input.evidence));
+    }));
+}
+function gradeDeterministic(grader, task, attempt) {
+    let passed = false;
+    const config = grader.config;
+    if (grader.kind === "content") {
+        const outputField = string(config.outputField) ?? "text";
+        const actual = string(attempt.output[outputField]);
+        const expected = string(config.expectedValue) ?? string(task.expectedOutput?.[string(config.expectedField) ?? "text"]);
+        passed = actual !== null && expected !== null && normalize(actual) === normalize(expected);
+    }
+    else if (grader.kind === "schema") {
+        passed = strings(config.requiredKeys).every((key) => Object.hasOwn(attempt.output, key));
+    }
+    else if (grader.kind === "artifact") {
+        const contains = string(config.refIncludes) ?? "";
+        passed = attempt.artifactRefs.some((ref) => ref.includes(contains));
+    }
+    else if (grader.kind === "runtime_event") {
+        passed = strings(config.requiredEvents).every((required) => attempt.runtimeEventRefs.some((ref) => ref.includes(required)));
+    }
+    else {
+        const fields = strings(config.fields);
+        const compared = fields.length ? fields : Object.keys(task.expectedOutput ?? {});
+        passed = compared.every((field) => Object.is(attempt.output[field], task.expectedOutput?.[field]));
+    }
+    return {
+        score: passed ? 1 : 0,
+        passed,
+        rewardEligible: grader.rewardEligible && passed,
+        failureClass: passed ? null : "policy_failure",
+        feedback: [passed ? "Deterministic grader passed." : "Deterministic grader failed."],
+        visibleEvidenceRefs: [...attempt.runtimeEventRefs, ...attempt.artifactRefs],
+        privilegedEvidenceRefs: grader.privileged ? [task.privilegedContextRef].filter((ref) => ref !== null) : [],
+    };
+}
+function evidence(grader, result) {
+    const content = GraderEvidenceContentSchema.parse({ schemaVersion: "openpond.graderEvidence.v1", graderId: grader.id, graderVersion: grader.version, ...result });
+    return GraderEvidenceSchema.parse({ ...content, contentHash: contentHash(content) });
+}
+function unavailable(message) {
+    return { score: null, passed: false, rewardEligible: false, failureClass: "grader_failure", feedback: [message], visibleEvidenceRefs: [], privilegedEvidenceRefs: [] };
+}
+function string(value) { return typeof value === "string" ? value : null; }
+function strings(value) { return Array.isArray(value) ? value.filter((item) => typeof item === "string") : []; }
+function normalize(value) { return value.normalize("NFKC").trim().replace(/[,，]/g, "").replace(/[.\s]+$/g, "").replace(/\s+/g, " "); }

package/dist/harness.js

@@ -0,0 +1,206 @@
+import { z } from "zod";
+import { FailureClassSchema, ImmutableArtifactRefSchema, ImmutableAssetRefSchema, ImmutableReleaseRefSchema, MetadataSchema, ReleaseHashSchema, ReleaseIdSchema, contentHash, } from "./common.js";
+import { EnvironmentContractSchema, PolicyBoundarySchema, ToolDeclarationSchema } from "./tasksets.js";
+import { createAttemptReceipt } from "./runs.js";
+export const PortabilityReportSchema = z.object({
+    portable: z.boolean(),
+    blockers: z.array(z.string().trim().min(1).max(2_000)).max(1_000),
+    localOnlyAssetRefs: z.array(ReleaseIdSchema).max(10_000),
+    hostPrivateAssetRefs: z.array(ReleaseIdSchema).max(10_000),
+}).strict();
+export const AgentSnapshotContentSchema = z.object({
+    schemaVersion: z.literal("openpond.agentSnapshot.v1"),
+    id: ReleaseIdSchema,
+    profileRelease: ImmutableReleaseRefSchema.nullable(),
+    instructions: z.array(ImmutableAssetRefSchema).max(10_000),
+    skills: z.array(ImmutableAssetRefSchema).max(10_000),
+    agents: z.array(ImmutableAssetRefSchema).max(10_000),
+    toolDeclarations: z.array(ToolDeclarationSchema).max(200),
+    capabilityRequirements: z.array(z.object({
+        id: ReleaseIdSchema,
+        required: z.boolean(),
+        scopes: z.array(z.string().trim().min(1).max(500)).max(100),
+    }).strict()).max(200),
+    dependencyLock: ImmutableAssetRefSchema,
+    portability: PortabilityReportSchema,
+    metadata: MetadataSchema,
+}).strict();
+export const AgentSnapshotSchema = AgentSnapshotContentSchema.extend({ contentHash: ReleaseHashSchema }).strict();
+export const LifecycleContractSchema = z.object({
+    create: z.literal(true),
+    reset: z.literal(true),
+    step: z.literal(true),
+    collect: z.literal(true),
+    destroy: z.literal(true),
+    resetScope: z.enum(["task", "attempt"]),
+}).strict();
+export const GraderInterfaceContractSchema = z.object({
+    visibleEvidence: z.array(ReleaseIdSchema).max(1_000),
+    privilegedEvidence: z.array(ReleaseIdSchema).max(1_000),
+    privateVerifierIsolation: z.boolean(),
+}).strict();
+export const HarnessReleaseContentSchema = z.object({
+    schemaVersion: z.literal("openpond.harnessRelease.v1"),
+    id: ReleaseIdSchema,
+    agentSnapshot: ImmutableReleaseRefSchema,
+    program: ImmutableAssetRefSchema,
+    environment: EnvironmentContractSchema,
+    tools: z.array(ToolDeclarationSchema).max(200),
+    lifecycle: LifecycleContractSchema,
+    graderInterface: GraderInterfaceContractSchema,
+    policy: PolicyBoundarySchema,
+    files: z.array(ImmutableAssetRefSchema).max(100_000),
+    metadata: MetadataSchema,
+}).strict();
+export const HarnessReleaseSchema = HarnessReleaseContentSchema.extend({ contentHash: ReleaseHashSchema }).strict();
+export const ModelActionSchema = z.object({
+    id: ReleaseIdSchema,
+    turn: z.number().int().nonnegative(),
+    kind: z.enum(["message", "tool_call", "terminal"]),
+    name: ReleaseIdSchema.nullable(),
+    arguments: z.record(z.string(), z.unknown()),
+    content: z.string().max(1_000_000).nullable(),
+}).strict();
+export const ToolObservationSchema = z.object({
+    actionId: ReleaseIdSchema,
+    turn: z.number().int().nonnegative(),
+    terminal: z.boolean(),
+    output: z.record(z.string(), z.unknown()),
+    artifactRefs: z.array(ImmutableArtifactRefSchema).max(100_000),
+}).strict();
+export async function executeRuntimeProtocol(input) {
+    const now = input.now ?? (() => new Date().toISOString());
+    const startedAt = now();
+    const controller = new AbortController();
+    const abort = () => controller.abort(input.signal?.reason ?? new Error("Harness execution cancelled."));
+    input.signal?.addEventListener("abort", abort, { once: true });
+    const timer = setTimeout(() => controller.abort(new Error("Harness execution timed out.")), input.manifest.limits.timeoutMs);
+    timer.unref?.();
+    const events = [];
+    const actions = [];
+    const observations = [];
+    let sequence = 0;
+    let lease = null;
+    let terminal = false;
+    let failureClass = null;
+    let output = {};
+    let artifactRefs = [];
+    const event = (type, payload, metadata = {}) => {
+        events.push({ sequence: sequence++, type, payloadHash: contentHash(payload), metadata });
+    };
+    try {
+        lease = await input.runtime.create({ manifest: input.manifest, taskId: input.taskId, seed: input.seed, signal: controller.signal });
+        event("created", { leaseId: lease.id });
+        await input.runtime.reset(lease, { seed: input.seed, signal: controller.signal });
+        event("reset", { seed: input.seed });
+        for (const candidate of input.actions.slice(0, input.manifest.limits.maxTurns)) {
+            if (controller.signal.aborted)
+                throw controller.signal.reason;
+            const action = ModelActionSchema.parse(candidate);
+            actions.push(action);
+            event("action", action);
+            const observation = ToolObservationSchema.parse(await input.runtime.step(lease, action, controller.signal));
+            if (observation.actionId !== action.id || observation.turn !== action.turn) {
+                throw new Error(`Observation ${observation.actionId} does not match action ${action.id}.`);
+            }
+            observations.push(observation);
+            output = observation.output;
+            event("observation", observation);
+            if (observation.terminal) {
+                terminal = true;
+                event("terminal", observation);
+                break;
+            }
+        }
+    }
+    catch (error) {
+        failureClass = controller.signal.aborted
+            ? input.signal?.aborted ? "cancelled" : "timeout"
+            : "infrastructure_failure";
+        event("failure", { message: error instanceof Error ? error.message : String(error) }, { rewardEligible: false });
+    }
+    finally {
+        clearTimeout(timer);
+        input.signal?.removeEventListener("abort", abort);
+        if (lease) {
+            try {
+                const collected = await input.runtime.collect(lease, new AbortController().signal);
+                artifactRefs = collected.artifactRefs;
+                event("collected", collected);
+            }
+            catch (error) {
+                failureClass = "infrastructure_failure";
+                event("failure", { message: error instanceof Error ? error.message : String(error) }, { phase: "collect", rewardEligible: false });
+            }
+            try {
+                await input.runtime.destroy(lease);
+                event("destroyed", { leaseId: lease.id });
+            }
+            catch (error) {
+                failureClass = "infrastructure_failure";
+                event("failure", { message: error instanceof Error ? error.message : String(error) }, { phase: "destroy", rewardEligible: false });
+            }
+        }
+    }
+    const traceContent = {
+        schemaVersion: "openpond.harnessTrace.v1",
+        manifest: { id: input.manifest.id, contentHash: input.manifest.contentHash },
+        taskId: input.taskId,
+        seed: input.seed,
+        events,
+        actions,
+        observations,
+        terminal,
+        failureClass,
+        output,
+    };
+    const trace = HarnessTraceSchema.parse({ ...traceContent, contentHash: contentHash(traceContent) });
+    const completedAt = now();
+    const receipt = createAttemptReceipt({
+        schemaVersion: "openpond.attemptReceipt.v1",
+        id: `attempt-${contentHash([input.manifest.contentHash, input.taskId, input.seed, trace.contentHash]).slice(0, 24)}`,
+        runManifest: { id: input.manifest.id, contentHash: input.manifest.contentHash },
+        taskId: input.taskId,
+        seed: input.seed,
+        terminal,
+        failureClass,
+        outputHash: terminal ? contentHash(output) : null,
+        traceHash: trace.contentHash,
+        artifactRefs,
+        graderEvidenceRefs: [],
+        startedAt,
+        completedAt,
+        latencyMs: Math.max(0, Date.parse(completedAt) - Date.parse(startedAt)),
+        costUsd: null,
+        legacyAttemptRef: null,
+        metadata: {},
+    });
+    return { receipt, trace, output };
+}
+export function createAgentSnapshot(input) {
+    const content = AgentSnapshotContentSchema.parse(input);
+    return AgentSnapshotSchema.parse({ ...content, contentHash: contentHash(content) });
+}
+export function createHarnessRelease(input) {
+    const content = HarnessReleaseContentSchema.parse(input);
+    return HarnessReleaseSchema.parse({ ...content, contentHash: contentHash(content) });
+}
+export const HarnessLifecycleEventSchema = z.object({
+    sequence: z.number().int().nonnegative(),
+    type: z.enum(["created", "reset", "action", "observation", "terminal", "failure", "collected", "destroyed"]),
+    payloadHash: ReleaseHashSchema,
+    metadata: MetadataSchema,
+}).strict();
+export const HarnessTraceSchema = z.object({
+    schemaVersion: z.literal("openpond.harnessTrace.v1"),
+    manifest: ImmutableReleaseRefSchema,
+    taskId: ReleaseIdSchema,
+    seed: z.string().trim().min(1).max(500),
+    events: z.array(HarnessLifecycleEventSchema).max(1_000_000),
+    actions: z.array(ModelActionSchema).max(100_000),
+    observations: z.array(ToolObservationSchema).max(100_000),
+    terminal: z.boolean(),
+    failureClass: FailureClassSchema.nullable(),
+    output: z.record(z.string(), z.unknown()),
+    contentHash: ReleaseHashSchema,
+}).strict();

package/dist/index.js

@@ -0,0 +1,5 @@
+export * from "./common.js";
+export * from "./graders.js";
+export * from "./harness.js";
+export * from "./runs.js";
+export * from "./tasksets.js";