@openpond/evals 0.1.0 → 0.2.0

package/CONTRACT.md

@@ -13,6 +13,9 @@ authorization, model streaming, artifact bytes, and runtime processes.
 | `GradeResult` | `GraderEvidence[]` | Preserve component score, pass, reward eligibility, failure class, feedback, and visible/private evidence references. Aggregate UI results remain host projections. |
 | managed-RL local receipt | `AttemptReceipt` | Submit canonical manifest/task/trace/artifact/grader identities; policy token responses and provider request IDs remain host-private trace data. |
 | resolved training bundle | `HarnessRelease` + host training binding | Environment, tools, program, policy, files, and grader interface belong to the Harness. Dataset/evidence, recipe, compute, engine, approval, and opaque leases remain explicit host-side bindings. |
+| completed Work or Development turn | `WorkEvidenceReceipt` | Project the authoritative terminal turn, immutable Agent snapshot when available, model/runtime identity, sanitized trace reference, exact output revisions, validation evidence, interventions, timing, usage, and explicit consent provenance. Keep the raw source and trace host-private. |
+| Agent plus environment runtime events | `WorkProcessTrace` | Emit one ordered trace with `agent` and `environment` layers. Bind every environment step to its outer Agent tool call or stable Agent-turn receipt hash. Hash inputs/outputs and expose only enumerated, bounded attributes. |
+| user feedback on Work output | `WorkFeedbackReceipt` | Append a new receipt bound to the evidence receipt and, when selected, the exact content-addressed output-revision descriptor. Corrections are separate artifacts and never mutate prior receipts. |
 
 ## Compatibility policy
 
@@ -29,6 +32,40 @@ authorization, model streaming, artifact bytes, and runtime processes.
   upper bounds.
 - Immutable content never contains secret values, opaque lease values, mutable
   provider resource IDs, database keys, UI state, or process handles.
+- The Work evidence schemas first ship as additive `0.2.x` package exports. The
+  package version does not replace the `openpond.workEvidenceReceipt.v1`,
+  `openpond.workProcessTrace.v1`, `openpond.workFeedbackReceipt.v1`, or
+  `openpond.workEvidenceEligibility.v1` schema literals.
+
+## Work evidence boundary
+
+- One `WorkEvidenceReceipt` spans the Agent and its environment or sandbox. A
+  later Taskset/Harness replay emits one `AttemptReceipt`, not separate Agent and
+  sandbox attempts.
+- Portable traces contain no private-trace path or reference. Source identities
+  are opaque SHA-256 URNs; artifacts and output-revision descriptors are
+  content-addressed. Hidden reasoning, prompts, raw command arguments/output,
+  local paths, credentials, provider handles, and raw validation text do not
+  cross the portable boundary.
+- Step attributes are closed and bounded: tool/validation/transition categories,
+  intervention outcomes, artifact counts, exit status, duration, available CPU
+  time and peak memory, and error class. Adding an arbitrary metadata bag would
+  be a privacy-boundary change.
+- Explicit process-and-artifact consent is required. Existing transcript consent
+  is not upgraded implicitly. Workspace-owned or multi-participant evidence must
+  satisfy the host's ownership and participant policy before projection.
+- Receipts are immutable observations; current consent status is not. Revoked,
+  expired, or source-deleted evidence is ineligible for all downstream uses.
+  Hosts enforce source-bound deletion and distribution withdrawal outside the
+  immutable receipt.
+- An incomplete trace remains valid observational evidence but is blocked from
+  eval, training, reward, and qualitative use according to the eligibility
+  report. Reward candidacy additionally requires a verified and reward-eligible
+  `AttemptReceipt` bound by hash.
+- Feedback is append-only. `outputRevisionRef` identifies an exact revision
+  descriptor even when two revisions contain identical bytes; correction
+  content remains a separate private artifact unless a later policy explicitly
+  permits disclosure.
 
 ## Runtime interfaces
 

package/README.md

@@ -1,8 +1,8 @@
 # `@openpond/evals`
 
 Portable OpenPond contracts and reference helpers for immutable Agent snapshots,
-Harness releases, Taskset releases, run manifests, attempt receipts, deterministic
-graders, and conformance fixtures.
+Harness releases, Taskset releases, run manifests, attempt receipts, observational
+Work evidence, deterministic graders, and conformance fixtures.
 
 ```ts
 import {
@@ -12,13 +12,38 @@ import {
   validateTasksetRelease,
   verifyAttemptReceipt,
 } from "@openpond/evals";
+
+import {
+  WorkEvidenceReceiptSchema,
+  classifyWorkEvidence,
+  workEvidenceConformance,
+} from "@openpond/evals/evidence";
 ```
 
 Subpath exports are available at `/harness`, `/tasksets`, `/graders`, `/runs`,
-and `/conformance`. The package is a protocol library, not a hosted client. It
+`/conformance`, and `/evidence`. The package is a protocol library, not a hosted client. It
 does not execute OpenPond Desktop or Sandbox sessions, resolve credentials, or
 persist artifacts.
 
+## Work evidence
+
+`WorkEvidenceReceipt` records one completed Work or Development turn as a single
+observational evidence unit. Its sanitized `WorkProcessTrace` combines Agent
+actions with environment or sandbox actions. Every environment step is bound to
+the outer Agent tool-call receipt hash or, for automatic lifecycle work, the
+stable Agent-turn receipt hash. Commands, exit status, lifecycle transitions,
+output and validation references, timing, and available CPU or peak-memory
+observations stay in that one trace.
+
+This receipt is not an evaluation result. A Taskset/Harness replay still emits
+exactly one `AttemptReceipt`; callers may classify Work evidence as an eval,
+training, reward, or qualitative candidate only through the explicit eligibility
+report. Reward candidacy requires a verified, reward-eligible Attempt receipt.
+
+Feedback is append-only and can bind an exact content-addressed output revision.
+Corrections remain separate artifacts and are valid only for
+`needs_correction` feedback.
+
 ## Privacy boundary
 
 Release content may include approved policy-visible assets and immutable hashes
@@ -28,6 +53,14 @@ mutable local UI state. Private graders receive only their declared evidence and
 must execute outside the policy-controlled Agent environment when tamper
 resistance matters.
 
+Work evidence requires explicit `work_process_and_artifacts` consent; transcript
+consent is not sufficient. Portable receipts expose only opaque source/workspace
+URNs, content hashes, bounded enums and counters, and sanitized artifact
+references. Raw traces and correction bytes stay host-private. Consent,
+revocation, expiry, deletion, ownership, participant policy, and source-bound
+retention remain authoritative host policy state; an immutable receipt never
+overrides that state.
+
 ## Runtime adapters
 
 Implement `HarnessRuntime` for environment state and `HarnessExecutor` around
@@ -55,6 +88,5 @@ pnpm --dir packages/evals run check
 pnpm run release:evals:patch -- --dry-run
 ```
 
-The first npm publication must be bootstrapped manually before trusted publishing
-can take over. Publication uses package-specific `evals-vX.Y.Z` tags and npm
-provenance; see the release workflow for the exact one-time setup.
+Publication uses package-specific `evals-vX.Y.Z` tags, npm trusted publishing,
+and provenance. See the release workflow for release preparation and verification.

package/RELEASING.md

@@ -1,7 +1,8 @@
 # Releasing `@openpond/evals`
 
 The package version is independent from OpenPond application and schema
-versions. Version `0.1.0` supports these initial schema literals:
+versions. Version `0.1.1` is the current provenance-backed baseline and supports
+these initial schema literals:
 
 - `openpond.agentSnapshot.v1`
 - `openpond.harnessRelease.v1`
@@ -12,38 +13,42 @@ versions. Version `0.1.0` supports these initial schema literals:
 - `openpond.graderEvidence.v1`
 - `openpond.evaluationResult.v1`
 
-## One-time npm bootstrap
+This branch prepares the additive Work evidence schemas for `0.2.0`:
 
-Trusted publishing can only be configured after the scoped package exists. Once
-this feature is merged, use a clean current `master` checkout:
+- `openpond.workEvidenceReceipt.v1`
+- `openpond.workProcessTrace.v1`
+- `openpond.workFeedbackReceipt.v1`
+- `openpond.workEvidenceEligibility.v1`
+
+## Trusted publishing
+
+The package already exists on npm and the `npm-production` environment is
+configured for trusted publishing through `release-evals.yml`. Do not run a
+manual `npm publish` or repeat the original bootstrap. The workflow uses its own
+pinned npm version for trusted publication and provenance.
+
+To inspect an already-published version:
 
 ```bash
 pnpm evals:check
-npm login
-npm publish ./packages/evals --access public --ignore-scripts --provenance=false
-npm install --global npm@^11.15.0
-npm trust github @openpond/evals \
-  --repo openpond/openpond \
-  --file release-evals.yml \
-  --environment npm-production \
-  --allow-publish
 npm trust list @openpond/evals
-gh workflow run release-evals.yml --ref master
+npm view @openpond/evals version dist.integrity dist.attestations
 ```
 
-The last command is idempotent recovery: because `0.1.0` is already on npm, the
-workflow skips publication and creates the missing `evals-v0.1.0` GitHub release.
-
 ## Later releases
 
-From a clean current `master` checkout:
+For releases that do not already carry an intentional version bump, merge
+feature work without changing the package version. When the intended package
+changes are on `master`, prepare a separate release from a clean current
+`master` checkout:
 
 ```bash
 pnpm release:evals:patch
 # or release:evals:minor / release:evals:major
 ```
 
-The release helper creates an Evals-only release PR. Merging it triggers trusted
-publishing with provenance, registry verification, and a package-specific tag.
-Record the registry integrity/provenance evidence and merge commit in the
-release notes after publication.
+The release helper creates an Evals-only release PR and updates the package
+version and lockfile. Merging it triggers trusted publishing with provenance,
+registry verification, and a package-specific tag. Record the registry
+integrity/provenance evidence and merge commit in the release notes after
+publication.

package/dist/evidence/authoring.js

@@ -0,0 +1,40 @@
+import { z } from "zod";
+import { ImmutableReleaseRefSchema, ReleaseHashSchema } from "../common.js";
+import { EvidenceArtifactRefSchema, WorkEvidenceReceiptSchema, evidenceArtifactRef, } from "./contracts.js";
+import { WorkEvidenceEligibilitySchema } from "./eligibility.js";
+export const WorkEvidenceAuthoringInputSchema = z.object({
+    schemaVersion: z.literal("openpond.workEvidenceAuthoringInput.v1"),
+    evidenceReceiptRef: EvidenceArtifactRefSchema,
+    inputHash: ReleaseHashSchema,
+    agentSnapshot: ImmutableReleaseRefSchema.nullable(),
+    sanitizedTraceRef: EvidenceArtifactRefSchema,
+    outputRefs: z.array(EvidenceArtifactRefSchema).max(10_000),
+    validationEvidenceRefs: z.array(EvidenceArtifactRefSchema).max(10_000),
+    incomplete: z.boolean(),
+    evalCandidate: z.boolean(),
+    blockerCodes: z.array(z.string().trim().min(1).max(120)).max(32),
+}).strict();
+export function toWorkEvidenceAuthoringInput(evidenceInput, eligibilityInput) {
+    const evidence = WorkEvidenceReceiptSchema.parse(evidenceInput);
+    const eligibility = WorkEvidenceEligibilitySchema.parse(eligibilityInput);
+    if (eligibility.evidenceReceiptHash !== evidence.contentHash) {
+        throw new Error("Eligibility report belongs to a different Work evidence receipt.");
+    }
+    const receiptRef = evidenceArtifactRef({
+        contentHash: evidence.contentHash,
+        mediaType: "application/vnd.openpond.work-evidence+json",
+        sizeBytes: null,
+    });
+    return WorkEvidenceAuthoringInputSchema.parse({
+        schemaVersion: "openpond.workEvidenceAuthoringInput.v1",
+        evidenceReceiptRef: receiptRef,
+        inputHash: evidence.inputHash,
+        agentSnapshot: evidence.agentSnapshot,
+        sanitizedTraceRef: evidence.trace.sanitizedRef,
+        outputRefs: evidence.outputRefs,
+        validationEvidenceRefs: evidence.validationEvidenceRefs,
+        incomplete: evidence.trace.incomplete,
+        evalCandidate: eligibility.decisions.eval_candidate.eligible,
+        blockerCodes: eligibility.decisions.eval_candidate.blockers,
+    });
+}

package/dist/evidence/conformance.js

@@ -0,0 +1,194 @@
+import { canonicalJson, contentHash } from "../common.js";
+import { createWorkEvidenceReceipt, createWorkFeedbackReceipt, createWorkProcessTrace, evidenceArtifactRef, workEvidenceReceiptRef, workSourceOpaqueRef, } from "./contracts.js";
+import { classifyWorkEvidence } from "./eligibility.js";
+const startedAt = "2026-08-04T12:00:00.000Z";
+const completedAt = "2026-08-04T12:00:02.500Z";
+const sourceRevisionHash = contentHash({ fixture: "desktop-work", revision: 1 });
+const output = evidenceArtifactRef({
+    contentHash: contentHash("fixture output"),
+    mediaType: "text/plain",
+    sizeBytes: 14,
+});
+const validation = evidenceArtifactRef({
+    contentHash: contentHash({ kind: "test", status: "passed" }),
+    mediaType: "application/json",
+    sizeBytes: null,
+});
+const consent = evidenceArtifactRef({
+    contentHash: contentHash({ scope: "work_process_and_artifacts", grantedAt: startedAt }),
+    mediaType: "application/json",
+    sizeBytes: null,
+});
+export const completeWorkProcessTraceFixture = createWorkProcessTrace({
+    schemaVersion: "openpond.workProcessTrace.v1",
+    sourceRevisionHash,
+    sanitationPolicyVersion: "openpond.desktop-work-sanitizer.v1",
+    incomplete: false,
+    incompleteReasons: [],
+    droppedEventCount: 1,
+    steps: [
+        step(0, startedAt, "agent", "state_transition", "turn_started", "started", {
+            transitionState: "running",
+        }),
+        step(1, "2026-08-04T12:00:00.500Z", "agent", "tool", "tool_invoked", "started", {
+            toolCategory: "filesystem",
+            inputHash: contentHash({ path: "hashed-only" }),
+            receiptHash: contentHash("fixture-tool-call"),
+        }),
+        step(2, "2026-08-04T12:00:01.000Z", "environment", "artifact", "artifact_created", "completed", {
+            outputHash: output.contentHash,
+            artifacts: [output],
+            receiptHash: contentHash("fixture-environment-operation"),
+            parentReceiptHash: contentHash("fixture-tool-call"),
+        }),
+        step(3, "2026-08-04T12:00:01.500Z", "environment", "validation", "validation_completed", "completed", {
+            validationKind: "test",
+            outputHash: validation.contentHash,
+            receiptHash: contentHash("fixture-validation"),
+            parentReceiptHash: contentHash("fixture-tool-call"),
+        }),
+        step(4, completedAt, "agent", "state_transition", "turn_completed", "completed", {
+            transitionState: "completed",
+        }),
+    ],
+});
+const traceBytes = new TextEncoder().encode(canonicalJson(completeWorkProcessTraceFixture)).byteLength;
+const traceRef = evidenceArtifactRef({
+    contentHash: completeWorkProcessTraceFixture.contentHash,
+    mediaType: "application/vnd.openpond.work-process-trace+json",
+    sizeBytes: traceBytes,
+});
+export const completeWorkEvidenceFixture = createWorkEvidenceReceipt({
+    schemaVersion: "openpond.workEvidenceReceipt.v1",
+    id: `work-evidence-${contentHash([sourceRevisionHash, traceRef.contentHash]).slice(0, 24)}`,
+    source: {
+        surface: "desktop",
+        experience: "work",
+        opaqueRef: workSourceOpaqueRef(["fixture-session", "fixture-turn"]),
+        revisionHash: sourceRevisionHash,
+    },
+    agentSnapshot: { id: "fixture-agent-snapshot", contentHash: contentHash("fixture-agent") },
+    model: {
+        provider: "fixture",
+        model: "scripted",
+        revision: "1",
+        artifactHash: null,
+        tokenizerRevision: null,
+        chatTemplateHash: null,
+    },
+    runtime: {
+        adapterId: "desktop-work",
+        adapterVersion: "1",
+        capabilityRef: null,
+    },
+    inputHash: contentHash("fixture prompt"),
+    terminal: { status: "completed", failureClass: null },
+    trace: {
+        sanitizedRef: traceRef,
+        traceHash: traceRef.contentHash,
+        sanitationPolicyVersion: completeWorkProcessTraceFixture.sanitationPolicyVersion,
+        incomplete: false,
+    },
+    outputRefs: [output],
+    artifactRefs: [output],
+    validationEvidenceRefs: [validation],
+    interventions: { approvals: 0, questions: 0, steeringEvents: 0, otherUserInterventions: 0 },
+    timing: { startedAt, completedAt, latencyMs: 2_500 },
+    usage: { promptTokens: 12, completionTokens: 8, totalTokens: 20 },
+    costUsd: 0,
+    provenance: {
+        consentReceiptRef: consent,
+        consentScope: "work_process_and_artifacts",
+        consentGrantedAt: startedAt,
+        policyVersion: "openpond.work-evidence-policy.v1",
+        projectorVersion: "openpond.desktop-work-evidence-projector.v1",
+        disclosure: "portable_sanitized",
+        ownershipScope: "personal",
+        workspaceRef: null,
+        participantPolicy: "creator_only",
+        retention: { policy: "source_bound", deleteWithSource: true, expiresAt: null },
+    },
+});
+export const acceptedWorkFeedbackFixture = createWorkFeedbackReceipt({
+    schemaVersion: "openpond.workFeedbackReceipt.v1",
+    id: `work-feedback-${contentHash([completeWorkEvidenceFixture.contentHash, "accepted"]).slice(0, 24)}`,
+    evidenceReceiptRef: workEvidenceReceiptRef(completeWorkEvidenceFixture),
+    outputRevisionRef: output,
+    verdict: "accepted",
+    reasonCodes: ["correct", "complete"],
+    correctionRef: null,
+    correctedOutputRevisionRef: null,
+    actor: "user",
+    createdAt: "2026-08-04T12:01:00.000Z",
+}, completeWorkEvidenceFixture);
+export const incompleteWorkProcessTraceFixture = createWorkProcessTrace({
+    schemaVersion: "openpond.workProcessTrace.v1",
+    sourceRevisionHash: contentHash({ fixture: "incomplete", revision: 1 }),
+    sanitationPolicyVersion: "openpond.desktop-work-sanitizer.v1",
+    incomplete: true,
+    incompleteReasons: ["missing_terminal", "unsupported_events_dropped"],
+    droppedEventCount: 2,
+    steps: [step(0, startedAt, "agent", "state_transition", "turn_started", "started", {
+            transitionState: "running",
+        })],
+});
+export const activeWorkEvidenceEligibilityFixture = classifyWorkEvidence({
+    evidence: completeWorkEvidenceFixture,
+    feedback: [acceptedWorkFeedbackFixture],
+    policyState: "active",
+    reconstructability: { input: true, environment: true, verifier: true },
+});
+export const revokedWorkEvidenceEligibilityFixture = classifyWorkEvidence({
+    evidence: completeWorkEvidenceFixture,
+    feedback: [acceptedWorkFeedbackFixture],
+    policyState: "revoked",
+    reconstructability: { input: true, environment: true, verifier: true },
+});
+export const invalidRawEvidenceFixture = {
+    ...completeWorkEvidenceFixture,
+    source: {
+        ...completeWorkEvidenceFixture.source,
+        opaqueRef: "raw-database-turn-id",
+    },
+    privateRef: "/private/runtime/trace.json",
+    publicData: {
+        reasoning: "hidden model reasoning",
+        apiKey: "secret-like-value",
+    },
+};
+export const workEvidenceConformance = {
+    trace: completeWorkProcessTraceFixture,
+    receipt: completeWorkEvidenceFixture,
+    feedback: acceptedWorkFeedbackFixture,
+    activeEligibility: activeWorkEvidenceEligibilityFixture,
+    revokedEligibility: revokedWorkEvidenceEligibilityFixture,
+    incompleteTrace: incompleteWorkProcessTraceFixture,
+    invalidRawEvidence: invalidRawEvidenceFixture,
+};
+function step(sequence, timestamp, layer, kind, action, status, input) {
+    return {
+        sequence,
+        timestamp,
+        layer,
+        kind,
+        action,
+        status,
+        inputHash: input.inputHash ?? null,
+        outputHash: input.outputHash ?? null,
+        receiptHash: input.receiptHash ?? null,
+        parentReceiptHash: input.parentReceiptHash ?? null,
+        artifactRefs: input.artifacts ?? [],
+        attributes: {
+            toolCategory: input.toolCategory ?? null,
+            validationKind: input.validationKind ?? null,
+            transitionState: input.transitionState ?? null,
+            interventionOutcome: null,
+            artifactCount: input.artifacts?.length ?? 0,
+            exitCode: null,
+            durationMs: null,
+            cpuTimeMs: null,
+            memoryPeakBytes: null,
+            errorClass: null,
+        },
+    };
+}