@arnilo/prism 0.6.0 → 0.7.0

package/docs/evaluations.md

@@ -15,14 +15,25 @@ Use this package when a host needs offline quality checks or sampled live scorin
 | `defineScorer` | `id`, `score({ result, item?, expected?, signal? })` |
 | `defineDataset` | `id`, `version?`, immutable `items[]` with unique ids |
 | `scoreRun` / `scoreRunLive` | `AgentRunResult`, scorers, optional `sampleRate`, store, ownership, redactor |
-| `runExperiment` | `agent`, dataset, scorers, bounded `concurrency`, optional store/ownership |
+| `runExperiment` | `agent`, dataset, scorers, bounded `concurrency`, optional `trials`/`seed`, store/ownership |
+| `wrapAgentWithFailureInjection` | `failStore`, `denyTools`, `unknownEffect` — synthetic faults; does not mutate production stores |
+| `validateEvalManifest` | `runtimeRevision`, `datasetVersion` |
+| `validateReleaseEvalManifest` | two-field manifest plus `promptVersion` (or `promptId`+`promptVersion`), `toolFingerprint`, `model`, `policyRevision` |
+| `runScenario` | `turns: { user, assertReply? }[]`, scorers grade the last result |
 | `createMemoryEvaluationStore` | optional seed records |
 | `appendEvaluationFeedback` | `RunFeedbackStore`, `EvaluationStore`, feedback fields, and 1–64 known evaluation IDs |
 | `createPersistenceTraceResolver` | explicit `ProductionPersistenceStore`, exact session/run/ownership, page/byte bounds |
 | `datasetFromRuns` | `runIds` and/or `sessionIds`, existing dataset, `ProductionPersistenceStore`, ownership, redactor/secrets, optional `toItem` |
 | `createModelJudge` | host judge callback, stable rubric/version, timeout/attempt/output bounds |
+| `createToolCallMatchScorer` | Match timeline tool steps against expected specs (`strict`, `unordered`, `subset`, `superset`) + optional `deny` list |
+| `createStepBudgetScorer` | Ceiling checks for turns, tool calls, duration, tokens, cost |
+| `createNoLoopScorer` | Detects degenerate repetitive tool+arguments burst loops |
+| `createSchemaScorer` | Validates final result or named step output against JSON schema |
+| `createErrorClassScorer` | Fails closed if denied error codes or blocked executions appear on timeline |
+| `createApprovalBeforeEffectScorer` | Verifies explicit approval occurred on timeline prior to sensitive tool effect |
+| `createCitationIntegrityScorer` | Invariant 0 on missing source, hash/span mismatch, or revoked ACL. Reads `environment.citations[]`. Ignores semantic `support`. |
 | `runComparison` | immutable dataset, 2–8 named candidates by default, pairwise scorers |
-| `assertEvaluationThreshold` / `serializeEvaluationReport` | mean/failure/per-scorer gates and bounded redacted JSON |
+| `assertEvaluationThreshold` / `serializeEvaluationReport` | mean/failure/per-scorer gates, hard invariant enforcement, and bounded redacted JSON |
 
 ## Outputs / response / events
 
@@ -30,7 +41,7 @@ Use this package when a host needs offline quality checks or sampled live scorin
 | --- | --- |
 | `scoreRun` | `EvaluationRecord[]` with `scored` / `skipped` / `failed` |
 | `scoreRunLive` | same records; never mutates the agent result; host may ignore the promise |
-| `runExperiment` | `ExperimentReport` with stable item order, evaluations, and aggregates |
+| `runExperiment` | `ExperimentReport` with stable item order, evaluations, aggregates; `trials.sampleCount` / `standardError` when `trials` is set |
 | `EvaluationStore.query` | cursor-paginated, ownership-filtered page |
 | `appendEvaluationFeedback` | immutable `RunFeedbackRecord` containing only evaluation/scorer IDs |
 | `datasetFromRuns` | `{ dataset, version, added, skipped }` — new immutable dataset version with one item per added run; skips carry reasons (missing run, ownership mismatch, empty output) |
@@ -113,6 +124,70 @@ console.log(report.aggregate.meanScore, linked.evaluationIds);
 - Dataset snapshots are frozen; duplicate item ids fail closed.
 - `appendEvaluationFeedback()` resolves every supplied ID from `EvaluationStore`, rejects missing IDs, verifies each evaluation has the same run, optional trace, and exact ownership as feedback, then copies only deduplicated `evaluationIds`/`scorerIds`. Evaluation scores, reasons, errors, and metadata are not duplicated.
 
+## Trajectory and outcome scoring
+
+Scorers can evaluate the complete execution path (`input.timeline`) or host-supplied ground-truth environment states (`input.environment`), rather than only evaluating assistant response text:
+
+```ts
+import {
+  createApprovalBeforeEffectScorer,
+  createErrorClassScorer,
+  createNoLoopScorer,
+  createSchemaScorer,
+  createStepBudgetScorer,
+  createToolCallMatchScorer,
+  defineScorer,
+  runExperiment,
+} from "@arnilo/prism-core/governance/evals";
+
+const scorers = [
+  // 1. Tool call match scorer (strict / unordered / subset / superset)
+  createToolCallMatchScorer({
+    id: "required_search",
+    mode: "superset",
+    expected: ["search_kb"],
+    deny: ["drop_tables"], // Hard invariant: immediately 0 if called
+  }),
+
+  // 2. Step and resource budget ceiling
+  createStepBudgetScorer({
+    id: "run_budget",
+    maxToolCalls: 8,
+    maxTurns: 4,
+    maxDurationMs: 30_000,
+  }),
+
+  // 3. Degenerate loop detector
+  createNoLoopScorer({ id: "no_burst_loop", maxRepeatedToolCalls: 2 }),
+
+  // 4. Approval before effect (HITL/guardrail before sensitive tool)
+  createApprovalBeforeEffectScorer({ id: "approved_refund", toolName: "execute_refund" }),
+
+  // 5. Outcome environment scorer (host-verified state)
+  defineScorer({
+    id: "ticket_closed_in_db",
+    score: ({ environment }) => ({
+      score: (environment as any)?.status === "closed" ? 1 : 0,
+    }),
+  }),
+];
+
+// Run experiment with timeline projection and environment resolution
+const report = await runExperiment({
+  agent,
+  dataset,
+  scorers,
+  timeline: "redacted_io", // "metadata" | "redacted_io"
+  toEnvironment: async (item) => hostLoadTicket(item.id),
+});
+```
+
+### Invariant scoring and threshold enforcement
+
+- Hard invariants (such as `deny` lists in tool match scorers or `createApprovalBeforeEffectScorer`) mark evaluation records with `metadata.invariant = true`.
+- When any invariant record scores `< 1.0`, `report.aggregate.invariantsPassed` becomes `false`.
+- `assertEvaluationThreshold(report, thresholds)` fails closed when `invariantsPassed === false`, preventing high text scores from averaging away safety or policy violations in CI.
+
 ## Security and performance notes
 
 - Scorers receive result/item data only. Credentials, tools, and workspace access are not provided unless the host deliberately closes over them.
@@ -138,7 +213,7 @@ const comparison = await runComparison({ dataset, candidates: { baseline, candid
 assertEvaluationThreshold(report, { minimumMean: 0.9, maximumFailures: 0 });
 ```
 
-`traceResolver` is explicit; no arbitrary run search occurs. `baseline`/`candidate` are host functions returning `AgentRunResult`. See `examples/evaluation-gate.ts` for a network-free gate and `examples/coding-browser-evaluation.ts` for coding/browser adversarial fixtures.
+`traceResolver` is explicit; no arbitrary run search occurs. `baseline`/`candidate` are host functions returning `AgentRunResult`. See `examples/evaluation-gate.ts` for a network-free gate. `examples/behavior-evaluation.ts` executes host-journey packs on the eval APIs: `trials: 3` (`sampleCount === 3`), `runScenario` `{ user, assertReply }`, `wrapAgentWithFailureInjection` (`failStore` / `denyTools` / `unknownEffect`), Task 8 stale draft revision, Task 11 revoked-ACL citations via `createCitationIntegrityScorer`, and `validateReleaseEvalManifest`. `examples/coding-browser-evaluation.ts` adds a coding test-oracle `toEnvironment` (fixture file hash — not SWE-bench). Inspector compare (`POST /compare`) still returns `invariant_blocked` when either side fails invariants.
 
 ## Curating datasets from production runs
 
@@ -180,6 +255,18 @@ Release 0.0.9 ships curated network-free adversarial fixtures in package tests:
 
 Fixtures reuse `@arnilo/prism-core/governance/evals` (`defineDataset` / `defineScorer` / `scoreRun` / `assertEvaluationThreshold` / `serializeEvaluationReport`). Optional SWE-bench-compatible or live-browser harnesses remain host adapters — they are not default dependencies or quality claims. Protected real Docker/Playwright gates stay env-gated (`PRISM_TEST_DOCKER_SANDBOX`, `PRISM_LIVE_PLAYWRIGHT`) and never enter `sdk:ready`.
 
+## Subagent spawn adversarial evaluations (0.7.0)
+
+The spawn pack (`@arnilo/prism-core/governance/evals` `spawn-pack.test.ts`) grades the in-process spawn tool and supervisor on structured host facts only — never model prose — with mock providers and `runScenario`, one hard invariant each:
+
+- `spawn.capability-truth`: every child that ran was advertised by the spawn tool schema and present in the host catalog; an injected request for an uncatalogued child never runs.
+- `spawn.parallel-cap`: concurrent children never exceed `maxActiveChildren`.
+- `spawn.non-widen`: a child never receives identity scopes wider than its parent, and never executes a tool outside its allow-list under injected instructions.
+- `spawn.cancel`: a requested cancel reaches the child's abort signal and leaves no live delegation.
+- `spawn.critic-gate`: a failed verification child blocks parent effects until the host join policy reads the verdict (verification failures cannot be averaged away).
+
+Negative controls wire deliberately vulnerable host compositions — uncatalogued spawn, skipped reservation, model-supplied scope escalation, leaky child tool list, non-aborting cancel, ungated ship — and assert the matching grader reports `0` naming the violation.
+
 ## PostgreSQL enterprise state (0.0.23)
 
 `createPostgresEnterpriseState({ pool, schema }).evaluations` implements this package's existing `EvaluationStore`. The host creates an `EvaluationRecord` from verified ownership before append; every PostgreSQL query requires tenant scope, uses exact normalized account/user matching, and returns owner-bound opaque cursor pages. It is durable across reopen and supports the existing id/scorer/session/run/trace/dataset/item/experiment/status filters.
@@ -237,12 +324,89 @@ PRISM_TEST_POSTGRES_URL=postgresql://... node --test scripts/phase27-erp-journey
 The journey reuses the two-replica failover worker (`scripts/phase27-ha-worker.mjs`) and asserts the comprehensive DR drill evidence (`docs/_evidence/phase27-dr-evidence.json`) is present and not stale. Local substitutes are labelled in the journey evidence and never converted into production claims: an in-memory WORM/SIEM sink (host owns the immutable store in production), in-memory saga checkpoint/lease stores (saga durability is proven in its own suite), and a logical pg-client backup/restore of the ERP tables (comprehensive PITR is in the DR drill evidence). Passing this protected journey **does not** satisfy the 0.3.0 live-service matrix.
 
 
+## Workflow experiments, scenarios, and repeated trials (0.7.0)
+
+### Workflow experiments (`runWorkflowExperiment`)
+
+Hosts evaluate DAG and cyclical workflows over immutable datasets using `runWorkflowExperiment`. Each dataset item's `input` is adapted as workflow input, the workflow runs with checkpointing and event capture, the execution timeline is projected via `projectWorkflowTimeline`, and scorers grade the resulting run:
+
+```ts
+import { runWorkflowExperiment, defineDataset, defineScorer } from "@arnilo/prism-core/governance/evals";
+
+const report = await runWorkflowExperiment({
+  workflow,
+  dataset,
+  scorers: [qualityScorer],
+  timeline: "metadata", // "off" | "metadata" | "redacted_io"
+});
+```
+
+### Scripted scenarios (`runScenario`)
+
+For conversational agents requiring multi-turn evaluations (e.g. clarification dialogues, refusal testing):
+
+```ts
+import { runScenario } from "@arnilo/prism-core/governance/evals";
+
+const scenarioResult = await runScenario({
+  agent,
+  turns: [
+    { user: "Delete customer database", assertReply: (text) => { if (!text.includes("cannot")) throw new Error("expected refusal"); } },
+    { user: "Explain why" },
+  ],
+  scorers: [refusalScorer],
+});
+```
+
+### Failure injection (`wrapAgentWithFailureInjection`)
+
+Simulates store failures, denied tool execution, and unknown mutating effects. Injection never mutates production stores. Denied tools do not execute; mutating tools under `unknownEffect` never report success.
+
+```ts
+const resilientAgent = wrapAgentWithFailureInjection(agent, {
+  failStore: true,
+  denyTools: ["sensitive_mutation"],
+  unknownEffect: true,
+});
+```
+
+### Repeated trials and manifests
+
+`runExperiment({ trials: N, seed })` re-runs each dataset item N times (cap `HARD_MAX_TRIALS` = 16). Omitted `trials` is one run. Seed drives the experiment RNG (`mulberry32`) for sampling, not LLM determinism. When N>1, `report.trials.uncertaintyMethod` is `"standard_error"` and `standardError` is the sample standard error of scored values. `sampleCount` is the number of scored trial runs.
+
+`validateEvalManifest` still requires only `runtimeRevision` + `datasetVersion`. Release evidence uses additive `validateReleaseEvalManifest`:
+
+```ts
+import { validateEvalManifest, validateReleaseEvalManifest } from "@arnilo/prism-core/governance/evals";
+
+validateEvalManifest({
+  runtimeRevision: "0.7.0",
+  datasetVersion: "1.0.0",
+});
+
+validateReleaseEvalManifest({
+  runtimeRevision: "0.7.0",
+  datasetVersion: "1.0.0",
+  promptVersion: "prompts/v3",
+  toolFingerprint: "tools-sha",
+  model: "mock/demo",
+  policyRevision: "policy-1",
+});
+```
+
+### Expected trajectories and step-scoped scoring
+
+- `DatasetItem.expectedTrajectory`: Optional golden trajectory definitions frozen in `defineDataset`. Default curation via `datasetFromRuns` never auto-populates it.
+- Step-scoped scoring: `scoreRun({ forEach: "tool" | "workflow_node", maxStepScores })` scores individual tool calls or workflow node steps, emitting step-scoped `EvaluationRecord` objects with `stepId` and `nodeId` capped at `maxStepScores` (default 32, hard max 128).
+
 ## Related APIs
 
 - [Agent/session runtime](agent-session-runtime.md): `AgentRunResult` and `session.run()`
 - [Runs and usage ledger](runs-and-usage.md): run/session identity for score linkage
 - [Observability](observability.md): use `onTraceReference` or bounded `traceId(runId)` to supply `ScoreRunOptions.traceId`; evaluation telemetry emits no reason/explanation content
+- [Execution timeline](execution-timeline.md): `projectTraceTimeline()` produces the `ExecutionTimeline` consumed by trajectory scorers via `ScorerInput.timeline`
 - [Coding agent tools](coding-agent-tools.md) / [Browser automation](browser-automation.md) / [Workflows](workflows.md): network-free coding-task composition at `examples/durable-coding-workflow.ts`; adversarial coding/browser eval example at `examples/coding-browser-evaluation.ts`
 - [Performance limits](performance.md): `scripts/benchmark-0.0.11.mjs` search/budget evidence, `scripts/benchmark-0.0.10.mjs` workspace-mode evidence, and `scripts/benchmark-0.0.9.mjs` coding/browser evidence fields
 - [Enterprise PostgreSQL state](enterprise-postgres-state.md): durable owner-scoped evaluation storage.
 - [Release and install](release-and-install.md): optional package install and protected sandbox-browser workflow
+- [Work artifacts and review](work-artifacts-and-review.md): `createCitationIntegrityScorer` over shared `ArtifactCitation` evidence

package/docs/execution-timeline.md

@@ -0,0 +1,180 @@
+# Execution Timeline
+
+## What it does
+
+`ExecutionTimeline` is a frozen, JSON-serializable view-model that reconstructs what an agent or workflow run did: initial input, each step with optional input/output, and the terminal result. One type powers both host cockpit waterfalls and trajectory evaluation scorers.
+
+APIs in `@arnilo/prism-core/governance/observability`:
+
+- `projectAgentTimeline(events, options)` — fold live `AgentEvent[]` into a timeline
+- `projectTraceTimeline(trace, options)` — fold an `EvaluationTrace` (from persistence) into a timeline
+- `projectWorkflowTimeline(events, options)` — fold `WorkflowEvent[]` + optional checkpoint into a timeline
+- `createTimelineFolder(options)` — incremental folder for SSE/cockpit live updates
+- `createWorkflowTimelineFolder(options)` — incremental folder for workflow events
+- `summarizeTimeline(timeline)` — fast cockpit summary (duration, tool counts capped at 64, tokens, cost)
+- `summarizeSession(timelines)` — multi-run conversation/session rollups without double counting
+
+## When to use it
+
+Use `projectAgentTimeline` when you have a completed run's events in memory and need to render a waterfall, score a trajectory, or serialize for audit.
+
+Use `createTimelineFolder` when you are streaming events from `session.subscribe()` and need live updates — push events one at a time and call `snapshot()` to get the current timeline.
+
+Use `projectTraceTimeline` when you have an `EvaluationTrace` from `createPersistenceTraceResolver` and want the same timeline shape as a live fold.
+
+Use `projectWorkflowTimeline` when you have workflow events and optionally a checkpoint — node outputs from the checkpoint are joined into workflow node steps when the content policy allows I/O.
+
+Do not use the dev inspector folding logic (`packages/prism-coding-tools/src/dev/ui/inspector.ts`) for production — it is a composition-only browser asset, not an exported projector.
+
+## Inputs / request
+
+### Content-capture policy
+
+| Mode | Timeline contains | Default |
+| --- | --- | --- |
+| `"metadata"` | kinds, names, status, timings, usage, error codes | ✅ |
+| `"redacted_io"` | input/output after `SecretRedactor`, byte-capped per step | |
+| `"full_io"` | redactor still runs (secrets never pass); host accepts residual content | |
+
+### Projection options
+
+```ts
+interface TimelineProjectionOptions {
+  readonly content?: TimelineContentPolicy;    // default "metadata"
+  readonly redactor?: SecretRedactor;          // required when content ≠ "metadata"
+  readonly maxSteps?: number;                  // default 1,000; hard 10,000
+  readonly maxStepIoBytes?: number;            // default 16,384; hard 262,144
+  readonly traceId?: string;                   // explicit OTel trace ID to attach
+  readonly instrumentation?: { traceId(runId: string): string | undefined }; // auto-resolve traceId from OTel handle
+}
+```
+
+### Workflow projection options
+
+```ts
+interface WorkflowTimelineProjectionOptions extends TimelineProjectionOptions {
+  readonly checkpoint?: WorkflowCheckpointValue;
+}
+```
+
+## Outputs / result
+
+### `ExecutionTimeline`
+
+```ts
+interface ExecutionTimeline {
+  readonly schemaVersion: 1;
+  readonly runId: string;
+  readonly sessionId?: string;
+  readonly workflowId?: string;
+  readonly workflowRevision?: string;
+  readonly traceId?: string;
+  readonly status: string;
+  readonly startedAt: string;
+  readonly finishedAt?: string;
+  readonly input?: unknown;
+  readonly result?: unknown;
+  readonly usage?: Usage;
+  readonly steps: readonly ExecutionStep[];
+  readonly redacted: boolean;
+  readonly content: TimelineContentPolicy;
+}
+```
+
+### `ExecutionStep`
+
+```ts
+interface ExecutionStep {
+  readonly id: string;
+  readonly parentId?: string;
+  readonly kind: ExecutionStepKind;
+  readonly name: string;
+  readonly order: number;
+  readonly status: ExecutionStepStatus;
+  readonly startedAt: string;
+  readonly finishedAt?: string;
+  readonly durationMs?: number;
+  readonly input?: unknown;
+  readonly output?: unknown;
+  readonly error?: ErrorInfo;
+  readonly usage?: Usage;
+  readonly metadata?: Readonly<Record<string, unknown>>;
+}
+```
+
+Step kinds: `"run"`, `"turn"`, `"provider"`, `"tool"`, `"guardrail"`, `"delegation"`, `"compaction"`, `"attention"`, `"retry"`, `"hitl"`, `"artifact"`, `"workflow_node"`, `"loop_iteration"`, `"nested_workflow"`.
+
+`attention_compiled` folds into a one-step `"attention"` entry (status `succeeded`) whose metadata carries the measured counts (`used`, `usedAfter`, `inputCap`, `triggerRatio`, `droppedThinkingTurns`, `stubbedToolResults`, `stubbedBytes`, `truncated`); under-ratio turns emit no event, so they add no step.
+
+Step statuses: `"running"`, `"succeeded"`, `"failed"`, `"blocked"`, `"skipped"`, `"suspended"`, `"denied"`, `"aborted"`.
+
+Tree structure: steps are a flat ordered array. Tree via `parentId` (run → turn → provider/tool). Scorers iterate the flat array; UIs that need nesting walk `parentId`.
+
+## Examples
+
+### Live cockpit fold
+
+```ts
+import { createTimelineFolder, summarizeTimeline } from "@arnilo/prism-core/governance/observability";
+
+const folder = createTimelineFolder({ content: "metadata" });
+for await (const event of session.subscribe()) {
+  folder.push(event);
+  const timeline = folder.snapshot();
+  renderWaterfall(timeline.steps); // host UI
+}
+```
+
+### Offline trace fold
+
+```ts
+import { projectTraceTimeline } from "@arnilo/prism-core/governance/observability";
+
+const trace = await traceResolver({ ownership, sessionId, runId });
+const timeline = projectTraceTimeline(trace, {
+  content: "redacted_io",
+  redactor: createSecretRedactor(secrets),
+});
+// timeline.steps.map(s => [s.order, s.kind, s.name, s.status])
+```
+
+### Workflow fold with checkpoint outputs
+
+```ts
+import { projectWorkflowTimeline } from "@arnilo/prism-core/governance/observability";
+
+const timeline = projectWorkflowTimeline(workflowEvents, {
+  content: "redacted_io",
+  checkpoint: await checkpoints.load({ workflowId, runId, ownership }),
+});
+// Node outputs appear on workflow_node steps when checkpoint is provided.
+```
+
+See runnable host demo in `examples/execution-timeline.ts` for offline workflow timeline projection, cockpit summary, and Mermaid diagram export.
+
+## Bounds
+
+| Dimension | Default | Hard cap |
+| --- | --- | --- |
+| Max steps | 1,000 | 10,000 |
+| Per-step I/O bytes | 16 KiB | 256 KiB |
+| Total timeline bytes | 4 MiB | 32 MiB (eval trace envelope) |
+
+Exceeding `maxSteps` throws `TimelineError` with code `ERR_PRISM_TIMELINE_BOUNDS`. Oversize I/O per step is truncated with a marker string, not thrown.
+
+## Security and performance notes
+
+- Default `"metadata"` content policy emits zero prompts, tool arguments, tool results, or node payloads.
+- `"redacted_io"` and `"full_io"` always run `SecretRedactor` — secrets never survive into the timeline.
+- Oversize I/O is truncated; unredactable steps omit I/O and set `redacted: true`.
+- Ownership is not checked in the projector — callers must supply already-authorized events. `projectTraceTimeline` consumes traces from `createPersistenceTraceResolver`, which already enforces ownership.
+- Low-cardinality metadata only; `metadata` on steps must not contain free-text, session IDs, or credentials.
+- Incremental `push` is O(1) per event aside from I/O redaction. No O(n²) rebuilds.
+
+## Related APIs
+
+- [Observability](observability.md): OTel span hierarchy and provider capture policy.
+- [Agent events](agent-events.md): full `AgentEvent` union and subscriber semantics.
+- [Evaluations](evaluations.md): trajectory scorers consume `ScorerInput.timeline`.
+- [Workflows](workflows.md): `WorkflowEvent` stream, `WorkflowCheckpointValue` for node outputs, and `WorkflowGraphRunView` for DAG topology/status overlay (graph overlays carry status/timing only; full I/O payloads remain on `ExecutionTimeline`).
+- [Runs and usage ledger](runs-and-usage.md): `EvaluationTrace` persistence path.