@warlock.js/ai 4.4.0 → 4.6.0

package/skills/durable-agent-runs/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: durable-agent-runs
+description: 'Mid-run crash-resume for agents AND planners — opt in with durable: { store, deleteOnComplete? } on the config, pass a stable runId to execute(), and call agent.resume(runId) / planner.resume(runId) after a crash to continue from the last settled trip / plan node. Reuses the ai.snapshot.{memory,pg,redis} stores; checkpoints per-trip (agent) / per-node (planner); completed trips + nodes never re-run their tools and usage is never double-counted; a drifted definition throws AgentDriftError / PlannerDriftError (bypass with { force: true }). Triggers: `durable`, `agent.resume`, `planner.resume`, `resume(runId)`, `runId`, `AgentSnapshot`, `PlannerSnapshot`, `AgentSnapshotStatus`, `PlannerSnapshotStatus`, `AgentDriftError`, `PlannerDriftError`, `computeAgentSignature`, `agent.signature`, `deleteOnComplete`, `defaultSnapshotStore`, `ai.snapshot.pg`, `ai.snapshot.memory`, `SnapshotStore`, `force: true`; ''resume an agent after a crash'', ''durable agent run'', ''continue a planner from where it crashed'', ''checkpoint agent state'', ''idempotent tool re-run on resume'', ''signature drift on resume''; typical import `import { ai } from "@warlock.js/ai"`. Skip: durable human-in-the-loop approval resume (ai.human.resume of a PendingInterrupt) — `@warlock.js/ai/durable-resume/SKILL.md`; supervisor/workflow iterate-mid-turn snapshot resume + the store contracts themselves — `@warlock.js/ai/manage-ai-stores/SKILL.md`; competing libs `temporal`, `inngest`, `restate`.'
+---
+
+# Durable agent + planner runs — resume from the last checkpoint
+
+Opt-in mid-run crash-resume for the two long-running primitives. Turn it on, give the run a stable `runId`, and after a process crash `resume(runId)` re-hydrates the persisted state and continues from where it stopped — never re-issuing a settled trip's model call or re-invoking a completed node's capability.
+
+> **Not the same as [[durable-resume]].** That skill is `ai.human.resume(interruptId, decision)` — resuming a **gated tool call** hours later after a human rules (a `PendingInterrupt` in an `InterruptStore`). *This* skill is **crash-resume of an in-flight run** (an `AgentSnapshot` / `PlannerSnapshot` in a `SnapshotStore`): the process died mid-run, you restart, and continue the same trip / plan. Different trigger (a crash, not a human), different store, different verb (`agent.resume` / `planner.resume`, not `ai.human.resume`).
+
+## Opt in — `durable` on the config
+
+```ts
+import { ai } from "@warlock.js/ai";
+
+const writer = ai.agent({
+  name: "writer",
+  model,
+  tools: [searchTool, draftTool],
+  durable: {
+    store: ai.snapshot.pg({ client: pgPool }), // reuses the ai.snapshot.* stores
+    deleteOnComplete: false,                    // default — keep for the completed-run short-circuit + audit
+  },
+});
+```
+
+`durable` shape (identical on the agent and planner config):
+
+- **`store?`** — a `SnapshotStore`. Falls back to `ai.config({ defaultSnapshotStore })`. When neither resolves, snapshot writes **silently skip** and `resume()` throws.
+- **`deleteOnComplete?`** — drop the snapshot once the run completes successfully. Default `false`.
+
+**Absent `durable` ⇒ zero behavior change** — the loop starts at trip 0 / the first node, never writes a snapshot, and runs byte-for-byte as before.
+
+## Run with a stable `runId`, then resume
+
+The `runId` is the store key. Pass a stable one to `execute()` (or read the generated one off `result.report.runId`) so a later `resume()` can find the snapshot:
+
+```ts
+const result = await writer.execute("research X", { runId: "run-42" });
+
+// ...process crashes mid-run, restarts...
+
+const recovered = await writer.resume("run-42");
+// continues from the next unsettled trip; `recovered.report.status === "completed"`
+```
+
+Planners are the mirror image — `durable` on the config, `runId` on `execute(goal)`, `planner.resume(runId)`:
+
+```ts
+const research = ai.planner({
+  name: "research-assistant",
+  model,
+  capabilities: [{ name: "search", executable: searchAgent }, { name: "write", executable: writerAgent }],
+  durable: { store: ai.snapshot.pg({ client: pgPool }) },
+});
+
+const first = await research.execute("compare A vs B", { runId: "plan-7" });
+// ...crash...
+const done = await research.resume("plan-7");
+```
+
+## Checkpoint granularity
+
+| Primitive | Written | Contains | Resume continues at |
+|---|---|---|---|
+| **agent** | after every settled **trip** (`runTrip` end) | `messages`, `trips`, `toolCalls`, `usage`, resolved `systemPrompt` / `responseSchema`, `signature`, `status` | `trips.length` (the next trip index) |
+| **planner** | after every settled **plan node** (`executeStep` end) | the frozen `plan`, `executedSteps` ledger, `usage`, child `children` reports, `replanCount`, `signature`, `status` | the unfinished frontier (from `executedSteps`) |
+
+The write happens only where the persisted arrays are mutually consistent — for the agent, after every tool a trip requested has been dispatched and its result appended. A crash **mid-trip** loses only that in-flight trip (never checkpointed), which the resume re-issues cleanly. The planner **never re-calls the planning LLM** on resume — the plan is frozen on the first run; re-asking would burn tokens and risk a plan that no longer matches the ledger. Every field on both snapshots is JSON-serializable, so they round-trip through any `ai.snapshot.{memory,pg,redis}` backend verbatim.
+
+## Idempotency — what does and doesn't re-run
+
+```ts
+// Completed run: resume is a no-op that re-returns the stored result.
+const again = await writer.resume("run-42"); // runs nothing when status === "completed"
+```
+
+- **Completed trips / nodes never re-run their tools.** On agent resume, `trips.length` is the starting trip index — earlier trips' model calls are not replayed and their tool dispatches are not re-invoked. On planner resume, a completed node's capability dispatch is skipped (the sequential skip-guard / DAG re-seed derive the completed set from `executedSteps`).
+- **Usage is never double-counted.** The running `usage` total is restored from the snapshot; only the newly-executed trips / nodes add to it.
+- **Caveat — a crash MID-trip re-runs that trip's tools.** The in-flight trip was never checkpointed, so on resume its tools fire again. **Side-effectful tools (charging a card, sending an email) must be idempotent** — the same caller-responsibility boundary the supervisor and workflow primitives document. Guard them with your own dedupe key (e.g. `${runId}:${toolCallId}`).
+
+## Drift — definition changed since the snapshot
+
+Every agent / planner carries a structural `signature` (`agent.signature` — computed at factory time by `computeAgentSignature`), stamped on each snapshot. `resume()` compares the stored signature against the current definition; a mismatch throws before executing anything:
+
+- **agent** covers: model name + provider, sorted tool names, `maxTrips`, whether a default `output` schema is set, `version`. It does **not** cover system-prompt text, middleware, per-event handlers, placeholders, or `modelOptions` — runtime knobs that don't change a resumable run's shape.
+- **planner** covers: name + ordered capability names. A mid-run **re-plan is NOT drift** (the plan changed, not the definition); `replanCount` is persisted so the replan budget survives a resume.
+
+```ts
+import { AgentDriftError } from "@warlock.js/ai";
+
+try {
+  await writer.resume("run-42");
+} catch (error) {
+  if (error instanceof AgentDriftError) {
+    // The definition changed (a tool was added, the model swapped). Either roll the
+    // definition back, or — only when you've verified the change is snapshot-safe:
+    await writer.resume("run-42", { force: true }); // bypasses the drift check
+  }
+}
+```
+
+`{ force: true }` is the escape hatch (mirror `PlannerDriftError` for planners). `resume()` also throws `AgentExecutionError` / `PlannerFailedError` when no store is configured or no snapshot exists for the `runId`.
+
+## Pattern — a boot-drain resume loop
+
+On restart, resume every run the store still has in flight. Snapshots carry a `status` (`"running" | "completed" | "cancelled" | "failed"`), so you only resume the live ones:
+
+```ts
+const store = ai.snapshot.pg({ client: pgPool });
+const runIds = (await store.list?.()) ?? [];
+
+for (const runId of runIds) {
+  const snapshot = await store.load(runId);
+  if (snapshot?.status === "running") {
+    await writer.resume(runId); // completed/failed snapshots short-circuit or re-throw — skip them
+  }
+}
+```
+
+Pair `deleteOnComplete: true` with this loop when you don't need the completed-run audit trail — the store then holds only genuinely-unfinished runs, so the drain never touches settled ones.
+
+## Cost + testing
+
+- **Checkpointing cost is one store write per settled trip / node** — a `JSONB` upsert on `pg`, an in-process `Map` set on `memory`. A failed checkpoint is surfaced via logs, not thrown: it loses resume-ability from that point but never breaks an otherwise-healthy run.
+- **Resume saves the tokens of every settled trip / node** — their model calls are not replayed. A completed-run resume spends nothing (it rebuilds the result from the snapshot). The planning LLM is never re-called on planner resume.
+- **Test with `ai.snapshot.memory()`.** Drive `execute(input, { runId })` against a flaky model that throws once, assert the tool spy was called once, flip the failure off, `resume(runId)`, and assert (a) `status === "completed"`, (b) the tool spy count is unchanged (no re-invoke), and (c) `usage.total` counts each trip's tokens exactly once. Drift is testable by mutating the definition (add a tool) between `execute` and `resume` and asserting `AgentDriftError` — then `{ force: true }` proceeds.
+
+## See also
+
+- [[handle-ai-errors]] — the typed `AgentDriftError` / `PlannerDriftError` / `AgentExecutionError` / `PlannerFailedError` and how `result.error` surfaces a failed run.
+- [[manage-ai-stores]] — the `ai.snapshot.{memory,pg,redis}()` factories, the `SnapshotStore` contract, dev-owned `pg` / `redis` clients, and never-auto-migrated `schema()`.
+- [[persist-ai-data]] — supervisor / workflow snapshot resume (the sibling `iterate`-style durability) and the SnapshotStore migration notes.
+- [[durable-resume]] — the OTHER resume: `ai.human.resume` of a gated tool call (human-in-the-loop), not a crash.

package/skills/durable-resume/SKILL.md

@@ -0,0 +1,128 @@
+---
+name: durable-resume
+description: 'Persist a gated tool call and resume it from another process hours later — ships in @warlock.js/ai core: `ai.human.resume(interruptId, decision, options)`, the `InterruptStore` (`ai.human.interrupt.{memory,pg,redis}()`), `PendingInterrupt`, and the `InterruptSuspendedError` suspend sentinel. Triggers: `ai.human.resume`, `resume(interruptId, decision)`, `InterruptStore`, `ai.human.interrupt.memory`, `ai.human.interrupt.pg`, `ai.human.interrupt.redis`, `interruptMemory`, `interruptPg`, `interruptRedis`, `PendingInterrupt`, `InterruptSuspendedError`, `ResumeOptions`, `ResumeResult`, `PgClientLike`, `RedisClientLike`; ''approve hours later from a webhook'', ''persist the approval request and resume in another process'', ''durable human-in-the-loop'', ''store the interrupt in Postgres/Redis'', ''re-run the agent turn once the human approves''. Typical import `import { ai, InterruptSuspendedError } from "@warlock.js/ai"`. Skip: the in-process await gate and the policy/decision shapes — `@warlock.js/ai/approve-tool-calls/SKILL.md`.'
+---
+
+# Durable resume — persist the interrupt, approve from another process
+
+Interactive approval `await`s the operator in-process. **Durable** approval is for when the reviewer rules out-of-band — a Slack button, a webhook, hours later, in a different process. The flow: the handler **persists** the request to an `InterruptStore` and **throws** `InterruptSuspendedError` to suspend the run; the caller surfaces the `interruptId`; later, `ai.human.resume(interruptId, decision, { store })` applies the ruling.
+
+> **v1 durable resume re-runs the turn** with the decision pre-seeded — it does **not** rehydrate an in-flight supervisor mid-call (that's the deferred v2 lift). Re-running is idempotent because the prompt and the seeded decision fully determine the gated call's outcome.
+
+## Process A — suspend and surface the id
+
+```ts
+import { ai, InterruptSuspendedError } from "@warlock.js/ai";
+
+const store = ai.human.interrupt.memory(); // swap for pg / redis in production
+
+const agent = ai.agent({
+  model,
+  tools: [deleteAccount],
+  middleware: [
+    ai.human.approval({
+      policy: { type: "predicate", requiresApproval: (c) => c.toolName === "deleteAccount" },
+      store,
+      handler: async (req) => {
+        // 1. persist the pending interrupt
+        await store.save({
+          interruptId: req.interruptId,
+          request: req,
+          status: "pending",
+          savedAt: new Date().toISOString(),
+        });
+        // 2. notify the reviewer out-of-band
+        await slack.postApproval(req);
+        // 3. suspend the run — the middleware recognizes its OWN sentinel
+        throw new InterruptSuspendedError("Awaiting human approval", {
+          interruptId: req.interruptId,
+        });
+      },
+    }),
+  ],
+});
+
+const result = await agent.execute("Delete account #88");
+
+// execute() never throws — the suspend rides result.error:
+if (result.error instanceof InterruptSuspendedError) {
+  return { status: "awaiting-approval", interruptId: result.error.interruptId };
+}
+```
+
+The middleware catches the **sentinel** (`instanceof InterruptSuspendedError`) and short-circuits a failed `ToolInvokeResult` carrying it, so `error.interruptId` is on `result.error`. Hand that id to the reviewer.
+
+## Process B — resume hours later
+
+```ts
+import { ai } from "@warlock.js/ai";
+
+// Re-run the turn with the decision pre-seeded:
+const outcome = await ai.human.resume(
+  interruptId,
+  { type: "edit", args: { confirm: true } },
+  { store, agent },
+);
+
+if (outcome.type === "applied" && outcome.result) {
+  console.log(outcome.result.text); // the re-run completed with the ruling applied
+}
+```
+
+`ai.human.resume(interruptId, decision, options)` loads the `PendingInterrupt`, validates the decision shape, deletes the record, and — when an `agent` is supplied — re-executes the original prompt with the decision **pre-seeded** so the gated tool call resolves to the ruling instead of pausing again. The prompt comes from `request.context.originalInput`; pass `options.input` to override (e.g. to append the reviewer's note), and `options.executeOptions` to forward history / output schema / signal to the re-run.
+
+### Two resume shapes
+
+| Shape | Pass | Behavior |
+|---|---|---|
+| **re-run** | `{ store, agent }` | Loads, deletes, re-executes the turn; `ResultResult.result` carries the `AgentResult`. |
+| **apply-only** | `{ store }` (no `agent`) | Loads, validates, deletes; returns `{ type: "applied", decision }` for a caller-owned re-drive (custom transport). No turn re-run. |
+
+### Idempotent by construction
+
+```ts
+type ResumeResult =
+  | { type: "applied"; interruptId: string; decision: ApprovalDecision; result?: AgentResult }
+  | { type: "already-resolved"; interruptId: string };
+```
+
+A second resume of an already-resolved (deleted) or never-raised interrupt returns `{ type: "already-resolved" }` — it never double-applies the decision or re-runs the turn. The record is deleted **before** the re-run, so even a re-run that itself raises a fresh interrupt can't collide with the one being resolved. A malformed decision (`reject` with no `reason`, `edit` with no `args`, an unknown `type`) throws a `TypeError` loudly rather than silently mis-driving the re-run.
+
+## The `InterruptStore`
+
+`ai.human.interrupt.{memory,pg,redis}()` build the store. The contract mirrors `@warlock.js/ai`'s `CheckpointStore` / `SnapshotStore` — `save` / `load` / `delete` / optional `list(prefix?)` / `schema()` — so a consumer already running an orchestrator can reuse the **same** pool for the interrupt table.
+
+| Factory | Backing | Deps |
+|---|---|---|
+| `ai.human.interrupt.memory()` | process-local `Map` | none — zero runtime deps |
+| `ai.human.interrupt.pg(options)` | one Postgres row per interrupt, keyed by `interrupt_id` | lazily imports the optional `pg` peer |
+| `ai.human.interrupt.redis(options)` | one namespaced JSON value + a self-maintained id index | lazily imports the optional `redis` peer |
+
+```ts
+// Memory — dev / tests / single-process:
+const store = ai.human.interrupt.memory();
+
+// Postgres — pass a live pool (core never imports pg in that case):
+import { Pool } from "pg";
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+const store = ai.human.interrupt.pg({ client: pool });
+// Once, via your migration tool — the framework never auto-migrates:
+// await pool.query(store.schema());
+
+// …or let the store build its own pool (lazily import("pg")):
+const store = ai.human.interrupt.pg({ connectionString: process.env.DATABASE_URL });
+
+// Redis — pass a connected client, or a url:
+const store = ai.human.interrupt.redis({ url: process.env.REDIS_URL });
+```
+
+### Optional peers are lazy
+
+`pg` and `redis` are **optional** peer dependencies — neither is a hard dependency. The driver is imported only inside the store that needs it, and only when you pass a `connectionString` / `url` (passing a live `client` imports nothing). If the driver is absent, a **curated install string** surfaces on first use, never a raw module-resolution stack trace at import — so a memory-only consumer always loads cleanly. `PgClientLike` / `RedisClientLike` are structural interfaces, so any compatible pool/client satisfies them.
+
+`schema()` returns the reference DDL for the Postgres store (run it through your migration tool once) and an empty string for memory / redis, so callers treat `schema()` uniformly across drivers.
+
+## See also
+
+- [`@warlock.js/ai/approve-tool-calls/SKILL.md`](@warlock.js/ai/approve-tool-calls/SKILL.md) — the gate itself: the interrupt policy, the approve / reject / edit decision union, and the interactive (in-process await) handler.
+- `@warlock.js/ai` — the `CheckpointStore` / `SnapshotStore` the `InterruptStore` mirrors, and the `ai.agent(...)` re-run target.

package/skills/escalate-block-to-human/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: escalate-block-to-human
+description: 'Route a hard guardrail block to a human-review surface with @warlock.js/ai-guard — the `escalation.onBlock` seam and an `escalate: true` verdict. Triggers: `escalation`, `onBlock`, `GuardrailEscalation`, `GuardrailBlockEvent`, `escalate: true`, `{ type: "block", escalate: true }`, ''escalate a block to a human'', ''human review queue for guardrail'', ''page an operator on a guardrail block'', ''human-in-the-loop guardrail'', ''compose a block with a review surface'', ''custom detector that escalates''; typical import `import "@warlock.js/ai-guard"` then `ai.guardrail({ escalation: { onBlock } })`. Skip: composing the guard / phases / verdict model — `@warlock.js/ai-guard/guard-input-output/SKILL.md`; PII/moderation detectors — `@warlock.js/ai-guard/detect-and-redact-pii/SKILL.md`; durable suspend/resume human-step machinery (deferred) — not in this package.'
+---
+
+# Escalate a block to a human
+
+A `block` verdict can carry `escalate: true`. When it does, the guard `await`s your `escalation.onBlock(...)` handler **before** throwing the `GuardrailViolationError` — the seam to a human-review queue, an operator page, or any out-of-band approval surface.
+
+```ts
+import { ai } from "@warlock.js/ai";
+import "@warlock.js/ai-guard";
+
+const policy = ai.guardrail({
+  output: [ai.guardrail.moderation({ blockOn: ["self-harm"] })],
+  escalation: {
+    async onBlock(event) {
+      await reviewQueue.enqueue({
+        phase: event.phase,   // "input" | "output" | "tool"
+        reason: event.reason, // the detector's human-readable reason
+      });
+    },
+  },
+});
+
+const agent = ai.agent({ model, middleware: [policy] });
+```
+
+## When `onBlock` fires
+
+`onBlock` fires **only** for a verdict of `{ type: "block", escalate: true }` — not for an ordinary `block`, and never for `allow` / `redact` / `flag`. It is **awaited before** the `GuardrailViolationError` is thrown, so your handler runs to completion (enqueue succeeds, the page is sent) before the error surfaces on `result.error`. The run still aborts: escalation is a *signal*, not a recovery — `execute()` returns with `result.error` populated as usual.
+
+## The `GuardrailBlockEvent` payload
+
+`onBlock(event)` receives:
+
+| Field | Type | Meaning |
+|---|---|---|
+| `phase` | `"input" \| "output" \| "tool"` | where the block fired |
+| `reason` | `string` | the detector's human-readable reason |
+| `matches` | `readonly GuardrailMatch[] \| undefined` | what tripped the rule (rule id, span, label), when reported |
+| `ctx` | `MiddlewareTripContext` | the live trip context — `state`, `messages`, `agent`, `model`, `signal` |
+
+`ctx` lets the handler enrich the review item with run context (session id from `ctx.state`, the offending messages, etc.).
+
+## Producing an escalating verdict
+
+The built-in detectors return ordinary `block` verdicts (no `escalate`). To escalate, author a tiny custom `GuardrailDetector` that sets `escalate: true` on its `block`:
+
+```ts
+import type { GuardrailDetector } from "@warlock.js/ai-guard";
+
+const wirePolicy: GuardrailDetector = {
+  name: "wire-transfer",
+  check(text) {
+    if (/wire \$?\d{5,}/i.test(text)) {
+      return {
+        type: "block",
+        reason: "large wire transfer requires human approval",
+        escalate: true, // <- routes through escalation.onBlock
+        matches: [{ rule: "wire-transfer.large", label: "wire" }],
+      };
+    }
+
+    return { type: "allow" };
+  },
+};
+
+const policy = ai.guardrail({
+  tool: [wirePolicy],
+  toolNames: ["initiate_transfer"],
+  escalation: { async onBlock(e) { await approvals.request(e); } },
+});
+```
+
+A `check()` may be sync or async (async = call an external service); the guard awaits either.
+
+## A plain callback by design
+
+`escalation.onBlock` is a **plain callback** — `ai-guard` takes **no** dependency on the deferred durable human-step machinery (suspend/resume). The callback is the decoupling seam: inside it you wire your own review queue, and (where your stack supports it) a `workflow.resume(...)` loop. This package only emits the *signal*; it does not own durable suspension. When the typed human-step handoff ships, `onBlock` upgrades to it without a breaking change here.
+
+## See also
+
+- [`@warlock.js/ai-guard/guard-input-output/SKILL.md`](@warlock.js/ai-guard/guard-input-output/SKILL.md) — composing the guard, the phases, the verdict model, and how a `block` surfaces on `result.error`.
+- [`@warlock.js/ai-guard/detect-and-redact-pii/SKILL.md`](@warlock.js/ai-guard/detect-and-redact-pii/SKILL.md) — the `pii` detector and the optional `moderation` peer that commonly drives an escalation.

package/skills/eval-datasets-and-ci/SKILL.md

@@ -0,0 +1,117 @@
+---
+name: eval-datasets-and-ci
+description: 'Datasets + regression-gated eval CI with ai.dataset({...}) feeding agent.eval({cases,baseline,tolerance}). Covers the immutable filterable/shardable dataset (cases / fromFile JSONL), DatasetEntry tags, EvalReport.regression (regressed/added/removed/passed) against a baseline, and the ai.eval reporters toJUnit / toJSON / fromJSON for CI artifacts + committed baselines. Triggers: `ai.dataset`, `DatasetContract`, `DatasetEntry`, `DatasetOptions`, `dataset.filter`, `dataset.shard`, `fromFile`, `agent.eval`, `EvalOptions`, `EvalReport`, `EvalCaseResult`, `EvalRegression`, `baseline`, `tolerance`, `ai.eval.toJUnit`, `ai.eval.toJSON`, `ai.eval.fromJSON`, `diff`, JSONL; ''eval dataset from a JSONL file'', ''shard an eval suite across CI jobs'', ''fail CI on an eval regression'', ''emit a JUnit report'', ''snapshot an eval baseline''; typical import `import { ai } from "@warlock.js/ai"`. Skip: the scorers + LLM-as-judge + Vitest matchers themselves — `@warlock.js/ai/ai-dx-helpers/SKILL.md` (registerAiMatchers / ai.eval.exact|contains|predicate|judge); record/replay of model calls for deterministic tests — `@warlock.js/ai/record-replay-llm/SKILL.md`; competing libs `promptfoo`, `braintrust`.'
+---
+
+# `ai.dataset()` + `agent.eval()` regression CI
+
+Turn a corpus of cases into a regression-gated CI signal. `ai.dataset(...)` wraps cases into an immutable, filterable, shardable collection; `agent.eval({ cases, baseline, tolerance })` runs them, scores them, and diffs against a prior report; the `ai.eval.*` reporters serialize the result for CI ingestion and tomorrow's baseline.
+
+> This skill is the **dataset + CI** layer. The scorers, LLM-as-judge config, and Vitest matchers live in [`@warlock.js/ai/ai-dx-helpers/SKILL.md`](@warlock.js/ai/ai-dx-helpers/SKILL.md); `agent.eval`'s core scoring loop is in [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md).
+
+## `ai.dataset()` — immutable, filterable, shardable
+
+```ts
+import { ai } from "@warlock.js/ai";
+
+const ds = ai.dataset({
+  name: "support",
+  cases: [{ name: "greeting", input: "hi", expected: "Hello" }], // inline entries
+  fromFile: "./eval/support.jsonl",  // JSONL read ONCE, synchronously, at construction
+});
+
+ds.name;   // "support"
+ds.cases;  // DatasetEntry[] (inline first, then file entries appended)
+```
+
+- A `DatasetEntry` is an `EvalCase` plus optional `tags?: string[]` for filtering / sharding.
+- **`fromFile`** reads a JSONL file (one JSON object per line; blank lines skipped) synchronously at construction — mirroring `SystemPrompt.fromFile`. A malformed line throws an **`InvalidRequestError` naming the 1-based line number**; a missing/unreadable path throws too. `cases` and `fromFile` combine (file entries append after inline).
+
+### `filter` / `shard` — derive new datasets
+
+```ts
+const smoke = ds.filter((entry) => entry.tags?.includes("smoke"));
+const shard = ds.shard(0, 4); // first of four parallel CI shards
+```
+
+Both return a **fresh dataset sharing nothing mutable**. `shard(index, total)` is deterministic round-robin by position: every entry lands in exactly one shard, so the union of all `total` shards reproduces the full list with no gaps or overlaps. `shard` validates its args (positive integer `total`, `index` in `[0, total)`) and throws `InvalidRequestError` otherwise.
+
+## `agent.eval({ cases })` — accepts a dataset directly
+
+```ts
+const report = await myAgent.eval({
+  cases: ds,                       // a DatasetContract OR a raw EvalCase[]
+  scorers: [ai.eval.contains()],
+});
+
+expect(report.passed).toBe(true); // true only when EVERY case passed
+```
+
+The runner reads `.cases` off a dataset. Full `EvalReport`: `{ agentName, total, passedCount, failedCount, passRate, meanScore, passed, cases, duration, regression? }`. Each `EvalCaseResult` carries the case, the full `AgentResult`, every scorer's `scores`, the mean `score`, `passed`, and `duration`.
+
+## Regression gating — `baseline` + `tolerance`
+
+```ts
+import { readFile, writeFile } from "node:fs/promises";
+
+const baseline = ai.eval.fromJSON(await readFile("./eval/baseline.json", "utf8"));
+
+const report = await myAgent.eval({
+  cases: ds,
+  scorers: [ai.eval.exact()],
+  baseline,         // a prior EvalReport to diff against
+  tolerance: 0.05,  // max allowed per-case score DROP before it regresses. default 0 (any drop)
+});
+
+if (report.regression && !report.regression.passed) {
+  console.error("Regressed:", report.regression.regressed); // [{ name, before, after }]
+  process.exit(1);
+}
+```
+
+When `baseline` is set the report carries a `regression` block (`EvalRegression`), joining cases by `name`:
+
+- **`regressed`** — `[{ name, before, after }]` for cases whose new score fell more than `tolerance` below baseline.
+- **`added`** / **`removed`** — case names present in only one report. Adding or dropping a case **never fails the gate by itself**.
+- **`passed`** — `true` when `regressed` is empty.
+
+The pure `diff(report, baseline, tolerance)` function (exported as `diff`) is the same logic, decoupled from the runner — depends only on the two reports and the tolerance, mutates neither.
+
+## CI reporters — `ai.eval.toJUnit` / `toJSON` / `fromJSON`
+
+Pure functions over a finished `EvalReport`:
+
+```ts
+// JUnit-XML artifact for CI ingestion — one <testsuite> (the agent), one <testcase> per case,
+// a <failure> on each non-passing case (joined scorer reasons), times in SECONDS.
+await writeFile("./report.junit.xml", ai.eval.toJUnit(report));
+
+// Round-trippable snapshot — today's report becomes tomorrow's baseline.
+await writeFile("./eval/baseline.json", ai.eval.toJSON(report));
+const restored = ai.eval.fromJSON(await readFile("./eval/baseline.json", "utf8"));
+```
+
+`toJSON`/`fromJSON` preserve `result` payloads, per-case `scores`, timings, and any attached `regression` block, so a parsed report drives regression diffing exactly as the in-memory one. `toJUnit` hand-emits XML (no `xml` dependency) and entity-escapes every dynamic value.
+
+## Typical CI shard job
+
+```ts
+const shard = ai.dataset({ name: "support", fromFile: "./eval/support.jsonl" })
+  .shard(Number(process.env.SHARD_INDEX), Number(process.env.SHARD_TOTAL));
+
+const report = await agent.eval({
+  cases: shard,
+  scorers: [ai.eval.contains()],
+  baseline: ai.eval.fromJSON(await readFile("./eval/baseline.json", "utf8")),
+  tolerance: 0.05,
+});
+
+await writeFile(`./out/report-${process.env.SHARD_INDEX}.junit.xml`, ai.eval.toJUnit(report));
+if (report.regression && !report.regression.passed) process.exit(1);
+```
+
+## See also
+
+- [`@warlock.js/ai/ai-dx-helpers/SKILL.md`](@warlock.js/ai/ai-dx-helpers/SKILL.md) — `ai.eval.{exact,contains,predicate,judge}` scorers + Vitest matchers
+- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — `agent.eval` scoring loop, `EvalCase` / `EvalJudge`
+- [`@warlock.js/ai/record-replay-llm/SKILL.md`](@warlock.js/ai/record-replay-llm/SKILL.md) — `ai.vcr` for deterministic, offline eval runs

package/skills/generate-images/SKILL.md

@@ -0,0 +1,138 @@
+---
+name: generate-images
+description: 'Text-to-image via ai.image({ model: sdk.image({ name }), prompt }) — the image-OUTPUT verb (Theme I), returning the uniform never-throws { data, error, usage, report } envelope with cost-truth + panoptic observation. Models come from an adapter''s image() factory: OpenAI gpt-image-* (token-metered) / dall-e-* (per-image), Google imagen-* (per-image). Result images are a discriminated GeneratedImage = { type: "base64" } | { type: "url" }. Triggers: `ai.image`, `sdk.image`, `openai.image`, `google.image`, `ImageModelContract`, `GeneratedImage`, `ImageModelPricing`; ''generate an image'', ''text to image'', ''gpt-image'', ''dall-e'', ''imagen'', ''product thumbnail'', ''image output''; typical import `import { ai } from "@warlock.js/ai"` + `import { OpenAISDK } from "@warlock.js/ai-openai"`. Skip: image INPUT / vision attachments to a chat agent — `@warlock.js/ai/run-ai-agent/SKILL.md`; embeddings — `@warlock.js/ai/embed-text/SKILL.md`; competing libs raw `openai.images.generate`, `langchain` image tools.'
+---
+
+# Generate images — the image-output verb (`ai.image`)
+
+`ai.image()` is the output counterpart to `ai.agent` for the image modality (the first verb of the output-modality track, Theme I). Prompt-in / images-out, wrapped in the same uniform result contract every executable returns — so it slots into cost dashboards and panoptic traces exactly like an agent run.
+
+This is image **output** (generation). For image/PDF/audio **input** to a chat agent (vision), see [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md).
+
+## Shape
+
+```ts
+// 1. Build an image model from an adapter's image() factory.
+const model = openai.image({ name: "gpt-image-1" });   // ImageModelContract
+
+// 2. Run the verb — never throws; failures land on result.error.
+const { data, error, usage, report } = await ai.image({ model, prompt: "a red bicycle" });
+
+if (error) {
+  console.warn(error.code);          // typed AIError (auth / rate-limit / content-filter / …)
+} else {
+  for (const img of data.images) {   // GeneratedImage[]
+    if (img.type === "base64") save(Buffer.from(img.base64, "base64"), img.mediaType);
+    else download(img.url);
+  }
+}
+```
+
+`ImageModelContract` mirrors `EmbedderContract` — a peer primitive on the SDK adapter, produced by the optional `image?()` factory. An adapter without an image API simply doesn't define `image()`, so `ai.anthropic.image(...)` is a **compile-time** error, not a silent runtime failure.
+
+## The result envelope
+
+```ts
+type ImageResult = {
+  type: "image";
+  data?: { images: GeneratedImage[] };  // undefined on failure
+  error?: AIError;                       // undefined on success — NEVER thrown
+  usage: Usage;                          // tokens (gpt-image) + cost when priced
+  report: ImageReport;                   // type:"image", model, imageCount, lineage
+};
+
+type GeneratedImage =
+  | { type: "base64"; base64: string; mediaType: string; revisedPrompt?: string }
+  | { type: "url"; url: string; mediaType?: string; revisedPrompt?: string };
+```
+
+## Generation options (provider-neutral)
+
+```ts
+await ai.image({
+  model,
+  prompt: "an isometric office desk, soft studio lighting",
+  count: 2,            // n images
+  size: "1024x1024",   // OpenAI WxH (also resolves perImageBySize pricing)
+  quality: "high",     // OpenAI quality tier
+  aspectRatio: "16:9", // Imagen ratio
+  negativePrompt: "blurry, watermark", // Imagen
+  format: "png",       // output container hint
+  signal,              // AbortSignal
+  observe: collector,  // route the report to an Observer (panoptic), like agents
+  sessionId: "checkout-123",
+  options: { background: "transparent" }, // provider-specific passthrough
+});
+```
+
+Each adapter maps the options its API supports and ignores the rest. `options` is the escape hatch for provider-specific knobs (OpenAI `background`, DALL·E `responseFormat: "url"`, Imagen `imageSize` / `personGeneration`).
+
+## OpenAI — gpt-image (token-metered) + DALL·E (per-image)
+
+```ts
+import { OpenAISDK } from "@warlock.js/ai-openai";
+
+const openai = new OpenAISDK({ apiKey: process.env.OPENAI_API_KEY! });
+
+// gpt-image-1 always returns base64 bytes; priced per TOKEN.
+const gpt = openai.image({ name: "gpt-image-1", pricing: { input: 5, output: 40 } });
+
+// DALL·E 3 — per-image pricing; defaults to base64 (opt into url with options).
+const dalle = openai.image({ name: "dall-e-3", pricing: { perImage: 0.04 } });
+```
+
+A non-image model id (`openai.image({ name: "gpt-4o" })`) throws `InvalidRequestError` **at construction** — fail fast, like the embedder/vision guards.
+
+## Google — Imagen (per-image)
+
+```ts
+import { GoogleSDK } from "@warlock.js/ai-google";
+
+const google = new GoogleSDK({ apiKey: process.env.GEMINI_API_KEY! });
+const imagen = google.image({ name: "imagen-4.0-generate-001", pricing: { perImage: 0.04 } });
+
+const { data } = await ai.image({ model: imagen, prompt: "a watercolor lighthouse at dawn", aspectRatio: "3:4" });
+```
+
+Imagen returns base64 bytes (no hosted URL). When every candidate is safety-filtered, `ai.image` surfaces a typed `ContentFilterError` on `result.error`.
+
+## Cost-truth — one rollup, two metering models
+
+`ai.image` fills `usage.cost` (a `ModelPricing`-shaped USD breakdown) so image spend folds into the **same** `Usage.cost` rollup as text — no second accounting path:
+
+- **Token-metered** (gpt-image-1): `{ input, output }` USD-per-1M-tokens → standard `computeCost` against the returned token usage.
+- **Per-image** (DALL·E, Imagen): `{ perImage }` (or `perImageBySize["1792x1024"]`) × image count → `cost.output`.
+
+Unpriced model → `usage.cost` stays `undefined` (honest "cost unknown", never a false zero). A pre-priced adapter response is honored, not overwritten.
+
+## Pattern — catalog thumbnail in a workflow `run` step
+
+```ts
+ai.step({
+  name: "thumbnail",
+  run: async (ctx) => {
+    const { data, error } = await ai.image({
+      model: openai.image({ name: "gpt-image-1" }),
+      prompt: `product photo, white background: ${ctx.steps.extract.output.title}`,
+      size: "1024x1024",
+    });
+    if (error) throw error; // step retry/backoff handles transient provider faults
+    ctx.state.thumb = data.images[0];
+  },
+});
+```
+
+## Observability
+
+The completed `ImageReport` routes to any registered `Observer` (panoptic, OTel, …) through the shared `observe` seam — pass `observe: true` (global), an `Observer` object (flow-local), or rely on observe-all. Cost + latency attribute to `report.model` for free. See [`@warlock.js/ai/observe-ai-flows/SKILL.md`](@warlock.js/ai/observe-ai-flows/SKILL.md).
+
+## Testing
+
+`MockSDK({ imageResponses, imagePricing }).image({ name })` returns a deterministic `MockImageModel` — no HTTP. Script images/usage/errors and inspect `model.calls`.
+
+```ts
+import { MockSDK } from "@warlock.js/ai";
+
+const mock = MockSDK({ imageResponses: [{}], imagePricing: { perImage: 0.04 } });
+const { data, usage } = await ai.image({ model: mock.image({ name: "mock-image" }), prompt: "x" });
+```