@directive-run/claude-plugin 1.16.0

package/skills/testing-ai-systems/ai-testing-evals.md

@@ -0,0 +1,485 @@
+# AI testing and evaluations
+
+> Covers `@directive-run/ai/testing` and `@directive-run/ai/evals` — `createMockAgentRunner`, `createTestOrchestrator`, simulators, `createEvalSuite` + `eval*` criteria.
+
+Mock runners, test orchestrators, snapshot helpers, simulators, and a dataset-driven evaluation framework. Import all testing utilities from `@directive-run/ai/testing`; eval criteria factories come from `@directive-run/ai/evals`.
+
+## Decision tree
+
+```
+What are you testing?
+├── Single-agent runs (unit/integration) → createTestOrchestrator
+├── Multi-agent patterns                  → createTestMultiAgentOrchestrator
+├── Just the runner contract              → createMockAgentRunner
+├── Approval workflow                     → createApprovalSimulator (built into createTestOrchestrator)
+├── Breakpoint pauses                     → createBreakpointSimulator
+├── DAG topology                          → createTestDag + assertDagExecution
+├── Failure modes (timeouts, errors)      → createFailingRunner
+├── Snapshot constraint behavior          → createConstraintRecorder
+└── Time-dependent logic                  → createTimeController
+
+What are you evaluating?
+├── Output quality                       → createEvalSuite + eval* criteria
+├── Token / latency budgets              → evalCost + evalLatency
+├── Output shape / size                  → evalOutputLength + evalStructure
+├── Safety / PII                         → evalSafety
+├── Reference match                      → evalMatch
+├── Faithfulness / relevance / coherence → evalFaithfulness / evalRelevance / evalCoherence
+├── LLM-as-judge                          → evalJudge
+└── Custom assertion                     → evalAssert
+```
+
+## Mock runner
+
+`createMockAgentRunner` returns `{ run, getCalls, getCallsFor, clearCalls, setResponse, setDefaultResponse }`. The factory takes a SINGLE options object — responses are keyed by **agent name**, NOT input pattern. (For input-pattern matching, use `generate` inside a response.) **The function is named `createMockAgentRunner`, not `createMockRunner`**.
+
+```typescript
+import { createMockAgentRunner } from "@directive-run/ai/testing";
+
+const mock = createMockAgentRunner({
+  responses: {
+    analyst: { output: "Analysis: positive trend", totalTokens: 100 },
+    writer:  { output: "Draft article…",            totalTokens: 200, delay: 50 },
+    failing: { output: "n/a", error: new Error("simulated 503") },
+  },
+  defaultResponse: { output: "mock response", totalTokens: 10 },
+  recordCalls: true,
+});
+
+const orchestrator = createAgentOrchestrator({ runner: mock.run });
+
+await orchestrator.run({ name: "analyst", instructions: "…", model: "x" }, "Analyze this");
+
+mock.getCalls();                // RecordedCall[] — every recorded run
+mock.getCallsFor("analyst");    // RecordedCall[] — only this agent
+mock.setResponse("analyst", { output: "different reply", totalTokens: 80 });
+mock.clearCalls();
+```
+
+For dynamic responses keyed on input, use the `generate` callback inside the config:
+
+```typescript
+const mock = createMockAgentRunner({
+  responses: {
+    classifier: {
+      output: "fallback",
+      totalTokens: 10,
+      generate: (input) => /summarize/i.test(input)
+        ? { output: "Summary: …", totalTokens: 50 }
+        : { output: "Default reply", totalTokens: 10 },
+    },
+  },
+});
+```
+
+## Test orchestrator
+
+`createTestOrchestrator` extends `AgentOrchestrator` with mock-runner wiring, an approval simulator, and call recording. Pass `mockResponses` and Directive constraints/resolvers exactly as you would to the production factory.
+
+```typescript
+import { createTestOrchestrator } from "@directive-run/ai/testing";
+import { t } from "@directive-run/core";
+
+const test = createTestOrchestrator({
+  mockResponses: {
+    analyst: { output: "Analysis: positive trend", totalTokens: 100 },
+  },
+  factsSchema: {
+    result: t.string(),
+    confidence: t.number(),
+  },
+  init: (facts) => {
+    facts.result = "";
+    facts.confidence = 0;
+  },
+  constraints: {
+    lowConfidence: {
+      when: (facts) => facts.confidence < 0.5,
+      require: { type: "RE_ANALYZE" },
+    },
+  },
+  resolvers: {
+    reAnalyze: {
+      requirement: (r): r is { type: "RE_ANALYZE" } => r.type === "RE_ANALYZE",
+      resolve: async (req, context) => { context.facts.confidence = 0.8; },
+    },
+  },
+});
+
+const result = await test.run({ name: "analyst", instructions: "…", model: "x" }, "Analyze");
+
+test.getCalls();             // RecordedCall[]
+test.getApprovalRequests();  // ApprovalRequest[] from the built-in simulator
+test.mockRunner;             // the underlying MockAgentRunner
+test.approvalSimulator;      // the built-in ApprovalSimulator
+test.resetAll();             // reset orchestrator + clear calls + clear approvals
+```
+
+## Test multi-agent orchestrator
+
+```typescript
+import {
+  createTestMultiAgentOrchestrator,
+  createMockAgentRunner,
+  assertMultiAgentState,
+} from "@directive-run/ai/testing";
+import { sequential } from "@directive-run/ai/multi-agent";
+
+const mock = createMockAgentRunner({
+  responses: {
+    researcher: { output: "research notes", totalTokens: 150 },
+    writer:     { output: "draft article", totalTokens: 200 },
+  },
+});
+
+const orch = createTestMultiAgentOrchestrator({
+  agents: {
+    researcher: { name: "researcher", instructions: "research", model: "x" },
+    writer:     { name: "writer",     instructions: "write",    model: "x" },
+  },
+  patterns: {
+    pipeline: sequential(["researcher", "writer"]),
+  },
+  runner: mock.run,
+});
+
+orch.start();
+await orch.runPattern("pipeline", "Write about AI");
+
+assertMultiAgentState(orch, {
+  status: "completed",
+});
+```
+
+## Snapshot / introspection helpers
+
+| Helper | Purpose |
+|---|---|
+| `createConstraintRecorder()` | Records every constraint evaluation as a `ConstraintSnapshot`; pair with `assertOrchestratorState`. |
+| `assertOrchestratorState(orch, expected)` | Diffs facts, status, and constraint state against an expected shape. |
+| `assertMultiAgentState(orch, expected)` | Same shape, for multi-agent. |
+| `assertScratchpadState(orch, expected)` | Checks the multi-agent scratchpad contents. |
+| `assertDerivedValues(orch, expected)` | Diffs derivation values. |
+| `assertTimelineEvents(timeline, expected)` | Asserts a sequence of timeline event types/payloads. |
+| `assertDagExecution(events, expected)` | Asserts DAG node execution order. |
+| `assertBreakpointHit(events, expected)` | Asserts a breakpoint was hit with given metadata. |
+| `assertRerouted(events, expected)` | Asserts a self-healing reroute fired (when configured). |
+| `assertAgentHealth(monitor, agentId, expected)` | Asserts a health score range / passing failed counts. |
+| `assertCheckpoint(cp, expected)` | Asserts checkpoint shape (state version, conversation length, etc.). |
+| `assertMultiplexedStream(stream, expected)` | Asserts a multiplexed stream produced expected per-agent chunks. |
+
+There is no `assertAgentCalled` / `assertAgentNotCalled` / `assertTokensUsed` / `assertGuardrailPassed` / `assertGuardrailBlocked` — those don't exist. Use `mock.getCalls()` + `expect()` from your test framework, and inspect `result.tokenUsage` and `result.guardrailEvents` directly.
+
+```typescript
+import { expect } from "vitest";
+
+const calls = mock.getCallsFor("analyst");
+expect(calls).toHaveLength(1);
+expect(calls[0].input).toMatch(/analyze/i);
+
+expect(result.tokenUsage).toBeLessThan(500);
+```
+
+## Simulators
+
+```typescript
+import {
+  createFailingRunner,
+  createApprovalSimulator,
+  createBreakpointSimulator,
+  createTimeController,
+  createTestCheckpointStore,
+  createTestReflectionEvaluator,
+  createTestEmbedder,
+} from "@directive-run/ai/testing";
+```
+
+### `createFailingRunner(error?, options?)`
+
+Returns an `AgentRunner` that throws (or fails after N successes). Replaces the imagined `createErrorSimulator` / `createLatencySimulator`.
+
+```typescript
+const failing = createFailingRunner(new Error("rate_limit"), {
+  delay: 100,
+  failAfter: 2, // first 2 calls succeed, then it starts throwing
+});
+
+const orchestrator = createAgentOrchestrator({
+  runner: failing,
+  selfHealing: { fallbackRunners: [backupRunner] },
+});
+```
+
+For latency-only simulation, set `delay` on a `MockAgentConfig`:
+
+```typescript
+const slow = createMockAgentRunner({
+  responses: { writer: { output: "slow reply", totalTokens: 50, delay: 800 } },
+});
+```
+
+### `createApprovalSimulator(options?)`
+
+Drives the approval workflow deterministically — auto-approve, auto-reject, or queue.
+
+```typescript
+const sim = createApprovalSimulator({ defaultAction: "approve" });
+
+const orchestrator = createAgentOrchestrator({
+  runner,
+  autoApproveToolCalls: false,
+  onApprovalRequest: (req) => sim.handle(req),
+});
+
+const requests = sim.getRequests();
+sim.clearRequests();
+```
+
+### `createBreakpointSimulator(options?)`
+
+Resolves breakpoints automatically, with optional modifications. Pair with the orchestrator's `breakpoints:` config.
+
+### `createTimeController(startTime?)`
+
+Drives `Date.now()` and `setTimeout` deterministically — required when testing budget windows, retry backoffs, or memory expiry.
+
+### `createTestCheckpointStore()`
+
+In-memory checkpoint store for round-tripping `checkpoint()` + `restore()` without disk IO.
+
+### `createTestReflectionEvaluator(options?)`
+
+Deterministic evaluator for `reflect()` patterns — scores can be hard-coded per iteration.
+
+### `createTestEmbedder()`
+
+Deterministic embedder (no network calls) for `createSemanticCache` testing.
+
+## Evaluation framework
+
+`createEvalSuite` runs a dataset of test cases through one or more agents, scores each output with one or more `EvalCriterion`, and returns aggregate results. Criteria are NOT a `criteria.*()` namespace — they're top-level `eval*` factories.
+
+```typescript
+import {
+  createEvalSuite,
+  evalCost,
+  evalLatency,
+  evalOutputLength,
+  evalSafety,
+  evalStructure,
+  evalMatch,
+  evalJudge,
+  evalFaithfulness,
+  evalRelevance,
+  evalCoherence,
+  evalAssert,
+} from "@directive-run/ai/evals";
+
+const suite = createEvalSuite({
+  agents: [analystAgent, writerAgent],
+  runner,
+  dataset: [
+    { id: "ts-basics",   input: "What is TypeScript?",   expected: "TypeScript is…",       tags: ["basics"] },
+    { id: "monads",      input: "Explain monads",         expected: "A monad is…",         tags: ["advanced"] },
+    { id: "ssr-tradeoffs", input: "Server vs client rendering", expected: "Tradeoffs…",    tags: ["advanced"] },
+  ],
+  criteria: {
+    cost:      evalCost({ maxTokensPerRun: 500 }),
+    latency:   evalLatency({ maxMs: 5000 }),
+    safety:    evalSafety(),
+    relevance: evalRelevance({ embedder: embedderFn, minSimilarity: 0.7 }),
+    judge:     evalJudge({ runner, judge: judgeAgent }),
+  },
+  concurrency: 4,
+  onCaseComplete: (caseResult) => console.log(`✓ ${caseResult.caseId} — ${caseResult.score.toFixed(2)}`),
+});
+
+const results = await suite.run();
+
+console.log(results.summary.averageScore);  // 0.82
+console.log(results.summary.passRate);      // 0.90
+console.log(results.cases);                 // EvalCaseResult[]
+console.log(results.agentSummaries);        // EvalAgentSummary[] (per-agent breakdown)
+console.log(results.totalTokens, results.durationMs);
+```
+
+### Custom criterion
+
+A criterion is `{ name, fn, threshold?, weight? }` where `fn` returns `{ score, passed?, reason?, durationMs? }`. **Scorers must be pure** — no side effects.
+
+```typescript
+const codeQuality = {
+  name: "code-quality",
+  threshold: 0.6,
+  weight: 1.0,
+  fn: (ctx) => {
+    const start = Date.now();
+    const output = String(ctx.result.output);
+    const hasCode = /\b(function|const|class)\b/.test(output);
+    const hasExplanation = output.length > 100;
+
+    const score = hasCode && hasExplanation ? 1.0 : hasCode ? 0.7 : 0.2;
+    return {
+      score,
+      passed: score >= 0.6,
+      reason: hasCode && hasExplanation ? "code with explanation"
+            : hasCode ? "code only"
+            : "no code",
+      durationMs: Date.now() - start,
+    };
+  },
+};
+
+const suite = createEvalSuite({
+  agents: [coderAgent],
+  runner,
+  dataset,
+  criteria: { codeQuality },
+});
+```
+
+### LLM-as-judge
+
+```typescript
+import { evalJudge } from "@directive-run/ai/evals";
+
+const judge = evalJudge({
+  runner,                     // any AgentRunner — typically a smaller/cheaper model
+  judge: judgeAgent,          // the AgentLike to invoke
+  promptTemplate: `…`,        // optional — defaults to a JSON-output rubric
+  timeoutMs: 30_000,
+});
+```
+
+The judge returns `{ score: number, reason?: string }` parsed from JSON. The default prompt enforces "Respond with ONLY a JSON object: …" — override `promptTemplate` if you need a different rubric.
+
+### CI quality gates
+
+```typescript
+const results = await suite.run();
+
+if (results.summary.averageScore < 0.75) {
+  console.error(`Quality gate failed: ${results.summary.averageScore} < 0.75`);
+  process.exit(1);
+}
+
+for (const [criterionName, summary] of Object.entries(results.summary.byCriterion)) {
+  if (summary.averageScore < 0.6) {
+    console.error(`${criterionName} regressed: ${summary.averageScore}`);
+    process.exit(1);
+  }
+}
+```
+
+## Anti-patterns
+
+### Calling `createMockRunner` (singular, with input-pattern array)
+
+```typescript
+// WRONG — createMockRunner does not exist; the array-pattern form does not exist
+createMockRunner([{ input: /analyze/, output: "…" }])
+
+// CORRECT — keyed by agent name
+createMockAgentRunner({
+  responses: { analyst: { output: "…", totalTokens: 100 } },
+})
+```
+
+### Importing hallucinated assertions
+
+```typescript
+// WRONG — none of these exist
+import {
+  assertAgentCalled,
+  assertAgentNotCalled,
+  assertTokensUsed,
+  assertGuardrailPassed,
+  assertGuardrailBlocked,
+} from "@directive-run/ai/testing";
+
+// CORRECT — use the mock's recorder + your test framework's assertions
+const calls = mock.getCallsFor("analyst");
+expect(calls).toHaveLength(1);
+expect(result.tokenUsage).toBeLessThan(200);
+```
+
+### Importing `createEvaluator`, `criteria.*()`, `createLLMJudge`, `createEvaluationSuite`
+
+```typescript
+// WRONG — these are all hallucinated names
+import { createEvaluator, criteria, createLLMJudge, createEvaluationSuite } from "@directive-run/ai/testing";
+const evaluator = createEvaluator({ criteria: [criteria.relevance(), criteria.coherence()] });
+const judge = createLLMJudge({ runner, model: "…", criteria: ["accuracy"], rubric: "…" });
+
+// CORRECT — top-level eval factories + createEvalSuite
+import { createEvalSuite, evalRelevance, evalCoherence, evalJudge } from "@directive-run/ai/evals";
+const suite = createEvalSuite({
+  agents: [agent],
+  runner,
+  dataset,
+  criteria: {
+    relevance: evalRelevance({ embedder, minSimilarity: 0.7 }),
+    coherence: evalCoherence({ embedder, minSimilarity: 0.5 }),
+    judge:     evalJudge({ runner, judge: judgeAgent }),
+  },
+});
+```
+
+### Importing `createErrorSimulator` / `createLatencySimulator`
+
+```typescript
+// WRONG — neither exists
+import { createErrorSimulator, createLatencySimulator } from "@directive-run/ai/testing";
+
+// CORRECT — createFailingRunner for errors; delay in MockAgentConfig for latency
+import { createFailingRunner, createMockAgentRunner } from "@directive-run/ai/testing";
+const failing = createFailingRunner(new Error("503"), { failAfter: 2, delay: 100 });
+const slow = createMockAgentRunner({ responses: { writer: { output: "…", delay: 800 } } });
+```
+
+### Side-effecting scorers
+
+```typescript
+// WRONG — scorers run inside the eval suite and must be pure
+{
+  name: "quality",
+  fn: (ctx) => {
+    fs.writeFileSync("eval.log", String(ctx.result.output)); // ← side effect
+    metrics.increment("evals");                                // ← side effect
+    return { score: 0.8 };
+  },
+}
+
+// CORRECT — log/metrics in onCaseComplete or after suite.run()
+const suite = createEvalSuite({
+  …,
+  onCaseComplete: (caseResult) => {
+    metrics.increment("evals");
+    fs.appendFileSync("eval.log", JSON.stringify(caseResult) + "\n");
+  },
+});
+```
+
+## Quick reference
+
+| API | Import path | Purpose |
+|---|---|---|
+| `createMockAgentRunner` | `@directive-run/ai/testing` | Deterministic mock runner with call recording |
+| `createTestOrchestrator` | `@directive-run/ai/testing` | Single-agent orchestrator with mock + approval simulator |
+| `createTestMultiAgentOrchestrator` | `@directive-run/ai/testing` | Multi-agent orchestrator wired to a mock runner |
+| `createFailingRunner` | `@directive-run/ai/testing` | Runner that throws (or fails after N successes) |
+| `createApprovalSimulator` | `@directive-run/ai/testing` | Deterministic approve/reject driver |
+| `createBreakpointSimulator` | `@directive-run/ai/testing` | Deterministic breakpoint resolver |
+| `createTimeController` | `@directive-run/ai/testing` | Virtual clock for budgets / backoffs / TTLs |
+| `createTestCheckpointStore` | `@directive-run/ai/testing` | In-memory checkpoint store |
+| `createTestReflectionEvaluator` | `@directive-run/ai/testing` | Deterministic evaluator for `reflect()` |
+| `createTestEmbedder` | `@directive-run/ai/testing` | Deterministic embedder for cache tests |
+| `assertOrchestratorState` / `assertMultiAgentState` / `assertScratchpadState` / `assertDerivedValues` / `assertTimelineEvents` / `assertDagExecution` / `assertBreakpointHit` / `assertRerouted` / `assertAgentHealth` / `assertCheckpoint` / `assertMultiplexedStream` | `@directive-run/ai/testing` | State/topology/event assertions |
+| `createEvalSuite` | `@directive-run/ai/evals` | Dataset-driven multi-criterion eval runner |
+| `evalCost` / `evalLatency` / `evalOutputLength` / `evalSafety` / `evalStructure` / `evalMatch` / `evalJudge` / `evalFaithfulness` / `evalRelevance` / `evalCoherence` / `evalAssert` | `@directive-run/ai/evals` | Built-in criterion factories |
+
+## See also
+
+- [`ai-orchestrator.md`](./ai-orchestrator.md) — `createTestOrchestrator` wraps this orchestrator; the prod surface is what your tests are exercising
+- [`ai-multi-agent.md`](./ai-multi-agent.md) — `createTestMultiAgentOrchestrator` for testing pattern execution
+- [`ai-debug-observability.md`](./ai-debug-observability.md) — recorded debug events feed `assertTimelineEvents` / `assertDagExecution` / `assertBreakpointHit`

package/skills/testing-ai-systems/api-skeleton.md

@@ -0,0 +1,5 @@
+# API Skeleton
+
+> Auto-generated. Do not edit.
+
+> Source JSON not found. Build website docs first.