@directive-run/knowledge 1.13.0 → 1.15.0

package/ai/ai-debug-observability.md

@@ -1,243 +1,323 @@
-# AI Debug and Observability
+# AI debug + observability
 
-Debug timeline with 25+ event types, breakpoints, checkpoints, and OpenTelemetry integration for AI orchestrators.
+> Covers `@directive-run/ai/devtools` and `@directive-run/core/plugins` — `createDebugTimeline`, breakpoints, `createOtelPlugin`, `createOTLPExporter`.
 
-## Decision Tree: "How do I observe what's happening?"
+Debug timeline with typed event recording, breakpoints, checkpoints, an OTel plugin for span emission, and an OTLP exporter for sending metrics/traces to any OTel-compatible backend.
+
+## Decision tree
 
 ```
 What do you need?
-├── Event stream of all AI activity → createDebugTimeline()
-├── Pause execution at specific points → breakpoints config
-├── Save/restore orchestrator state → checkpoint() / restore
-├── Export traces to observability platform → createOTLPExporter()
-│
-Which events to watch?
-├── Agent lifecycle → agent_start, agent_complete, agent_error, agent_retry
-├── Guardrails → guardrail_check
-├── Constraints/Resolvers → constraint_evaluate, resolver_start, resolver_complete
-├── Approval workflow → approval_request, approval_response
-├── Multi-agent patterns → pattern_start, pattern_complete, dag_node_update
-├── Composition patterns → race_start, race_winner, race_cancelled,
-│                          debate_round, reflection_iteration
-├── Handoffs → handoff_start, handoff_complete, reroute
-├── Tasks → task_start, task_complete, task_error, task_progress, goal_step
-├── Checkpoints → checkpoint_save, checkpoint_restore
-└── Breakpoints → breakpoint_hit, breakpoint_resumed
-```
-
-## Debug Timeline
-
-Subscribe to a real-time event stream of all orchestrator activity:
+├── Event stream of all AI activity  → createDebugTimeline()
+├── Pause execution at specific points → breakpoints (top-level option on orchestrator)
+├── Save / restore orchestrator state → orchestrator.checkpoint() / restore()
+├── Span emission to OTel             → createOtelPlugin({ serviceName })
+├── Ship metrics + traces to backend  → createOTLPExporter({ endpoint })
+└── DevTools panel + websocket export → createDevtoolsServer + createDevtoolsExport
+```
+
+## `createDebugTimeline(options?)`
+
+Returns a `DebugTimeline` with typed event recording, multi-axis querying, snapshot-aware forking, and JSON import/export. Import from `@directive-run/ai/devtools` — the main `@directive-run/ai` barrel re-exports it with `@deprecated` notices for v2.
 
 ```typescript
-import { createDebugTimeline } from "@directive-run/ai";
+import { createDebugTimeline, type DebugTimelineListener } from "@directive-run/ai/devtools";
 
 const timeline = createDebugTimeline({
-  maxEvents: 2000, // Ring buffer size (oldest evicted first)
+  maxEvents: 2000,                          // default 2000 — ring buffer, oldest evicted first
+  getSnapshotId: () => system.history?.currentId ?? null, // for snapshot-aware querying
+  goToSnapshot: (id) => system.history?.goTo(id),         // for forkFrom
 });
 
-// Subscribe to all events
-const unsubscribe = timeline.subscribe((event) => {
+// Subscribe — listener takes a single DebugEvent. There is NO options arg / filter shape.
+const unsubscribe: () => void = timeline.subscribe((event) => {
   console.log(`[${event.timestamp}] ${event.type}`, event);
 });
 
-// Filter by event type
-const agentEvents = timeline.subscribe(
-  (event) => {
-    console.log(`Agent: ${event.agentName} → ${event.type}`);
-  },
-  { filter: (event) => event.type.startsWith("agent_") },
-);
+// Filter at the listener — the API has no built-in filter option
+const unsubAgents = timeline.subscribe((event) => {
+  if (event.type.startsWith("agent_")) {
+    console.log(`agent: ${event.type}`);
+  }
+});
 
-// Query past events
-const errors = timeline.query({ type: "agent_error" });
-const recentAgentStarts = timeline.query({
+// Record an event manually
+timeline.record({
   type: "agent_start",
-  since: Date.now() - 60000,
+  timestamp: Date.now(),
+  agentId: "researcher",
+  snapshotId: null,
+  inputLength: 42,
 });
+
+// Length
+console.log(timeline.length);
 ```
 
-## Event Types Reference
+### Querying past events
 
-All 25+ event types emitted by the timeline:
+The query surface is method-based — there is no `timeline.query({ type, since })`. Use the typed helpers:
 
 ```typescript
-// Agent lifecycle
-type AgentEvents =
-  | { type: "agent_start"; agentName: string; prompt: string }
-  | { type: "agent_complete"; agentName: string; output: string; tokens: number; duration: number }
-  | { type: "agent_error"; agentName: string; error: Error }
-  | { type: "agent_retry"; agentName: string; attempt: number; maxRetries: number; error: Error };
-
-// Guardrails
-type GuardrailEvents =
-  | { type: "guardrail_check"; guardrailName: string; phase: "input" | "output"; passed: boolean; reason?: string };
-
-// Constraints and resolvers
-type ConstraintEvents =
-  | { type: "constraint_evaluate"; constraintId: string; result: boolean }
-  | { type: "resolver_start"; resolverType: string; requirementKey: string }
-  | { type: "resolver_complete"; resolverType: string; duration: number };
-
-// Approval workflow
-type ApprovalEvents =
-  | { type: "approval_request"; agentName: string; prompt: string }
-  | { type: "approval_response"; agentName: string; approved: boolean; reason?: string };
-
-// Multi-agent patterns
-type PatternEvents =
-  | { type: "pattern_start"; patternName: string; agents: string[] }
-  | { type: "pattern_complete"; patternName: string; duration: number }
-  | { type: "dag_node_update"; nodeId: string; status: "pending" | "running" | "complete" | "error" };
-
-// Composition patterns
-type CompositionEvents =
-  | { type: "race_start"; agents: string[] }
-  | { type: "race_winner"; agentName: string; duration: number }
-  | { type: "race_cancelled"; agentName: string; reason: string }
-  | { type: "debate_round"; round: number; agentName: string; position: string }
-  | { type: "reflection_iteration"; iteration: number; agentName: string };
-
-// Handoffs and routing
-type HandoffEvents =
-  | { type: "handoff_start"; from: string; to: string }
-  | { type: "handoff_complete"; from: string; to: string; duration: number }
-  | { type: "reroute"; from: string; to: string; reason: string };
-
-// Checkpoints and breakpoints
-type CheckpointEvents =
-  | { type: "checkpoint_save"; checkpointId: string }
-  | { type: "checkpoint_restore"; checkpointId: string }
-  | { type: "breakpoint_hit"; breakpointId: string; agentName: string }
-  | { type: "breakpoint_resumed"; breakpointId: string };
-
-// Tasks
-type TaskEvents =
-  | { type: "task_start"; taskId: string; label?: string }
-  | { type: "task_complete"; taskId: string; duration: number }
-  | { type: "task_error"; taskId: string; error: Error }
-  | { type: "task_progress"; taskId: string; percent: number; message?: string }
-  | { type: "goal_step"; iteration: number; goalMet: boolean };
-```
-
-## Attaching Timeline to Orchestrator
+const all          = timeline.getEvents();                                // DebugEvent[]
+const agentErrors  = timeline.getEventsByType("agent_error");              // typed-narrowed
+const recentSpan   = timeline.getEventsInRange(Date.now() - 60_000, Date.now());
+const researcher   = timeline.getEventsForAgent("researcher");
+const atSnapshot   = timeline.getEventsAtSnapshot(7);
+```
+
+`getEventsByType<T>(type)` narrows the union — `agentErrors` above is typed as `Extract<DebugEvent, { type: "agent_error" }>[]`, so `err.error.message` autocompletes.
+
+### Persistence + fork
 
 ```typescript
-import { createAgentOrchestrator, createDebugTimeline } from "@directive-run/ai";
+const json = timeline.export();        // string — full JSON dump
+timeline.clear();
+timeline.import(json);                  // re-hydrate
+
+timeline.forkFrom(snapshotId);          // truncate post-snapshot events + call goToSnapshot
+```
 
-const timeline = createDebugTimeline({ maxEvents: 2000 });
+## Attaching the timeline to an orchestrator
+
+`OrchestratorDebugConfig` has exactly one option: `verboseTimeline?: boolean`. There is NO `timeline:` / `breakpoints:` / `exporter:` field inside `debug:` — `breakpoints` and `onBreakpoint` are top-level options on the orchestrator, and the timeline is *read off* the orchestrator via `orchestrator.timeline` after construction.
+
+```typescript
+import { createAgentOrchestrator } from "@directive-run/ai";
 
 const orchestrator = createAgentOrchestrator({
   runner,
-  debug: {
-    timeline,
-    verbose: true, // Log all events to console
+  debug: { verboseTimeline: true },     // or just `debug: true`
+  // Top-level — NOT inside debug
+  breakpoints: [
+    { id: "before-write", before: "agent_start" },
+  ],
+  onBreakpoint: (req) => {
+    console.log("breakpoint hit", req.id);
+    orchestrator.resumeBreakpoint(req.id);
   },
 });
+
+// Timeline is exposed on the orchestrator (null when debug: false)
+const timeline = orchestrator.timeline;
+timeline?.subscribe((event) => console.log(event));
 ```
 
 ## Breakpoints
 
-Pause execution at specific points for human inspection:
+Configure breakpoints as a top-level option on the orchestrator. Each entry pauses execution before/after a specific event type; `onBreakpoint` fires with a `BreakpointRequest` you resume via `orchestrator.resumeBreakpoint(id)` (with optional input modifications) or cancel via `orchestrator.cancelBreakpoint(id, reason?)`.
 
 ```typescript
+import type { BreakpointConfig, BreakpointModifications } from "@directive-run/ai";
+
 const orchestrator = createMultiAgentOrchestrator({
   agents: { researcher, writer, editor },
   runner,
-  debug: {
-    timeline,
-    breakpoints: [
-      {
-        id: "before-write",
-        // Pause before the writer agent runs
-        when: (event) => {
-          return event.type === "agent_start" && event.agentName === "writer";
-        },
-        onHit: async (event, resume) => {
-          console.log("Paused before writer. Review researcher output.");
-          console.log("Press enter to continue...");
-
-          await waitForInput();
-          resume();
-        },
-      },
-      {
-        id: "on-error",
-        when: (event) => {
-          return event.type === "agent_error";
-        },
-        onHit: async (event, resume) => {
-          console.error("Agent error:", event.error);
-          // Decide whether to continue or abort
-          resume();
-        },
-      },
-    ],
+  breakpoints: [
+    { id: "before-write", before: "agent_start", filter: (e) => e.agentName === "writer" },
+    { id: "on-error",     after:  "agent_error" },
+  ],
+  onBreakpoint: async (req) => {
+    if (req.id === "on-error") {
+      // Log + cancel
+      console.error("breakpoint on error:", req.event);
+      orchestrator.cancelBreakpoint(req.id, "aborting on error");
+
+      return;
+    }
+
+    await waitForUserClick();
+    orchestrator.resumeBreakpoint(req.id);
   },
+  breakpointTimeoutMs: 5 * 60 * 1000,
 });
 ```
 
+There is NO `breakpoint.when(event)` / `breakpoint.onHit(event, resume)` shape — those don't exist. Filter on a typed `before:` / `after:` event-type key plus an optional `filter:` predicate; resume/cancel via the orchestrator's methods.
+
 ## Checkpoints
 
-Save and restore full orchestrator state for debugging or recovery:
+See `ai-orchestrator.md` and `ai-multi-agent.md` for the full checkpoint surface. Quick form:
 
 ```typescript
-// Save checkpoint
-const checkpoint = orchestrator.checkpoint();
-const serialized = JSON.stringify(checkpoint);
+const cp = await orchestrator.checkpoint({ label: "before-risky-op" });  // async!
+const serialized = JSON.stringify(cp);
 
-// Store to disk, database, etc.
-await fs.writeFile("checkpoint.json", serialized);
+// Restore on an EXISTING instance — there is no `checkpoint` constructor option
+const fresh = createAgentOrchestrator({ runner /* same config */ });
+fresh.restore(JSON.parse(serialized));
+```
+
+## OpenTelemetry plugin
+
+`createOtelPlugin({ serviceName, … })` returns a Directive plugin you pass to `createSystem({ plugins: [otel] })` (or to the orchestrator via its `plugins:` option). It emits spans for agent runs, resolver execution, guardrails, and pattern boundaries — mapped to GenAI semantic conventions.
+
+```typescript
+import { createOtelPlugin, OtelStatusCode } from "@directive-run/ai";
 
-// Restore from checkpoint
-const saved = JSON.parse(await fs.readFile("checkpoint.json", "utf-8"));
-const restored = createMultiAgentOrchestrator({
-  agents,
+const otel = createOtelPlugin({
+  serviceName: "my-ai-app",
+  serviceVersion: "1.0.0",
+  // Optional: a custom OtelTracer. Defaults to an in-memory tracer suitable for the OTLP exporter.
+  tracer: customTracer,
+});
+
+const orchestrator = createAgentOrchestrator({
   runner,
-  checkpoint: saved,
+  plugins: [otel],
 });
-restored.start();
 ```
 
-## OpenTelemetry Integration
+## OTLP exporter
 
-Export traces to any OpenTelemetry-compatible backend:
+`createOTLPExporter({ endpoint, … })` returns `{ exportMetrics, exportTraces }` — paired with an observability config that streams collected data to any OTel-compatible backend (Jaeger, Tempo, Grafana Cloud, Honeycomb, etc.).
 
 ```typescript
 import { createOTLPExporter } from "@directive-run/ai";
+// (re-exported from @directive-run/core/plugins)
 
 const exporter = createOTLPExporter({
-  endpoint: "http://localhost:4318/v1/traces",
+  endpoint: "http://localhost:4318",
   serviceName: "my-ai-app",
-  // Uses GenAI semantic conventions
-  // https://opentelemetry.io/docs/specs/semconv/gen-ai/
+  serviceVersion: "1.0.0",
+  headers: { Authorization: `Bearer ${process.env.OTEL_TOKEN}` },
+  timeoutMs: 10_000,
+  onError: (err, kind) => console.error(`[otlp] ${kind} export failed:`, err),
+});
+
+// Wire into the observability plugin (see core observability docs)
+const obs = createObservabilityPlugin({
+  metrics: { exporter: { export: exporter.exportMetrics } },
+  tracing: { exporter: { export: exporter.exportTraces } },
 });
 
 const orchestrator = createAgentOrchestrator({
   runner,
-  debug: {
-    timeline,
-    exporter, // Automatically exports spans
-  },
+  plugins: [obs, otel],
+});
+```
+
+The exporter maps Directive events to GenAI semantic conventions. The exact mapping (gen_ai.chat, gen_ai.tool, gen_ai.guardrail, gen_ai.orchestration) lives in `@directive-run/core/plugins/otlp-exporter.ts` and tracks the OpenTelemetry GenAI spec.
+
+## DevTools server (live websocket export)
+
+For browser-based devtools panels, pair `createDebugTimeline` with `createDevtoolsServer` (also from `@directive-run/ai/devtools`) — same data the in-process listener gets, streamed over a WebSocket.
+
+```typescript
+import { createDebugTimeline, createDevtoolsServer } from "@directive-run/ai/devtools";
+
+const timeline = createDebugTimeline({ maxEvents: 5000 });
+const server = createDevtoolsServer({ timeline, port: 9229 });
+
+server.start();
+// Connect a devtools UI to ws://localhost:9229
+```
+
+## Anti-patterns
+
+### `timeline.subscribe(listener, { filter })`
+
+```typescript
+// WRONG — subscribe takes only a listener
+timeline.subscribe((e) => console.log(e), { filter: (e) => e.type === "agent_error" })
+
+// CORRECT — filter inside the listener
+timeline.subscribe((e) => {
+  if (e.type === "agent_error") console.log(e);
+});
+```
+
+### `timeline.query({ type, since })`
+
+```typescript
+// WRONG — there is no .query() method
+timeline.query({ type: "agent_error", since: Date.now() - 60_000 })
+
+// CORRECT — typed getters
+timeline.getEventsByType("agent_error")
+  .filter((e) => e.timestamp >= Date.now() - 60_000);
+
+// Or use range directly
+timeline.getEventsInRange(Date.now() - 60_000, Date.now());
+```
+
+### Nesting timeline / breakpoints inside `debug:`
+
+```typescript
+// WRONG — only verboseTimeline lives inside debug; breakpoints + timeline are top-level
+createAgentOrchestrator({
+  runner,
+  debug: { timeline, verbose: true, breakpoints: [...] },
+})
+
+// CORRECT
+createAgentOrchestrator({
+  runner,
+  debug: { verboseTimeline: true },           // OR just `debug: true`
+  breakpoints: [...],                          // top-level
+  onBreakpoint: (req) => orchestrator.resumeBreakpoint(req.id),
 });
+// timeline is read off `orchestrator.timeline` after construction
+```
+
+### `breakpoint.when(event)` / `breakpoint.onHit(event, resume)`
+
+```typescript
+// WRONG — these shapes don't exist
+breakpoints: [
+  { id: "x", when: (e) => e.type === "agent_start", onHit: (e, resume) => resume() },
+]
+
+// CORRECT — declarative before:/after: + typed filter, then orchestrator.resumeBreakpoint()
+breakpoints: [
+  { id: "x", before: "agent_start", filter: (e) => e.agentName === "writer" },
+],
+onBreakpoint: (req) => orchestrator.resumeBreakpoint(req.id),
 ```
 
-The exporter maps Directive events to GenAI semantic conventions:
+### Restoring a checkpoint via the constructor
 
-| Directive Event | OTel Span | GenAI Attribute |
-|---|---|---|
-| `agent_start` → `agent_complete` | `gen_ai.chat` | `gen_ai.system`, `gen_ai.request.model` |
-| `resolver_start` → `resolver_complete` | `gen_ai.tool` | `gen_ai.tool.name` |
-| `guardrail_check` | `gen_ai.guardrail` | `gen_ai.guardrail.name`, `gen_ai.guardrail.passed` |
-| `pattern_start` → `pattern_complete` | `gen_ai.orchestration` | `gen_ai.orchestration.pattern` |
+```typescript
+// WRONG — there is no `checkpoint:` constructor option
+createMultiAgentOrchestrator({ agents, patterns, runner, checkpoint: saved })
+
+// CORRECT — orch.restore() on an existing instance
+const orch = createMultiAgentOrchestrator({ agents, patterns, runner });
+orch.restore(saved);
+orch.start();
+```
 
-## Quick Reference
+### Assuming `agent_complete.output` is `string`
+
+```typescript
+// WRONG — output is unknown, not string
+timeline.subscribe((e) => {
+  if (e.type === "agent_complete") console.log(e.output.length); // type error
+});
+
+// CORRECT — narrow before reading
+timeline.subscribe((e) => {
+  if (e.type === "agent_complete" && typeof e.output === "string") {
+    console.log(e.output.length);
+  }
+});
+```
 
-| API | Purpose | Key Options |
-|---|---|---|
-| `createDebugTimeline()` | Event stream for all activity | `maxEvents` |
-| `timeline.subscribe()` | Listen to events in real time | callback, filter |
-| `timeline.query()` | Search past events | type, since |
-| `orchestrator.checkpoint()` | Serialize full state | returns JSON-safe object |
-| `createOTLPExporter()` | Export traces to OTel backend | `endpoint`, `serviceName` |
-| breakpoints config | Pause at specific events | `when`, `onHit`, `resume` |
+## Quick reference
+
+| API | Purpose |
+|---|---|
+| `createDebugTimeline(options?)` | Typed event recorder + queryable history |
+| `timeline.subscribe(listener)` | Live event stream (no filter option) |
+| `timeline.getEventsByType<T>(type)` | Typed-narrowed query |
+| `timeline.getEventsInRange(start, end)` | Time-bounded query |
+| `timeline.getEventsForAgent(agentId)` | Per-agent stream |
+| `timeline.getEventsAtSnapshot(id)` | Snapshot-aware stream |
+| `timeline.forkFrom(id)` | Truncate post-snapshot + reroute |
+| `timeline.export()` / `import(json)` | JSON round-trip |
+| `orchestrator.timeline` | Read the timeline back off the orchestrator (`null` when `debug: false`) |
+| `breakpoints: [...]` + `onBreakpoint` | Top-level orchestrator options for pause points |
+| `orchestrator.resumeBreakpoint(id, mods?)` / `cancelBreakpoint(id, reason?)` | Resolve a paused breakpoint |
+| `createOtelPlugin(config)` | Plugin that emits spans for orchestrator activity |
+| `createOTLPExporter(config)` | Returns `{ exportMetrics, exportTraces }` for OTel backends |
+| `createDevtoolsServer({ timeline, port })` | WebSocket-bridged timeline for browser devtools |