@directive-run/claude-plugin 1.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +6 -0
- package/LICENSE +26 -0
- package/README.md +113 -0
- package/dist/index.cjs +2 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +66 -0
- package/dist/index.d.ts +66 -0
- package/dist/index.js +2 -0
- package/dist/index.js.map +1 -0
- package/package.json +64 -0
- package/skills/README.md +9 -0
- package/skills/building-ai-agents/SKILL.md +414 -0
- package/skills/building-ai-agents/ai-adapters.md +257 -0
- package/skills/building-ai-agents/ai-agents-streaming.md +307 -0
- package/skills/building-ai-agents/ai-communication.md +304 -0
- package/skills/building-ai-agents/api-skeleton.md +5 -0
- package/skills/building-ai-agents/examples.md +849 -0
- package/skills/building-ai-orchestrators/SKILL.md +401 -0
- package/skills/building-ai-orchestrators/ai-multi-agent.md +375 -0
- package/skills/building-ai-orchestrators/ai-orchestrator.md +466 -0
- package/skills/building-ai-orchestrators/ai-tasks.md +268 -0
- package/skills/building-ai-orchestrators/api-skeleton.md +5 -0
- package/skills/building-ai-orchestrators/examples.md +1609 -0
- package/skills/building-directive-systems/SKILL.md +499 -0
- package/skills/building-directive-systems/api-skeleton.md +5 -0
- package/skills/building-directive-systems/examples.md +4149 -0
- package/skills/building-directive-systems/multi-module.md +327 -0
- package/skills/building-directive-systems/plugins.md +361 -0
- package/skills/building-directive-systems/react-adapter.md +376 -0
- package/skills/building-directive-systems/system-api.md +463 -0
- package/skills/getting-started-with-directive/SKILL.md +198 -0
- package/skills/getting-started-with-directive/api-skeleton.md +5 -0
- package/skills/getting-started-with-directive/core-patterns.md +240 -0
- package/skills/getting-started-with-directive/examples.md +87 -0
- package/skills/getting-started-with-directive/sitemap.md +202 -0
- package/skills/hardening-ai-systems/SKILL.md +377 -0
- package/skills/hardening-ai-systems/ai-budget-resilience.md +408 -0
- package/skills/hardening-ai-systems/ai-guardrails-memory.md +370 -0
- package/skills/hardening-ai-systems/ai-security.md +400 -0
- package/skills/hardening-ai-systems/api-skeleton.md +5 -0
- package/skills/hardening-ai-systems/examples.md +628 -0
- package/skills/migrating-to-directive/SKILL.md +249 -0
- package/skills/migrating-to-directive/anti-patterns.md +382 -0
- package/skills/migrating-to-directive/api-skeleton.md +5 -0
- package/skills/migrating-to-directive/core-patterns.md +240 -0
- package/skills/migrating-to-directive/examples.md +511 -0
- package/skills/migrating-to-directive/schema-types.md +273 -0
- package/skills/reviewing-directive-code/SKILL.md +232 -0
- package/skills/reviewing-directive-code/anti-patterns.md +382 -0
- package/skills/reviewing-directive-code/api-skeleton.md +5 -0
- package/skills/reviewing-directive-code/core-patterns.md +240 -0
- package/skills/reviewing-directive-code/examples.md +463 -0
- package/skills/reviewing-directive-code/naming.md +321 -0
- package/skills/scaffolding-directive-modules/SKILL.md +218 -0
- package/skills/scaffolding-directive-modules/api-skeleton.md +5 -0
- package/skills/scaffolding-directive-modules/core-patterns.md +240 -0
- package/skills/scaffolding-directive-modules/examples.md +1386 -0
- package/skills/scaffolding-directive-modules/naming.md +321 -0
- package/skills/scaffolding-directive-modules/schema-types.md +273 -0
- package/skills/testing-ai-systems/SKILL.md +398 -0
- package/skills/testing-ai-systems/ai-debug-observability.md +329 -0
- package/skills/testing-ai-systems/ai-mcp-rag.md +293 -0
- package/skills/testing-ai-systems/ai-testing-evals.md +485 -0
- package/skills/testing-ai-systems/api-skeleton.md +5 -0
- package/skills/testing-ai-systems/examples.md +1261 -0
- package/skills/testing-directive-code/SKILL.md +479 -0
- package/skills/testing-directive-code/api-skeleton.md +5 -0
- package/skills/testing-directive-code/examples.md +156 -0
- package/skills/testing-directive-code/history.md +351 -0
- package/skills/testing-directive-code/testing.md +412 -0
- package/skills/writing-directive-constraints/SKILL.md +486 -0
- package/skills/writing-directive-constraints/api-skeleton.md +5 -0
- package/skills/writing-directive-constraints/constraints.md +274 -0
- package/skills/writing-directive-constraints/error-boundaries.md +330 -0
- package/skills/writing-directive-constraints/examples.md +1361 -0
- package/skills/writing-directive-constraints/resolvers.md +367 -0
- package/skills/writing-directive-modules/SKILL.md +342 -0
- package/skills/writing-directive-modules/anti-patterns.md +382 -0
- package/skills/writing-directive-modules/api-skeleton.md +5 -0
- package/skills/writing-directive-modules/core-patterns.md +240 -0
- package/skills/writing-directive-modules/examples.md +1639 -0
- package/skills/writing-directive-modules/naming.md +321 -0
- package/skills/writing-directive-modules/schema-types.md +273 -0
|
@@ -0,0 +1,329 @@
|
|
|
1
|
+
# AI debug + observability
|
|
2
|
+
|
|
3
|
+
> Covers `@directive-run/ai/devtools` and `@directive-run/core/plugins` — `createDebugTimeline`, breakpoints, `createOtelPlugin`, `createOTLPExporter`.
|
|
4
|
+
|
|
5
|
+
Debug timeline with typed event recording, breakpoints, checkpoints, an OTel plugin for span emission, and an OTLP exporter for sending metrics/traces to any OTel-compatible backend.
|
|
6
|
+
|
|
7
|
+
## Decision tree
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
What do you need?
|
|
11
|
+
├── Event stream of all AI activity → createDebugTimeline()
|
|
12
|
+
├── Pause execution at specific points → breakpoints (top-level option on orchestrator)
|
|
13
|
+
├── Save / restore orchestrator state → orchestrator.checkpoint() / restore()
|
|
14
|
+
├── Span emission to OTel → createOtelPlugin({ serviceName })
|
|
15
|
+
├── Ship metrics + traces to backend → createOTLPExporter({ endpoint })
|
|
16
|
+
└── DevTools panel + websocket export → createDevtoolsServer + createDevtoolsExport
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## `createDebugTimeline(options?)`
|
|
20
|
+
|
|
21
|
+
Returns a `DebugTimeline` with typed event recording, multi-axis querying, snapshot-aware forking, and JSON import/export. Import from `@directive-run/ai/devtools` — the main `@directive-run/ai` barrel re-exports it with `@deprecated` notices for v2.
|
|
22
|
+
|
|
23
|
+
```typescript
|
|
24
|
+
import { createDebugTimeline, type DebugTimelineListener } from "@directive-run/ai/devtools";
|
|
25
|
+
|
|
26
|
+
const timeline = createDebugTimeline({
|
|
27
|
+
maxEvents: 2000, // default 2000 — ring buffer, oldest evicted first
|
|
28
|
+
getSnapshotId: () => system.history?.currentId ?? null, // for snapshot-aware querying
|
|
29
|
+
goToSnapshot: (id) => system.history?.goTo(id), // for forkFrom
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
// Subscribe — listener takes a single DebugEvent. There is NO options arg / filter shape.
|
|
33
|
+
const unsubscribe: () => void = timeline.subscribe((event) => {
|
|
34
|
+
console.log(`[${event.timestamp}] ${event.type}`, event);
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
// Filter at the listener — the API has no built-in filter option
|
|
38
|
+
const unsubAgents = timeline.subscribe((event) => {
|
|
39
|
+
if (event.type.startsWith("agent_")) {
|
|
40
|
+
console.log(`agent: ${event.type}`);
|
|
41
|
+
}
|
|
42
|
+
});
|
|
43
|
+
|
|
44
|
+
// Record an event manually
|
|
45
|
+
timeline.record({
|
|
46
|
+
type: "agent_start",
|
|
47
|
+
timestamp: Date.now(),
|
|
48
|
+
agentId: "researcher",
|
|
49
|
+
snapshotId: null,
|
|
50
|
+
inputLength: 42,
|
|
51
|
+
});
|
|
52
|
+
|
|
53
|
+
// Length
|
|
54
|
+
console.log(timeline.length);
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### Querying past events
|
|
58
|
+
|
|
59
|
+
The query surface is method-based — there is no `timeline.query({ type, since })`. Use the typed helpers:
|
|
60
|
+
|
|
61
|
+
```typescript
|
|
62
|
+
const all = timeline.getEvents(); // DebugEvent[]
|
|
63
|
+
const agentErrors = timeline.getEventsByType("agent_error"); // typed-narrowed
|
|
64
|
+
const recentSpan = timeline.getEventsInRange(Date.now() - 60_000, Date.now());
|
|
65
|
+
const researcher = timeline.getEventsForAgent("researcher");
|
|
66
|
+
const atSnapshot = timeline.getEventsAtSnapshot(7);
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
`getEventsByType<T>(type)` narrows the union — `agentErrors` above is typed as `Extract<DebugEvent, { type: "agent_error" }>[]`, so `err.error.message` autocompletes.
|
|
70
|
+
|
|
71
|
+
### Persistence + fork
|
|
72
|
+
|
|
73
|
+
```typescript
|
|
74
|
+
const json = timeline.export(); // string — full JSON dump
|
|
75
|
+
timeline.clear();
|
|
76
|
+
timeline.import(json); // re-hydrate
|
|
77
|
+
|
|
78
|
+
timeline.forkFrom(snapshotId); // truncate post-snapshot events + call goToSnapshot
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Attaching the timeline to an orchestrator
|
|
82
|
+
|
|
83
|
+
`OrchestratorDebugConfig` has exactly one option: `verboseTimeline?: boolean`. There is NO `timeline:` / `breakpoints:` / `exporter:` field inside `debug:` — `breakpoints` and `onBreakpoint` are top-level options on the orchestrator, and the timeline is *read off* the orchestrator via `orchestrator.timeline` after construction.
|
|
84
|
+
|
|
85
|
+
```typescript
|
|
86
|
+
import { createAgentOrchestrator } from "@directive-run/ai";
|
|
87
|
+
|
|
88
|
+
const orchestrator = createAgentOrchestrator({
|
|
89
|
+
runner,
|
|
90
|
+
debug: { verboseTimeline: true }, // or just `debug: true`
|
|
91
|
+
// Top-level — NOT inside debug
|
|
92
|
+
breakpoints: [
|
|
93
|
+
{ id: "before-write", before: "agent_start" },
|
|
94
|
+
],
|
|
95
|
+
onBreakpoint: (req) => {
|
|
96
|
+
console.log("breakpoint hit", req.id);
|
|
97
|
+
orchestrator.resumeBreakpoint(req.id);
|
|
98
|
+
},
|
|
99
|
+
});
|
|
100
|
+
|
|
101
|
+
// Timeline is exposed on the orchestrator (null when debug: false)
|
|
102
|
+
const timeline = orchestrator.timeline;
|
|
103
|
+
timeline?.subscribe((event) => console.log(event));
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
## Breakpoints
|
|
107
|
+
|
|
108
|
+
Configure breakpoints as a top-level option on the orchestrator. Each entry pauses execution before/after a specific event type; `onBreakpoint` fires with a `BreakpointRequest` you resume via `orchestrator.resumeBreakpoint(id)` (with optional input modifications) or cancel via `orchestrator.cancelBreakpoint(id, reason?)`.
|
|
109
|
+
|
|
110
|
+
```typescript
|
|
111
|
+
import type { BreakpointConfig, BreakpointModifications } from "@directive-run/ai";
|
|
112
|
+
|
|
113
|
+
const orchestrator = createMultiAgentOrchestrator({
|
|
114
|
+
agents: { researcher, writer, editor },
|
|
115
|
+
runner,
|
|
116
|
+
breakpoints: [
|
|
117
|
+
{ id: "before-write", before: "agent_start", filter: (e) => e.agentName === "writer" },
|
|
118
|
+
{ id: "on-error", after: "agent_error" },
|
|
119
|
+
],
|
|
120
|
+
onBreakpoint: async (req) => {
|
|
121
|
+
if (req.id === "on-error") {
|
|
122
|
+
// Log + cancel
|
|
123
|
+
console.error("breakpoint on error:", req.event);
|
|
124
|
+
orchestrator.cancelBreakpoint(req.id, "aborting on error");
|
|
125
|
+
|
|
126
|
+
return;
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
await waitForUserClick();
|
|
130
|
+
orchestrator.resumeBreakpoint(req.id);
|
|
131
|
+
},
|
|
132
|
+
breakpointTimeoutMs: 5 * 60 * 1000,
|
|
133
|
+
});
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
There is NO `breakpoint.when(event)` / `breakpoint.onHit(event, resume)` shape — those don't exist. Filter on a typed `before:` / `after:` event-type key plus an optional `filter:` predicate; resume/cancel via the orchestrator's methods.
|
|
137
|
+
|
|
138
|
+
## Checkpoints
|
|
139
|
+
|
|
140
|
+
See `ai-orchestrator.md` and `ai-multi-agent.md` for the full checkpoint surface. Quick form:
|
|
141
|
+
|
|
142
|
+
```typescript
|
|
143
|
+
const cp = await orchestrator.checkpoint({ label: "before-risky-op" }); // async!
|
|
144
|
+
const serialized = JSON.stringify(cp);
|
|
145
|
+
|
|
146
|
+
// Restore on an EXISTING instance — there is no `checkpoint` constructor option
|
|
147
|
+
const fresh = createAgentOrchestrator({ runner /* same config */ });
|
|
148
|
+
fresh.restore(JSON.parse(serialized));
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
## OpenTelemetry plugin
|
|
152
|
+
|
|
153
|
+
`createOtelPlugin({ serviceName, … })` returns a Directive plugin you pass to `createSystem({ plugins: [otel] })` (or to the orchestrator via its `plugins:` option). It emits spans for agent runs, resolver execution, guardrails, and pattern boundaries — mapped to GenAI semantic conventions.
|
|
154
|
+
|
|
155
|
+
```typescript
|
|
156
|
+
import { createOtelPlugin, OtelStatusCode } from "@directive-run/ai";
|
|
157
|
+
|
|
158
|
+
const otel = createOtelPlugin({
|
|
159
|
+
serviceName: "my-ai-app",
|
|
160
|
+
serviceVersion: "1.0.0",
|
|
161
|
+
// Optional: a custom OtelTracer. Defaults to an in-memory tracer suitable for the OTLP exporter.
|
|
162
|
+
tracer: customTracer,
|
|
163
|
+
});
|
|
164
|
+
|
|
165
|
+
const orchestrator = createAgentOrchestrator({
|
|
166
|
+
runner,
|
|
167
|
+
plugins: [otel],
|
|
168
|
+
});
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
## OTLP exporter
|
|
172
|
+
|
|
173
|
+
`createOTLPExporter({ endpoint, … })` returns `{ exportMetrics, exportTraces }` — paired with an observability config that streams collected data to any OTel-compatible backend (Jaeger, Tempo, Grafana Cloud, Honeycomb, etc.).
|
|
174
|
+
|
|
175
|
+
```typescript
|
|
176
|
+
import { createOTLPExporter } from "@directive-run/ai";
|
|
177
|
+
// (re-exported from @directive-run/core/plugins)
|
|
178
|
+
|
|
179
|
+
const exporter = createOTLPExporter({
|
|
180
|
+
endpoint: "http://localhost:4318",
|
|
181
|
+
serviceName: "my-ai-app",
|
|
182
|
+
serviceVersion: "1.0.0",
|
|
183
|
+
headers: { Authorization: `Bearer ${process.env.OTEL_TOKEN}` },
|
|
184
|
+
timeoutMs: 10_000,
|
|
185
|
+
onError: (err, kind) => console.error(`[otlp] ${kind} export failed:`, err),
|
|
186
|
+
});
|
|
187
|
+
|
|
188
|
+
// Wire into the observability plugin (see core observability docs)
|
|
189
|
+
const obs = createObservabilityPlugin({
|
|
190
|
+
metrics: { exporter: { export: exporter.exportMetrics } },
|
|
191
|
+
tracing: { exporter: { export: exporter.exportTraces } },
|
|
192
|
+
});
|
|
193
|
+
|
|
194
|
+
const orchestrator = createAgentOrchestrator({
|
|
195
|
+
runner,
|
|
196
|
+
plugins: [obs, otel],
|
|
197
|
+
});
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
The exporter maps Directive events to GenAI semantic conventions. The exact mapping (gen_ai.chat, gen_ai.tool, gen_ai.guardrail, gen_ai.orchestration) lives in `@directive-run/core/plugins/otlp-exporter.ts` and tracks the OpenTelemetry GenAI spec.
|
|
201
|
+
|
|
202
|
+
## DevTools server (live websocket export)
|
|
203
|
+
|
|
204
|
+
For browser-based devtools panels, pair `createDebugTimeline` with `createDevtoolsServer` (also from `@directive-run/ai/devtools`) — same data the in-process listener gets, streamed over a WebSocket.
|
|
205
|
+
|
|
206
|
+
```typescript
|
|
207
|
+
import { createDebugTimeline, createDevtoolsServer } from "@directive-run/ai/devtools";
|
|
208
|
+
|
|
209
|
+
const timeline = createDebugTimeline({ maxEvents: 5000 });
|
|
210
|
+
const server = createDevtoolsServer({ timeline, port: 9229 });
|
|
211
|
+
|
|
212
|
+
server.start();
|
|
213
|
+
// Connect a devtools UI to ws://localhost:9229
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
## Anti-patterns
|
|
217
|
+
|
|
218
|
+
### `timeline.subscribe(listener, { filter })`
|
|
219
|
+
|
|
220
|
+
```typescript
|
|
221
|
+
// WRONG — subscribe takes only a listener
|
|
222
|
+
timeline.subscribe((e) => console.log(e), { filter: (e) => e.type === "agent_error" })
|
|
223
|
+
|
|
224
|
+
// CORRECT — filter inside the listener
|
|
225
|
+
timeline.subscribe((e) => {
|
|
226
|
+
if (e.type === "agent_error") console.log(e);
|
|
227
|
+
});
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
### `timeline.query({ type, since })`
|
|
231
|
+
|
|
232
|
+
```typescript
|
|
233
|
+
// WRONG — there is no .query() method
|
|
234
|
+
timeline.query({ type: "agent_error", since: Date.now() - 60_000 })
|
|
235
|
+
|
|
236
|
+
// CORRECT — typed getters
|
|
237
|
+
timeline.getEventsByType("agent_error")
|
|
238
|
+
.filter((e) => e.timestamp >= Date.now() - 60_000);
|
|
239
|
+
|
|
240
|
+
// Or use range directly
|
|
241
|
+
timeline.getEventsInRange(Date.now() - 60_000, Date.now());
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
### Nesting timeline / breakpoints inside `debug:`
|
|
245
|
+
|
|
246
|
+
```typescript
|
|
247
|
+
// WRONG — only verboseTimeline lives inside debug; breakpoints + timeline are top-level
|
|
248
|
+
createAgentOrchestrator({
|
|
249
|
+
runner,
|
|
250
|
+
debug: { timeline, verbose: true, breakpoints: [...] },
|
|
251
|
+
})
|
|
252
|
+
|
|
253
|
+
// CORRECT
|
|
254
|
+
createAgentOrchestrator({
|
|
255
|
+
runner,
|
|
256
|
+
debug: { verboseTimeline: true }, // OR just `debug: true`
|
|
257
|
+
breakpoints: [...], // top-level
|
|
258
|
+
onBreakpoint: (req) => orchestrator.resumeBreakpoint(req.id),
|
|
259
|
+
});
|
|
260
|
+
// timeline is read off `orchestrator.timeline` after construction
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
### `breakpoint.when(event)` / `breakpoint.onHit(event, resume)`
|
|
264
|
+
|
|
265
|
+
```typescript
|
|
266
|
+
// WRONG — these shapes don't exist
|
|
267
|
+
breakpoints: [
|
|
268
|
+
{ id: "x", when: (e) => e.type === "agent_start", onHit: (e, resume) => resume() },
|
|
269
|
+
]
|
|
270
|
+
|
|
271
|
+
// CORRECT — declarative before:/after: + typed filter, then orchestrator.resumeBreakpoint()
|
|
272
|
+
breakpoints: [
|
|
273
|
+
{ id: "x", before: "agent_start", filter: (e) => e.agentName === "writer" },
|
|
274
|
+
],
|
|
275
|
+
onBreakpoint: (req) => orchestrator.resumeBreakpoint(req.id),
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
### Restoring a checkpoint via the constructor
|
|
279
|
+
|
|
280
|
+
```typescript
|
|
281
|
+
// WRONG — there is no `checkpoint:` constructor option
|
|
282
|
+
createMultiAgentOrchestrator({ agents, patterns, runner, checkpoint: saved })
|
|
283
|
+
|
|
284
|
+
// CORRECT — orch.restore() on an existing instance
|
|
285
|
+
const orch = createMultiAgentOrchestrator({ agents, patterns, runner });
|
|
286
|
+
orch.restore(saved);
|
|
287
|
+
orch.start();
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
### Assuming `agent_complete.output` is `string`
|
|
291
|
+
|
|
292
|
+
```typescript
|
|
293
|
+
// WRONG — output is unknown, not string
|
|
294
|
+
timeline.subscribe((e) => {
|
|
295
|
+
if (e.type === "agent_complete") console.log(e.output.length); // type error
|
|
296
|
+
});
|
|
297
|
+
|
|
298
|
+
// CORRECT — narrow before reading
|
|
299
|
+
timeline.subscribe((e) => {
|
|
300
|
+
if (e.type === "agent_complete" && typeof e.output === "string") {
|
|
301
|
+
console.log(e.output.length);
|
|
302
|
+
}
|
|
303
|
+
});
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
## Quick reference
|
|
307
|
+
|
|
308
|
+
| API | Purpose |
|
|
309
|
+
|---|---|
|
|
310
|
+
| `createDebugTimeline(options?)` | Typed event recorder + queryable history |
|
|
311
|
+
| `timeline.subscribe(listener)` | Live event stream (no filter option) |
|
|
312
|
+
| `timeline.getEventsByType<T>(type)` | Typed-narrowed query |
|
|
313
|
+
| `timeline.getEventsInRange(start, end)` | Time-bounded query |
|
|
314
|
+
| `timeline.getEventsForAgent(agentId)` | Per-agent stream |
|
|
315
|
+
| `timeline.getEventsAtSnapshot(id)` | Snapshot-aware stream |
|
|
316
|
+
| `timeline.forkFrom(id)` | Truncate post-snapshot + reroute |
|
|
317
|
+
| `timeline.export()` / `import(json)` | JSON round-trip |
|
|
318
|
+
| `orchestrator.timeline` | Read the timeline back off the orchestrator (`null` when `debug: false`) |
|
|
319
|
+
| `breakpoints: [...]` + `onBreakpoint` | Top-level orchestrator options for pause points |
|
|
320
|
+
| `orchestrator.resumeBreakpoint(id, mods?)` / `cancelBreakpoint(id, reason?)` | Resolve a paused breakpoint |
|
|
321
|
+
| `createOtelPlugin(config)` | Plugin that emits spans for orchestrator activity |
|
|
322
|
+
| `createOTLPExporter(config)` | Returns `{ exportMetrics, exportTraces }` for OTel backends |
|
|
323
|
+
| `createDevtoolsServer({ timeline, port })` | WebSocket-bridged timeline for browser devtools |
|
|
324
|
+
|
|
325
|
+
## See also
|
|
326
|
+
|
|
327
|
+
- [`ai-orchestrator.md`](./ai-orchestrator.md) — `debug: { verboseTimeline }` config + `orchestrator.timeline` access + breakpoint surface
|
|
328
|
+
- [`ai-budget-resilience.md`](./ai-budget-resilience.md) — retry storms, circuit-breaker state changes, and budget warnings all flow through the debug timeline
|
|
329
|
+
- [`ai-testing-evals.md`](./ai-testing-evals.md) — `createTestTimeline` and `assertTimelineEvents` for asserting against recorded events
|
|
@@ -0,0 +1,293 @@
|
|
|
1
|
+
# AI MCP + RAG
|
|
2
|
+
|
|
3
|
+
> Covers `@directive-run/ai/mcp` and `@directive-run/ai` — MCP adapter (`createMCPAdapter`), RAG enricher, embedder utilities.
|
|
4
|
+
|
|
5
|
+
Model Context Protocol (MCP) server integration and Retrieval-Augmented Generation (RAG) enrichment for Directive AI agents. Import from `@directive-run/ai/mcp` for the MCP adapter (the main barrel re-exports it with `@deprecated` notices for v2 removal), and from `@directive-run/ai` for the RAG enricher + embedder utilities.
|
|
6
|
+
|
|
7
|
+
## Decision tree
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
What do you need?
|
|
11
|
+
├── External tool servers (MCP) → createMCPAdapter({ servers, ... })
|
|
12
|
+
│ ├── stdio transport → command-based MCP servers
|
|
13
|
+
│ └── SSE transport → HTTP-based MCP servers
|
|
14
|
+
│
|
|
15
|
+
├── Knowledge retrieval (RAG) → createRAGEnricher({ embedder, storage })
|
|
16
|
+
│ ├── User-supplied embeddings → EmbedderFn = (text) => Promise<number[]>
|
|
17
|
+
│ ├── Batched embedding calls → createBatchedEmbedder({ embed, batchSize, ... })
|
|
18
|
+
│ ├── Vector storage → createJSONFileStore(opts) or your own RAGStorage
|
|
19
|
+
│ └── Ingest documents → enricher.ingest(documents)
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## MCP server integration
|
|
23
|
+
|
|
24
|
+
The MCP adapter manages connections to one or more MCP servers, exposes their tools to your agents, and provides constraint-driven approval + risk-scoring for tool calls.
|
|
25
|
+
|
|
26
|
+
```typescript
|
|
27
|
+
import { createMCPAdapter, type MCPTool } from "@directive-run/ai/mcp";
|
|
28
|
+
|
|
29
|
+
const mcp = createMCPAdapter({
|
|
30
|
+
servers: [
|
|
31
|
+
{ name: "tools", transport: "stdio", command: "npx", args: ["mcp-server-tools"] },
|
|
32
|
+
{ name: "data", transport: "sse", url: "http://localhost:3001/sse" },
|
|
33
|
+
],
|
|
34
|
+
toolConstraints: {
|
|
35
|
+
"tools/dangerous-tool": { requireApproval: true, maxAttempts: 3 },
|
|
36
|
+
"tools/read-only": { requireApproval: false },
|
|
37
|
+
},
|
|
38
|
+
autoConnect: false, // default false — opt-in to connecting on creation
|
|
39
|
+
autoReconnect: true, // default false — opt-in to reconnect on disconnect
|
|
40
|
+
approvalTimeoutMs: 5 * 60 * 1000, // default 300_000
|
|
41
|
+
allowDirectCalls: false, // default false — must be true to use callToolDirect()
|
|
42
|
+
clientFactory: (cfg) => new Client(cfg), // optional — supply a real MCP SDK client
|
|
43
|
+
events: {
|
|
44
|
+
onConnect: ({ server }) => console.log(`mcp:${server} up`),
|
|
45
|
+
onDisconnect: ({ server, reason }) => console.warn(`mcp:${server} down — ${reason}`),
|
|
46
|
+
onToolCall: ({ server, tool, args }) => log("mcp:tool", { server, tool }),
|
|
47
|
+
onApproval: (request) => showApprovalDialog(request),
|
|
48
|
+
},
|
|
49
|
+
});
|
|
50
|
+
|
|
51
|
+
await mcp.connect();
|
|
52
|
+
|
|
53
|
+
// Get available tools — returns Map<serverName, MCPTool[]>, NOT a flat array
|
|
54
|
+
const toolsMap: Map<string, MCPTool[]> = mcp.getTools();
|
|
55
|
+
const flatTools = Array.from(toolsMap.values()).flat();
|
|
56
|
+
|
|
57
|
+
// Use tools with an agent
|
|
58
|
+
const agent = {
|
|
59
|
+
name: "researcher",
|
|
60
|
+
instructions: "Use available tools to research topics.",
|
|
61
|
+
model: "claude-sonnet-4-5",
|
|
62
|
+
tools: flatTools,
|
|
63
|
+
};
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## MCP server lifecycle
|
|
67
|
+
|
|
68
|
+
The lifecycle verbs are `connect` / `connectServer(name)` for opening connections, and `disconnect()` / `disconnectServer(name)` for closing them. There is no `mcp.disconnect("name")` mixed form, no `mcp.disconnectAll()`, and no `mcp.getStatus()`.
|
|
69
|
+
|
|
70
|
+
```typescript
|
|
71
|
+
// Connect everything
|
|
72
|
+
await mcp.connect();
|
|
73
|
+
|
|
74
|
+
// Or one server at a time
|
|
75
|
+
await mcp.connectServer("tools");
|
|
76
|
+
|
|
77
|
+
// Status — typed per-server
|
|
78
|
+
const single: MCPServerState | undefined = mcp.getServerStatus("tools");
|
|
79
|
+
const all: Map<string, MCPServerState> = mcp.getAllServerStatuses();
|
|
80
|
+
|
|
81
|
+
// Disconnect one server
|
|
82
|
+
await mcp.disconnectServer("tools");
|
|
83
|
+
|
|
84
|
+
// Disconnect all
|
|
85
|
+
await mcp.disconnect();
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Calling MCP tools with constraints
|
|
89
|
+
|
|
90
|
+
`callTool(server, tool, args, facts)` applies per-tool constraints (rate limits, approvals, argument-size caps). Use this in resolvers. `callToolDirect(server, tool, args)` bypasses the constraint pipeline — only available when `allowDirectCalls: true`.
|
|
91
|
+
|
|
92
|
+
```typescript
|
|
93
|
+
const result = await mcp.callTool(
|
|
94
|
+
"tools",
|
|
95
|
+
"search-docs",
|
|
96
|
+
{ query: "directive constraints" },
|
|
97
|
+
context.facts, // for constraint evaluation
|
|
98
|
+
);
|
|
99
|
+
|
|
100
|
+
// For trusted internal calls only:
|
|
101
|
+
const direct = await mcp.callToolDirect("tools", "internal-ping", {});
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
## Approval workflow for sensitive tools
|
|
105
|
+
|
|
106
|
+
When `toolConstraints[…].requireApproval: true`, the adapter queues an `MCPApprovalRequest` instead of executing immediately. Surface it via `events.onApproval` and resolve via the instance methods.
|
|
107
|
+
|
|
108
|
+
```typescript
|
|
109
|
+
const pending = mcp.getPendingApprovals();
|
|
110
|
+
mcp.approve(request.id);
|
|
111
|
+
mcp.reject(request.id, "violates data policy");
|
|
112
|
+
|
|
113
|
+
// Read the rejection reason for a previously-resolved request
|
|
114
|
+
const reason = mcp.getRejectionReason(request.id);
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
## Resource sync
|
|
118
|
+
|
|
119
|
+
MCP also exposes resources (read-only content the agent can pull). `syncResources` materializes them into Directive facts so constraints can react to them.
|
|
120
|
+
|
|
121
|
+
```typescript
|
|
122
|
+
const resourcesMap = mcp.getResources(); // Map<serverName, MCPResource[]>
|
|
123
|
+
const oneResource = await mcp.readResource("data", "file://config.yaml");
|
|
124
|
+
await mcp.syncResources(system.facts);
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## RAG enrichment
|
|
128
|
+
|
|
129
|
+
`createRAGEnricher` wires an embedder + a vector store into an agent's input pipeline, retrieving relevant context chunks before the agent runs.
|
|
130
|
+
|
|
131
|
+
```typescript
|
|
132
|
+
import {
|
|
133
|
+
createRAGEnricher,
|
|
134
|
+
createJSONFileStore,
|
|
135
|
+
createBatchedEmbedder,
|
|
136
|
+
createTestEmbedder,
|
|
137
|
+
type EmbedderFn,
|
|
138
|
+
type RAGEnricher,
|
|
139
|
+
} from "@directive-run/ai";
|
|
140
|
+
|
|
141
|
+
// 1. Supply an embedder. EmbedderFn = (text: string) => Promise<number[]>
|
|
142
|
+
// There is no createOpenAIEmbedder / createAnthropicEmbedder factory —
|
|
143
|
+
// you bring your own and pass it through createBatchedEmbedder for production.
|
|
144
|
+
const rawEmbed: EmbedderFn = async (text) => {
|
|
145
|
+
const res = await myEmbedAPI.embed(text);
|
|
146
|
+
return res.embedding;
|
|
147
|
+
};
|
|
148
|
+
|
|
149
|
+
const { embed } = createBatchedEmbedder({
|
|
150
|
+
embed: rawEmbed,
|
|
151
|
+
batchSize: 16,
|
|
152
|
+
flushIntervalMs: 50,
|
|
153
|
+
});
|
|
154
|
+
|
|
155
|
+
// 2. Storage — a JSON file store for prototyping, or your own RAGStorage adapter
|
|
156
|
+
const storage = createJSONFileStore({ path: "./rag-store.json" });
|
|
157
|
+
|
|
158
|
+
const enricher: RAGEnricher = createRAGEnricher({
|
|
159
|
+
embedder: embed,
|
|
160
|
+
storage,
|
|
161
|
+
topK: 5,
|
|
162
|
+
minSimilarity: 0.7,
|
|
163
|
+
});
|
|
164
|
+
|
|
165
|
+
// 3. Ingest documents
|
|
166
|
+
await enricher.ingest([
|
|
167
|
+
{ id: "doc-1", text: "Directive is a constraint-driven runtime…", metadata: { source: "README.md" } },
|
|
168
|
+
{ id: "doc-2", text: "Auto-tracked derivations recompute on read…", metadata: { source: "concepts" } },
|
|
169
|
+
]);
|
|
170
|
+
|
|
171
|
+
// 4. Use the enricher in your runner pipeline
|
|
172
|
+
const enrichedRunner: AgentRunner = async (agent, input, opts) => {
|
|
173
|
+
const enriched = await enricher.enrich(input);
|
|
174
|
+
return baseRunner(agent, enriched, opts);
|
|
175
|
+
};
|
|
176
|
+
|
|
177
|
+
const orchestrator = createAgentOrchestrator({ runner: enrichedRunner });
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
For testing without API calls, use `createTestEmbedder(dimensions?)` — deterministic, no network.
|
|
181
|
+
|
|
182
|
+
## Anti-patterns
|
|
183
|
+
|
|
184
|
+
### `mcp.disconnect(name)` / `mcp.disconnectAll()`
|
|
185
|
+
|
|
186
|
+
```typescript
|
|
187
|
+
// WRONG — neither shape exists
|
|
188
|
+
await mcp.disconnect("tools");
|
|
189
|
+
await mcp.disconnectAll();
|
|
190
|
+
|
|
191
|
+
// CORRECT
|
|
192
|
+
await mcp.disconnectServer("tools");
|
|
193
|
+
await mcp.disconnect();
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
### `mcp.getStatus()`
|
|
197
|
+
|
|
198
|
+
```typescript
|
|
199
|
+
// WRONG — no flat-object status method
|
|
200
|
+
const status = mcp.getStatus(); // ?? { tools: "connected", data: "connected" }
|
|
201
|
+
|
|
202
|
+
// CORRECT — typed per-server
|
|
203
|
+
const all = mcp.getAllServerStatuses(); // Map<name, MCPServerState>
|
|
204
|
+
const one = mcp.getServerStatus("tools"); // MCPServerState | undefined
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
### Treating `mcp.getTools()` as a flat array
|
|
208
|
+
|
|
209
|
+
```typescript
|
|
210
|
+
// WRONG — getTools() returns Map<serverName, MCPTool[]>
|
|
211
|
+
const flat: MCPTool[] = mcp.getTools();
|
|
212
|
+
agent.tools = flat;
|
|
213
|
+
|
|
214
|
+
// CORRECT — flatten if your agent expects a flat list
|
|
215
|
+
const flat = Array.from(mcp.getTools().values()).flat();
|
|
216
|
+
agent.tools = flat;
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
### Importing from `@directive-run/ai` only
|
|
220
|
+
|
|
221
|
+
```typescript
|
|
222
|
+
// WORKS, but fires v2-deprecation notices
|
|
223
|
+
import { createMCPAdapter } from "@directive-run/ai";
|
|
224
|
+
|
|
225
|
+
// PREFERRED — the subpath barrel is the v2-stable import
|
|
226
|
+
import { createMCPAdapter } from "@directive-run/ai/mcp";
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
### `MCPAdapterConfig` option names from a different library
|
|
230
|
+
|
|
231
|
+
```typescript
|
|
232
|
+
// WRONG — these options don't exist
|
|
233
|
+
createMCPAdapter({
|
|
234
|
+
servers,
|
|
235
|
+
connectionTimeout: 10_000,
|
|
236
|
+
reconnect: true,
|
|
237
|
+
})
|
|
238
|
+
|
|
239
|
+
// CORRECT — the real names
|
|
240
|
+
createMCPAdapter({
|
|
241
|
+
servers,
|
|
242
|
+
autoConnect: true,
|
|
243
|
+
autoReconnect: true,
|
|
244
|
+
approvalTimeoutMs: 300_000,
|
|
245
|
+
allowDirectCalls: false,
|
|
246
|
+
clientFactory: (cfg) => new Client(cfg),
|
|
247
|
+
})
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
### `createOpenAIEmbedder` / `createAnthropicEmbedder`
|
|
251
|
+
|
|
252
|
+
```typescript
|
|
253
|
+
// WRONG — no provider-specific embedder factories ship from @directive-run/ai
|
|
254
|
+
import { createOpenAIEmbedder } from "@directive-run/ai/openai";
|
|
255
|
+
const embedder = createOpenAIEmbedder({ apiKey });
|
|
256
|
+
|
|
257
|
+
// CORRECT — supply your own EmbedderFn, optionally batched
|
|
258
|
+
const rawEmbed: EmbedderFn = async (text) => (await openai.embeddings.create({ input: text, model: "text-embedding-3-small" })).data[0].embedding;
|
|
259
|
+
const { embed } = createBatchedEmbedder({ embed: rawEmbed, batchSize: 16 });
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
### Treating `Embedder` as an object with `embed(texts[]): number[][]`
|
|
263
|
+
|
|
264
|
+
```typescript
|
|
265
|
+
// WRONG — there is no Embedder interface; the EmbedderFn signature takes ONE string
|
|
266
|
+
type Embedder = { embed(texts: string[]): Promise<number[][]>; dimensions: number };
|
|
267
|
+
|
|
268
|
+
// CORRECT — single string in, single vector out
|
|
269
|
+
type EmbedderFn = (text: string) => Promise<number[]>;
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
## Quick reference
|
|
273
|
+
|
|
274
|
+
| API | Path | Returns | Purpose |
|
|
275
|
+
|---|---|---|---|
|
|
276
|
+
| `createMCPAdapter(config)` | `@directive-run/ai/mcp` | `MCPAdapter` | Connect to MCP servers, expose tools |
|
|
277
|
+
| `mcp.connect()` / `connectServer(name)` | instance method | `Promise<void>` | Open one or all connections |
|
|
278
|
+
| `mcp.disconnect()` / `disconnectServer(name)` | instance method | `Promise<void>` | Close one or all connections |
|
|
279
|
+
| `mcp.getTools()` | instance method | `Map<string, MCPTool[]>` | All tools, keyed by server |
|
|
280
|
+
| `mcp.getResources()` | instance method | `Map<string, MCPResource[]>` | All resources, keyed by server |
|
|
281
|
+
| `mcp.callTool(server, tool, args, facts)` | instance method | `Promise<MCPToolResult>` | Constraint-checked tool call |
|
|
282
|
+
| `mcp.callToolDirect(server, tool, args)` | instance method | `Promise<MCPToolResult>` | Bypass constraints (requires `allowDirectCalls: true`) |
|
|
283
|
+
| `mcp.getServerStatus(name)` / `getAllServerStatuses()` | instance method | `MCPServerState` | Per-server connection state |
|
|
284
|
+
| `mcp.approve(id)` / `reject(id, reason?)` | instance method | `void` | Resolve a pending approval |
|
|
285
|
+
| `createRAGEnricher(config)` | `@directive-run/ai` | `RAGEnricher` | Embedding-driven context retrieval |
|
|
286
|
+
| `createBatchedEmbedder(config)` | `@directive-run/ai` | `{ embed, flush, dispose }` | Wrap an EmbedderFn with batching |
|
|
287
|
+
| `createTestEmbedder(dim?)` | `@directive-run/ai` | `EmbedderFn` | Deterministic embedder for tests |
|
|
288
|
+
| `createJSONFileStore(options)` | `@directive-run/ai` | `RAGStorage` | File-backed vector store for prototyping |
|
|
289
|
+
|
|
290
|
+
## See also
|
|
291
|
+
|
|
292
|
+
- [`ai-orchestrator.md`](./ai-orchestrator.md) — where MCP-supplied tools and RAG-enriched runners flow into agent runs
|
|
293
|
+
- [`ai-security.md`](./ai-security.md) — `toolConstraints[…].requireApproval` and the per-tool approval workflow the MCP adapter exposes
|