@directive-run/knowledge 0.2.0

package/ai/ai-mcp-rag.md

@@ -0,0 +1,288 @@
+# AI MCP and RAG
+
+MCP (Model Context Protocol) server integration and RAG (Retrieval-Augmented Generation) enrichment for Directive AI agents.
+
+## Decision Tree: "How do I connect external tools or knowledge?"
+
+```
+What do you need?
+├── External tool servers (MCP) → createMCPAdapter({ servers: [...] })
+│   ├── stdio transport      → command-based MCP servers
+│   └── SSE transport        → HTTP-based MCP servers
+│
+├── Knowledge retrieval (RAG) → createRAGEnricher({ embedder, storage })
+│   ├── Need embeddings       → createOpenAIEmbedder() or createAnthropicEmbedder()
+│   ├── Need vector storage   → createJSONFileStore() or custom VectorStore
+│   └── Need chunk ingestion  → enricher.ingest(documents)
+│
+Where do I import from?
+├── MCP adapter → import { createMCPAdapter } from '@directive-run/ai'
+├── RAG enricher → import { createRAGEnricher } from '@directive-run/ai'
+└── Embedders → import from '@directive-run/ai/openai' (subpath)
+```
+
+## MCP Server Integration
+
+Connect to MCP servers to give agents access to external tools:
+
+```typescript
+import { createMCPAdapter } from "@directive-run/ai";
+
+const mcp = createMCPAdapter({
+  servers: [
+    // stdio transport — runs a local process
+    {
+      name: "tools",
+      transport: "stdio",
+      command: "npx mcp-server-tools",
+    },
+    // SSE transport — connects to an HTTP server
+    {
+      name: "data",
+      transport: "sse",
+      url: "http://localhost:3001/sse",
+    },
+  ],
+
+  // Per-tool constraints
+  toolConstraints: {
+    "tools/dangerous-tool": {
+      requireApproval: true,
+      maxAttempts: 3,
+    },
+    "tools/read-only": {
+      requireApproval: false,
+    },
+  },
+
+  // Connection options
+  connectionTimeout: 10000,
+  reconnect: true,
+});
+
+// Connect to all servers
+await mcp.connect();
+
+// Get available tools (normalized for Directive agents)
+const tools = mcp.getTools();
+
+// Use tools with an agent
+const agent = {
+  name: "researcher",
+  instructions: "Use available tools to research topics.",
+  model: "claude-sonnet-4-5",
+  tools: tools,
+};
+```
+
+### Anti-Pattern #36: Importing MCP from subpath
+
+```typescript
+// WRONG — there is no /mcp subpath export
+import { createMCPToolProvider } from "@directive-run/ai/mcp";
+
+// CORRECT — MCP adapter is exported from the main package
+import { createMCPAdapter } from "@directive-run/ai";
+```
+
+## MCP Server Lifecycle
+
+```typescript
+// Connect to all configured servers
+await mcp.connect();
+
+// Check server status
+const status = mcp.getStatus();
+// { tools: "connected", data: "connected" }
+
+// Disconnect a specific server
+await mcp.disconnect("tools");
+
+// Disconnect all servers
+await mcp.disconnectAll();
+
+// Reconnect after disconnect
+await mcp.connect();
+```
+
+## MCP with Orchestrator
+
+```typescript
+import { createAgentOrchestrator } from "@directive-run/ai";
+
+const mcp = createMCPAdapter({
+  servers: [
+    { name: "tools", transport: "stdio", command: "npx mcp-server-tools" },
+  ],
+});
+
+await mcp.connect();
+
+const orchestrator = createAgentOrchestrator({
+  runner,
+  hooks: {
+    onStart: async () => {
+      await mcp.connect();
+    },
+  },
+});
+
+const agent = {
+  name: "worker",
+  instructions: "Complete tasks using available tools.",
+  model: "claude-sonnet-4-5",
+  tools: mcp.getTools(),
+};
+
+const result = await orchestrator.run(agent, "Search for recent AI papers");
+```
+
+---
+
+## RAG Enrichment
+
+Augment agent prompts with relevant context from a knowledge base:
+
+```typescript
+import { createRAGEnricher } from "@directive-run/ai";
+import { createOpenAIEmbedder } from "@directive-run/ai/openai";
+
+const enricher = createRAGEnricher({
+  // Embedder for similarity search
+  embedder: createOpenAIEmbedder({
+    apiKey: process.env.OPENAI_API_KEY,
+  }),
+
+  // Vector storage backend
+  storage: createJSONFileStore({ filePath: "./chunks.json" }),
+
+  // Retrieval settings
+  topK: 5,              // Max chunks to retrieve
+  minSimilarity: 0.3,   // Minimum cosine similarity threshold
+
+  // Format each retrieved chunk
+  formatChunk: (chunk, similarity) => {
+    return `[${similarity.toFixed(2)}] ${chunk.content}`;
+  },
+});
+```
+
+## Ingesting Documents
+
+```typescript
+// Ingest raw text with metadata
+await enricher.ingest([
+  {
+    content: "Directive uses proxy-based facts for auto-tracking.",
+    metadata: { source: "docs", topic: "facts" },
+  },
+  {
+    content: "Derivations are auto-tracked computed values.",
+    metadata: { source: "docs", topic: "derivations" },
+  },
+]);
+
+// Ingest from files (chunks automatically)
+await enricher.ingestFile("./docs/architecture.md", {
+  chunkSize: 500,
+  chunkOverlap: 50,
+  metadata: { source: "architecture" },
+});
+```
+
+## Enriching Prompts
+
+```typescript
+// Basic enrichment — prepends relevant context
+const enrichedInput = await enricher.enrich("How do facts work?", {
+  prefix: "Use this context to answer:\n",
+});
+// Result: "Use this context to answer:\n[0.92] Directive uses proxy-based..."
+
+// With conversation history for better retrieval
+const enrichedInput = await enricher.enrich("Tell me more about that", {
+  prefix: "Use this context:\n",
+  history: messages,
+});
+```
+
+## RAG with Orchestrator
+
+```typescript
+import { createAgentOrchestrator } from "@directive-run/ai";
+
+const orchestrator = createAgentOrchestrator({
+  runner,
+  hooks: {
+    onBeforeRun: async (agent, prompt) => {
+      // Enrich every prompt with relevant context
+      const enriched = await enricher.enrich(prompt, {
+        prefix: "Relevant context:\n",
+      });
+
+      return { approved: true, modifiedPrompt: enriched };
+    },
+  },
+});
+```
+
+## Custom Embedders
+
+Implement the `Embedder` interface for any provider:
+
+```typescript
+import type { Embedder } from "@directive-run/ai";
+
+const customEmbedder: Embedder = {
+  embed: async (texts: string[]) => {
+    // Return float arrays, one per input text
+    const embeddings = await myEmbeddingAPI.embed(texts);
+
+    return embeddings.map((e) => e.vector);
+  },
+
+  dimensions: 1536, // Vector dimensions
+};
+
+const enricher = createRAGEnricher({
+  embedder: customEmbedder,
+  storage: createJSONFileStore({ filePath: "./chunks.json" }),
+  topK: 5,
+  minSimilarity: 0.3,
+});
+```
+
+## Custom Vector Storage
+
+Implement the `VectorStore` interface for any backend:
+
+```typescript
+import type { VectorStore } from "@directive-run/ai";
+
+const pgStore: VectorStore = {
+  add: async (chunks) => {
+    await db.query("INSERT INTO chunks ...", chunks);
+  },
+  search: async (vector, topK) => {
+    const results = await db.query(
+      "SELECT * FROM chunks ORDER BY embedding <=> $1 LIMIT $2",
+      [vector, topK],
+    );
+
+    return results.rows;
+  },
+  clear: async () => {
+    await db.query("DELETE FROM chunks");
+  },
+};
+```
+
+## Quick Reference
+
+| API | Import Path | Purpose |
+|---|---|---|
+| `createMCPAdapter` | `@directive-run/ai` | Connect to MCP tool servers |
+| `createRAGEnricher` | `@directive-run/ai` | RAG pipeline for prompt enrichment |
+| `createOpenAIEmbedder` | `@directive-run/ai/openai` | OpenAI text embeddings |
+| `createAnthropicEmbedder` | `@directive-run/ai/anthropic` | Anthropic text embeddings |
+| `createJSONFileStore` | `@directive-run/ai` | File-based vector storage |

package/ai/ai-multi-agent.md

@@ -0,0 +1,274 @@
+# AI Multi-Agent Orchestrator
+
+`createMultiAgentOrchestrator` coordinates multiple agents using 8 composition patterns. Each agent becomes a namespaced Directive module with a shared coordinator.
+
+## Decision Tree: "Which pattern do I need?"
+
+```
+How should agents interact?
+├── Independent, combine results → parallel()
+├── One feeds the next        → sequential()
+├── One agent delegates        → supervisor()
+├── Complex dependency graph   → dag()
+├── Agent critiques own output → reflect()
+├── First to finish wins      → race()
+├── Agents argue to consensus → debate()
+└── Iterate until goal met    → goal()
+```
+
+## Basic Setup
+
+```typescript
+import {
+  createMultiAgentOrchestrator,
+  parallel,
+  sequential,
+  supervisor,
+  dag,
+  reflect,
+  race,
+  debate,
+  goal,
+} from "@directive-run/ai";
+import { createAnthropicRunner } from "@directive-run/ai/anthropic";
+
+const runner = createAnthropicRunner({
+  apiKey: process.env.ANTHROPIC_API_KEY,
+});
+
+const orchestrator = createMultiAgentOrchestrator({
+  agents: {
+    researcher: {
+      name: "researcher",
+      instructions: "Research the topic thoroughly.",
+      model: "claude-sonnet-4-5",
+    },
+    writer: {
+      name: "writer",
+      instructions: "Write clear, engaging content.",
+      model: "claude-sonnet-4-5",
+    },
+    editor: {
+      name: "editor",
+      instructions: "Edit for clarity and correctness.",
+      model: "claude-haiku-3-5",
+    },
+  },
+  patterns: {
+    pipeline: sequential(["researcher", "writer", "editor"]),
+    brainstorm: parallel(["researcher", "writer"], mergeResults),
+    managed: supervisor("editor", ["researcher", "writer"]),
+    workflow: dag([
+      { id: "research", handler: "researcher" },
+      { id: "write", handler: "writer", dependencies: ["research"] },
+      { id: "edit", handler: "editor", dependencies: ["write"] },
+    ]),
+  },
+  runner,
+});
+
+// REQUIRED for multi-agent — must call start() before running patterns
+orchestrator.start();
+
+const result = await orchestrator.runPattern("pipeline", "Write about AI");
+```
+
+## Pattern Details
+
+### parallel — Run agents concurrently, merge results
+
+```typescript
+const brainstorm = parallel(
+  ["researcher", "writer"],
+  (results) => {
+    // results: Map<string, RunResult>
+    const combined = Array.from(results.values())
+      .map((r) => r.output)
+      .join("\n\n");
+
+    return combined;
+  },
+);
+```
+
+### sequential — Chain agents in order
+
+```typescript
+// Each agent receives the previous agent's output as its prompt
+const pipeline = sequential(["researcher", "writer", "editor"]);
+```
+
+### supervisor — One agent delegates to workers
+
+```typescript
+// Editor decides which worker to invoke and when to stop
+const managed = supervisor("editor", ["researcher", "writer"]);
+```
+
+### dag — Directed acyclic graph of dependencies
+
+```typescript
+// DagNode shape
+interface DagNode {
+  id: string;
+  handler: string;          // agent name
+  dependencies?: string[];  // node IDs this depends on
+  transform?: (input: string, depResults: Map<string, string>) => string;
+}
+
+const workflow = dag([
+  { id: "research", handler: "researcher" },
+  {
+    id: "write",
+    handler: "writer",
+    dependencies: ["research"],
+    transform: (input, deps) => {
+      const research = deps.get("research");
+
+      return `Based on research:\n${research}\n\nWrite about: ${input}`;
+    },
+  },
+  { id: "edit", handler: "editor", dependencies: ["write"] },
+]);
+```
+
+### reflect — Agent critiques and revises its own output
+
+```typescript
+const selfImprove = reflect("writer", {
+  maxIterations: 3,
+  stopWhen: (output, iteration) => {
+    return output.includes("FINAL") || iteration >= 3;
+  },
+});
+```
+
+### race — First agent to finish wins
+
+```typescript
+const fastest = race(["researcher", "writer"], {
+  minSuccess: 1,       // How many must complete (default: 1)
+  timeout: 30000,      // Cancel remaining after timeout
+});
+```
+
+### debate — Agents argue to consensus
+
+```typescript
+const consensus = debate(["researcher", "writer"], {
+  maxRounds: 5,
+  judge: "editor",     // Agent that decides when consensus is reached
+});
+```
+
+### goal — Iterate until a condition is met
+
+```typescript
+const iterative = goal("researcher", {
+  maxIterations: 10,
+  goalCheck: (output, facts) => {
+    return facts.confidence > 0.9;
+  },
+});
+```
+
+## Fact Propagation
+
+Each agent has its own namespaced facts. The coordinator module (`__coord`) holds shared state:
+
+```typescript
+// Read agent-specific facts
+orchestrator.system.facts.researcher.status;
+orchestrator.system.facts.writer.lastOutput;
+
+// Read coordinator facts
+orchestrator.system.facts.__coord.activePattern;
+orchestrator.system.facts.__coord.completedAgents;
+```
+
+## Checkpoint Serialization
+
+```typescript
+// Save entire multi-agent state
+const checkpoint = orchestrator.checkpoint();
+const serialized = JSON.stringify(checkpoint);
+
+// Restore from checkpoint
+const restored = createMultiAgentOrchestrator({
+  agents,
+  patterns,
+  runner,
+  checkpoint: JSON.parse(serialized),
+});
+restored.start();
+```
+
+## Tasks in Multi-Agent Patterns
+
+Tasks and agents share the DAG/sequential/parallel namespace. See `ai-tasks.md` for details.
+
+```typescript
+const workflow = dag([
+  { id: "research", handler: "researcher" },
+  { id: "format", handler: "formatter-task" },  // task, not agent
+  { id: "edit", handler: "editor", dependencies: ["research", "format"] },
+]);
+```
+
+## Anti-Patterns
+
+### #24: Forgetting start() for multi-agent
+
+```typescript
+// WRONG — multi-agent orchestrators require explicit start()
+const orchestrator = createMultiAgentOrchestrator({ agents, runner });
+const result = await orchestrator.runPattern("pipeline", "prompt");
+// Error: Orchestrator not started
+
+// CORRECT
+const orchestrator = createMultiAgentOrchestrator({ agents, runner });
+orchestrator.start();
+const result = await orchestrator.runPattern("pipeline", "prompt");
+```
+
+### #30: race minSuccess greater than agent count
+
+```typescript
+// WRONG — minSuccess cannot exceed the number of agents
+const broken = race(["researcher", "writer"], {
+  minSuccess: 3, // Only 2 agents, will never satisfy
+});
+
+// CORRECT — minSuccess <= agents.length
+const working = race(["researcher", "writer"], {
+  minSuccess: 1,
+});
+```
+
+### Reusing agent names across patterns
+
+```typescript
+// WRONG — agent names must match keys in the agents config
+patterns: {
+  pipeline: sequential(["research-agent", "write-agent"]),
+  // These don't match the keys "researcher", "writer"
+},
+
+// CORRECT — use the exact keys from agents config
+patterns: {
+  pipeline: sequential(["researcher", "writer"]),
+},
+```
+
+## Quick Reference
+
+| Pattern | Use Case | Key Option |
+|---|---|---|
+| `parallel()` | Independent work, merge results | merge function |
+| `sequential()` | Pipeline, each feeds next | agent order |
+| `supervisor()` | Dynamic delegation | supervisor agent |
+| `dag()` | Complex dependencies | DagNode[] |
+| `reflect()` | Self-improvement loop | maxIterations, stopWhen |
+| `race()` | Fastest wins | minSuccess, timeout |
+| `debate()` | Consensus building | maxRounds, judge |
+| `goal()` | Condition-driven iteration | goalCheck |