@directive-run/knowledge 1.14.0 → 1.16.0

package/ai/ai-budget-resilience.md

@@ -1,235 +1,408 @@
-# AI Budget and Resilience
+# AI budget + resilience
 
-Budget wrappers, retry policies, fallback chains, circuit breakers, health monitors, semantic caching, and constraint-driven provider routing for AI runners.
+> Covers `@directive-run/ai` — `withBudget`, `withRetry`, `withFallback`, `createCircuitBreaker`, `createHealthMonitor`, `createSemanticCache`, `createConstraintRouter`.
 
-## Decision Tree: "How do I protect my AI calls?"
+Runner wrappers and supporting utilities for cost control, retry, fallback, circuit breaking, fleet health tracking, semantic caching, and constraint-driven provider routing. Compose these around any `AgentRunner` before passing it to an orchestrator.
+
+## Decision tree
 
 ```
 What failure mode are you guarding against?
-├── Cost overruns → withBudget(runner, { budgets: [...] })
-├── Transient errors → withRetry(runner, { maxRetries, backoff })
-├── Provider outage → withFallback(primaryRunner, fallbackRunner)
-├── Repeated failures → createCircuitBreaker({ failureThreshold })
-├── Fleet monitoring → createHealthMonitor({ agents: {...} })
-│
-Want to avoid redundant LLM calls?
-├── Yes, similar inputs → createSemanticCache({ embedder, similarity })
-│
-Need dynamic provider selection?
-└── Yes → createConstraintRouter({ providers: [...], constraints: [...] })
+├── Cost overruns           → withBudget(runner, config)
+├── Transient errors        → withRetry(runner, config)
+├── Provider outage         → withFallback([primary, backup], config?)
+├── Repeated cascading fails → createCircuitBreaker(config) + breaker.execute(fn)
+├── Fleet-wide health       → createHealthMonitor() + recordSuccess/Failure
+├── Redundant LLM calls    → createSemanticCache(config) + createSemanticCacheGuardrail
+└── Dynamic provider pick  → createConstraintRouter({ providers, constraints })
 ```
 
-## Budget Wrapping
+## `withBudget(runner, config)` — cost caps
 
-Wrap any runner with cost tracking and enforcement per time window:
+`BudgetConfig` controls per-call and per-window cost limits. The `onBudgetExceeded` callback fires before the wrapper throws `BudgetExceededError`. There is NO `budgetWarningThreshold` or `onWarning` option on this wrapper — for percentage-based warnings, use `onBudgetWarning` on `createAgentOrchestrator` instead.
 
 ```typescript
 import { withBudget } from "@directive-run/ai";
 
+const pricing = { inputPerMillion: 3, outputPerMillion: 15 };
+
 const budgetRunner = withBudget(baseRunner, {
-  // Hard cap per single LLM call
   maxCostPerCall: 0.10,
-
-  // Time-window budgets (multiple allowed)
+  pricing,                       // required when maxCostPerCall is set
+  estimatedOutputMultiplier: 1.0, // for summarization use 0.3; for generation use 3.0
+  charsPerToken: 4,
   budgets: [
-    {
-      window: "hour",
-      maxCost: 1.0,
-      pricing: { inputPerMillion: 3, outputPerMillion: 15 },
-    },
-    {
-      window: "day",
-      maxCost: 10.0,
-      pricing: { inputPerMillion: 3, outputPerMillion: 15 },
-    },
+    { window: "hour", maxCost: 5.00, pricing },
+    { window: "day",  maxCost: 50.00, pricing },
   ],
-
-  // Callback when approaching limit (0-1 percentage)
-  budgetWarningThreshold: 0.8,
-  onWarning: (usage) => {
-    console.warn(`Budget at ${(usage.percentage * 100).toFixed(0)}%`);
+  onBudgetExceeded: ({ estimated, remaining, window }) => {
+    console.warn(`[budget] ${window} would overshoot — est $${estimated.toFixed(4)}, remaining $${remaining.toFixed(4)}`);
   },
 });
 ```
 
-### Anti-Pattern #29: budgetWarningThreshold out of range
+Catching the throw:
 
 ```typescript
-// WRONG – threshold must be a 0-1 percentage
-const budgetRunner = withBudget(baseRunner, {
-  budgetWarningThreshold: 80, // Not a percentage!
-  budgets: [{ window: "hour", maxCost: 1.0, pricing: { inputPerMillion: 3, outputPerMillion: 15 } }],
-});
-
-// CORRECT – use a decimal between 0 and 1
-const budgetRunner = withBudget(baseRunner, {
-  budgetWarningThreshold: 0.8, // 80%
-  budgets: [{ window: "hour", maxCost: 1.0, pricing: { inputPerMillion: 3, outputPerMillion: 15 } }],
-});
+import { BudgetExceededError } from "@directive-run/ai";
+
+try {
+  await budgetRunner(agent, prompt);
+} catch (err) {
+  if (err instanceof BudgetExceededError) {
+    console.log(err.window, err.estimated, err.remaining);
+  }
+}
 ```
 
-## Retry Policies
+## `withRetry(runner, config)` — transient-error retry
 
-Wrap a runner with automatic retry on transient failures:
+Exponential backoff with jitter is hard-coded; `backoff: "linear"` / `"none"` do NOT exist. The retry decision goes through built-in HTTP-status checks (non-retryable: 400 / 401 / 403 / 404 / 422) then your `isRetryable` predicate.
 
 ```typescript
 import { withRetry } from "@directive-run/ai";
 
 const retryRunner = withRetry(baseRunner, {
-  maxRetries: 3,
-  backoff: "exponential", // "exponential" | "linear" | "none"
-  baseDelayMs: 100,
-
-  // Only retry specific errors
-  shouldRetry: (error) => {
-    return error.status === 429 || error.status >= 500;
+  maxRetries: 3,                    // default 3
+  baseDelayMs: 1000,                // default 1000
+  maxDelayMs: 30_000,               // default 30000
+  isRetryable: (error) => {
+    const msg = error.message.toLowerCase();
+    return msg.includes("rate_limit") || msg.includes("503") || msg.includes("504");
+  },
+  onRetry: (attempt, error, delayMs) => {
+    console.log(`retry ${attempt} in ${delayMs}ms — ${error.message}`);
   },
 });
 ```
 
-## Fallback Chains
+Catching the exhausted throw:
+
+```typescript
+import { RetryExhaustedError } from "@directive-run/ai";
+
+try {
+  await retryRunner(agent, prompt);
+} catch (err) {
+  if (err instanceof RetryExhaustedError) {
+    console.log(`gave up after ${err.retryCount} retries; last: ${err.lastError.message}`);
+  }
+}
+```
+
+## `withFallback(runners[], config?)` — provider chain
 
-Automatically switch to a backup runner when the primary fails:
+**`withFallback` takes an ARRAY of runners as positional arg 1**, then an optional config. The chain tries each runner in order; the first success wins. When all fail, `AllProvidersFailedError` is thrown carrying all per-runner errors.
 
 ```typescript
-import { withFallback } from "@directive-run/ai";
+import { withFallback, withRetry } from "@directive-run/ai";
 import { createAnthropicRunner } from "@directive-run/ai/anthropic";
 import { createOpenAIRunner } from "@directive-run/ai/openai";
+import { createOllamaRunner } from "@directive-run/ai/ollama";
 
-const primary = createAnthropicRunner({ apiKey: process.env.ANTHROPIC_API_KEY });
-const backup = createOpenAIRunner({ apiKey: process.env.OPENAI_API_KEY });
+const resilient = withFallback(
+  [
+    withRetry(createAnthropicRunner({ apiKey: process.env.ANTHROPIC_API_KEY! }), { maxRetries: 2 }),
+    withRetry(createOpenAIRunner({ apiKey: process.env.OPENAI_API_KEY! }),     { maxRetries: 2 }),
+    createOllamaRunner({ baseUrl: "http://localhost:11434" }),
+  ],
+  {
+    shouldFallback: (error) => !error.message.includes("invalid_api_key"),
+    onFallback: (fromIndex, toIndex, error) => {
+      console.log(`falling back from runner ${fromIndex} to ${toIndex}: ${error.message}`);
+    },
+  },
+);
+```
 
-// Falls back to backup when primary throws
-const resilientRunner = withFallback(primary, backup);
+Catching all-failed:
+
+```typescript
+import { AllProvidersFailedError } from "@directive-run/ai";
+
+try {
+  await resilient(agent, prompt);
+} catch (err) {
+  if (err instanceof AllProvidersFailedError) {
+    err.errors.forEach((e, i) => console.log(`[${i}] ${e.message}`));
+  }
+}
 ```
 
-## Circuit Breaker
+## `createCircuitBreaker(config)` — failure isolation
 
-Prevent repeated calls to a failing provider. Opens after N failures, resets after a timeout:
+The circuit breaker is from `@directive-run/core` (it's not AI-specific). Use `breaker.execute(fn)` to run an arbitrary async closure inside the breaker; there is NO `breaker.wrap(runner)` shortcut.
 
 ```typescript
-import { createCircuitBreaker } from "@directive-run/ai";
+import { createCircuitBreaker, CircuitBreakerOpenError } from "@directive-run/core";
 
 const breaker = createCircuitBreaker({
-  failureThreshold: 3,     // Open after 3 consecutive failures
-  resetTimeout: 30000,     // Try again after 30s (half-open state)
-  halfOpenMaxAttempts: 1,  // Allow 1 test request in half-open
+  name: "openai",
+  failureThreshold: 5,         // default 5
+  recoveryTimeMs: 30_000,      // default 30000
+  halfOpenMaxRequests: 3,      // default 3 — concurrent test requests when half-open
+  failureWindowMs: 60_000,     // default 60000
 });
 
-// Use with a runner
-const protectedRunner = breaker.wrap(baseRunner);
+// Use with a runner: wrap the call inside execute()
+async function callAgent(agent: AgentLike, input: string) {
+  return await breaker.execute(() => baseRunner(agent, input));
+}
+
+// Inspect state
+console.log(breaker.getState());       // "CLOSED" | "OPEN" | "HALF_OPEN"
+console.log(breaker.getStats());       // { successCount, failureCount, lastFailureTime, ... }
+console.log(breaker.isAllowed());      // boolean — would a request be allowed now?
 
-// Check state
-console.log(breaker.state); // "closed" | "open" | "half-open"
-breaker.reset();            // Force back to closed
+breaker.reset();                       // force back to CLOSED, clear stats
+breaker.forceState("OPEN");            // testing helper
 ```
 
-### Anti-Pattern #28: Sharing a CircuitBreaker across unrelated agents
+Catching the OPEN throw:
 
 ```typescript
-// WRONG – one failing agent opens the breaker for all agents
-const sharedBreaker = createCircuitBreaker({ failureThreshold: 3, resetTimeout: 30000 });
-const researchRunner = sharedBreaker.wrap(baseRunner);
-const writerRunner = sharedBreaker.wrap(baseRunner); // Same breaker!
+try {
+  await breaker.execute(() => baseRunner(agent, prompt));
+} catch (err) {
+  if (err instanceof CircuitBreakerOpenError) {
+    console.log(err.code);         // "CIRCUIT_OPEN"
+    console.log(err.state);        // "OPEN" | "HALF_OPEN"
+    console.log(err.retryAfterMs); // wait this long before next attempt
+  }
+}
+```
 
-// CORRECT – each agent gets its own breaker instance
-const researchBreaker = createCircuitBreaker({ failureThreshold: 3, resetTimeout: 30000 });
-const writerBreaker = createCircuitBreaker({ failureThreshold: 3, resetTimeout: 30000 });
+### Don't share a breaker across unrelated runners
 
-const researchRunner = researchBreaker.wrap(baseRunner);
-const writerRunner = writerBreaker.wrap(baseRunner);
+```typescript
+// WRONG — one failing runner trips the breaker for all of them
+const shared = createCircuitBreaker({ name: "shared" });
+async function research(agent, input) { return shared.execute(() => researchRunner(agent, input)); }
+async function write(agent, input)    { return shared.execute(() => writerRunner(agent, input));   }
+
+// CORRECT — one breaker per failure domain
+const researchBreaker = createCircuitBreaker({ name: "anthropic-research" });
+const writerBreaker   = createCircuitBreaker({ name: "openai-writer" });
 ```
 
-## Health Monitor
+## `createHealthMonitor(config?)` — fleet health metrics
 
-Monitor agent health across the system, track circuit breaker states, and report status:
+`createHealthMonitor` is a metrics tracker: you record successes/failures into it; it computes per-agent health scores you can use to drive routing or observability. It is NOT an auto-polling daemon. There is no `monitor.start()`, no `monitor.stop()`, no `monitor.getReport()`, no `agents:` config, no `checkInterval`, and no `onStatusChange` callback — those don't exist.
 
 ```typescript
 import { createHealthMonitor } from "@directive-run/ai";
 
 const monitor = createHealthMonitor({
-  agents: {
-    researcher: { runner: researchRunner, circuitBreaker: researchBreaker },
-    writer: { runner: writerRunner, circuitBreaker: writerBreaker },
-  },
-  checkInterval: 60000, // Health check every 60s
-
-  onStatusChange: (agent, status) => {
-    console.log(`${agent}: ${status}`); // "healthy" | "degraded" | "unhealthy"
+  windowMs: 60_000,           // rolling window for metrics (default 60s)
+  maxNormalLatencyMs: 5000,   // above this counts as "slow" for the latency component
+  weights: {                  // weights for the composite health score
+    successRate: 0.5,
+    latency:     0.3,
+    circuitState: 0.2,
   },
 });
 
-monitor.start();
-const report = monitor.getReport();
-monitor.stop();
+// Record outcomes from your runner calls
+const start = Date.now();
+try {
+  const result = await baseRunner(agent, input);
+  monitor.recordSuccess("researcher", Date.now() - start);
+} catch (err) {
+  monitor.recordFailure("researcher", Date.now() - start, err as Error);
+}
+
+// If you also use a circuit breaker, mirror its state into the monitor
+monitor.updateCircuitState("researcher", breaker.getState() === "OPEN" ? "open" : "closed");
+
+// Read
+console.log(monitor.getHealthScore("researcher"));  // 0-100 (50 when no data)
+console.log(monitor.getMetrics("researcher"));      // { successRate, avgLatency, ... }
+console.log(monitor.getAllMetrics());                // Record<agentId, metrics>
+
+monitor.reset();
 ```
 
-## Semantic Cache
+## `createSemanticCache(config)` — embedding-based response cache
 
-Cache LLM responses by semantic similarity to avoid redundant calls:
+Semantic caching avoids redundant LLM calls when a new input is close enough to one already seen. Wire it into an orchestrator via the cache guardrail factory — there is NO `cache.wrap(runner)` method.
 
 ```typescript
-import { createSemanticCache } from "@directive-run/ai";
-import { createOpenAIEmbedder } from "@directive-run/ai/openai";
+import { createSemanticCache, createSemanticCacheGuardrail, type EmbedderFn } from "@directive-run/ai";
+
+// You supply the embedder — no createOpenAIEmbedder / createAnthropicEmbedder exists.
+// `EmbedderFn = (text: string) => Promise<number[]>`
+const embedder: EmbedderFn = async (text) => {
+  // call your embedding model (OpenAI, Voyage, local model, …)
+  return await myEmbedAPI(text);
+};
 
 const cache = createSemanticCache({
-  embedder: createOpenAIEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
-  similarity: 0.98,   // Minimum cosine similarity to count as a hit
-  maxSize: 1000,       // Max cached entries (LRU eviction)
-  ttl: 3600000,        // Cache entry TTL in ms (1 hour)
+  embedder,
+  similarityThreshold: 0.95,
+  maxCacheSize: 1000,
+  ttlMs: 60 * 60 * 1000, // 1 hour
+  onHit:  (entry, sim) => console.log(`cache hit @ ${(sim * 100).toFixed(1)}%`),
+  onMiss: (query) => console.log(`miss: ${query.slice(0, 40)}…`),
 });
 
-// Wrap a runner with caching
-const cachedRunner = cache.wrap(baseRunner);
+const cacheGuardrail = createSemanticCacheGuardrail({ cache });
+
+const orchestrator = createAgentOrchestrator({
+  runner: baseRunner,
+  guardrails: {
+    input: [cacheGuardrail], // short-circuits with the cached response when a hit fires
+  },
+});
 ```
 
-## Constraint-Driven Provider Routing
+For testing, a `createTestEmbedder` is exported from `@directive-run/ai` that produces deterministic embeddings without API calls.
+
+## `createConstraintRouter(config)` — constraint-driven provider selection
 
-Route LLM calls to different providers based on runtime constraints:
+Routes each runner call to a provider based on runtime context.
 
 ```typescript
 import { createConstraintRouter } from "@directive-run/ai";
 
 const router = createConstraintRouter({
   providers: [
+    { name: "ollama",    runner: ollamaRunner,    costPerMillion: 0 },
     { name: "anthropic", runner: anthropicRunner, costPerMillion: 3 },
-    { name: "openai", runner: openaiRunner, costPerMillion: 5 },
-    { name: "ollama", runner: ollamaRunner, costPerMillion: 0 },
+    { name: "openai",    runner: openaiRunner,    costPerMillion: 5 },
   ],
   constraints: [
-    // Use cheapest provider when budget is low
-    { when: (context) => context.budgetRemaining < 1.0, prefer: "ollama" },
-    // Use best provider for high-priority tasks
-    { when: (context) => context.priority === "high", prefer: "anthropic" },
+    { when: (ctx) => ctx.budgetRemaining < 1.0, prefer: "ollama" },
+    { when: (ctx) => ctx.priority === "high",   prefer: "anthropic" },
   ],
 });
 ```
 
-## Combining Wrappers
+`router` returns an `AgentRunner` you pass directly to an orchestrator. The router's context fields depend on your config; check the `ConstraintRouterConfig` type for the full surface.
 
-Wrappers compose – apply them inside-out (innermost runs first):
+## Composing wrappers
+
+Wrappers are pure functions — compose them inside-out (innermost runs first per call).
 
 ```typescript
 import { withBudget, withRetry, withFallback } from "@directive-run/ai";
 
-// Order: retry → budget → fallback
-const resilientRunner = withFallback(
-  withBudget(
-    withRetry(primaryRunner, { maxRetries: 3, backoff: "exponential", baseDelayMs: 100 }),
-    { budgets: [{ window: "hour", maxCost: 1.0, pricing: { inputPerMillion: 3, outputPerMillion: 15 } }] },
-  ),
-  fallbackRunner,
+// Per call: retry first → budget cap on retried call → fallback to backup if all retries fail
+const resilient = withFallback(
+  [
+    withBudget(
+      withRetry(primaryRunner, { maxRetries: 3 }),
+      {
+        pricing: { inputPerMillion: 3, outputPerMillion: 15 },
+        budgets: [{ window: "hour", maxCost: 5.0, pricing: { inputPerMillion: 3, outputPerMillion: 15 } }],
+      },
+    ),
+    fallbackRunner,
+  ],
 );
 ```
 
-## Quick Reference
+## Anti-patterns
+
+### `withFallback(primary, backup)` (two positional args)
+
+```typescript
+// WRONG — withFallback takes a single ARRAY of runners, then an optional config
+withFallback(primary, backup)
+
+// CORRECT
+withFallback([primary, backup], { onFallback: (from, to, err) => {} })
+```
+
+### `withRetry` with `backoff: "linear" | "none"` or `shouldRetry`
+
+```typescript
+// WRONG — backoff is always exponential+jitter (not configurable); shouldRetry is named isRetryable
+withRetry(runner, { maxRetries: 3, backoff: "exponential", shouldRetry: (e) => true })
+
+// CORRECT
+withRetry(runner, { maxRetries: 3, baseDelayMs: 500, isRetryable: (e) => /5\d{2}/.test(e.message) })
+```
+
+### `createCircuitBreaker({ resetTimeout, halfOpenMaxAttempts })`
+
+```typescript
+// WRONG — option names from a different library
+createCircuitBreaker({ resetTimeout: 30_000, halfOpenMaxAttempts: 1 })
+
+// CORRECT
+createCircuitBreaker({ recoveryTimeMs: 30_000, halfOpenMaxRequests: 1 })
+```
+
+### `breaker.wrap(runner)` / `breaker.state`
+
+```typescript
+// WRONG — these don't exist on CircuitBreaker
+const protected = breaker.wrap(baseRunner);
+console.log(breaker.state);
+
+// CORRECT
+const result = await breaker.execute(() => baseRunner(agent, input));
+console.log(breaker.getState());
+```
+
+### `createHealthMonitor({ agents, checkInterval, onStatusChange })` / `monitor.start()` / `monitor.getReport()`
+
+```typescript
+// WRONG — none of these exist
+const monitor = createHealthMonitor({
+  agents: { researcher: { runner, circuitBreaker } },
+  checkInterval: 60_000,
+  onStatusChange: (id, s) => {},
+});
+monitor.start();
+const report = monitor.getReport();
+
+// CORRECT — push metrics in from your call sites
+const monitor = createHealthMonitor({ windowMs: 60_000 });
+monitor.recordSuccess("researcher", latencyMs);
+monitor.recordFailure("researcher", latencyMs, error);
+const score = monitor.getHealthScore("researcher");
+```
+
+### `createOpenAIEmbedder` / `createAnthropicEmbedder`
+
+```typescript
+// WRONG — no embedder factories ship in @directive-run/ai
+import { createOpenAIEmbedder } from "@directive-run/ai/openai";
+
+// CORRECT — pass your own EmbedderFn
+const embedder: EmbedderFn = async (text) => {
+  const res = await myEmbedAPI.embed(text);
+  return res.embedding;
+};
+const cache = createSemanticCache({ embedder, similarityThreshold: 0.95 });
+```
+
+### `cache.wrap(runner)`
+
+```typescript
+// WRONG — SemanticCache doesn't have a wrap() method
+const cachedRunner = cache.wrap(baseRunner);
+
+// CORRECT — wire it in as a guardrail
+const cacheGuardrail = createSemanticCacheGuardrail({ cache });
+const orchestrator = createAgentOrchestrator({ runner: baseRunner, guardrails: { input: [cacheGuardrail] } });
+```
+
+## Quick reference
 
-| Utility | Purpose | Key Options |
+| Utility | Shape | Throws |
 |---|---|---|
-| `withBudget` | Cost caps per time window | `budgets`, `maxCostPerCall` |
-| `withRetry` | Retry transient failures | `maxRetries`, `backoff`, `shouldRetry` |
-| `withFallback` | Switch to backup runner | primary, fallback runners |
-| `createCircuitBreaker` | Stop calling failing providers | `failureThreshold`, `resetTimeout` |
-| `createHealthMonitor` | Fleet health tracking | `agents`, `checkInterval` |
-| `createSemanticCache` | Avoid redundant LLM calls | `similarity`, `maxSize`, `ttl` |
-| `createConstraintRouter` | Dynamic provider selection | `providers`, `constraints` |
+| `withBudget(runner, BudgetConfig)` | wraps an AgentRunner | `BudgetExceededError` |
+| `withRetry(runner, RetryConfig)` | wraps an AgentRunner | `RetryExhaustedError` |
+| `withFallback(runners[], config?)` | wraps an array of AgentRunners | `AllProvidersFailedError` |
+| `createCircuitBreaker(config)` | returns `CircuitBreaker` with `execute(fn)` | `CircuitBreakerOpenError` |
+| `createHealthMonitor(config?)` | returns `HealthMonitor`; push outcomes in via `recordSuccess/Failure` | — |
+| `createSemanticCache(config)` | returns `SemanticCache`; use via `createSemanticCacheGuardrail({ cache })` | — |
+| `createConstraintRouter(config)` | returns an `AgentRunner` that picks providers at call time | — |
+
+## See also
+
+- [`ai-orchestrator.md`](./ai-orchestrator.md) — `circuitBreaker` / `selfHealing` / `agentRetry` / `maxTokenBudget` options on `createAgentOrchestrator`
+- [`ai-guardrails-memory.md`](./ai-guardrails-memory.md) — `createSemanticCacheGuardrail({ cache })` wires this file's `SemanticCache` into the orchestrator
+- [`ai-debug-observability.md`](./ai-debug-observability.md) — recording retry storms, circuit-breaker state changes, and budget warnings in the debug timeline