@falai/agent 1.1.3 → 1.2.0

package/docs/core/tools/streaming-execution.md

@@ -0,0 +1,161 @@
+# Streaming Tool Execution
+
+The `StreamingToolExecutor` executes tools as they arrive from the LLM stream rather than waiting for the full response. It provides concurrency control, abort handling, and ordered result yielding.
+
+## Overview
+
+- Tools begin executing immediately as they are parsed from the LLM stream
+- Read-only (concurrency-safe) tools run in parallel
+- Write (non-concurrency-safe) tools run serially with exclusive access
+- Results are always yielded in the original request order
+- Progress messages bypass ordering and are delivered immediately
+
+## Concurrency Control
+
+The executor enforces a strict invariant at all times:
+
+> Either **all** executing tools have `isConcurrencySafe === true`, **or** exactly **one** tool is executing with `isConcurrencySafe === false`.
+
+Tools without the `isConcurrencySafe` method default to `false` (serial execution), preserving backward compatibility with plain `Tool` objects.
+
+A configurable `maxParallel` limit (default: 10) caps the number of concurrently executing tools regardless of concurrency safety.
+
+### Example: Mixed Read/Write Tools
+
+```typescript
+import { Agent, EnhancedTool } from "@falai/agent";
+
+const readFile: EnhancedTool = {
+  id: "read-file",
+  name: "read_file",
+  description: "Read a file from disk",
+  parameters: {
+    type: "object",
+    properties: { path: { type: "string" } },
+    required: ["path"],
+  },
+  handler: async (ctx, args) => {
+    const content = await fs.readFile(args?.path as string, "utf-8");
+    return { data: content, success: true };
+  },
+  isConcurrencySafe: () => true,   // safe to run in parallel
+  isReadOnly: () => true,
+  maxResultSizeChars: 50_000,
+};
+
+const writeFile: EnhancedTool = {
+  id: "write-file",
+  name: "write_file",
+  description: "Write content to a file",
+  parameters: {
+    type: "object",
+    properties: {
+      path: { type: "string" },
+      content: { type: "string" },
+    },
+    required: ["path", "content"],
+  },
+  handler: async (ctx, args) => {
+    await fs.writeFile(args?.path as string, args?.content as string);
+    return { success: true };
+  },
+  isConcurrencySafe: () => false,  // must run exclusively
+  interruptBehavior: () => "block",
+};
+
+const agent = new Agent({
+  name: "CodeAssistant",
+  provider: anthropicProvider,
+  tools: [readFile, writeFile],
+});
+```
+
+When the LLM requests `read_file` three times followed by `write_file`, the three reads execute in parallel. Once all reads complete, the write executes alone.
+
+## Abort Behavior
+
+### Sibling Abort
+
+When a tool in a concurrent batch fails, all sibling tools in the same batch receive an abort signal. Each tool's `interruptBehavior` determines the response:
+
+- `'cancel'` — tool is immediately aborted
+- `'block'` (default) — tool is allowed to finish
+
+### Parent AbortSignal
+
+A parent `AbortSignal` can be passed via `StreamingToolExecutorOptions`. When it fires:
+
+1. Tools with `interruptBehavior() === 'cancel'` are aborted immediately
+2. Tools with `interruptBehavior() === 'block'` complete normally
+3. No new queued tools are started
+
+```typescript
+const controller = new AbortController();
+
+// Pass signal through agent options or directly to the executor
+for await (const chunk of agent.respondStream({
+  history,
+  signal: controller.signal,
+})) {
+  process.stdout.write(chunk.delta);
+}
+
+// Cancel from user action
+controller.abort();
+```
+
+## Progress Reporting
+
+Tools can emit progress messages during execution. These are yielded immediately to the caller without being buffered behind result ordering.
+
+```typescript
+for await (const chunk of agent.respondStream({ history })) {
+  if (chunk.toolExecution?.progress) {
+    console.log(`[progress] ${chunk.toolExecution.toolCallId}: ${chunk.toolExecution.progress}`);
+  }
+  if (chunk.toolExecution?.result) {
+    console.log(`[result] ${chunk.toolExecution.toolCallId}: done`);
+  }
+  process.stdout.write(chunk.delta);
+}
+```
+
+## Result Ordering
+
+Results are always yielded in the same order as the original tool call requests, regardless of actual completion order. If tool B finishes before tool A, tool B's result is buffered until tool A's result is yielded first.
+
+## API Reference
+
+### Constructor
+
+```typescript
+new StreamingToolExecutor<TContext, TData>(
+  toolContext: ToolContext<TContext, TData>,
+  options?: {
+    maxParallel?: number;   // default: 10
+    signal?: AbortSignal;   // parent abort signal
+  }
+)
+```
+
+### Methods
+
+| Method | Description |
+|---|---|
+| `addTool(toolCall, tool)` | Queue a tool for execution. Concurrency safety is evaluated once at queue time. |
+| `getCompletedResults()` | Synchronous generator yielding available results in request order. |
+| `getRemainingResults()` | Async generator yielding all results (waits for pending tools). |
+| `discard()` | Stop processing new queued tools. Running tools continue per their `interruptBehavior`. |
+| `getUpdatedContext()` | Return accumulated context updates from completed tools. |
+| `hasUnfinishedTools()` | `true` if any tools are still queued or executing. |
+
+### Default Behaviors for Plain `Tool` Objects
+
+| Property | Default |
+|---|---|
+| `isConcurrencySafe` | `false` |
+| `isReadOnly` | `false` |
+| `isDestructive` | `false` |
+| `interruptBehavior` | `'block'` |
+
+Plain `Tool` objects work without modification — they execute serially and are allowed to complete on abort.

package/docs/guides/context-compaction.md

@@ -0,0 +1,96 @@
+# Context Compaction
+
+The `CompactionEngine` automatically manages conversation history size when approaching token limits. It applies multi-layered strategies in order of cost, from cheap truncation to LLM-powered summarization.
+
+## Compaction Strategies
+
+Strategies are applied in order until the history fits within the token budget:
+
+| Strategy | Cost | Description |
+|---|---|---|
+| `none` | Free | History is under threshold — no action taken |
+| `tool_result_budget` | Free | Truncate oversized tool results with a notice |
+| `micro_compact` | Free | Collapse whitespace in verbose tool outputs |
+| `auto_compact` | LLM call | Summarize old messages via the configured AI provider |
+
+If the LLM summarization fails, the engine falls back to aggressive truncation (removing oldest messages) and logs a warning. The next compaction attempt will retry summarization.
+
+## Configuration
+
+Compaction is configured at the agent level via the `compaction` option:
+
+```typescript
+import { Agent } from "@falai/agent";
+
+const agent = new Agent({
+  name: "LongConversationAgent",
+  provider: anthropicProvider,
+  compaction: {
+    maxTokens: 100_000,
+    compactionThreshold: 0.8,    // trigger at 80% of budget
+    preserveRecentCount: 10,     // always keep last 10 messages
+    maxToolResultChars: 5_000,   // truncate tool results over 5k chars
+    provider: anthropicProvider, // provider for LLM summarization
+  },
+});
+```
+
+### CompactionOptions
+
+| Option | Type | Constraint | Description |
+|---|---|---|---|
+| `maxTokens` | `number` | > 0 | Maximum token budget for the conversation |
+| `compactionThreshold` | `number` | 0.5 – 0.95 | Ratio at which compaction triggers |
+| `preserveRecentCount` | `number` | ≥ 2 | Recent messages that are never modified |
+| `maxToolResultChars` | `number` | > 0 | Per-tool-result character limit before truncation |
+| `provider` | `AiProvider` | — | Provider used for LLM summarization |
+
+Invalid options throw at construction time.
+
+## How It Works
+
+When the `SessionManager` detects that estimated tokens exceed `maxTokens * compactionThreshold`, the `CompactionEngine` runs:
+
+1. **Token estimation** — character-based heuristic (~4 chars/token), no external tokenizer needed
+2. **Tool result budget** — truncate any tool result exceeding `maxToolResultChars`, append a notice like `[Truncated: 12000 chars total, showing first 5000]`
+3. **Micro-compact** — collapse whitespace in tool outputs for the compactable portion of history
+4. **Auto-compact** — summarize old messages via the AI provider, replacing them with a `[Conversation Summary]` system message
+
+The last `preserveRecentCount` messages are never modified or removed by any strategy.
+
+## Manual Compaction
+
+You can also use the `CompactionEngine` directly:
+
+```typescript
+import { CompactionEngine } from "@falai/agent";
+
+const result = await CompactionEngine.checkAndCompact(history, {
+  maxTokens: 100_000,
+  compactionThreshold: 0.8,
+  preserveRecentCount: 10,
+  maxToolResultChars: 5_000,
+  provider: anthropicProvider,
+});
+
+console.log(result.strategy);        // 'none' | 'tool_result_budget' | 'micro_compact' | 'auto_compact'
+console.log(result.estimatedTokens); // tokens after compaction
+console.log(result.messagesCompacted);
+```
+
+### Standalone Utilities
+
+```typescript
+// Estimate tokens for a history
+const tokens = CompactionEngine.estimateTokens(history);
+
+// Truncate tool results only
+const budgeted = CompactionEngine.applyToolResultBudget(history, 5_000);
+```
+
+## Key Properties
+
+- **Idempotent** — compacting already-compacted history with the same options produces the same result
+- **Deterministic estimation** — `estimateTokens` always returns the same value for the same input
+- **Preservation guarantee** — the last `preserveRecentCount` messages are never touched
+- **Graceful degradation** — LLM failure falls back to truncation, never crashes

package/docs/guides/prompt-optimization.md

@@ -0,0 +1,164 @@
+# Prompt Optimization
+
+The `PromptSectionCache` optimizes prompt generation by memoizing static sections across turns and recomputing only dynamic sections per-turn. Combined with the native history format change, this reduces redundant computation and token usage.
+
+## Section Types
+
+Prompt sections are classified as either static or dynamic:
+
+| Type | Behavior | Examples |
+|---|---|---|
+| `static` | Cached after first computation, reused across turns | Agent identity, glossary, knowledge base, route descriptions, scoring rules |
+| `dynamic` | Recomputed on every `resolveAll()` call | Instructions, directives, available tools, guidelines |
+
+Static sections only change when the underlying state changes (context update, session switch, route change). Dynamic sections depend on per-turn state and are always fresh.
+
+## Configuration
+
+Prompt caching is enabled by default. Configure it via the `promptCache` option on the agent:
+
+```typescript
+import { Agent } from "@falai/agent";
+
+const agent = new Agent({
+  name: "MyAgent",
+  provider: anthropicProvider,
+  promptCache: {
+    enabled: true,        // default: true
+    volatileKeys: [],     // keys that always recompute, even if registered as static
+  },
+});
+```
+
+### PromptCacheConfig
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `enabled` | `boolean` | `true` | Enable/disable section memoization |
+| `volatileKeys` | `string[]` | `[]` | Section keys forced to recompute every turn |
+
+Set `enabled: false` to disable caching entirely (useful for debugging):
+
+```typescript
+const agent = new Agent({
+  name: "DebugAgent",
+  provider: anthropicProvider,
+  promptCache: { enabled: false },
+});
+```
+
+## PromptSectionCache API
+
+### `register(key, type, compute)`
+
+Register a section with a unique key, type (`'static'` or `'dynamic'`), and a compute function.
+
+```typescript
+cache.register("agentMeta", "static", async () => {
+  return "## Agent Identity\nYou are MyAgent.";
+});
+
+cache.register("directives", "dynamic", () => {
+  return "## Directives\n- Address the user's question";
+});
+```
+
+### `get(key)`
+
+Retrieve a section's value. Static sections return the cached value when available; dynamic sections always recompute.
+
+### `resolveAll()`
+
+Resolve all registered sections in registration order. Returns `(string | null)[]`.
+
+### `invalidate(key)`
+
+Force a specific section to recompute on the next `resolveAll()` call.
+
+```typescript
+cache.invalidate("knowledgeBase");
+```
+
+### `invalidateAll()`
+
+Force all sections to recompute. Called automatically on session change or `/clear`.
+
+```typescript
+cache.invalidateAll();
+```
+
+## Automatic Cache Invalidation
+
+The framework invalidates relevant caches automatically when state changes:
+
+| Event | Sections Invalidated |
+|---|---|
+| `agent.updateContext()` | `agentMeta`, `knowledgeBase` |
+| Session change / clear | All sections (`invalidateAll()`) |
+| Route switch | Route-dependent sections (active routes, route rules, route knowledge base) |
+
+No manual cache management is needed for typical usage.
+
+## Native History Format
+
+History is now sent as native provider messages via `GenerateMessageInput.history` instead of being JSON-serialized into the system prompt. This saves tokens (no JSON overhead) and lets providers optimize for their native message format.
+
+### Migration from `addInteractionHistory()`
+
+The `PromptComposer.addInteractionHistory()` method is deprecated. If you were calling it directly:
+
+**Before:**
+```typescript
+const pc = new PromptComposer(context);
+await pc.addAgentMeta(agentOptions);
+await pc.addInteractionHistory(history);  // embedded in prompt string
+await pc.addLastMessage(lastMessage);
+const prompt = await pc.build();
+
+const response = await provider.generateMessage({ prompt, history: [] });
+```
+
+**After:**
+```typescript
+const pc = new PromptComposer(context, cache);
+await pc.addAgentMeta(agentOptions);
+// No addInteractionHistory() — history flows natively
+const prompt = await pc.build();
+
+const response = await provider.generateMessage({ prompt, history });
+```
+
+The `addInteractionHistory()` method still works for backward compatibility but is marked `@deprecated` and will be removed in a future version.
+
+## Manual Cache Usage
+
+You can use `PromptSectionCache` directly for custom prompt pipelines:
+
+```typescript
+import { PromptSectionCache } from "@falai/agent";
+
+const cache = new PromptSectionCache({ enabled: true });
+
+cache.register("identity", "static", () => "You are a helpful assistant.");
+cache.register("tools", "dynamic", () => "Available: search, calculate");
+
+// First call: both sections computed
+const sections1 = await cache.resolveAll(); // ["You are a helpful assistant.", "Available: search, calculate"]
+
+// Second call: identity served from cache, tools recomputed
+const sections2 = await cache.resolveAll();
+
+// Invalidate a specific section
+cache.invalidate("identity");
+
+// Next call: identity recomputed, tools recomputed (always)
+const sections3 = await cache.resolveAll();
+```
+
+## Key Properties
+
+- **Static sections cache** — computed once per session, reused across turns until invalidated
+- **Dynamic sections recompute** — always fresh on every `resolveAll()` call
+- **Automatic invalidation** — context updates, session changes, and route switches trigger targeted invalidation
+- **Configurable** — disable caching or mark specific keys as volatile
+- **Backward compatible** — `addInteractionHistory()` still works, just deprecated

package/examples/advanced-patterns/context-compaction.ts

@@ -0,0 +1,223 @@
+/**
+ * Context Compaction Example
+ *
+ * Demonstrates the CompactionEngine for managing conversation history size.
+ * Shows agent-level compaction config and how long conversations trigger
+ * automatic compaction through layered strategies.
+ *
+ * Key concepts:
+ * - Agent-level compaction configuration via `AgentCompactionConfig`
+ * - Token estimation using character-based heuristic (~4 chars/token)
+ * - Layered compaction strategies: tool_result_budget → micro_compact → auto_compact
+ * - Preservation of recent messages during compaction
+ * - Manual compaction via CompactionEngine API
+ */
+
+import {
+    Agent,
+    CompactionEngine,
+    GeminiProvider,
+    type HistoryItem,
+    type CompactionOptions,
+} from "../../src/index";
+
+// --- Agent-level compaction config ---
+
+async function demonstrateAgentCompaction() {
+    console.log("=== Agent-Level Compaction Config ===\n");
+
+    const provider = new GeminiProvider({
+        apiKey: process.env.GEMINI_API_KEY || "demo-key",
+        model: "models/gemini-2.5-flash",
+    });
+
+    // Compaction is configured at the agent level.
+    // The agent validates options on construction and wires the CompactionEngine
+    // into the SessionManager so compaction happens transparently.
+    const agent = new Agent({
+        name: "LongConversationAgent",
+        description: "An agent that handles long conversations gracefully",
+        provider,
+        compaction: {
+            maxTokens: 100_000,
+            compactionThreshold: 0.8,   // compact at 80% of budget
+            preserveRecentCount: 10,    // always keep last 10 messages
+            maxToolResultChars: 5_000,  // truncate tool results over 5k chars
+            enabled: true,
+        },
+    });
+
+    console.log("Agent created with compaction config:");
+    console.log("  maxTokens:           100,000");
+    console.log("  compactionThreshold: 0.8 (triggers at 80k tokens)");
+    console.log("  preserveRecentCount: 10");
+    console.log("  maxToolResultChars:  5,000");
+    console.log();
+    console.log("Compaction runs automatically in SessionManager when history grows.");
+    console.log("No manual intervention needed for typical usage.\n");
+}
+
+// --- Manual CompactionEngine usage ---
+
+async function demonstrateManualCompaction() {
+    console.log("=== Manual CompactionEngine Usage ===\n");
+
+    // Build a synthetic history with large tool results
+    const history: HistoryItem[] = [
+        { role: "user", content: "Analyze the codebase for security issues." },
+        { role: "assistant", content: "I'll scan the files for common vulnerabilities." },
+        { role: "tool", tool_call_id: "tc_1", name: "scan_files", content: "x".repeat(20_000) },
+        { role: "assistant", content: "Found some issues. Let me check more files." },
+        { role: "tool", tool_call_id: "tc_2", name: "scan_files", content: "y".repeat(15_000) },
+        { role: "user", content: "What about SQL injection?" },
+        { role: "assistant", content: "Let me search for raw SQL queries." },
+        { role: "tool", tool_call_id: "tc_3", name: "search_code", content: "z".repeat(10_000) },
+        { role: "user", content: "Summarize the findings." },
+        { role: "assistant", content: "Here is a summary of the security audit." },
+    ];
+
+    // 1. Token estimation
+    const tokens = CompactionEngine.estimateTokens(history);
+    console.log(`Estimated tokens: ${tokens}`);
+    console.log(`Total messages:   ${history.length}\n`);
+
+    // 2. Tool result budgeting (no LLM call needed)
+    const budgeted = CompactionEngine.applyToolResultBudget(history, 5_000);
+    const budgetedTokens = CompactionEngine.estimateTokens(budgeted);
+    console.log("After tool result budget (maxChars=5000):");
+    console.log(`  Tokens: ${tokens} → ${budgetedTokens}`);
+
+    for (let i = 0; i < budgeted.length; i++) {
+        if (budgeted[i].role === "tool") {
+            const truncated = budgeted[i].content.length < history[i].content.length;
+            console.log(`  Message ${i} (tool): ${truncated ? "truncated" : "unchanged"} (${budgeted[i].content.length} chars)`);
+        }
+    }
+    console.log();
+
+    // 3. Full compaction with a mock provider
+    // In real usage you'd pass the agent's provider for LLM summarization.
+    // Here we show the layered strategy selection.
+    const mockProvider = {
+        generateMessage: async () => ({
+            content: "Security audit found 3 potential SQL injection points and 2 XSS vulnerabilities.",
+            toolCalls: [],
+        }),
+    };
+
+    const options: CompactionOptions = {
+        maxTokens: 5_000,           // tight budget to force compaction
+        compactionThreshold: 0.8,
+        preserveRecentCount: 4,
+        maxToolResultChars: 2_000,
+        provider: mockProvider as any,
+    };
+
+    const result = await CompactionEngine.checkAndCompact(history, options);
+
+    console.log("Full compaction result:");
+    console.log(`  Strategy:          ${result.strategy}`);
+    console.log(`  Estimated tokens:  ${result.estimatedTokens}`);
+    console.log(`  Messages compacted: ${result.messagesCompacted}`);
+    console.log(`  History length:    ${result.history.length} (was ${history.length})`);
+    if (result.summary) {
+        console.log(`  Summary:           "${result.summary}"`);
+    }
+}
+
+// --- Demonstrating compaction strategies ---
+
+async function demonstrateStrategies() {
+    console.log("\n=== Compaction Strategy Ladder ===\n");
+
+    const smallHistory: HistoryItem[] = [
+        { role: "user", content: "Hello" },
+        { role: "assistant", content: "Hi there!" },
+    ];
+
+    const mockProvider = {
+        generateMessage: async () => ({
+            content: "Conversation summary.",
+            toolCalls: [],
+        }),
+    };
+
+    const baseOptions: CompactionOptions = {
+        maxTokens: 10_000,
+        compactionThreshold: 0.8,
+        preserveRecentCount: 2,
+        maxToolResultChars: 1_000,
+        provider: mockProvider as any,
+    };
+
+    // Strategy: 'none' — history is well under budget
+    const r1 = await CompactionEngine.checkAndCompact(smallHistory, baseOptions);
+    console.log(`Small history (${CompactionEngine.estimateTokens(smallHistory)} tokens):`);
+    console.log(`  → Strategy: ${r1.strategy}\n`);
+
+    // Strategy: 'tool_result_budget' — large tool results push over threshold
+    const mediumHistory: HistoryItem[] = [
+        { role: "user", content: "Analyze this." },
+        { role: "tool", tool_call_id: "tc_m1", name: "analyze", content: "a".repeat(30_000) },
+        { role: "user", content: "Thanks." },
+        { role: "assistant", content: "You're welcome." },
+    ];
+
+    const r2 = await CompactionEngine.checkAndCompact(mediumHistory, {
+        ...baseOptions,
+        maxTokens: 2_000,
+    });
+    console.log(`Medium history with large tool result (${CompactionEngine.estimateTokens(mediumHistory)} tokens):`);
+    console.log(`  → Strategy: ${r2.strategy}`);
+    console.log(`  → Tokens after: ${r2.estimatedTokens}\n`);
+
+    // Strategy: 'auto_compact' — many messages push well over budget
+    const longHistory: HistoryItem[] = Array.from({ length: 50 }, (_, i) => ({
+        role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant",
+        content: `Message ${i}: ${"lorem ipsum ".repeat(100)}`,
+    }));
+
+    const r3 = await CompactionEngine.checkAndCompact(longHistory, {
+        ...baseOptions,
+        maxTokens: 5_000,
+    });
+    console.log(`Long history (${CompactionEngine.estimateTokens(longHistory)} tokens, ${longHistory.length} messages):`);
+    console.log(`  → Strategy: ${r3.strategy}`);
+    console.log(`  → Tokens after: ${r3.estimatedTokens}`);
+    console.log(`  → Messages compacted: ${r3.messagesCompacted}`);
+}
+
+// --- Validation demo ---
+
+function demonstrateValidation() {
+    console.log("\n=== CompactionOptions Validation ===\n");
+
+    const invalidConfigs = [
+        { label: "threshold too low (0.3)", opts: { compactionThreshold: 0.3, preserveRecentCount: 4, maxToolResultChars: 1000, maxTokens: 10000 } },
+        { label: "threshold too high (0.99)", opts: { compactionThreshold: 0.99, preserveRecentCount: 4, maxToolResultChars: 1000, maxTokens: 10000 } },
+        { label: "preserveRecentCount < 2", opts: { compactionThreshold: 0.8, preserveRecentCount: 1, maxToolResultChars: 1000, maxTokens: 10000 } },
+        { label: "maxToolResultChars <= 0", opts: { compactionThreshold: 0.8, preserveRecentCount: 4, maxToolResultChars: 0, maxTokens: 10000 } },
+    ];
+
+    for (const { label, opts } of invalidConfigs) {
+        try {
+            CompactionEngine.validateOptions(opts as any);
+            console.log(`  ${label}: accepted (unexpected)`);
+        } catch (e) {
+            console.log(`  ${label}: rejected — ${(e as Error).message}`);
+        }
+    }
+}
+
+async function main() {
+    await demonstrateAgentCompaction();
+    await demonstrateManualCompaction();
+    await demonstrateStrategies();
+    demonstrateValidation();
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+    main().catch(console.error);
+}
+
+export { main };