@falai/agent 1.1.2 → 1.2.0

package/docs/guides/prompt-optimization.md

@@ -0,0 +1,164 @@
+# Prompt Optimization
+
+The `PromptSectionCache` optimizes prompt generation by memoizing static sections across turns and recomputing only dynamic sections per-turn. Combined with the native history format change, this reduces redundant computation and token usage.
+
+## Section Types
+
+Prompt sections are classified as either static or dynamic:
+
+| Type | Behavior | Examples |
+|---|---|---|
+| `static` | Cached after first computation, reused across turns | Agent identity, glossary, knowledge base, route descriptions, scoring rules |
+| `dynamic` | Recomputed on every `resolveAll()` call | Instructions, directives, available tools, guidelines |
+
+Static sections only change when the underlying state changes (context update, session switch, route change). Dynamic sections depend on per-turn state and are always fresh.
+
+## Configuration
+
+Prompt caching is enabled by default. Configure it via the `promptCache` option on the agent:
+
+```typescript
+import { Agent } from "@falai/agent";
+
+const agent = new Agent({
+  name: "MyAgent",
+  provider: anthropicProvider,
+  promptCache: {
+    enabled: true,        // default: true
+    volatileKeys: [],     // keys that always recompute, even if registered as static
+  },
+});
+```
+
+### PromptCacheConfig
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `enabled` | `boolean` | `true` | Enable/disable section memoization |
+| `volatileKeys` | `string[]` | `[]` | Section keys forced to recompute every turn |
+
+Set `enabled: false` to disable caching entirely (useful for debugging):
+
+```typescript
+const agent = new Agent({
+  name: "DebugAgent",
+  provider: anthropicProvider,
+  promptCache: { enabled: false },
+});
+```
+
+## PromptSectionCache API
+
+### `register(key, type, compute)`
+
+Register a section with a unique key, type (`'static'` or `'dynamic'`), and a compute function.
+
+```typescript
+cache.register("agentMeta", "static", async () => {
+  return "## Agent Identity\nYou are MyAgent.";
+});
+
+cache.register("directives", "dynamic", () => {
+  return "## Directives\n- Address the user's question";
+});
+```
+
+### `get(key)`
+
+Retrieve a section's value. Static sections return the cached value when available; dynamic sections always recompute.
+
+### `resolveAll()`
+
+Resolve all registered sections in registration order. Returns `(string | null)[]`.
+
+### `invalidate(key)`
+
+Force a specific section to recompute on the next `resolveAll()` call.
+
+```typescript
+cache.invalidate("knowledgeBase");
+```
+
+### `invalidateAll()`
+
+Force all sections to recompute. Called automatically on session change or `/clear`.
+
+```typescript
+cache.invalidateAll();
+```
+
+## Automatic Cache Invalidation
+
+The framework invalidates relevant caches automatically when state changes:
+
+| Event | Sections Invalidated |
+|---|---|
+| `agent.updateContext()` | `agentMeta`, `knowledgeBase` |
+| Session change / clear | All sections (`invalidateAll()`) |
+| Route switch | Route-dependent sections (active routes, route rules, route knowledge base) |
+
+No manual cache management is needed for typical usage.
+
+## Native History Format
+
+History is now sent as native provider messages via `GenerateMessageInput.history` instead of being JSON-serialized into the system prompt. This saves tokens (no JSON overhead) and lets providers optimize for their native message format.
+
+### Migration from `addInteractionHistory()`
+
+The `PromptComposer.addInteractionHistory()` method is deprecated. If you were calling it directly:
+
+**Before:**
+```typescript
+const pc = new PromptComposer(context);
+await pc.addAgentMeta(agentOptions);
+await pc.addInteractionHistory(history);  // embedded in prompt string
+await pc.addLastMessage(lastMessage);
+const prompt = await pc.build();
+
+const response = await provider.generateMessage({ prompt, history: [] });
+```
+
+**After:**
+```typescript
+const pc = new PromptComposer(context, cache);
+await pc.addAgentMeta(agentOptions);
+// No addInteractionHistory() — history flows natively
+const prompt = await pc.build();
+
+const response = await provider.generateMessage({ prompt, history });
+```
+
+The `addInteractionHistory()` method still works for backward compatibility but is marked `@deprecated` and will be removed in a future version.
+
+## Manual Cache Usage
+
+You can use `PromptSectionCache` directly for custom prompt pipelines:
+
+```typescript
+import { PromptSectionCache } from "@falai/agent";
+
+const cache = new PromptSectionCache({ enabled: true });
+
+cache.register("identity", "static", () => "You are a helpful assistant.");
+cache.register("tools", "dynamic", () => "Available: search, calculate");
+
+// First call: both sections computed
+const sections1 = await cache.resolveAll(); // ["You are a helpful assistant.", "Available: search, calculate"]
+
+// Second call: identity served from cache, tools recomputed
+const sections2 = await cache.resolveAll();
+
+// Invalidate a specific section
+cache.invalidate("identity");
+
+// Next call: identity recomputed, tools recomputed (always)
+const sections3 = await cache.resolveAll();
+```
+
+## Key Properties
+
+- **Static sections cache** — computed once per session, reused across turns until invalidated
+- **Dynamic sections recompute** — always fresh on every `resolveAll()` call
+- **Automatic invalidation** — context updates, session changes, and route switches trigger targeted invalidation
+- **Configurable** — disable caching or mark specific keys as volatile
+- **Backward compatible** — `addInteractionHistory()` still works, just deprecated

package/examples/advanced-patterns/context-compaction.ts

@@ -0,0 +1,223 @@
+/**
+ * Context Compaction Example
+ *
+ * Demonstrates the CompactionEngine for managing conversation history size.
+ * Shows agent-level compaction config and how long conversations trigger
+ * automatic compaction through layered strategies.
+ *
+ * Key concepts:
+ * - Agent-level compaction configuration via `AgentCompactionConfig`
+ * - Token estimation using character-based heuristic (~4 chars/token)
+ * - Layered compaction strategies: tool_result_budget → micro_compact → auto_compact
+ * - Preservation of recent messages during compaction
+ * - Manual compaction via CompactionEngine API
+ */
+
+import {
+    Agent,
+    CompactionEngine,
+    GeminiProvider,
+    type HistoryItem,
+    type CompactionOptions,
+} from "../../src/index";
+
+// --- Agent-level compaction config ---
+
+async function demonstrateAgentCompaction() {
+    console.log("=== Agent-Level Compaction Config ===\n");
+
+    const provider = new GeminiProvider({
+        apiKey: process.env.GEMINI_API_KEY || "demo-key",
+        model: "models/gemini-2.5-flash",
+    });
+
+    // Compaction is configured at the agent level.
+    // The agent validates options on construction and wires the CompactionEngine
+    // into the SessionManager so compaction happens transparently.
+    const agent = new Agent({
+        name: "LongConversationAgent",
+        description: "An agent that handles long conversations gracefully",
+        provider,
+        compaction: {
+            maxTokens: 100_000,
+            compactionThreshold: 0.8,   // compact at 80% of budget
+            preserveRecentCount: 10,    // always keep last 10 messages
+            maxToolResultChars: 5_000,  // truncate tool results over 5k chars
+            enabled: true,
+        },
+    });
+
+    console.log("Agent created with compaction config:");
+    console.log("  maxTokens:           100,000");
+    console.log("  compactionThreshold: 0.8 (triggers at 80k tokens)");
+    console.log("  preserveRecentCount: 10");
+    console.log("  maxToolResultChars:  5,000");
+    console.log();
+    console.log("Compaction runs automatically in SessionManager when history grows.");
+    console.log("No manual intervention needed for typical usage.\n");
+}
+
+// --- Manual CompactionEngine usage ---
+
+async function demonstrateManualCompaction() {
+    console.log("=== Manual CompactionEngine Usage ===\n");
+
+    // Build a synthetic history with large tool results
+    const history: HistoryItem[] = [
+        { role: "user", content: "Analyze the codebase for security issues." },
+        { role: "assistant", content: "I'll scan the files for common vulnerabilities." },
+        { role: "tool", tool_call_id: "tc_1", name: "scan_files", content: "x".repeat(20_000) },
+        { role: "assistant", content: "Found some issues. Let me check more files." },
+        { role: "tool", tool_call_id: "tc_2", name: "scan_files", content: "y".repeat(15_000) },
+        { role: "user", content: "What about SQL injection?" },
+        { role: "assistant", content: "Let me search for raw SQL queries." },
+        { role: "tool", tool_call_id: "tc_3", name: "search_code", content: "z".repeat(10_000) },
+        { role: "user", content: "Summarize the findings." },
+        { role: "assistant", content: "Here is a summary of the security audit." },
+    ];
+
+    // 1. Token estimation
+    const tokens = CompactionEngine.estimateTokens(history);
+    console.log(`Estimated tokens: ${tokens}`);
+    console.log(`Total messages:   ${history.length}\n`);
+
+    // 2. Tool result budgeting (no LLM call needed)
+    const budgeted = CompactionEngine.applyToolResultBudget(history, 5_000);
+    const budgetedTokens = CompactionEngine.estimateTokens(budgeted);
+    console.log("After tool result budget (maxChars=5000):");
+    console.log(`  Tokens: ${tokens} → ${budgetedTokens}`);
+
+    for (let i = 0; i < budgeted.length; i++) {
+        if (budgeted[i].role === "tool") {
+            const truncated = budgeted[i].content.length < history[i].content.length;
+            console.log(`  Message ${i} (tool): ${truncated ? "truncated" : "unchanged"} (${budgeted[i].content.length} chars)`);
+        }
+    }
+    console.log();
+
+    // 3. Full compaction with a mock provider
+    // In real usage you'd pass the agent's provider for LLM summarization.
+    // Here we show the layered strategy selection.
+    const mockProvider = {
+        generateMessage: async () => ({
+            content: "Security audit found 3 potential SQL injection points and 2 XSS vulnerabilities.",
+            toolCalls: [],
+        }),
+    };
+
+    const options: CompactionOptions = {
+        maxTokens: 5_000,           // tight budget to force compaction
+        compactionThreshold: 0.8,
+        preserveRecentCount: 4,
+        maxToolResultChars: 2_000,
+        provider: mockProvider as any,
+    };
+
+    const result = await CompactionEngine.checkAndCompact(history, options);
+
+    console.log("Full compaction result:");
+    console.log(`  Strategy:          ${result.strategy}`);
+    console.log(`  Estimated tokens:  ${result.estimatedTokens}`);
+    console.log(`  Messages compacted: ${result.messagesCompacted}`);
+    console.log(`  History length:    ${result.history.length} (was ${history.length})`);
+    if (result.summary) {
+        console.log(`  Summary:           "${result.summary}"`);
+    }
+}
+
+// --- Demonstrating compaction strategies ---
+
+async function demonstrateStrategies() {
+    console.log("\n=== Compaction Strategy Ladder ===\n");
+
+    const smallHistory: HistoryItem[] = [
+        { role: "user", content: "Hello" },
+        { role: "assistant", content: "Hi there!" },
+    ];
+
+    const mockProvider = {
+        generateMessage: async () => ({
+            content: "Conversation summary.",
+            toolCalls: [],
+        }),
+    };
+
+    const baseOptions: CompactionOptions = {
+        maxTokens: 10_000,
+        compactionThreshold: 0.8,
+        preserveRecentCount: 2,
+        maxToolResultChars: 1_000,
+        provider: mockProvider as any,
+    };
+
+    // Strategy: 'none' — history is well under budget
+    const r1 = await CompactionEngine.checkAndCompact(smallHistory, baseOptions);
+    console.log(`Small history (${CompactionEngine.estimateTokens(smallHistory)} tokens):`);
+    console.log(`  → Strategy: ${r1.strategy}\n`);
+
+    // Strategy: 'tool_result_budget' — large tool results push over threshold
+    const mediumHistory: HistoryItem[] = [
+        { role: "user", content: "Analyze this." },
+        { role: "tool", tool_call_id: "tc_m1", name: "analyze", content: "a".repeat(30_000) },
+        { role: "user", content: "Thanks." },
+        { role: "assistant", content: "You're welcome." },
+    ];
+
+    const r2 = await CompactionEngine.checkAndCompact(mediumHistory, {
+        ...baseOptions,
+        maxTokens: 2_000,
+    });
+    console.log(`Medium history with large tool result (${CompactionEngine.estimateTokens(mediumHistory)} tokens):`);
+    console.log(`  → Strategy: ${r2.strategy}`);
+    console.log(`  → Tokens after: ${r2.estimatedTokens}\n`);
+
+    // Strategy: 'auto_compact' — many messages push well over budget
+    const longHistory: HistoryItem[] = Array.from({ length: 50 }, (_, i) => ({
+        role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant",
+        content: `Message ${i}: ${"lorem ipsum ".repeat(100)}`,
+    }));
+
+    const r3 = await CompactionEngine.checkAndCompact(longHistory, {
+        ...baseOptions,
+        maxTokens: 5_000,
+    });
+    console.log(`Long history (${CompactionEngine.estimateTokens(longHistory)} tokens, ${longHistory.length} messages):`);
+    console.log(`  → Strategy: ${r3.strategy}`);
+    console.log(`  → Tokens after: ${r3.estimatedTokens}`);
+    console.log(`  → Messages compacted: ${r3.messagesCompacted}`);
+}
+
+// --- Validation demo ---
+
+function demonstrateValidation() {
+    console.log("\n=== CompactionOptions Validation ===\n");
+
+    const invalidConfigs = [
+        { label: "threshold too low (0.3)", opts: { compactionThreshold: 0.3, preserveRecentCount: 4, maxToolResultChars: 1000, maxTokens: 10000 } },
+        { label: "threshold too high (0.99)", opts: { compactionThreshold: 0.99, preserveRecentCount: 4, maxToolResultChars: 1000, maxTokens: 10000 } },
+        { label: "preserveRecentCount < 2", opts: { compactionThreshold: 0.8, preserveRecentCount: 1, maxToolResultChars: 1000, maxTokens: 10000 } },
+        { label: "maxToolResultChars <= 0", opts: { compactionThreshold: 0.8, preserveRecentCount: 4, maxToolResultChars: 0, maxTokens: 10000 } },
+    ];
+
+    for (const { label, opts } of invalidConfigs) {
+        try {
+            CompactionEngine.validateOptions(opts as any);
+            console.log(`  ${label}: accepted (unexpected)`);
+        } catch (e) {
+            console.log(`  ${label}: rejected — ${(e as Error).message}`);
+        }
+    }
+}
+
+async function main() {
+    await demonstrateAgentCompaction();
+    await demonstrateManualCompaction();
+    await demonstrateStrategies();
+    demonstrateValidation();
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+    main().catch(console.error);
+}
+
+export { main };

package/examples/advanced-patterns/streaming-responses.ts

@@ -14,6 +14,7 @@ import {
   AnthropicProvider,
   OpenAIProvider,
   GeminiProvider,
+  type EnhancedTool,
 } from "../../src/index";
 
 // Custom context type
@@ -141,8 +142,8 @@ async function legacyStreamingWithAnthropic() {
 
     // Legacy respondStream API - requires manual session management
     let fullMessage = "";
-    for await (const chunk of agent.respondStream({ 
-      history: agent.session.getHistory() 
+    for await (const chunk of agent.respondStream({
+      history: agent.session.getHistory()
     })) {
       if (chunk.delta) {
         process.stdout.write(chunk.delta);
@@ -157,7 +158,7 @@ async function legacyStreamingWithAnthropic() {
         );
         console.log(`   - Data:`, agent.session.getData() || "None");
         console.log(`   - Tool Calls: ${chunk.toolCalls?.length || 0}`);
-        
+
         // Manual session history management required
         await agent.session.addMessage("assistant", fullMessage);
         console.log(`   - Session Messages: ${agent.session.getHistory().length}`);
@@ -220,7 +221,7 @@ async function modernStreamingWithOpenAI() {
           `   - Route: ${chunk.session?.currentRoute?.title || "None"}`
         );
         console.log(`   - Data:`, agent.session.getData() || "None");
-        
+
         // Session automatically updated - no manual work needed!
         console.log(`   - Session Messages: ${agent.session.getHistory().length}`);
       }
@@ -267,10 +268,10 @@ async function modernStreamingComparison() {
 
     // Manual session management
     await agent.session.addMessage("user", userMessage);
-    
+
     let oldWayMessage = "";
-    for await (const chunk of agent.respondStream({ 
-      history: agent.session.getHistory() 
+    for await (const chunk of agent.respondStream({
+      history: agent.session.getHistory()
     })) {
       if (chunk.delta) {
         process.stdout.write(chunk.delta);
@@ -528,6 +529,81 @@ async function logFeedback(data: { rating: number; comments: string }) {
   console.log("✨ Feedback logged successfully!");
 }
 
+async function streamingWithToolExecution() {
+  console.log("\n🤖 Example 7: Streaming with Tool Execution\n");
+
+  const provider = new AnthropicProvider({
+    apiKey: process.env.ANTHROPIC_API_KEY || "",
+    model: "claude-sonnet-4-5",
+  });
+
+  // Define EnhancedTools with concurrency metadata
+  const readFileTool: EnhancedTool = {
+    id: "read_file",
+    name: "Read File",
+    description: "Read a file from disk",
+    parameters: {
+      type: "object",
+      properties: { path: { type: "string" } },
+      required: ["path"],
+    },
+    handler: async (_ctx, args) => {
+      await new Promise((r) => setTimeout(r, 200));
+      return { data: `Contents of ${args?.path}`, success: true };
+    },
+    isConcurrencySafe: () => true,
+    isReadOnly: () => true,
+    interruptBehavior: () => "cancel",
+  };
+
+  const writeFileTool: EnhancedTool = {
+    id: "write_file",
+    name: "Write File",
+    description: "Write content to a file",
+    parameters: {
+      type: "object",
+      properties: { path: { type: "string" }, content: { type: "string" } },
+      required: ["path", "content"],
+    },
+    handler: async (_ctx, args) => {
+      await new Promise((r) => setTimeout(r, 150));
+      return { data: `Wrote to ${args?.path}`, success: true };
+    },
+    isConcurrencySafe: () => false,
+    isDestructive: () => true,
+    interruptBehavior: () => "block",
+    maxResultSizeChars: 1_000,
+  };
+
+  const agent = new Agent({
+    name: "ToolStreamingAssistant",
+    description: "Demonstrates streaming with concurrent tool execution",
+    provider,
+    tools: [readFileTool, writeFileTool],
+  });
+
+  try {
+    console.log("📤 Streaming with tool execution...\n");
+    console.log("When the LLM calls multiple read-only tools, they execute in parallel.");
+    console.log("Write tools wait for exclusive access.\n");
+    console.log("Response: ");
+
+    for await (const chunk of agent.stream("Read index.ts and utils.ts, then write output.ts")) {
+      if (chunk.delta) {
+        process.stdout.write(chunk.delta);
+      }
+
+      if (chunk.done) {
+        console.log("\n\n✅ Stream complete!");
+        console.log(`   Tool Calls: ${chunk.toolCalls?.length || 0}`);
+        console.log(`   Session Messages: ${agent.session.getHistory().length}`);
+      }
+    }
+  } catch (error) {
+    console.error("❌ Error:", error);
+  }
+}
+
 async function main() {
   console.log("🚀 Starting Streaming Examples\n");
   console.log("=".repeat(60));
@@ -539,6 +615,7 @@ async function main() {
     { name: "API Comparison (Gemini)", fn: modernStreamingComparison },
     { name: "Modern Streaming with Routes", fn: modernStreamingWithRoutes },
     { name: "Modern Streaming with Abort", fn: modernStreamingWithAbortSignal },
+    { name: "Streaming with Tool Execution", fn: streamingWithToolExecution },
   ];
 
   console.log("\nAvailable Examples:");
@@ -553,6 +630,7 @@ async function main() {
   console.log("   - Streaming provides real-time responses for better UX");
   console.log("   - Use AbortSignal to cancel long-running streams");
   console.log("   - Access chunk.route and chunk.step for flow information");
+  console.log("   - NEW: EnhancedTool metadata enables parallel read-only tool execution");
 
   console.log("\n" + "=".repeat(60));