@minded-ai/mindedjs 1.0.128 → 1.0.130

package/docs/sdk/agent-api.md

@@ -0,0 +1,524 @@
+# Agent API
+
+The Agent class is the core component of MindedJS that orchestrates conversations, manages state, and coordinates flows. This document covers the key methods available on the Agent instance.
+
+## Overview
+
+The Agent class provides methods for:
+
+- Managing conversation state
+- Updating session data
+- Retrieving state information
+- Handling events and triggers
+- Managing voice sessions
+
+## Key Methods
+
+### updateState
+
+Updates the state of an active session. This method allows you to programmatically modify the conversation state, including messages, memory, and other state properties.
+
+#### Signature
+
+```typescript
+async updateState({
+  sessionId: string,
+  state: Partial<State>
+}): Promise<void>
+```
+
+#### Parameters
+
+- `sessionId` (string): The unique identifier of the session to update
+- `state` (Partial<State>): An object containing the state properties to update. Can include:
+  - `messages`: Array of messages to add or update
+  - `memory`: Partial memory object to merge with existing memory
+  - `overrideStartFromNodeId`: Node ID to override the next execution point
+  - Any other state properties defined in your agent
+
+#### Usage Examples
+
+##### Update Memory
+
+```typescript
+// Update specific memory fields
+await agent.updateState({
+  sessionId: 'session-123',
+  state: {
+    memory: {
+      user: {
+        name: 'John Doe',
+        preferences: {
+          language: 'en',
+        },
+      },
+    },
+  },
+});
+```
+
+##### Add Messages
+
+```typescript
+import { HumanMessage, AIMessage } from '@langchain/core/messages';
+
+// Add a new message to the conversation
+await agent.updateState({
+  sessionId: 'session-123',
+  state: {
+    messages: [new HumanMessage('User input'), new AIMessage('Agent response')],
+  },
+});
+```
+
+##### Update Multiple State Properties
+
+```typescript
+// Update both memory and messages
+await agent.updateState({
+  sessionId: 'session-123',
+  state: {
+    memory: {
+      conversation: {
+        topic: 'Order inquiry',
+        urgency: 'high',
+      },
+    },
+    messages: [new HumanMessage('I need help with my order')],
+  },
+});
+```
+
+#### Common Use Cases
+
+1. **Tool State Updates**: Tools can update state after execution
+
+   ```typescript
+   // Inside a tool's execute method
+   async execute({ input, state, agent }) {
+     // Perform tool logic
+     const result = await processOrder(input.orderId);
+
+     // Update state with results
+     await agent.updateState({
+       sessionId: state.sessionId,
+       state: {
+         memory: {
+           order: {
+             id: result.orderId,
+             status: result.status
+           }
+         }
+       }
+     });
+
+     return { result };
+   }
+   ```
+
+2. **External Event Handling**: Update state based on external events
+
+   ```typescript
+   // Handle webhook or external trigger
+   async handleWebhook(sessionId: string, webhookData: any) {
+     await agent.updateState({
+       sessionId,
+       state: {
+         memory: {
+           externalData: webhookData,
+           lastWebhookTime: new Date().toISOString()
+         }
+       }
+     });
+   }
+   ```
+
+3. **Voice Session Corrections**: Update messages during voice conversations
+   ```typescript
+   // Correct a voice transcription
+   await agent.updateState({
+     sessionId: voiceSessionId,
+     state: {
+       messages: [correctedMessage],
+       overrideStartFromNodeId: currentNodeId,
+     },
+   });
+   ```
+
+### getState
+
+Retrieves the current state of a session.
+
+#### Signature
+
+```typescript
+async getState(sessionId: string): Promise<State<Memory>>
+```
+
+#### Parameters
+
+- `sessionId` (string): The unique identifier of the session
+
+#### Returns
+
+A `State` object containing:
+
+- `sessionId`: The session identifier
+- `memory`: The current memory state
+- `messages`: Array of conversation messages
+- `sessionType`: The type of session (TEXT, VOICE, etc.)
+- Other state properties
+
+#### Usage Example
+
+```typescript
+// Get current state
+const state = await agent.getState('session-123');
+
+// Access memory
+console.log('User name:', state.memory.user?.name);
+
+// Check conversation history
+console.log('Message count:', state.messages.length);
+
+// Get session type
+console.log('Session type:', state.sessionType);
+```
+
+### compiledGraph
+
+Access to the underlying LangGraph compiled graph for advanced operations.
+
+#### Usage Example
+
+```typescript
+// Get LangGraph state directly
+const graphState = await agent.compiledGraph.getState(agent.getLangraphConfig(sessionId));
+
+// Update state with LangGraph API
+await agent.compiledGraph.updateState(agent.getLangraphConfig(sessionId), {
+  messages: [newMessage],
+  overrideStartFromNodeId: 'specific-node',
+});
+```
+
+## State Management Best Practices
+
+### 1. Partial Updates
+
+Always use partial updates to avoid overwriting existing state:
+
+```typescript
+// ✅ Good - Partial update
+await agent.updateState({
+  sessionId,
+  state: {
+    memory: {
+      user: { name: 'Jane' },
+    },
+  },
+});
+
+// ❌ Avoid - Don't replace entire memory
+await agent.updateState({
+  sessionId,
+  state: {
+    memory: completeMemoryObject, // This might lose other fields
+  },
+});
+```
+
+### 2. Atomic Operations
+
+Group related updates together:
+
+```typescript
+// ✅ Good - Single atomic update
+await agent.updateState({
+  sessionId,
+  state: {
+    memory: {
+      order: { id: '123', status: 'confirmed' },
+      lastAction: 'order_confirmed',
+    },
+    messages: [confirmationMessage],
+  },
+});
+
+// ❌ Avoid - Multiple separate updates
+await agent.updateState({ sessionId, state: { memory: { order: { id: '123' } } } });
+await agent.updateState({ sessionId, state: { memory: { order: { status: 'confirmed' } } } });
+await agent.updateState({ sessionId, state: { messages: [confirmationMessage] } });
+```
+
+### 3. State Validation
+
+The state updates are validated against your memory schema:
+
+```typescript
+try {
+  await agent.updateState({
+    sessionId,
+    state: {
+      memory: {
+        user: {
+          email: 'invalid-email', // Will fail validation if email has format validation
+        },
+      },
+    },
+  });
+} catch (error) {
+  console.error('State validation failed:', error);
+}
+```
+
+### 4. Tool State Updates
+
+Tools should return state updates rather than calling `updateState` directly when possible:
+
+```typescript
+// ✅ Preferred - Return state updates
+async execute({ input, state }) {
+  const result = await processData(input);
+  return {
+    result,
+    state: {
+      memory: {
+        processedData: result.data
+      }
+    }
+  };
+}
+
+// ⚠️ Use only when necessary - Direct update
+async execute({ input, state, agent }) {
+  const result = await processData(input);
+  await agent.updateState({
+    sessionId: state.sessionId,
+    state: {
+      memory: { processedData: result.data }
+    }
+  });
+  return { result };
+}
+```
+
+## Tool Execution API
+
+### ToolExecutor
+
+The `ToolExecutor` class enables standalone tool execution, particularly useful for browser automation and external tool invocations.
+
+#### Setting Up Tool Execution
+
+```typescript
+import { ToolExecutor } from '@minded-ai/mindedjs';
+
+// Create a tool executor
+const toolExecutor = new ToolExecutor(agent);
+
+// Register tools that can be executed via requests
+// Only tools with allowExecutionRequests: true will be registered
+toolExecutor.registerTools(tools);
+```
+
+#### Tool Configuration for Execution
+
+Tools must explicitly opt-in to be executable via requests:
+
+```typescript
+import { Tool } from '@minded-ai/mindedjs';
+
+const myBrowserTool: Tool<typeof schema, Memory> = {
+  name: 'browserAction',
+  description: 'Performs browser automation',
+  input: schema,
+  allowExecutionRequests: true, // Required for ToolExecutor
+  execute: async ({ input, state, agent }) => {
+    // Access CDP URL if available (browser-use context)
+    if (state.cdpUrl) {
+      // Connect to browser using Playwright or similar
+      const browser = await chromium.connectOverCDP(state.cdpUrl);
+      const page = browser.contexts()[0].pages()[0];
+
+      // Perform browser automation
+      await page.goto(input.url);
+      const title = await page.title();
+
+      return {
+        result: { pageTitle: title },
+        state: {
+          memory: {
+            lastVisitedUrl: input.url,
+          },
+        },
+      };
+    }
+
+    // Fallback for non-browser context
+    return { result: 'Browser context not available' };
+  },
+};
+```
+
+#### Executing Tools
+
+```typescript
+const response = await toolExecutor.executeTool({
+  toolName: 'browserAction',
+  toolParams: { url: 'https://example.com' },
+  sessionId: 'session-123',
+  requestId: 'req-456',
+  cdpUrl: 'ws://localhost:9222/devtools/browser/...', // Optional, for browser-use
+});
+
+if (response.error) {
+  console.error('Tool execution failed:', response.error);
+} else {
+  console.log('Tool result:', response.result);
+}
+```
+
+### CDP URL and Browser Automation
+
+When tools are executed in a browser-use context, the Chrome DevTools Protocol (CDP) URL is available in the state:
+
+#### Accessing CDP URL in Tools
+
+```typescript
+const browserTool: Tool<typeof schema, Memory> = {
+  name: 'advancedBrowserAction',
+  description: 'Advanced browser automation with direct CDP access',
+  input: schema,
+  allowExecutionRequests: true,
+  execute: async ({ input, state, agent }) => {
+    const { cdpUrl } = state;
+
+    if (!cdpUrl) {
+      return {
+        result: 'This tool requires browser context',
+        error: 'CDP URL not available',
+      };
+    }
+
+    // Use Playwright with CDP
+    const browser = await playwright.chromium.connectOverCDP(cdpUrl);
+    const context = browser.contexts()[0];
+    const page = context.pages()[0];
+
+    // Perform complex browser automation
+    await page.evaluate(() => {
+      // Execute JavaScript in the browser
+      document.querySelector('#submit')?.click();
+    });
+
+    // Take screenshot
+    const screenshot = await page.screenshot();
+
+    return {
+      result: {
+        success: true,
+        screenshot: screenshot.toString('base64'),
+      },
+      state: {
+        memory: {
+          lastAction: 'browser_automation',
+          timestamp: new Date().toISOString(),
+        },
+      },
+    };
+  },
+};
+```
+
+#### Browser-Use Integration Flow
+
+1. Browser-use requests tool execution via socket
+2. ToolExecutor receives request with CDP URL
+3. CDP URL is injected into state (`state.cdpUrl`)
+4. Tool connects to browser using CDP URL
+5. Tool performs automation and returns results
+6. State updates are automatically applied
+
+#### Security Considerations
+
+- Tools must have `allowExecutionRequests: true` to be executable via ToolExecutor
+- CDP URLs are only available during browser-use execution context
+- Tools should validate CDP URL availability before attempting browser operations
+- Always handle connection failures gracefully
+
+## Integration with Other Features
+
+### With Browser Use Tools
+
+When using browser automation tools, state updates happen automatically through the ToolExecutor:
+
+```typescript
+// Browser tools can update state via the ToolExecutor
+const toolExecutor = new ToolExecutor(agent);
+toolExecutor.registerTools(browserTools);
+
+// State updates are handled internally when tools return state changes
+// The cdpUrl is automatically injected when available
+```
+
+### With Voice Sessions
+
+Voice sessions use `updateState` for real-time corrections:
+
+```typescript
+// Voice session correction flow
+const voiceSession = new VoiceSession(agent, sessionId);
+// Corrections use updateState internally
+```
+
+### With Events
+
+State can be updated in event handlers:
+
+```typescript
+agent.on(AgentEvents.TRIGGER_EVENT, async ({ triggerName, sessionId }) => {
+  if (triggerName === 'external_update') {
+    await agent.updateState({
+      sessionId,
+      state: {
+        memory: {
+          lastTrigger: triggerName,
+          triggerTime: new Date().toISOString(),
+        },
+      },
+    });
+  }
+});
+```
+
+## Error Handling
+
+Always handle potential errors when updating state:
+
+```typescript
+try {
+  await agent.updateState({
+    sessionId,
+    state: stateUpdates,
+  });
+} catch (error) {
+  if (error.message.includes('Session not found')) {
+    // Handle missing session
+    console.error('Session does not exist:', sessionId);
+  } else if (error.message.includes('Validation')) {
+    // Handle validation errors
+    console.error('Invalid state update:', error);
+  } else {
+    // Handle other errors
+    console.error('Failed to update state:', error);
+  }
+}
+```
+
+## Related Documentation
+
+- [Memory](./memory.md) - Learn about memory schemas and management
+- [Events](./events.md) - Handle agent events and triggers
+- [Debugging](./debugging.md) - Debug state updates and transitions
+- [Tools](../low-code-editor/tools.md) - Create tools that update state