@nextsparkjs/plugin-langchain 0.1.0-beta.1

package/docs/03-orchestration/01-graph-orchestrator.md

@@ -0,0 +1,809 @@
+# Graph-Based Orchestration
+
+This guide covers the **LangGraph-based orchestrator**, the recommended approach for multi-agent systems. It replaces inefficient ReAct loops with an explicit state machine, achieving 25-50x faster execution with deterministic flow.
+
+> **Note**: For the legacy ReAct-based approach, see [Legacy Orchestration](./02-legacy-react.md) (deprecated).
+
+## Why Graph-Based Orchestration?
+
+The traditional ReAct (Reasoning + Acting) pattern has significant drawbacks at scale:
+
+### The Problem with ReAct Loops
+
+```
+User: "Show my tasks and find StartupXYZ account number"
+
+┌─────────────────────────────────────────────────────────────┐
+│ ORCHESTRATOR ReAct Loop (~10-20 iterations)                 │
+│ ├─ Think → Act → Observe → Think → Act...                   │
+│ └─ Decide: route_to_task                                    │
+│     ↓                                                       │
+│     ┌─────────────────────────────────────────────────┐     │
+│     │ TASK-ASSISTANT ReAct Loop (~10-15 iterations)   │     │
+│     │ └─ Think → Act → Observe → Think...             │     │
+│     └─────────────────────────────────────────────────┘     │
+│     ↓ (returns to orchestrator)                             │
+│ ├─ Think → decide second part                               │
+│ └─ Decide: route_to_customer                                │
+│     ↓                                                       │
+│     ┌─────────────────────────────────────────────────┐     │
+│     │ CUSTOMER-ASSISTANT ReAct Loop (~10-15 iter)     │     │
+│     └─────────────────────────────────────────────────┘     │
+└─────────────────────────────────────────────────────────────┘
+
+TOTAL: 50+ LLM iterations → TIMEOUT (2-5 minutes)
+```
+
+**Issues:**
+- **50+ LLM calls** for a simple multi-intent request
+- **Recursion limit errors** from nested loops
+- **Non-deterministic** - different paths each time
+- **Expensive** - each iteration costs tokens
+- **Slow** - 2-5 minute response times
+
+### The Solution: Explicit State Machine
+
+```
+User: "Show my tasks and find StartupXYZ account number"
+
+┌─────────────────────────────────────────────────────────────┐
+│                         START                                │
+│                           ↓                                  │
+│                  ┌─────────────┐                            │
+│                  │   ROUTER    │ ← 1 LLM call               │
+│                  │   (Intent)  │   (structured output)       │
+│                  └──────┬──────┘                            │
+│                         ↓                                    │
+│         ┌───────────────┴───────────────┐                   │
+│         ↓                               ↓                   │
+│   ┌──────────────┐              ┌──────────────┐           │
+│   │ TASK_HANDLER │              │ CUST_HANDLER │           │
+│   │   (0 LLM)    │              │   (0 LLM)    │           │
+│   └───────┬──────┘              └───────┬──────┘           │
+│           ↓                             ↓                   │
+│           └─────────────┬───────────────┘                   │
+│                         ↓                                    │
+│                  ┌─────────────┐                            │
+│                  │  COMBINER   │ ← 1 LLM call               │
+│                  │ (Response)  │   (optional)               │
+│                  └──────┬──────┘                            │
+│                         ↓                                    │
+│                        END                                   │
+└─────────────────────────────────────────────────────────────┘
+
+TOTAL: 2 LLM calls → 2-3 seconds
+```
+
+**Benefits:**
+- **1-2 LLM calls** instead of 50+
+- **No recursion** - explicit transitions
+- **Deterministic** - same input = same path
+- **Cost effective** - 25-50x fewer tokens
+- **Fast** - 2-10 second responses
+
+---
+
+## Theoretical Foundations
+
+### What is LangGraph?
+
+LangGraph is a framework for building **stateful, multi-step AI applications** using explicit graphs. Unlike ReAct patterns where the LLM decides what to do next in a loop, LangGraph defines explicit states and transitions.
+
+Key concepts:
+
+| Concept | Description |
+|---------|-------------|
+| **State** | The data that flows through the graph |
+| **Node** | A function that transforms state |
+| **Edge** | Connection between nodes |
+| **Conditional Edge** | Edge that chooses next node based on state |
+| **Channel** | How state updates are merged |
+
+### State Machines vs ReAct Loops
+
+| Aspect | ReAct Loop | State Machine |
+|--------|------------|---------------|
+| **Control Flow** | LLM decides | Code defines |
+| **Predictability** | Variable | Deterministic |
+| **LLM Calls** | O(n) per step | O(1) total |
+| **Debugging** | Hard (black box) | Easy (explicit) |
+| **Parallelism** | Limited | Native support |
+
+### Why Deterministic Flow Matters
+
+1. **Reproducibility**: Same input produces same execution path
+2. **Debugging**: You can trace exactly which nodes executed
+3. **Testing**: Unit test each node independently
+4. **Cost Control**: Predictable token usage
+5. **SLA Compliance**: Guaranteed response times
+
+---
+
+## Architecture
+
+### Graph Overview
+
+```
+                    ┌──────────────────────────────────────────────────────┐
+                    │                    ORCHESTRATOR GRAPH                 │
+                    │                                                       │
+   User Input ──────┤                                                       │
+                    │   ┌─────────┐                                        │
+                    │   │  START  │                                        │
+                    │   └────┬────┘                                        │
+                    │        │                                             │
+                    │        ▼                                             │
+                    │   ┌──────────┐     Structured Output                 │
+                    │   │  ROUTER  │ ◄── (Zod Schema)                      │
+                    │   │  (LLM)   │     Intent Classification             │
+                    │   └────┬─────┘                                       │
+                    │        │                                             │
+                    │   ┌────┴────┬────────────┬─────────────┐            │
+                    │   ▼         ▼            ▼             ▼            │
+                    │ greeting  clarify    single       multi            │
+                    │   │         │        intent       intent           │
+                    │   │         │           │            │              │
+                    │   │         │           ▼            ▼              │
+                    │   │         │     ┌──────────┐  ┌──────────┐       │
+                    │   │         │     │ HANDLER  │  │ HANDLER  │ ...   │
+                    │   │         │     │ (No LLM) │  │ (No LLM) │       │
+                    │   │         │     └────┬─────┘  └────┬─────┘       │
+                    │   │         │          │             │              │
+                    │   │         │          └──────┬──────┘              │
+                    │   │         │                 ▼                     │
+                    │   │         │          ┌──────────┐                 │
+                    │   └─────────┴─────────►│ COMBINER │                 │
+                    │                        │  (LLM)   │                 │
+                    │                        └────┬─────┘                 │
+                    │                             │                       │
+                    │                             ▼                       │
+                    │                        ┌────────┐                   │
+   Response ◄───────┤                        │  END   │                   │
+                    │                        └────────┘                   │
+                    └──────────────────────────────────────────────────────┘
+```
+
+### Node Responsibilities
+
+| Node | Purpose | LLM Calls | Input | Output |
+|------|---------|-----------|-------|--------|
+| **Router** | Classify user intent | 1 | User message | Intent[] |
+| **Task Handler** | Execute task operations | 0 | Intent | JSON result |
+| **Customer Handler** | Execute customer operations | 0 | Intent | JSON result |
+| **Page Handler** | Execute page operations | 0 | Intent | JSON result |
+| **Combiner** | Synthesize response | 0-1 | All results | User text |
+
+### Data Flow
+
+1. **User Input** → Raw message string
+2. **Router** → Parses to structured `Intent[]`
+3. **Handlers** → Execute operations, return JSON
+4. **Combiner** → Converts JSON to natural language
+5. **Response** → Text for user
+
+---
+
+## Nodes Reference
+
+### Router Node
+
+The router is the "brain" that classifies user intent in a single LLM call.
+
+**Location**: `lib/graph/nodes/router.ts`
+
+**Features**:
+- **Structured Output**: Uses Zod schema for reliable JSON extraction
+- **Multi-Intent**: Detects multiple intents in one message
+- **Retry Logic**: Handles malformed responses from local models
+- **Provider Agnostic**: Works with OpenAI, Anthropic, Ollama, LM Studio
+
+**Output Schema**:
+
+```typescript
+const RouterOutputSchema = z.object({
+    intents: z.array(z.object({
+        type: z.enum(['task', 'customer', 'page', 'greeting', 'clarification']),
+        action: z.enum(['list', 'create', 'update', 'delete', 'search', 'get', 'unknown']),
+        parameters: z.record(z.unknown()),
+        originalText: z.string(),
+    })),
+    needsClarification: z.boolean(),
+    clarificationQuestion: z.string().nullable(),
+})
+```
+
+**Examples**:
+
+```
+Input: "Show my tasks"
+Output: {
+    intents: [{
+        type: "task",
+        action: "list",
+        parameters: {},
+        originalText: "Show my tasks"
+    }],
+    needsClarification: false
+}
+
+Input: "Create task 'Buy milk' high priority and find customer ABC"
+Output: {
+    intents: [
+        { type: "task", action: "create", parameters: { title: "Buy milk", priority: "high" } },
+        { type: "customer", action: "search", parameters: { query: "ABC" } }
+    ],
+    needsClarification: false
+}
+```
+
+### Handler Nodes
+
+Handlers execute operations **without LLM calls** - they directly call entity services.
+
+#### Task Handler
+
+**Location**: `lib/graph/nodes/task-handler.ts`
+
+**Supported Actions**:
+- `list` - Get all tasks with optional filters
+- `get` - Get single task by ID
+- `create` - Create new task
+- `update` - Update task fields
+- `search` - Find tasks by query
+- `delete` - Delete task
+
+**Example**:
+
+```typescript
+// Intent: { type: "task", action: "create", parameters: { title: "Buy milk", priority: "high" } }
+
+// Handler calls:
+await TasksService.create(userId, teamId, { title: "Buy milk", priority: "high" })
+
+// Returns:
+{
+    success: true,
+    operation: "create",
+    data: { id: "123", title: "Buy milk", priority: "high", status: "todo" },
+    message: "Created task: Buy milk"
+}
+```
+
+#### Customer Handler
+
+**Location**: `lib/graph/nodes/customer-handler.ts`
+
+**Supported Actions**:
+- `list` - Get all customers
+- `get` - Get single customer
+- `create` - Create new customer
+- `update` - Update customer fields
+- `search` - Find customers by name, email, phone, account
+- `delete` - Delete customer
+
+#### Page Handler
+
+**Location**: `lib/graph/nodes/page-handler.ts`
+
+**Supported Actions**:
+- `list` - Get published pages
+- `get` - Get page by ID or slug
+- `search` - Find pages by title
+
+> **Note**: Create/Update/Delete for pages returns error directing to Page Builder UI.
+
+### Combiner Node
+
+The combiner converts JSON handler results into natural language.
+
+**Location**: `lib/graph/nodes/combiner.ts`
+
+**Optimization**: For simple single-intent operations, uses **template-based responses** without LLM:
+
+```typescript
+// Template response (no LLM needed):
+// Intent: list 3 tasks
+"Found 3 task(s):
+• Buy milk (high) - todo
+• Call client (medium) - in-progress
+• Review docs (low) - done"
+
+// LLM response (multi-intent or complex):
+// Uses GPT-4o-mini to synthesize natural response
+```
+
+**When LLM is used**:
+- Multiple handler results (task + customer)
+- List/search with >5 items
+- Complex data requiring summarization
+
+**When templates are used**:
+- Single intent operations
+- Simple list/search with <=5 items
+- CRUD confirmations
+
+---
+
+## State Schema
+
+### OrchestratorState
+
+The complete state that flows through the graph:
+
+```typescript
+interface OrchestratorState {
+    // ---- Input (immutable) ----
+    input: string                          // User message
+    sessionId: string                      // Session identifier
+    context: AgentContext                  // userId, teamId
+    conversationHistory: BaseMessage[]     // Recent messages (last 5)
+
+    // ---- Router Output ----
+    intents: Intent[]                      // Classified intents
+    needsClarification: boolean            // If true, ask user
+    clarificationQuestion?: string         // Question to ask
+
+    // ---- Handler Outputs ----
+    handlerResults: {                      // JSON from each handler
+        task?: TaskHandlerResult
+        customer?: CustomerHandlerResult
+        page?: PageHandlerResult
+    }
+    completedHandlers: IntentType[]        // Which handlers ran
+
+    // ---- Final Output ----
+    finalResponse: string | null           // Text for user
+    error: string | null                   // Error message
+
+    // ---- Tracing ----
+    traceId?: string                       // Observability trace
+}
+```
+
+### Intent Type
+
+```typescript
+interface Intent {
+    type: 'task' | 'customer' | 'page' | 'greeting' | 'clarification'
+    action: 'list' | 'create' | 'update' | 'delete' | 'search' | 'get' | 'unknown'
+    parameters: Record<string, unknown>
+    originalText: string
+    confidence?: number
+}
+```
+
+### Handler Results
+
+```typescript
+interface TaskHandlerResult {
+    success: boolean
+    operation: 'list' | 'create' | 'update' | 'delete' | 'search' | 'get'
+    data: TaskData[] | TaskData | null
+    count?: number
+    message: string
+    error?: string
+}
+```
+
+---
+
+## Routing Logic
+
+### From Router to Handlers
+
+The `routeByIntents` function determines execution path:
+
+```typescript
+function routeByIntents(state: OrchestratorState): RouterRoute {
+    const intentTypes = state.intents.map(i => i.type)
+
+    // Handle special cases
+    if (state.error) return 'error'
+    if (state.needsClarification) return 'clarification'
+    if (intentTypes.length === 1 && intentTypes[0] === 'greeting') return 'greeting'
+
+    // Determine handler combination
+    const hasTask = intentTypes.includes('task')
+    const hasCustomer = intentTypes.includes('customer')
+    const hasPage = intentTypes.includes('page')
+
+    if (hasTask && hasCustomer && hasPage) return 'all_handlers'
+    if (hasTask && hasCustomer) return 'task_and_customer'
+    if (hasTask && hasPage) return 'task_and_page'
+    if (hasCustomer && hasPage) return 'customer_and_page'
+    if (hasTask) return 'task_only'
+    if (hasCustomer) return 'customer_only'
+    if (hasPage) return 'page_only'
+
+    return 'clarification'
+}
+```
+
+### Handler Sequencing
+
+For multi-intent requests, handlers execute sequentially:
+
+```
+task_and_customer: Task Handler → Customer Handler → Combiner
+task_and_page:     Task Handler → Page Handler → Combiner
+all_handlers:      Task Handler → Customer Handler → Page Handler → Combiner
+```
+
+Each handler's output is **merged** into `handlerResults`, preserving all results.
+
+### Conditional Transitions
+
+```typescript
+// After task handler
+function routeAfterTask(state): string {
+    if (needsCustomer && !completedCustomer) return 'customer_handler'
+    if (needsPage && !completedPage) return 'page_handler'
+    return 'combiner'
+}
+```
+
+---
+
+## Multi-Model Compatibility
+
+### Structured Output Methods
+
+Different LLM providers support different structured output methods:
+
+| Provider | Method | Notes |
+|----------|--------|-------|
+| OpenAI API | `functionCalling` | Most reliable |
+| Anthropic | `functionCalling` | Tool use |
+| Ollama | `functionCalling` | Most models |
+| LM Studio | `jsonSchema` | OpenAI-compatible servers |
+
+The `getStructuredOutputMethod()` helper auto-detects the best method:
+
+```typescript
+import { getStructuredOutputMethod } from '../../providers'
+
+const method = getStructuredOutputMethod({ provider: 'openai' })
+// Returns: 'functionCalling' for real OpenAI, 'jsonSchema' for LM Studio
+
+const structuredModel = model.withStructuredOutput(schema, { method })
+```
+
+### Retry Logic with Zod Validation
+
+Local models (LM Studio, Ollama) may produce malformed JSON. The router includes retry logic:
+
+```typescript
+async function invokeRouterWithRetry(model, messages, method) {
+    for (let attempt = 1; attempt <= 3; attempt++) {
+        // Try 1: Structured output
+        const result = await tryStructuredOutput(model, messages, method)
+        if (result) return result
+
+        // Try 2: Manual JSON parsing (fallback)
+        const manual = await tryManualJsonParsing(model, messages)
+        if (manual) return manual
+
+        // Exponential backoff before retry
+        await sleep(500 * Math.pow(2, attempt - 1))
+    }
+    throw new Error('Router failed after retries')
+}
+
+function tryManualJsonParsing(model, messages) {
+    const result = await model.invoke(messages)
+    const content = result.content as string
+
+    // Extract JSON from markdown code blocks or raw text
+    const json = extractJsonFromResponse(content)
+    const parsed = JSON.parse(json)
+
+    // Validate with Zod
+    return RouterOutputSchema.safeParse(parsed)
+}
+```
+
+### Provider Configuration
+
+Configure providers in theme's `langchain.config.ts`:
+
+```typescript
+'orchestrator': {
+    provider: 'anthropic',              // Claude for router
+    model: 'claude-3-haiku-20240307',
+    temperature: 0.1,                   // Low for consistency
+},
+'task-assistant': {
+    provider: 'openai',                 // GPT-4o-mini for handlers
+    model: 'gpt-4o-mini',
+    temperature: 0.3,
+}
+```
+
+---
+
+## Configuration
+
+### Environment Variables
+
+```env
+# Enable graph orchestrator (recommended)
+LANGCHAIN_USE_GRAPH_ORCHESTRATOR=true
+
+# Provider settings
+LANGCHAIN_DEFAULT_PROVIDER=openai
+
+# Debug mode
+LANGCHAIN_PLUGIN_DEBUG=true
+
+# Provider-specific
+OPENAI_API_KEY=sk-...
+ANTHROPIC_API_KEY=sk-ant-...
+```
+
+### Graph Configuration
+
+```typescript
+interface GraphConfig {
+    maxHistoryMessages: number     // Default: 5
+    routerTemperature: number      // Default: 0.1
+    combinerTemperature: number    // Default: 0.3
+    parallelExecution: boolean     // Default: true (future)
+    handlerTimeout: number         // Default: 30000ms
+}
+```
+
+### Usage
+
+```typescript
+import { invokeOrchestrator } from '@/contents/plugins/langchain/lib/graph'
+
+const result = await invokeOrchestrator(
+    message,
+    sessionId,
+    { userId, teamId },
+    conversationHistory,
+    {
+        config: {
+            maxHistoryMessages: 10,
+            routerTemperature: 0.05,
+        },
+        traceId: 'trace-123',
+    }
+)
+```
+
+---
+
+## Performance Comparison
+
+### Benchmarks
+
+| Metric | Graph Orchestrator | ReAct Loops | Improvement |
+|--------|-------------------|-------------|-------------|
+| **LLM Calls (single intent)** | 1 | 20-30 | 20-30x |
+| **LLM Calls (multi intent)** | 2 | 50+ | 25-50x |
+| **Response Time (single)** | 2-3s | 10-15s | 5x |
+| **Response Time (multi)** | 3-5s | 60-120s | 20-30x |
+| **Token Cost** | ~500 | ~15,000 | 30x |
+| **Recursion Errors** | Never | Frequent | - |
+| **Timeout Errors** | Never | Common | - |
+
+### Real-World Example
+
+**Query**: "Show my tasks and find the account number for StartupXYZ"
+
+| Metric | Graph | ReAct |
+|--------|-------|-------|
+| Total LLM calls | 2 | 47 |
+| Router calls | 1 | - |
+| Orchestrator iterations | - | 15 |
+| Task agent iterations | - | 12 |
+| Customer agent iterations | - | 20 |
+| Response time | 4.2s | 127s (timeout) |
+| Tokens used | 892 | 23,450 |
+| Estimated cost | $0.002 | $0.047 |
+
+---
+
+## Examples
+
+### Single Intent Flow
+
+```typescript
+// User: "List my tasks"
+
+// 1. Router (1 LLM call)
+{
+    intents: [{ type: "task", action: "list", parameters: {} }],
+    needsClarification: false
+}
+
+// 2. Route: task_only → task_handler
+
+// 3. Task Handler (0 LLM calls)
+{
+    handlerResults: {
+        task: {
+            success: true,
+            operation: "list",
+            data: [{ id: "1", title: "Task A" }, ...],
+            count: 5
+        }
+    }
+}
+
+// 4. Combiner (0 LLM calls - uses template)
+{
+    finalResponse: "Found 5 task(s):\n• Task A\n• Task B..."
+}
+
+// TOTAL: 1 LLM call
+```
+
+### Multi-Intent Flow
+
+```typescript
+// User: "Show tasks and find StartupXYZ phone number"
+
+// 1. Router (1 LLM call)
+{
+    intents: [
+        { type: "task", action: "list", parameters: {} },
+        { type: "customer", action: "search", parameters: { query: "StartupXYZ", fields: ["phone"] } }
+    ]
+}
+
+// 2. Route: task_and_customer → task_handler → customer_handler → combiner
+
+// 3. Task Handler (0 LLM)
+// 4. Customer Handler (0 LLM)
+{
+    handlerResults: {
+        task: { success: true, data: [...], count: 3 },
+        customer: { success: true, data: [{ name: "StartupXYZ", phone: "+1-555-1234" }] }
+    }
+}
+
+// 5. Combiner (1 LLM call - multi-result)
+{
+    finalResponse: "Found 3 tasks. StartupXYZ phone number is +1-555-1234."
+}
+
+// TOTAL: 2 LLM calls
+```
+
+### Error Handling
+
+```typescript
+// User: "Update task 999" (non-existent)
+
+// Task Handler returns error:
+{
+    handlerResults: {
+        task: {
+            success: false,
+            operation: "update",
+            data: null,
+            error: "Task not found",
+            message: "Could not find task with ID 999"
+        }
+    }
+}
+
+// Combiner generates friendly error:
+{
+    finalResponse: "I couldn't find that task. Would you like to see your task list?"
+}
+```
+
+---
+
+## Debugging
+
+### Enable Debug Mode
+
+```env
+LANGCHAIN_PLUGIN_DEBUG=true
+```
+
+### Debug Output
+
+```
+[Router] Classifying intent for: Show my tasks
+[Router] Classified intents: [{"type":"task","action":"list"}]
+[Task Handler] Processing intent: list
+[Task Handler] Found 5 tasks
+[Combiner] Using template-based response
+```
+
+### Observability
+
+Use the [Observability Dashboard](../04-advanced/01-observability.md) to trace execution:
+
+- View trace timeline with each node
+- See provider/model used per node
+- Inspect input/output at each step
+- Track token usage and costs
+
+---
+
+## Migration from ReAct
+
+### Step 1: Enable Graph Mode
+
+```env
+LANGCHAIN_USE_GRAPH_ORCHESTRATOR=true
+```
+
+### Step 2: Configure Providers
+
+Ensure providers are configured in `langchain.config.ts`:
+
+```typescript
+'orchestrator': {
+    provider: 'anthropic',  // Claude for reliable classification
+    model: 'claude-3-haiku-20240307',
+    temperature: 0.1,
+}
+```
+
+### Step 3: Test
+
+```bash
+# Run your test suite
+pnpm test
+
+# Monitor observability dashboard
+# /sector7/ai-observability
+```
+
+### Fallback
+
+The system supports automatic fallback:
+
+```typescript
+// In orchestrator.ts
+if (isGraphOrchestratorEnabled()) {
+    return processWithGraphOrchestrator(message, context)
+}
+return processWithOrchestrator(message, context)  // Legacy ReAct
+```
+
+---
+
+## Best Practices
+
+### 1. Keep Router Focused
+
+The router should only classify intent. Keep the system prompt simple and focused on intent extraction.
+
+### 2. Use Low Temperature
+
+```typescript
+routerTemperature: 0.1  // Consistent classification
+combinerTemperature: 0.3  // Slightly creative responses
+```
+
+### 3. Handle All Intent Types
+
+Ensure your handlers cover all possible actions for each entity type.
+
+### 4. Monitor Performance
+
+Use observability to track:
+- Intent classification accuracy
+- Handler execution times
+- Combiner usage (template vs LLM)
+
+### 5. Test Multi-Intent
+
+Always test with multi-intent queries to ensure proper sequencing.
+
+---
+
+## Related Documentation
+
+- [Observability & Tracing](../04-advanced/01-observability.md) - Debug and monitor execution
+- [Token Tracking](../04-advanced/02-token-tracking.md) - Cost management
+- [Configuration](../01-getting-started/03-configuration.md) - Provider setup
+- [Legacy Orchestration](./02-legacy-react.md) - ReAct-based approach (deprecated)