@nextsparkjs/plugin-langchain 0.1.0-beta.1

package/docs/01-getting-started/01-overview.md

@@ -0,0 +1,196 @@
+# LangChain Plugin - Overview
+
+## Introduction
+
+The LangChain Plugin is a comprehensive AI agent infrastructure that enables themes to integrate conversational AI capabilities into their applications. Built on top of [LangChain](https://langchain.com/) and [LangGraph](https://langchain-ai.github.io/langgraph/), this plugin provides a complete framework for creating, configuring, and deploying AI agents with:
+
+- **Multiple LLM Provider Support**: OpenAI, Anthropic, and Ollama (local)
+- **Persistent Conversation Memory**: Database-backed session management
+- **Tool-based Architecture**: Agents interact with your data through well-defined tools
+- **Multi-tenant Security**: Row-Level Security (RLS) for data isolation
+- **Orchestration Patterns**: Route requests to specialized agents
+
+## Core Concepts
+
+### What is an AI Agent?
+
+An AI agent is a system that uses a Large Language Model (LLM) to reason about problems and take actions through tools. Unlike simple chat interfaces, agents can:
+
+1. **Reason** about the user's intent
+2. **Plan** a sequence of actions
+3. **Execute** tools to retrieve or modify data
+4. **Respond** with accurate, data-driven answers
+
+```
+User Message → Agent (LLM + Tools) → Tool Execution → Response
+```
+
+### Plugin vs Theme Responsibility
+
+This plugin follows a clear separation of concerns:
+
+| Plugin Provides | Theme Implements |
+|-----------------|------------------|
+| Agent creation infrastructure | Agent definitions and configuration |
+| Memory persistence | Entity-specific tools |
+| Provider abstraction | System prompts |
+| Tool building utilities | API endpoints |
+| Message serialization | Orchestration logic |
+
+The plugin is **entity-agnostic** - it doesn't know about your tasks, customers, or pages. Themes define what data agents can access and how.
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                         THEME LAYER                              │
+├─────────────────────────────────────────────────────────────────┤
+│  langchain.config.ts    │  agents/*.md   │  tools/*.ts          │
+│  (Agent Definitions)    │  (Prompts)     │  (Entity Tools)      │
+├─────────────────────────────────────────────────────────────────┤
+│                         PLUGIN LAYER                             │
+├─────────────────────────────────────────────────────────────────┤
+│  createAgent()   │  memoryStore    │  providers     │  buildTools│
+│  (Factory)       │  (Persistence)  │  (LLM Access)  │  (Helpers) │
+├─────────────────────────────────────────────────────────────────┤
+│                       LANGCHAIN/LANGGRAPH                        │
+├─────────────────────────────────────────────────────────────────┤
+│  LLM Providers   │  ReAct Pattern  │  Message Types │  Tool Calls│
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Key Features
+
+### 1. Multi-Provider Support
+
+Switch between LLM providers without changing your agent code:
+
+```typescript
+// Use local Ollama for development
+{ provider: 'ollama', model: 'llama3.2:3b' }
+
+// Use OpenAI for production
+{ provider: 'openai', model: 'gpt-4o' }
+
+// Use Anthropic for specific agents
+{ provider: 'anthropic', model: 'claude-3-5-sonnet-20241022' }
+```
+
+### 2. Persistent Conversations
+
+All conversations are automatically persisted to PostgreSQL with:
+
+- **Session Management**: Create, list, rename, pin conversations
+- **Message Limits**: Configurable sliding window (default: 50 messages)
+- **Conversation Limits**: Maximum 50 conversations per user
+- **Multi-tenancy**: Isolated by userId + teamId
+
+### 3. Tool-Based Data Access
+
+Agents interact with your data through tools, not direct database access:
+
+```typescript
+const taskTools = [
+    {
+        name: 'list_tasks',
+        description: 'List all tasks for the current user',
+        schema: z.object({ status: z.string().optional() }),
+        func: async ({ status }) => TasksService.list(userId, { status })
+    }
+]
+```
+
+### 4. Orchestration Patterns
+
+Route requests to specialized agents based on user intent:
+
+```
+User: "Show me my tasks"
+  → Orchestrator analyzes intent
+  → Routes to Task Agent
+  → Task Agent uses task tools
+  → Returns task list
+```
+
+## Use Cases
+
+### Single Agent Architecture
+
+Best for simple applications with unified data access:
+
+- Personal productivity apps
+- Small business tools
+- Prototypes and MVPs
+
+```
+User → Single Agent (all tools) → Response
+```
+
+### Multi-Agent Orchestration
+
+Best for complex applications with specialized domains:
+
+- Enterprise CRM systems
+- Content management platforms
+- E-commerce with multiple entities
+
+```
+User → Orchestrator → [Task Agent | Customer Agent | Page Agent] → Response
+```
+
+## Getting Started
+
+To integrate the LangChain plugin into your theme:
+
+1. **Install Dependencies**: Run the database migration
+2. **Configure Environment**: Set up LLM provider credentials
+3. **Create Configuration**: Define agents in `langchain.config.ts`
+4. **Build Tools**: Create tools for your entities
+5. **Write Prompts**: Define agent behavior in markdown files
+6. **Expose API**: Create chat endpoints for your frontend
+
+Continue to the [Architecture](../02-core-concepts/01-architecture.md) section for detailed technical concepts, or jump to [Installation](./02-installation.md) to get started immediately.
+
+## Documentation Structure
+
+See the full [Documentation Index](../index.md) for navigation.
+
+### Getting Started
+
+| Section | Description |
+|---------|-------------|
+| [Overview](./01-overview.md) | Introduction and core concepts (you are here) |
+| [Installation](./02-installation.md) | Setup and dependencies |
+| [Configuration](./03-configuration.md) | Theme-level configuration |
+
+### Core Concepts
+
+| Section | Description |
+|---------|-------------|
+| [Architecture](../02-core-concepts/01-architecture.md) | Technical architecture and patterns |
+| [Agents](../02-core-concepts/02-agents.md) | Creating and customizing agents |
+| [Tools](../02-core-concepts/03-tools.md) | Building tools for agents |
+
+### Orchestration
+
+| Section | Description |
+|---------|-------------|
+| [Graph Orchestrator](../03-orchestration/01-graph-orchestrator.md) | **Recommended** - Modern state-machine orchestration |
+| [Legacy ReAct](../03-orchestration/02-legacy-react.md) | Deprecated ReAct-based approach |
+
+### Advanced Topics
+
+| Section | Description |
+|---------|-------------|
+| [Observability](../04-advanced/01-observability.md) | Tracing, metrics, and debugging dashboard |
+| [Token Tracking](../04-advanced/02-token-tracking.md) | Token usage and cost monitoring |
+| [Streaming](../04-advanced/03-streaming.md) | Real-time SSE streaming responses |
+| [Guardrails](../04-advanced/04-guardrails.md) | Security: injection detection, PII masking |
+
+### Reference
+
+| Section | Description |
+|---------|-------------|
+| [API Reference](../05-reference/01-api-reference.md) | Complete API documentation |
+| [Customization](../05-reference/02-customization.md) | Advanced customization guide |
+| [Examples](../05-reference/03-examples.md) | Real-world implementation examples |

package/docs/01-getting-started/02-installation.md

@@ -0,0 +1,368 @@
+# Installation
+
+This guide covers the complete setup process for integrating the LangChain plugin into your theme.
+
+## Prerequisites
+
+Before installing, ensure you have:
+
+- PostgreSQL database (Supabase recommended)
+- Node.js 18+ with pnpm
+- At least one LLM provider configured (Ollama recommended for development)
+
+## Step 1: Database Migration
+
+The plugin requires a `langchain_sessions` table for conversation persistence.
+
+### Run the Migration
+
+```bash
+# Navigate to the migrations folder
+cd contents/plugins/langchain/migrations
+
+# Apply the migration to your database
+psql $DATABASE_URL -f 001_langchain_memory.sql
+```
+
+Or using your migration tool:
+
+```bash
+pnpm db:migrate
+```
+
+### Migration Contents
+
+The migration creates:
+
+```sql
+-- Table for conversation storage
+CREATE TABLE public."langchain_sessions" (
+    id              TEXT PRIMARY KEY,
+    "userId"        TEXT NOT NULL REFERENCES users(id),
+    "teamId"        TEXT NOT NULL REFERENCES teams(id),
+    "sessionId"     TEXT NOT NULL,
+    name            TEXT DEFAULT NULL,
+    "isPinned"      BOOLEAN DEFAULT false,
+    messages        JSONB NOT NULL DEFAULT '[]',
+    metadata        JSONB DEFAULT '{}',
+    "maxMessages"   INTEGER DEFAULT 50,
+    "expiresAt"     TIMESTAMPTZ DEFAULT NULL,
+    "createdAt"     TIMESTAMPTZ DEFAULT now(),
+    "updatedAt"     TIMESTAMPTZ DEFAULT now(),
+    UNIQUE ("userId", "teamId", "sessionId")
+);
+
+-- Performance indexes
+CREATE INDEX idx_sessions_lookup ON langchain_sessions(...);
+CREATE INDEX idx_sessions_pinned ON langchain_sessions(...);
+CREATE INDEX idx_sessions_updated ON langchain_sessions(...);
+
+-- Row-Level Security
+ALTER TABLE langchain_sessions ENABLE ROW LEVEL SECURITY;
+```
+
+## Step 2: Environment Variables
+
+Configure your LLM provider(s) in `.env`:
+
+### Required Variables
+
+```env
+# Enable the plugin
+LANGCHAIN_PLUGIN_ENABLED=true
+
+# Optional: Enable debug console logging
+LANGCHAIN_PLUGIN_DEBUG=false
+
+# Enable file logging (core environment variable)
+LOG_ENABLED=true  # Logs to logger/ai/
+```
+
+### Provider Configuration
+
+#### Option A: Ollama (Recommended for Development)
+
+Ollama runs locally and is free. Install from [ollama.com](https://ollama.com):
+
+```bash
+# Install Ollama (macOS)
+brew install ollama
+
+# Pull a model
+ollama pull llama3.2:3b     # Fast, lightweight
+ollama pull qwen2.5:7b      # Better quality
+ollama pull mistral         # Good balance
+
+# Start Ollama server
+ollama serve
+```
+
+Configure in `.env`:
+
+```env
+LANGCHAIN_OLLAMA_BASE_URL=http://localhost:11434
+LANGCHAIN_OLLAMA_MODEL=qwen2.5:7b
+```
+
+#### Option B: OpenAI
+
+```env
+OPENAI_API_KEY=sk-...
+LANGCHAIN_OPENAI_MODEL=gpt-4o-mini
+```
+
+#### Option C: Anthropic
+
+```env
+ANTHROPIC_API_KEY=sk-ant-...
+LANGCHAIN_ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
+```
+
+#### Option D: LM Studio (OpenAI-Compatible)
+
+LM Studio provides a local server with OpenAI API compatibility:
+
+```env
+# Point OpenAI provider to LM Studio
+LANGCHAIN_OPENAI_BASE_URL=http://localhost:1234/v1
+LANGCHAIN_OPENAI_MODEL=local-model
+# API key not required for local
+```
+
+### Complete Example
+
+```env
+# Plugin settings
+LANGCHAIN_PLUGIN_ENABLED=true
+LANGCHAIN_PLUGIN_DEBUG=false
+
+# File logging (core)
+LOG_ENABLED=true  # Logs to logger/ai/
+
+# Ollama (development)
+LANGCHAIN_OLLAMA_BASE_URL=http://localhost:11434
+LANGCHAIN_OLLAMA_MODEL=qwen2.5:7b
+
+# OpenAI (production fallback)
+OPENAI_API_KEY=sk-...
+LANGCHAIN_OPENAI_MODEL=gpt-4o-mini
+
+# Anthropic (optional)
+ANTHROPIC_API_KEY=sk-ant-...
+LANGCHAIN_ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
+```
+
+## Step 3: Theme Structure
+
+Create the following structure in your theme:
+
+```
+contents/themes/your-theme/
+├── lib/
+│   └── langchain/
+│       ├── langchain.config.ts    # Agent configuration
+│       ├── orchestrator.ts        # (optional) Orchestration logic
+│       ├── agents/
+│       │   ├── index.ts           # Prompt loader
+│       │   └── *.md               # System prompts
+│       └── tools/
+│           ├── index.ts           # Tool exports
+│           └── *.ts               # Entity tools
+└── api/
+    └── ai/
+        └── chat/
+            └── route.ts           # Chat API endpoint
+```
+
+### Using Presets
+
+The plugin provides presets as starting points:
+
+```bash
+# Copy configuration preset
+cp contents/plugins/langchain/presets/lib/langchain.config.ts.preset \
+   contents/themes/your-theme/lib/langchain/langchain.config.ts
+
+# Copy API endpoint preset
+mkdir -p contents/themes/your-theme/api/ai/chat
+cp contents/plugins/langchain/presets/api/chat/route.ts.preset \
+   contents/themes/your-theme/api/ai/chat/route.ts
+
+# Copy tools preset
+mkdir -p contents/themes/your-theme/lib/langchain/tools
+cp contents/plugins/langchain/presets/lib/tools/entity-tools.ts.preset \
+   contents/themes/your-theme/lib/langchain/tools/example.ts
+
+# Copy prompt preset
+mkdir -p contents/themes/your-theme/lib/langchain/agents
+cp contents/plugins/langchain/presets/agents/entity-assistant.md.preset \
+   contents/themes/your-theme/lib/langchain/agents/assistant.md
+```
+
+## Step 4: Basic Configuration
+
+Create your agent configuration:
+
+```typescript
+// contents/themes/your-theme/lib/langchain/langchain.config.ts
+
+import type {
+    ThemeLangChainConfig,
+    AgentDefinition,
+    ToolContext,
+} from '@/contents/plugins/langchain/types/langchain.types'
+import { createAgentHelpers } from '@/contents/plugins/langchain/lib/agent-helpers'
+import { createMyTools } from './tools/my-entity'
+
+export const AGENTS: Record<string, AgentDefinition> = {
+    'my-assistant': {
+        provider: 'ollama',
+        temperature: 0.3,
+        description: 'Assistant for my entity',
+        systemPrompt: 'my-assistant',  // loads agents/my-assistant.md
+        createTools: (context: ToolContext) => createMyTools(context),
+    },
+}
+
+export const langchainConfig: ThemeLangChainConfig = {
+    defaultProvider: 'ollama',
+    defaultTemperature: 0.3,
+    agents: AGENTS,
+}
+
+// Export helpers
+const helpers = createAgentHelpers(AGENTS, {
+    provider: langchainConfig.defaultProvider,
+    temperature: langchainConfig.defaultTemperature,
+})
+
+export const {
+    getAgentConfig,
+    getAgentModelConfig,
+    getAgentTools,
+    getAgentPromptName,
+    hasAgent,
+    getAgentNames,
+} = helpers
+```
+
+## Step 5: Verify Installation
+
+### Test Provider Connection
+
+```typescript
+import { isProviderAvailable, getAvailableProviders } from '@/contents/plugins/langchain/plugin.config'
+
+// Check available providers
+console.log('Available:', getAvailableProviders())
+// ['ollama', 'openai'] (depends on env config)
+
+// Check specific provider
+console.log('Ollama:', isProviderAvailable('ollama'))
+// true/false
+```
+
+### Test Agent Creation
+
+```typescript
+import { createAgent } from '@/contents/plugins/langchain/lib/agent-factory'
+
+const agent = await createAgent({
+    sessionId: 'test-session',
+    systemPrompt: 'You are a helpful assistant. Respond with "Hello, I am working!"',
+    tools: [],
+    modelConfig: {
+        provider: 'ollama',
+        model: 'llama3.2:3b',
+        temperature: 0.3,
+    },
+})
+
+const response = await agent.chat('Test message')
+console.log(response.content)
+// Should receive a response from the LLM
+```
+
+### Test Memory Persistence
+
+```typescript
+import { memoryStore } from '@/contents/plugins/langchain/lib/memory-store'
+
+const context = { userId: 'test-user', teamId: 'test-team' }
+
+// Create session (sessionId is auto-generated)
+const { sessionId } = await memoryStore.createSession(context, 'Test Chat')
+console.log('Created session:', sessionId)
+
+// List sessions
+const sessions = await memoryStore.listSessions(context)
+console.log('Sessions:', sessions.length)
+
+// Clean up
+await memoryStore.clearSession(sessionId, context)
+```
+
+## Troubleshooting
+
+### Ollama Connection Failed
+
+```
+Error: Failed to connect to Ollama at http://localhost:11434
+```
+
+**Solution:**
+
+```bash
+# Check if Ollama is running
+curl http://localhost:11434/api/tags
+
+# Start Ollama
+ollama serve
+
+# Verify model is pulled
+ollama list
+```
+
+### OpenAI API Key Invalid
+
+```
+Error: Invalid API Key
+```
+
+**Solution:**
+- Verify `OPENAI_API_KEY` is set correctly
+- Check the key starts with `sk-`
+- Ensure the key has API access (not just ChatGPT access)
+
+### Database Connection Error
+
+```
+Error: relation "langchain_sessions" does not exist
+```
+
+**Solution:**
+
+```bash
+# Run the migration
+psql $DATABASE_URL -f contents/plugins/langchain/migrations/001_langchain_memory.sql
+```
+
+### Provider Not Available
+
+```
+Error: Provider 'openai' is not available. Configure OPENAI_API_KEY
+```
+
+**Solution:**
+- Set the required environment variables
+- Restart the development server
+- Use `getAvailableProviders()` to check what's configured
+
+## Next Steps
+
+Now that the plugin is installed:
+
+1. [Configure your agents](./03-configuration.md)
+2. [Create custom tools](../02-core-concepts/03-tools.md)
+3. [Set up graph orchestration](../03-orchestration/01-graph-orchestrator.md) (recommended)
+4. [Configure observability](../04-advanced/01-observability.md)