@nexo-labs/payload-typesense 1.3.0

package/README.md

@@ -0,0 +1,185 @@
+# @nexo-labs/payload-typesense
+
+A powerful, production-ready search plugin that integrates [Typesense](https://typesense.org/) with [Payload CMS](https://payloadcms.com/). This plugin provides lightning-fast, typo-tolerant search capabilities with real-time synchronization, and a comprehensive RAG (Retrieval Augmented Generation) system for building AI-powered conversational agents.
+
+## Features
+
+*   **Real-time Sync**: Automatically syncs your Payload collections to Typesense.
+*   **Flexible Indexing**: Map a single Payload collection to multiple Typesense indices (tables) with different configurations.
+*   **Advanced Search**:
+    *   **Semantic Search**: Vector-based search using embeddings.
+    *   **Keyword Search**: Traditional text matching with typo tolerance.
+    *   **Hybrid Search**: Combines vector and keyword search for optimal relevance.
+*   **RAG & AI Agents**:
+    *   Built-in support for conversational AI agents.
+    *   Integrates with OpenAI and Gemini.
+    *   Manages chat sessions and history.
+    *   Streaming responses (SSE).
+*   **Optimized Performance**: Configurable HNSW parameters for vector search and batch processing for sync.
+
+## Installation
+
+```bash
+pnpm add @nexo-labs/payload-typesense
+```
+
+## Configuration
+
+The plugin is configured in your `payload.config.ts`.
+
+### Basic Setup
+
+```typescript
+import { buildConfig } from 'payload/config';
+import { typesenseSearch } from '@nexo-labs/payload-typesense';
+
+export default buildConfig({
+  // ...
+  plugins: [
+    typesenseSearch({
+      typesense: {
+        apiKey: process.env.TYPESENSE_API_KEY,
+        nodes: [
+          {
+            host: process.env.TYPESENSE_HOST,
+            port: 443,
+            protocol: 'https',
+          },
+        ],
+      },
+      features: {
+        sync: { enabled: true },
+        search: { enabled: true },
+        embedding: {
+          type: 'openai',
+          apiKey: process.env.OPENAI_API_KEY,
+          model: 'text-embedding-3-small',
+        },
+      },
+      collections: {
+        posts: [
+          {
+            enabled: true,
+            tableSuffix: 'v1',
+            searchFields: ['title', 'content'],
+            chunkingStrategy: { type: 'markdown' },
+          },
+        ],
+      },
+    }),
+  ],
+});
+```
+
+### Sync Configuration
+
+You can define how each collection is synced to Typesense. The `collections` object maps collection slugs to an array of `CollectionTableConfig`.
+
+```typescript
+collections: {
+  posts: [
+    {
+      enabled: true,
+      // Optional suffix for the Typesense collection name
+      // Result: "posts_summary"
+      tableSuffix: 'summary',
+      // Fields to include in the Typesense document
+      searchFields: ['title', 'excerpt'],
+      // Chunking strategy for long text
+      chunkingStrategy: { type: 'simple', chunkSize: 500 },
+    },
+    {
+      enabled: true,
+      // Explicitly naming the table overrides the default naming convention
+      // Result: "my_custom_posts_index"
+      tableName: 'my_custom_posts_index',
+      searchFields: ['title', 'content'],
+      chunkingStrategy: { type: 'markdown' },
+    }
+  ]
+}
+```
+
+### RAG & Agents Configuration
+
+Enable RAG to build chat interfaces on top of your data.
+
+```typescript
+features: {
+  rag: {
+    enabled: true,
+    agents: [
+      {
+        slug: 'support-bot',
+        name: 'Support Assistant',
+        systemPrompt: 'You are a helpful support assistant.',
+        llmModel: 'gpt-4o-mini',
+        // Collections this agent can access
+        searchCollections: ['posts_full_text'],
+      }
+    ]
+  }
+}
+```
+
+## Usage
+
+### Search API
+
+The plugin exposes endpoints for searching your data.
+
+**Endpoint:** `/api/typesense/search`
+
+```typescript
+// Example using fetch
+const response = await fetch('/api/typesense/search', {
+  method: 'POST',
+  body: JSON.stringify({
+    query: 'how to install',
+    collections: ['posts_full_text'],
+  })
+});
+```
+
+### RAG Chat API
+
+**Endpoint:** `/api/typesense/rag/chat`
+
+This endpoint supports Server-Sent Events (SSE) for streaming AI responses.
+
+```typescript
+// Example client-side code (simplified)
+const eventSource = new EventSource('/api/typesense/rag/chat?message=hello');
+
+eventSource.onmessage = (event) => {
+  const data = JSON.parse(event.data);
+  if (data.type === 'token') {
+    console.log(data.data); // Streamed text
+  }
+};
+```
+
+## Roadmap
+
+We are actively working to improve the modularity and flexibility of the plugin.
+
+### 1. Modular Embedding Strategies (High Priority)
+**Goal:** Allow different embedding models for different collections (tables).
+*   **Current State:** Global embedding configuration only.
+*   **Future:**
+    *   Move `embedding` config to `CollectionTableConfig`.
+    *   Support "Table X" -> Gemini, "Table Y" -> OpenAI Large.
+    *   Update RAG agents to dynamically select the correct embedding model based on the collection being queried.
+
+### 2. Expanded Provider Support
+*   **Embeddings:** Add support for Cohere, HuggingFace, and local models (via Ollama).
+*   **LLMs:** Add support for Anthropic (Claude), Mistral, and local LLMs.
+
+### 4. Admin UI
+*   Dashboard within Payload Admin to view Typesense collection status.
+*   Buttons to manually trigger re-sync or re-indexing.
+*   Visual management of RAG agents.
+
+### 5. Advanced Sync Controls
+*   **Soft Deletes:** Option to mark documents as deleted in Typesense instead of removing them.
+*   **Conditional Sync:** specialized `syncCondition` function to control which documents are synced based on their content (e.g., only sync `published` posts).