@operor/knowledge 0.1.0

package/README.md

@@ -0,0 +1,457 @@
+# @operor/knowledge
+
+Knowledge base system with vector search, retrieval pipeline, and document ingestors for Operor.
+
+## Overview
+
+`@operor/knowledge` provides a complete knowledge base solution for AI agents, enabling them to retrieve relevant context from documents, FAQs, and web content. The system uses vector embeddings for semantic search and includes specialized pipelines for ingestion and retrieval.
+
+Inspired by [Astra Agent KB design](https://github.com/datastax/astra-assistants-api) and [OpenClaw's extensible plugin architecture](https://github.com/openclaw/openclaw), this package provides a modular, production-ready knowledge base with pluggable ingestors.
+
+## Quick Start
+
+### Installation
+
+```bash
+pnpm add @operor/knowledge
+```
+
+### Basic Usage
+
+```typescript
+import {
+  SQLiteKnowledgeStore,
+  EmbeddingService,
+  TextChunker,
+  IngestionPipeline,
+  RetrievalPipeline,
+} from '@operor/knowledge';
+
+// 1. Initialize components
+const store = new SQLiteKnowledgeStore('./knowledge.db', 1536);
+await store.initialize();
+
+const embedder = new EmbeddingService({
+  provider: 'openai',
+  apiKey: process.env.OPENAI_API_KEY,
+  model: 'text-embedding-3-small',
+});
+
+const chunker = new TextChunker({ chunkSize: 500, chunkOverlap: 50 });
+const ingestion = new IngestionPipeline(store, embedder, chunker);
+const retrieval = new RetrievalPipeline(store, embedder, 0.85);
+
+// 2. Ingest documents
+await ingestion.ingest({
+  sourceType: 'url',
+  content: 'Your document content here...',
+  title: 'Getting Started Guide',
+});
+
+// 3. Query the knowledge base
+const result = await retrieval.retrieve('How do I get started?', { limit: 5 });
+console.log(result.context); // Formatted context for LLM injection
+console.log(result.isFaqMatch); // true if FAQ fast-path matched
+```
+
+## Architecture
+
+### Core Components
+
+- **KnowledgeStore**: SQLite-backed storage with `sqlite-vec` for vector search
+- **EmbeddingService**: Multi-provider embeddings via Vercel AI SDK (OpenAI, Google, Mistral, Cohere, Ollama)
+- **TextChunker**: Document chunking using LangChain text splitters
+- **IngestionPipeline**: Orchestrates document ingestion (chunk → embed → store)
+- **RetrievalPipeline**: Query-time retrieval with FAQ fast-path optimization
+
+### Ingestors (Extensible)
+
+Following OpenClaw's plugin pattern, ingestors are modular and extensible:
+
+- **UrlIngestor**: Web crawling with Readability extraction
+- **FileIngestor**: PDF, DOCX, XLSX, CSV, TXT, MD, HTML parsing
+- **WatiFaqSync**: FAQ extraction from WATI conversations via LLM
+
+## Technology Stack
+
+All dependencies use the latest stable versions:
+
+- **Vector Search**: `sqlite-vec` (0.1.7-alpha.2) — Fast vector search in SQLite
+- **Embeddings**: `ai` (6.x) + `@ai-sdk/*` (3.x) — Vercel AI SDK with multi-provider support
+- **Text Splitting**: `@langchain/textsplitters` (1.x) — Recursive and markdown splitters
+- **Document Parsing**:
+  - PDF: `unpdf` (1.4.0)
+  - DOCX: `mammoth` (1.11.0)
+  - XLSX: `xlsx` (0.18.5)
+  - HTML: `@mozilla/readability` (0.6.0) + `linkedom` (0.18.12)
+- **Storage**: `better-sqlite3` (12.x) — Fast, synchronous SQLite
+
+## Embedding Providers
+
+The `EmbeddingService` supports multiple providers via the Vercel AI SDK:
+
+| Provider | Model (default) | Dimensions | API Key Required |
+|----------|----------------|------------|------------------|
+| OpenAI | `text-embedding-3-small` | 1536 | Yes |
+| Google | `text-embedding-004` | 768 | Yes |
+| Mistral | `mistral-embed` | 1024 | Yes |
+| Cohere | `embed-english-v3.0` | 1024 | Yes |
+| Ollama | `nomic-embed-text` | 768 | No (local) |
+
+**Important**: When switching embedding providers, you must re-ingest all documents. The vector dimensions must match what the store was initialized with, or search will fail.
+
+### Provider Examples
+
+```typescript
+// OpenAI (default)
+const embedder = new EmbeddingService({
+  provider: 'openai',
+  apiKey: process.env.OPENAI_API_KEY,
+  model: 'text-embedding-3-small', // optional
+  dimensions: 1536, // optional, defaults per provider
+});
+
+// Google Gemini
+const embedder = new EmbeddingService({
+  provider: 'google',
+  apiKey: process.env.GOOGLE_API_KEY,
+  model: 'text-embedding-004',
+});
+
+// Mistral
+const embedder = new EmbeddingService({
+  provider: 'mistral',
+  apiKey: process.env.MISTRAL_API_KEY,
+});
+
+// Cohere
+const embedder = new EmbeddingService({
+  provider: 'cohere',
+  apiKey: process.env.COHERE_API_KEY,
+});
+
+// Ollama (local, no API key)
+const embedder = new EmbeddingService({
+  provider: 'ollama',
+  model: 'nomic-embed-text',
+  baseURL: 'http://localhost:11434/v1', // optional
+});
+```
+
+## Usage Guide
+
+### Ingesting Documents
+
+#### From URL
+
+```typescript
+import { UrlIngestor } from '@operor/knowledge';
+
+const urlIngestor = new UrlIngestor(ingestion);
+
+// Single URL
+await urlIngestor.ingestUrl('https://docs.example.com/guide');
+
+// Sitemap (batch ingest)
+await urlIngestor.ingestSitemap('https://example.com/sitemap.xml', {
+  maxPages: 50,
+});
+
+// Crawl website (BFS traversal)
+await urlIngestor.crawl('https://docs.example.com', {
+  maxPages: 20,
+  maxDepth: 2,
+});
+```
+
+#### From File
+
+```typescript
+import { FileIngestor } from '@operor/knowledge';
+
+const fileIngestor = new FileIngestor(ingestion);
+
+// Supports: PDF, DOCX, XLSX, CSV, TXT, MD, HTML
+await fileIngestor.ingestFile('./docs/manual.pdf', 'User Manual');
+await fileIngestor.ingestFile('./data/products.xlsx', 'Product Catalog');
+```
+
+#### FAQ Entries
+
+```typescript
+// Manual FAQ entry
+await ingestion.ingestFaq(
+  'What is the return policy?',
+  'You can return items within 30 days of purchase.'
+);
+
+// Batch FAQ sync from WATI conversations
+import { WatiFaqSync } from '@operor/knowledge';
+
+const faqSync = new WatiFaqSync(ingestion, async (conversation) => {
+  // Your LLM extraction logic here
+  return [
+    { question: 'How do I reset my password?', answer: 'Click Forgot Password...' },
+  ];
+});
+
+await faqSync.syncFromConversations(conversations, {
+  minAnswerLength: 20,
+  maxPairs: 100,
+});
+```
+
+### Retrieving Context
+
+```typescript
+// Basic retrieval
+const result = await retrieval.retrieve('What is the return policy?', {
+  limit: 5,
+  scoreThreshold: 0.7,
+});
+
+console.log(result.isFaqMatch); // true if FAQ fast-path matched
+console.log(result.context); // Formatted context for LLM injection
+console.log(result.results); // Raw search results with scores
+
+// Filter by source type
+const result = await retrieval.retrieve('pricing info', {
+  limit: 3,
+  sourceTypes: ['url', 'file'], // exclude FAQs
+});
+```
+
+### FAQ Fast-Path
+
+The retrieval pipeline includes an optimization for FAQs:
+
+1. First searches FAQ documents only
+2. If score ≥ 0.85 (configurable), returns immediately (fast-path)
+3. Otherwise, searches full knowledge base
+
+This ensures instant responses for common questions while maintaining comprehensive search for complex queries.
+
+```typescript
+// Adjust FAQ threshold (default: 0.85)
+const retrieval = new RetrievalPipeline(store, embedder, 0.90);
+```
+
+## CLI Usage
+
+The `@operor/cli` package provides commands for managing the knowledge base:
+
+```bash
+# Add documents
+operor kb add-url https://docs.example.com/guide
+operor kb add-file ./manual.pdf
+operor kb add-faq "What are your hours?" "We are open 9-5 Mon-Fri"
+
+# Search
+operor kb search "return policy" -n 5
+
+# List all documents
+operor kb list
+
+# Delete a document
+operor kb delete <doc-id>
+
+# Show statistics
+operor kb stats
+```
+
+## API Reference
+
+### EmbeddingService
+
+```typescript
+class EmbeddingService {
+  constructor(config: EmbeddingServiceConfig);
+
+  embed(text: string): Promise<number[]>;
+  embedMany(texts: string[]): Promise<number[][]>;
+
+  get dimensions(): number;
+  get provider(): string;
+
+  static defaultDimensions(provider: string, model?: string): number;
+}
+
+interface EmbeddingServiceConfig {
+  provider: 'openai' | 'google' | 'mistral' | 'cohere' | 'ollama';
+  apiKey?: string;
+  model?: string;
+  baseURL?: string;
+  dimensions?: number;
+}
+```
+
+### SQLiteKnowledgeStore
+
+```typescript
+class SQLiteKnowledgeStore implements KnowledgeStore {
+  constructor(dbPath?: string, dimensions?: number);
+
+  initialize(): Promise<void>;
+  close(): Promise<void>;
+
+  addDocument(doc: KBDocument): Promise<void>;
+  getDocument(id: string): Promise<KBDocument | null>;
+  listDocuments(): Promise<KBDocument[]>;
+  deleteDocument(id: string): Promise<void>;
+
+  addChunks(chunks: KBChunk[]): Promise<void>;
+  search(query: string, embedding: number[], options?: KBSearchOptions): Promise<KBSearchResult[]>;
+  searchByEmbedding(embedding: number[], options?: KBSearchOptions): Promise<KBSearchResult[]>;
+
+  getDimensions(): number;
+}
+```
+
+### TextChunker
+
+```typescript
+class TextChunker {
+  constructor(options?: ChunkOptions);
+
+  chunk(text: string, options?: ChunkOptions): Promise<string[]>;
+  chunkMarkdown(markdown: string, options?: ChunkOptions): Promise<string[]>;
+}
+
+interface ChunkOptions {
+  chunkSize?: number; // default: 500
+  chunkOverlap?: number; // default: 50
+}
+```
+
+### IngestionPipeline
+
+```typescript
+class IngestionPipeline {
+  constructor(store: KnowledgeStore, embedder: EmbeddingService, chunker: TextChunker);
+
+  ingest(input: IngestInput): Promise<KBDocument>;
+  ingestFaq(question: string, answer: string, metadata?: Record<string, any>): Promise<KBDocument>;
+}
+
+interface IngestInput {
+  sourceType: 'url' | 'file' | 'faq' | 'annotation';
+  content: string;
+  title?: string;
+  sourceUrl?: string;
+  fileName?: string;
+  metadata?: Record<string, any>;
+}
+```
+
+### RetrievalPipeline
+
+```typescript
+class RetrievalPipeline {
+  constructor(store: KnowledgeStore, embedder: EmbeddingService, faqThreshold?: number);
+
+  retrieve(query: string, options?: KBSearchOptions): Promise<RetrievalResult>;
+}
+
+interface RetrievalResult {
+  results: KBSearchResult[];
+  context: string; // Formatted for LLM injection
+  isFaqMatch: boolean; // true if FAQ fast-path matched
+}
+```
+
+### Types
+
+```typescript
+interface KBDocument {
+  id: string;
+  sourceType: 'url' | 'file' | 'faq' | 'annotation';
+  sourceUrl?: string;
+  fileName?: string;
+  title?: string;
+  content: string;
+  metadata?: Record<string, any>;
+  createdAt: number;
+  updatedAt: number;
+}
+
+interface KBChunk {
+  id: string;
+  documentId: string;
+  content: string;
+  chunkIndex: number;
+  embedding?: number[];
+  metadata?: Record<string, any>;
+}
+
+interface KBSearchResult {
+  chunk: KBChunk;
+  document: KBDocument;
+  score: number; // 0-1, higher is better
+  distance: number; // raw vector distance
+}
+
+interface KBSearchOptions {
+  limit?: number;
+  scoreThreshold?: number;
+  sourceTypes?: ('url' | 'file' | 'faq' | 'annotation')[];
+  metadata?: Record<string, any>;
+}
+```
+
+## Extending with Custom Ingestors
+
+Following OpenClaw's plugin architecture pattern, you can create custom ingestors:
+
+```typescript
+export class CustomIngestor {
+  private pipeline: IngestionPipeline;
+
+  constructor(pipeline: IngestionPipeline) {
+    this.pipeline = pipeline;
+  }
+
+  async ingestCustomSource(source: string): Promise<KBDocument> {
+    // 1. Extract content from your source
+    const content = await this.extractContent(source);
+
+    // 2. Use pipeline to ingest
+    return this.pipeline.ingest({
+      sourceType: 'url', // or 'file', 'annotation'
+      content,
+      title: 'Custom Source',
+      metadata: { source: 'custom' },
+    });
+  }
+
+  private async extractContent(source: string): Promise<string> {
+    // Your extraction logic
+    return 'extracted content';
+  }
+}
+```
+
+## Troubleshooting
+
+### Dimension Mismatch Errors
+
+If you see errors like "Dimension mismatch for inserted vector", it means:
+- You switched embedding providers without re-ingesting documents
+- The store was initialized with different dimensions than the embedder produces
+
+**Solution**: Delete the KB database and re-ingest all documents with the new provider.
+
+```bash
+rm knowledge.db knowledge.db-shm knowledge.db-wal
+operor kb add-url https://docs.example.com  # re-ingest
+```
+
+### Performance Tips
+
+- Use batch ingestion for multiple documents (sitemap, crawl)
+- Adjust chunk size based on your content (smaller for FAQs, larger for docs)
+- Set appropriate `scoreThreshold` to filter low-quality results
+- Use `sourceTypes` filter to narrow search scope
+
+## License
+
+MIT