@sochdb/sochdb 0.4.1 → 0.4.3

package/README.md

@@ -1,8 +1,212 @@
 # SochDB Node.js SDK
 
-**Dual-mode architecture: Embedded (FFI) + Server (gRPC/IPC)**  
+**LLM-Optimized Embedded Database with Native Vector Search**
+
+---
+
+## Installation
+
+```bash
+npm install @sochdb/sochdb
+```
+
+Or from source:
+```bash
+cd sochdb-typescript-sdk
+npm install
+```
+
+---
+
+## Architecture: Flexible Deployment
+
+**Tri-mode architecture: Embedded + Concurrent + Server (gRPC/IPC)**  
 Choose the deployment mode that fits your needs.
 
+---
+
+# SochDB Node.js SDK Documentation
+
+**LLM-Optimized Embedded Database with Native Vector Search**
+
+---
+
+## Table of Contents
+
+1. [Quick Start](#1-quick-start)
+2. [Installation](#2-installation)
+3. [Features](#3-features)
+   - [Namespace API](#namespace-api---multi-tenant-isolation)
+   - [Priority Queue API](#priority-queue-api---task-processing)
+4. [Architecture Overview](#4-architecture-overview)
+5. [Core Key-Value Operations](#5-core-key-value-operations)
+6. [Transactions (ACID with SSI)](#6-transactions-acid-with-ssi)
+7. [Query Builder](#7-query-builder)
+8. [Prefix Scanning](#8-prefix-scanning)
+9. [SQL Operations](#9-sql-operations)
+10. [Table Management & Index Policies](#10-table-management--index-policies)
+11. [Namespaces & Collections](#11-namespaces--collections)
+12. [Priority Queues](#12-priority-queues)
+13. [Vector Search](#13-vector-search)
+14. [Hybrid Search (Vector + BM25)](#14-hybrid-search-vector--bm25)
+15. [Graph Operations](#15-graph-operations)
+16. [Temporal Graph (Time-Travel)](#16-temporal-graph-time-travel)
+17. [Semantic Cache](#17-semantic-cache)
+18. [Memory System](#18-memory-system)
+19. [Session Management](#19-session-management)
+20. [Context Query Builder (LLM Optimization)](#20-context-query-builder-llm-optimization)
+21. [Atomic Multi-Index Writes](#21-atomic-multi-index-writes)
+22. [Recovery & WAL Management](#22-recovery--wal-management)
+23. [Checkpoints & Snapshots](#23-checkpoints--snapshots)
+24. [Compression & Storage](#24-compression--storage)
+25. [Statistics & Monitoring](#25-statistics--monitoring)
+26. [Distributed Tracing](#26-distributed-tracing)
+27. [Workflow & Run Tracking](#27-workflow--run-tracking)
+28. [Server Mode (gRPC Client)](#28-server-mode-grpc-client)
+29. [IPC Client (Unix Sockets)](#29-ipc-client-unix-sockets)
+30. [Standalone VectorIndex](#30-standalone-vectorindex)
+31. [Vector Utilities](#31-vector-utilities)
+32. [Data Formats (TOON/JSON/Columnar)](#32-data-formats-toonjsoncolumnar)
+33. [Policy Service](#33-policy-service)
+34. [MCP (Model Context Protocol)](#34-mcp-model-context-protocol)
+35. [Configuration Reference](#35-configuration-reference)
+36. [Error Handling](#36-error-handling)
+37. [Async Support](#37-async-support)
+38. [Building & Development](#38-building--development)
+39. [Complete Examples](#39-complete-examples)
+40. [Migration Guide](#40-migration-guide)
+
+---
+
+## 1. Quick Start
+
+### Concurrent Embedded Mode
+
+For web applications with multiple Node.js processes (PM2 cluster, multiple workers):
+
+```typescript
+import { EmbeddedDatabase } from '@sochdb/sochdb';
+import express from 'express';
+
+// Open in concurrent mode - multiple processes can access simultaneously
+const db = EmbeddedDatabase.openConcurrent('./web_db');
+
+const app = express();
+
+app.get('/user/:id', async (req, res) => {
+  // Multiple concurrent requests can read simultaneously (~100ns)
+  const data = await db.get(Buffer.from(`user:${req.params.id}`));
+  if (!data) {
+    res.status(404).json({ error: 'not found' });
+    return;
+  }
+  res.send(data);
+});
+
+app.post('/user/:id', async (req, res) => {
+  // Writes are automatically coordinated (~60µs amortized)
+  await db.put(Buffer.from(`user:${req.params.id}`), req.body);
+  res.json({ status: 'ok' });
+});
+
+// Check concurrent mode status
+console.log(`Concurrent mode: ${db.isConcurrent}`);  // true
+
+// Start with PM2 cluster mode (multiple workers can access DB)
+// pm2 start app.js -i max
+app.listen(3000);
+```
+
+### Performance
+
+| Operation | Standard Mode | Concurrent Mode |
+|-----------|---------------|-----------------|
+| Read (single process) | ~100ns | ~100ns |
+| Read (multi-process) | **Blocked** ❌ | ~100ns ✅ |
+| Write | ~5ms (fsync) | ~60µs (amortized) |
+| Max concurrent readers | 1 | 1024 |
+
+### PM2 Cluster Example
+
+```bash
+# Install PM2
+npm install -g pm2
+
+# Start with automatic worker scaling
+pm2 start server.js -i max
+
+# All workers can access the same database concurrently!
+pm2 logs
+```
+
+### PM2 Ecosystem File
+
+```javascript
+// ecosystem.config.js
+module.exports = {
+  apps: [{
+    name: 'api-server',
+    script: './server.js',
+    instances: 'max',  // Scale across all CPU cores
+    exec_mode: 'cluster',
+    env: {
+      NODE_ENV: 'production',
+      DB_PATH: './shared_db'  // All workers use same DB
+    }
+  }]
+};
+```
+
+```bash
+# Deploy with ecosystem file
+pm2 start ecosystem.config.js
+
+# Monitor all workers
+pm2 monit
+```
+
+### Docker Compose with PM2
+
+```yaml
+version: '3.8'
+services:
+  app:
+    build: .
+    environment:
+      - NODE_ENV=production
+      - INSTANCES=4  # 4 PM2 workers
+    volumes:
+      - ./data:/app/data  # Shared database volume
+    ports:
+      - "3000:3000"
+    command: pm2-runtime start ecosystem.config.js
+```
+
+### Kubernetes Deployment
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sochdb-app
+spec:
+  replicas: 4  # 4 pods share the database
+  template:
+    spec:
+      containers:
+      - name: app
+        image: myapp:latest
+        volumeMounts:
+        - name: db-storage
+          mountPath: /app/data
+      volumes:
+      - name: db-storage
+        persistentVolumeClaim:
+          claimName: sochdb-pvc  # Shared PVC with ReadWriteMany
+```
+
+---
+
 ## Features
 
 ### Memory System - LLM-Native Memory for AI Agents
@@ -239,21 +443,354 @@ console.log(`Pending: ${stats.pending}, Completed: ${stats.completed}`);
 
 ---
 
-## Installation
+---
 
+## System Requirements
+
+### For Concurrent Mode
+
+- **SochDB Core**: Latest version
+- **Node.js**: 14.0+ (18.0+ recommended)
+- **Native Library**: `libsochdb_storage.{dylib,so}`
+- **FFI**: Koffi (automatically installed)
+
+**Operating Systems:**
+- ✅ Linux (Ubuntu 20.04+, RHEL 8+)
+- ✅ macOS (10.15+, both Intel and Apple Silicon)
+- ⚠️  Windows (requires native builds)
+
+**File Descriptors:**
+- Default limit: 1024 (sufficient for most workloads)
+- For high concurrency with PM2: `ulimit -n 4096`
+
+**Memory:**
+- Standard mode: ~50MB base + data
+- Concurrent mode: +4KB per concurrent reader slot (1024 slots = ~4MB overhead)
+- PM2 cluster: Each worker has independent memory
+
+---
+
+## Troubleshooting
+
+### "Database is locked" Error (Standard Mode)
+
+```
+Error: SQLITE_BUSY: database is locked
+```
+
+**Solution**: Use concurrent mode for multi-process access:
+
+```typescript
+// ❌ Standard mode - PM2 cluster will fail
+const db = new EmbeddedDatabase('./data.db');
+
+// ✅ Concurrent mode - PM2 cluster works!
+const db = EmbeddedDatabase.openConcurrent('./data.db');
+```
+
+### Library Not Found Error
+
+```
+Error: Dynamic library 'libsochdb_storage.dylib' not found
+```
+
+**macOS**:
 ```bash
-npm install @sochdb/sochdb
+# Build and install library
+cd /path/to/sochdb
+cargo build --release
+sudo cp target/release/libsochdb_storage.dylib /usr/local/lib/
 ```
 
-Or from source:
+**Linux**:
 ```bash
-cd sochdb-typescript-sdk
-npm install
+cd /path/to/sochdb
+cargo build --release
+sudo cp target/release/libsochdb_storage.so /usr/local/lib/
+sudo ldconfig
 ```
 
+**Development Mode** (no install):
+```bash
+export DYLD_LIBRARY_PATH=/path/to/sochdb/target/release  # macOS
+export LD_LIBRARY_PATH=/path/to/sochdb/target/release    # Linux
+```
+
+### PM2 Cluster Issues
+
+**Symptom**: Workers crash with "database locked"
+
+**Solution**: Ensure concurrent mode is used:
+```javascript
+// ecosystem.config.js
+module.exports = {
+  apps: [{
+    name: 'api',
+    script: './server.js',
+    instances: 4,
+    exec_mode: 'cluster',
+    env: {
+      USE_CONCURRENT_MODE: 'true'  // Flag to use openConcurrent()
+    }
+  }]
+};
+```
+
+```typescript
+// server.ts
+const db = process.env.USE_CONCURRENT_MODE 
+  ? EmbeddedDatabase.openConcurrent('./db')
+  : new EmbeddedDatabase('./db');
+
+console.log('Concurrent mode:', db.isConcurrent);  // Should be true
+```
+
+### Docker Volume Permissions
+
+**Symptom**: `EACCES: permission denied` when opening database
+
+**Solution**: Fix volume ownership:
+```dockerfile
+FROM node:18
+WORKDIR /app
+
+# Create data directory with correct permissions
+RUN mkdir -p /app/data && chown -R node:node /app
+
+# Switch to non-root user
+USER node
+
+COPY --chown=node:node . .
+RUN npm install
+
+CMD ["npm", "start"]
+```
+
+### Performance Issues
+
+**Symptom**: Concurrent reads slower than expected
+
+**Check 1** - Verify concurrent mode:
+```typescript
+if (!db.isConcurrent) {
+    console.error('Database is not in concurrent mode!');
+    process.exit(1);
+}
+```
+
+**Check 2** - Monitor PM2 workers:
+```bash
+pm2 monit  # Real-time monitoring
+pm2 logs --lines 200  # Check for errors
+```
+
+**Check 3** - Batch writes:
+```typescript
+// ❌ Slow - individual writes
+for (const item of items) {
+    await collection.insert(item);
+}
+
+// ✅ Fast - batch write
+await collection.insertBatch(items);
+```
+
+---
+
+## 🆕 Vector Search - Native HNSW
+
+SochDB now includes **native HNSW (Hierarchical Navigable Small World)** vector search for sub-millisecond similarity search across millions of vectors.
+
+### Quick Start - Vector Search
+
+```typescript
+import { HnswIndex } from '@sochdb/sochdb';
+
+// Create HNSW index
+const index = new HnswIndex({
+  dimension: 384,           // Vector dimension
+  maxConnections: 16,       // M parameter (default: 16)
+  efConstruction: 200,      // Build quality (default: 200)
+  efSearch: 100             // Search quality (default: 100)
+});
+
+// Insert vectors (batch is 10-100× faster)
+index.insertBatch(
+  ['doc1', 'doc2', 'doc3'],
+  [[1.0, 2.0, ...], [3.0, 4.0, ...], [5.0, 6.0, ...]]
+);
+
+// Search for similar vectors
+const results = index.search(queryVector, 10);
+console.log(results);
+// [{ id: 'doc1', distance: 0.15 }, { id: 'doc3', distance: 0.23 }, ...]
+
+// Clean up
+index.close();
+```
+
+### Performance Comparison
+
+| Implementation | 10K vectors | 100K vectors | 1M vectors |
+|----------------|-------------|--------------|------------|
+| **Linear Scan (old)** | ~50ms | ~500ms | ~5000ms |
+| **Native HNSW (new)** | <0.5ms | <1ms | <1ms |
+| **Speedup** | **100×** | **500×** | **5000×** |
+
+### Two Ways to Use Vector Search
+
+#### 1. Direct HNSW API (Recommended for Production)
+
+Best performance, full control:
+
+```typescript
+import { HnswIndex } from '@sochdb/sochdb';
+
+const index = new HnswIndex({ dimension: 1536 });
+index.insertBatch(ids, embeddings);
+const results = index.search(queryEmbedding, 10);
+```
+
+**✅ Use when:**
+- You need maximum performance
+- Working with large datasets (>10K vectors)
+- Building RAG/AI applications
+- Have existing embedding pipeline
+
+#### 2. Collection API (Simple, High-Level)
+
+Convenient API with metadata support:
+
+```typescript
+import { Database } from '@sochdb/sochdb';
+
+const db = await Database.open('./mydb');
+const ns = await db.createNamespace({ name: 'docs' });
+
+const collection = await ns.createCollection({
+  name: 'embeddings',
+  dimension: 384,
+  indexed: true  // Note: Currently uses linear search in embedded mode
+});
+
+await collection.insert([1.0, 2.0, ...], { title: 'Document 1' }, 'doc1');
+const results = await collection.search({ queryVector: [...], k: 10 });
+```
+
+**⚠️ Current Limitation:** Collection API uses O(n) linear search in embedded mode. For production use with >10K vectors, use:
+- Direct HNSW API (above), OR
+- gRPC Server Mode (see below)
+
+#### 3. gRPC Server Mode (Production-Ready)
+
+For distributed systems, multi-language support:
+
+```typescript
+import { SochDBClient } from '@sochdb/sochdb';
+
+// Start server: sochdb-grpc --port 50051
+const client = new SochDBClient({ address: 'localhost:50051' });
+
+// Create HNSW index
+await client.createIndex('docs', {
+  dimension: 1536,
+  config: { m: 16, ef_construction: 200 },
+  metric: 'cosine'
+});
+
+// Insert and search
+await client.insertBatch('docs', ids, vectors);
+const results = await client.search('docs', queryVector, 10);
+```
+
+**✅ Full HNSW support with:**
+- Native Rust implementation
+- Persistence
+- Distributed queries
+- Multi-language clients
+
+### Migration from Linear Search
+
+If you're using the Collection API with large datasets and experiencing slow search:
+
+**Before (slow):**
+```typescript
+// O(n) scan through all documents
+const results = await collection.search({ queryVector, k: 10 });
+```
+
+**After (fast) - Option 1: Use HnswIndex directly:**
+```typescript
+import { HnswIndex } from '@sochdb/sochdb';
+
+const index = new HnswIndex({ dimension: 384 });
+index.insertBatch(ids, vectors);
+const results = index.search(queryVector, 10); // <1ms
+```
+
+**After (fast) - Option 2: Use gRPC mode:**
+```bash
+# Terminal 1: Start server
+sochdb-grpc --port 50051
+
+# Terminal 2: Use client
+```
+```typescript
+const client = new SochDBClient({ address: 'localhost:50051' });
+await client.createIndex('docs', { dimension: 384 });
+const results = await client.search('docs', queryVector, 10);
+```
+
+### Complete Examples
+
+- **[06_native_vector_search.ts](https://github.com/sochdb/sochdb-nodejs-examples/blob/main/06_native_vector_search.ts)** - Direct HNSW usage with benchmarks
+- **[AI PDF Chatbot](https://github.com/sochdb/sochdb-nodejs-examples/tree/main/ai-pdf-chatbot-langchain)** - LangChain RAG example
+
+### API Reference
+
+```typescript
+// HnswIndex Configuration
+interface HnswConfig {
+  dimension: number;           // Required: vector dimension
+  maxConnections?: number;     // M parameter (default: 16)
+  efConstruction?: number;     // Build quality (default: 200)
+  efSearch?: number;           // Search quality (default: 100)
+}
+
+// Search Result
+interface SearchResult {
+  id: string;                  // Vector ID
+  distance: number;            // Distance (lower = more similar)
+}
+
+// Main Methods
+class HnswIndex {
+  constructor(config: HnswConfig)
+  insert(id: string, vector: number[]): void
+  insertBatch(ids: string[], vectors: number[][]): void
+  search(queryVector: number[], k: number, fast?: boolean): SearchResult[]
+  searchUltra(queryVector: number[], k: number): SearchResult[]
+  close(): void
+  
+  // Properties
+  get length(): number         // Number of vectors
+  get dimension(): number      // Vector dimension
+  get efSearch(): number
+  set efSearch(value: number)  // Adjust search quality
+}
+```
+
+### Roadmap
+
+- **Current**: Direct HNSW FFI bindings
+- **Next**: Collection API auto-uses HNSW in embedded mode
+- **Future**: Persistent HNSW indexes with disk storage
+
+---
+
 # SochDB Node.js SDK Documentation
 
-**Version 0.4.0** | LLM-Optimized Embedded Database with Native Vector Search
+**LLM-Optimized Embedded Database with Native Vector Search**
 
 ---
 
@@ -3484,7 +4021,7 @@ A:
 A: Yes! Both modes have the same API. Change `Database.open()` to `SochDBClient()` and vice versa.
 
 **Q: Do temporal graphs work in embedded mode?**  
-A: Yes! As of v0.4.0, temporal graphs work in both embedded and server modes with identical APIs.
+A: Yes! Temporal graphs work in both embedded and server modes with identical APIs.
 
 **Q: Is embedded mode slower than server mode?**  
 A: Embedded mode is faster for single-process use (no network overhead). Server mode is better for distributed deployments.