@cmdoss/memwal-sdk 0.8.0 → 0.9.0

package/README.md

@@ -5,6 +5,18 @@
 
 **MemWal** (Memory + Walrus) - TypeScript SDK for decentralized memory storage on Sui blockchain with Walrus.
 
+## Features
+
+| Feature | Description |
+|---------|-------------|
+| **Memory CRUD** | Create, read, search, delete memories on Sui + Walrus |
+| **Unified Search** | `pdw.memory.search()` - auto-embeds query and searches |
+| **SEAL Encryption** | Identity-based encryption (enabled by default) |
+| **Vector Search** | Fast HNSW search (768 dimensions default) |
+| **AI Classification** | Auto-categorize memories (fact, event, preference, etc.) |
+| **Batch Upload** | ~90% gas savings with Quilt batching |
+| **Knowledge Graph** | Entity and relationship extraction |
+
 ## Installation
 
 ```bash
@@ -31,17 +43,19 @@ const pdw = new SimplePDWClient({
   embedding: {
     provider: 'openrouter',
     apiKey: process.env.OPENROUTER_API_KEY!,
+    dimensions: 768  // Default: 768 (fast), 1536, or 3072 (highest quality)
   }
 });
 
 await pdw.ready();
 
-// Create memory
+// Create memory (auto: embed, classify, encrypt, upload, register, index)
 const memory = await pdw.memory.create('I work at CommandOSS as a developer');
 console.log('Memory ID:', memory.id);
 
-// Search memories
-const results = await pdw.search.vector('work experience', { limit: 5 });
+// Search memories (NEW: unified search API)
+const results = await pdw.memory.search('work experience', { limit: 5 });
+console.log('Found:', results.length, 'memories');
 ```
 
 ### Browser (with dapp-kit + Slush/Sui Wallet)
@@ -72,232 +86,475 @@ function MyComponent() {
       network: 'testnet',
       userAddress: account.address,
       sui: { packageId: process.env.NEXT_PUBLIC_PACKAGE_ID! },
+      embedding: {
+        provider: 'openrouter',
+        apiKey: process.env.NEXT_PUBLIC_OPENROUTER_API_KEY!,
+        dimensions: 768
+      },
       features: { enableLocalIndexing: false }, // Disable for browser
     });
 
+    await pdw.ready();
+
     // Create memory - wallet popup appears for signing
     const memory = await pdw.memory.create('Hello from browser!');
+
+    // Search memories
+    const results = await pdw.memory.search('hello', { limit: 5 });
   };
 }
 ```
 
-## Features
+## API Reference
 
-| Feature | Description |
-|---------|-------------|
-| Memory CRUD | Create, read, update, delete memories on Sui + Walrus |
-| Vector Search | Semantic search with HNSW (3072 dimensions) |
-| AI Classification | Auto-categorize memories (fact, event, preference, etc.) |
-| SEAL Encryption | Optional identity-based encryption |
-| Batch Upload | ~90% gas savings with Quilt batching |
-| Knowledge Graph | Entity and relationship extraction |
+### Core Namespaces
 
-## Environment Variables
+The SDK provides 5 core namespaces for common operations:
 
-```env
-SUI_PRIVATE_KEY=suiprivkey1...
-PACKAGE_ID=0x...
-OPENROUTER_API_KEY=sk-or-v1-...
+```typescript
+pdw.memory   // Create, get, search, list, delete memories
+pdw.ai       // Embed, classify, extract commands
+pdw.index    // HNSW index management
+pdw.wallet   // Wallet address and balance info
+pdw.advanced // Power user features (graph, analytics, etc.)
+```
 
-# Optional
-WALRUS_AGGREGATOR=https://aggregator.walrus-testnet.walrus.space
-WALRUS_PUBLISHER=https://publisher.walrus-testnet.walrus.space
+### `pdw.memory` - Memory Operations
+
+```typescript
+// CREATE - handles everything internally (embed, encrypt, upload, register, index)
+const memory = await pdw.memory.create(content, {
+  category?: string,      // Auto-classify if not provided
+  importance?: number,    // Auto-score if not provided (1-10)
+  embedding?: number[],   // Auto-generate if not provided
+});
+
+// CREATE BATCH - upload multiple memories efficiently
+const memories = await pdw.memory.createBatch([
+  { content: 'Memory 1', category: 'fact' },
+  { content: 'Memory 2', category: 'preference' }
+]);
+
+// SEARCH - unified semantic search (auto-embeds query)
+const results = await pdw.memory.search(query, {
+  limit?: number,         // Max results (default: 10)
+  threshold?: number,     // Min similarity 0-1 (default: 0.7)
+  category?: string,      // Filter by category
+  includeContent?: boolean // Include decrypted content (default: true)
+});
+
+// GET - retrieve by ID (auto-decrypts)
+const memory = await pdw.memory.get(memoryId);
+
+// LIST - list all memories with filters
+const memories = await pdw.memory.list({
+  category?: string,
+  limit?: number,
+  offset?: number
+});
+
+// DELETE
+await pdw.memory.delete(memoryId);
+await pdw.memory.deleteBatch([id1, id2, id3]);
 ```
 
-## API Overview
+### `pdw.ai` - AI Operations
 
 ```typescript
-// Memory operations
-await pdw.memory.create(content, { category, importance });
-await pdw.memory.createBatch([...contents]);
-await pdw.memory.get(id);
-await pdw.memory.delete(id);
-
-// Search
-await pdw.search.vector(query, { limit });
-await pdw.search.byCategory('fact');
-await pdw.search.hybrid(query, { category });
-
-// AI
-await pdw.ai.embed(text);
-await pdw.ai.classify(content);
-await pdw.ai.shouldSave(content);
+// Generate embeddings
+const embedding = await pdw.ai.embed(text);
+const embeddings = await pdw.ai.embedBatch([text1, text2]);
 
-// Knowledge Graph
-await pdw.graph.extract(content);
+// Classify content
+const { category, importance } = await pdw.ai.classify(content);
+
+// Extract memory commands from user input
+const memories = pdw.ai.extractMultipleMemories(userMessage);
+// Input: "remember I like pizza and my name is John"
+// Output: ["I like pizza", "my name is John"]
+
+// Check if content should be saved as memory
+const shouldSave = await pdw.ai.shouldSave(content);
 ```
 
-## Vector Search (HNSW)
+### `pdw.index` - HNSW Index Management
 
-The SDK uses HNSW for fast vector search:
+```typescript
+// Add vector to index
+await pdw.index.add(spaceId, vectorId, vector, {
+  content: string,
+  blobId: string,
+  category: string,
+  importance: number,
+  isEncrypted: boolean,
+  forceStoreContent: boolean  // Store content even when encrypted (for RAG)
+});
 
-| Implementation | Environment | Performance |
-|----------------|-------------|-------------|
-| `hnswlib-node` | Node.js | **Fastest** (native C++) |
-| `hnswlib-wasm` | Browser + Node.js | Good (fallback) |
+// Search vectors
+const results = await pdw.index.search(spaceId, queryVector, {
+  k: 10,
+  threshold: 0.7
+});
 
-The SDK auto-detects and uses the best available implementation.
+// Get index stats
+const stats = pdw.index.getStats(spaceId);
 
-**For best performance in Node.js:**
+// Rebuild index from blockchain
+await pdw.index.rebuild(userAddress);
 
-```bash
-# Requires C++ build tools
-npm install hnswlib-node
+// Clear index
+await pdw.index.clear(spaceId);
+
+// Flush to disk
+await pdw.index.flush(spaceId);
 ```
 
-<details>
-<summary>Build tools installation</summary>
+### `pdw.wallet` - Wallet Information
 
-| Platform | Command |
-|----------|---------|
-| Windows | Install [VS Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) with C++ workload |
-| macOS | `xcode-select --install` |
-| Linux | `sudo apt-get install build-essential python3` |
+```typescript
+// Get wallet address
+const address = pdw.wallet.address;
 
-</details>
+// Get SUI balance
+const balance = await pdw.wallet.balance();
+
+// Get memory count
+const count = await pdw.wallet.memoryCount();
+```
+
+### `pdw.advanced` - Power User Features
+
+```typescript
+// Knowledge Graph
+const graph = await pdw.advanced.graph.extract(content);
+// Returns: { entities: [...], relationships: [...] }
+
+// Analytics
+const insights = await pdw.advanced.analytics.getInsights();
+
+// Manual encryption/decryption
+const encrypted = await pdw.advanced.encryption.encrypt(data);
+const decrypted = await pdw.advanced.encryption.decrypt(encrypted);
+
+// Permissions
+await pdw.advanced.permissions.grant(memoryId, targetAddress);
+await pdw.advanced.permissions.revoke(memoryId, targetAddress);
+
+// Transaction building (low-level)
+const tx = pdw.advanced.blockchain.buildCreateMemoryTx(params);
+```
+
+### Legacy Search API (Deprecated)
+
+```typescript
+// Still works but deprecated - use pdw.memory.search() instead
+await pdw.search.vector(query, { limit });      // -> pdw.memory.search()
+await pdw.search.byCategory('fact');            // -> pdw.memory.list({ category: 'fact' })
+await pdw.search.hybrid(query, { category });   // -> pdw.memory.search(query, { category })
+```
 
 ## Configuration
 
-### Basic Configuration
+### Full Configuration
 
 ```typescript
 const pdw = new SimplePDWClient({
-  signer: keypair,               // Required: Sui keypair or wallet adapter
-  network: 'testnet',            // Required: 'testnet' | 'mainnet' | 'devnet'
+  // Required
+  signer: keypair,                // Sui keypair or wallet adapter
+  network: 'testnet',             // 'testnet' | 'mainnet' | 'devnet'
 
   // Embedding configuration (required for AI features)
   embedding: {
-    provider: 'openrouter',      // 'google' | 'openai' | 'openrouter' | 'cohere'
+    provider: 'openrouter',       // 'google' | 'openai' | 'openrouter' | 'cohere'
     apiKey: process.env.OPENROUTER_API_KEY,
-    modelName: 'google/gemini-embedding-001',  // Optional: defaults per provider
-    dimensions: 3072             // Optional: defaults per provider
+    modelName: 'google/text-embedding-004',  // Optional: defaults per provider
+    dimensions: 768               // Optional: 768 (default), 1536, or 3072
   },
 
   // Optional: Sui configuration
   sui: {
-    packageId: '0x...',          // Default: from env PACKAGE_ID
-    rpcUrl: 'https://...',       // Default: from network
+    packageId: '0x...',           // Default: from env PACKAGE_ID
+    rpcUrl: 'https://...',        // Default: from network
   },
 
   // Optional: Walrus configuration
   walrus: {
-    aggregator: 'https://...',   // Default: from network
-    publisher: 'https://...',    // Default: from network
+    aggregatorUrl: 'https://aggregator.walrus-testnet.walrus.space',
+    publisherUrl: 'https://publisher.walrus-testnet.walrus.space',
+  },
+
+  // Optional: AI configuration
+  ai: {
+    apiKey: process.env.OPENROUTER_API_KEY,
+    chatModel: 'google/gemini-2.5-flash',
   },
 
   // Optional: Feature flags
   features: {
-    enableEncryption: true,      // Default: true (SEAL encryption)
-    enableLocalIndexing: true,   // Default: true (HNSW vector search)
-    enableKnowledgeGraph: true   // Default: true (entity extraction)
+    enableEncryption: true,       // Default: true (SEAL encryption)
+    enableLocalIndexing: true,    // Default: true (HNSW vector search)
+    enableKnowledgeGraph: true    // Default: true (entity extraction)
+  },
+
+  // Optional: Encryption configuration
+  encryption: {
+    enabled: true,
+    keyServers: ['0x...', '0x...'],  // Default: testnet key servers
+    threshold: 2,                    // M of N key servers required
+    accessRegistryId: '0x...'
+  },
+
+  // Optional: Index backup to Walrus (cloud sync)
+  indexBackup: {
+    enabled: true,
+    aggregatorUrl: 'https://...',
+    publisherUrl: 'https://...',
+    autoSync: false,
+    epochs: 3
   }
 });
 ```
 
-### SEAL Encryption (Enabled by Default)
+### Environment Variables
+
+```bash
+# Required
+SUI_PRIVATE_KEY=suiprivkey1...
+PACKAGE_ID=0x...
+
+# Embedding (at least one API key required)
+EMBEDDING_PROVIDER=openrouter    # google, openai, openrouter, cohere
+EMBEDDING_API_KEY=               # Falls back to provider-specific keys
+EMBEDDING_MODEL=                 # Optional: override default model
+EMBEDDING_DIMENSIONS=768         # 768 (default), 1536, 3072
+
+# Provider-specific API keys (fallback)
+OPENROUTER_API_KEY=sk-or-v1-...
+GEMINI_API_KEY=...
+OPENAI_API_KEY=sk-...
+COHERE_API_KEY=...
+
+# Walrus (optional - defaults to testnet)
+WALRUS_NETWORK=testnet
+WALRUS_AGGREGATOR=https://aggregator.walrus-testnet.walrus.space
+WALRUS_PUBLISHER=https://publisher.walrus-testnet.walrus.space
+
+# SEAL Encryption (optional - defaults configured)
+ENABLE_ENCRYPTION=true
+SEAL_KEY_SERVERS=0xKEY1,0xKEY2
+SEAL_THRESHOLD=2
+ACCESS_REGISTRY_ID=0x...
+```
+
+### Embedding Providers
 
-**🔒 Encryption is enabled by default in v0.7.0+**
+| Provider | Model Default | Dimensions | API Key |
+|----------|---------------|------------|---------|
+| `openrouter` | `google/text-embedding-004` | 768 | `OPENROUTER_API_KEY` |
+| `google` | `text-embedding-004` | 768 | `GEMINI_API_KEY` |
+| `openai` | `text-embedding-3-small` | 1536 | `OPENAI_API_KEY` |
+| `cohere` | `embed-english-v3.0` | 1024 | `COHERE_API_KEY` |
+
+**Dimensions Trade-offs:**
+- **768** (default): Fast embedding + small storage + fast search
+- **1536**: Balance of quality and speed
+- **3072**: Highest quality, slowest
+
+**Recommendation**: Use `openrouter` with 768 dimensions for best performance.
+
+## SEAL Encryption
+
+Encryption is **enabled by default** in v0.8.0+. All memory content is automatically encrypted using SEAL (Secure Encrypted Access Layer).
 
-All memory content is automatically encrypted using SEAL (Secure Encrypted Access Layer) with identity-based encryption.
+### How It Works
 
-#### Default Encryption (Recommended)
+1. **Identity-based**: Uses Sui address as encryption identity
+2. **Threshold security**: Requires M of N key servers (default: 2 of 2)
+3. **Zero gas for decryption**: No blockchain transactions required
+4. **Privacy-preserving**: Content never exposed on-chain
+
+### Default Encryption (Recommended)
 
 ```typescript
 const pdw = new SimplePDWClient({
   signer: keypair,
   network: 'testnet',
   embedding: { provider: 'openrouter', apiKey: 'key' }
-  // Encryption automatically enabled with defaults!
+  // Encryption automatically enabled!
 });
 
-// Memories are encrypted on upload, decrypted on retrieval
+// Memories encrypted on upload, decrypted on retrieval
 await pdw.memory.create('My private data');
 ```
 
-#### Custom Encryption Configuration
+### Disabling Encryption (Development Only)
 
 ```typescript
 const pdw = new SimplePDWClient({
-  signer: keypair,
-  network: 'testnet',
-  embedding: { provider: 'openrouter', apiKey: 'key' },
-
-  // Encryption configuration (optional)
-  encryption: {
-    enabled: true,              // Default: true
-    keyServers: [               // Default: testnet key servers
-      '0x73d05d62c18d9374e3ea529e8e0ed6161da1a141a94d3f76ae3fe4e99356db75',
-      '0xf5d14a81a982144ae441cd7d64b09027f116a468bd36e7eca494f750591623c8'
-    ],
-    threshold: 2,               // Default: 2 (M of N key servers required)
-    accessRegistryId: '0x...'   // Default: testnet registry
+  // ...
+  features: {
+    enableEncryption: false  // Stores content in plaintext!
   }
 });
 ```
 
-#### Environment Variables
+## Server-Side RAG
 
-```bash
-# Disable encryption (not recommended for production)
-ENABLE_ENCRYPTION=false
+For server-side applications with RAG (Retrieval-Augmented Generation), content needs to be stored in the local index even when encrypted.
 
-# Custom key servers (comma-separated)
-SEAL_KEY_SERVERS=0xKEY1,0xKEY2,0xKEY3
+### Configuration
 
-# Custom threshold
-SEAL_THRESHOLD=2
+When indexing memories server-side, use `forceStoreContent: true`:
 
-# Custom access registry
-ACCESS_REGISTRY_ID=0x...
+```typescript
+await pdw.index.add(walletAddress, vectorId, embedding, {
+  content: plaintextContent,
+  blobId: blobId,
+  isEncrypted: true,
+  forceStoreContent: true  // Store content for RAG even when encrypted
+});
 ```
 
-#### How SEAL Encryption Works
+### API Route Example
 
-- **Zero gas for decryption** - No blockchain transactions required
-- **Identity-based** - Uses Sui address as identity (userAddress)
-- **Threshold security** - Requires M of N key servers (default: 2 of 2)
-- **Cross-app sharing** - Share memories between apps with same user
-- **Privacy-preserving** - Content never exposed on-chain or in plaintext
+```typescript
+// /api/memory/index/route.ts
+export async function POST(req: Request) {
+  const { walletAddress, memoryId, content, embedding, blobId } = await req.json();
+
+  const pdw = await getReadOnlyPDWClient(walletAddress);
+
+  await pdw.index.add(walletAddress, vectorId, embedding, {
+    content,
+    blobId,
+    isEncrypted: true,
+    forceStoreContent: true  // Enable RAG for encrypted memories
+  });
+}
+```
 
-#### Disabling Encryption (Development Only)
+### Chat API with Memory Search
 
-⚠️ **Not recommended for production**. Only use for non-sensitive test data:
+```typescript
+// /api/chat/route.ts
+export async function POST(req: Request) {
+  const { messages, walletAddress } = await req.json();
+  const pdw = await getReadOnlyPDWClient(walletAddress);
+
+  // Search memories for context
+  const memories = await pdw.memory.search(userMessage, {
+    limit: 10,
+    threshold: 0.3,
+    includeContent: true
+  });
+
+  // Build prompt with memory context
+  const systemPrompt = `
+    User's memories:
+    ${memories.map(m => m.content).join('\n')}
+  `;
+
+  // Call LLM with context
+  return streamText({ model, messages, system: systemPrompt });
+}
+```
+
+## Vector Search (HNSW)
+
+The SDK uses HNSW for fast approximate nearest neighbor search:
+
+| Implementation | Environment | Performance |
+|----------------|-------------|-------------|
+| `hnswlib-node` | Node.js | **Fastest** (native C++) |
+| `hnswlib-wasm` | Browser + Node.js | Good (fallback) |
+
+The SDK auto-detects and uses the best available implementation.
+
+### For Best Performance in Node.js
+
+```bash
+# Requires C++ build tools
+npm install hnswlib-node
+```
+
+<details>
+<summary>Build tools installation</summary>
+
+| Platform | Command |
+|----------|---------|
+| Windows | Install [VS Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) with C++ workload |
+| macOS | `xcode-select --install` |
+| Linux | `sudo apt-get install build-essential python3` |
+
+</details>
+
+## Index Rebuild
+
+When users log in on a new device or the local index is lost, rebuild from blockchain:
 
 ```typescript
-const pdw = new SimplePDWClient({
-  signer: keypair,
-  network: 'testnet',
-  embedding: { provider: 'openrouter', apiKey: 'key' },
-  features: {
-    enableEncryption: false  // ⚠️ Stores content in plaintext!
-  }
-});
+import { rebuildIndexNode, hasExistingIndexNode } from '@cmdoss/memwal-sdk';
+import { SuiClient, getFullnodeUrl } from '@mysten/sui/client';
+
+const client = new SuiClient({ url: getFullnodeUrl('testnet') });
+
+// Check if index exists
+const hasIndex = await hasExistingIndexNode(userAddress);
+
+if (!hasIndex) {
+  // Rebuild from blockchain + Walrus
+  const result = await rebuildIndexNode({
+    userAddress,
+    client,
+    packageId: process.env.PACKAGE_ID!,
+    network: 'testnet',
+    force: false,  // Set true to force rebuild
+    fetchConcurrency: 10,  // Parallel blob fetches
+    onProgress: (current, total, status) => {
+      console.log(`${current}/${total}: ${status}`);
+    }
+  });
+
+  console.log(`Indexed ${result.indexedMemories}/${result.totalMemories} memories`);
+}
 ```
 
-### Embedding Providers
+## Exports
 
-| Provider | Model Default | Dimensions | API Key |
-|----------|---------------|------------|---------|
-| `openrouter` | `google/gemini-embedding-001` | 3072 | `OPENROUTER_API_KEY` |
-| `google` | `text-embedding-004` | 3072 | `GEMINI_API_KEY` |
-| `openai` | `text-embedding-3-small` | 1536 | `OPENAI_API_KEY` |
-| `cohere` | `embed-english-v3.0` | 1024 | `COHERE_API_KEY` |
+```typescript
+// Main exports
+import { SimplePDWClient } from '@cmdoss/memwal-sdk';
+
+// Browser-safe exports
+import { SimplePDWClient, DappKitSigner } from '@cmdoss/memwal-sdk/browser';
+
+// React hooks
+import { usePDWClient, useMemory } from '@cmdoss/memwal-sdk/hooks';
 
-**Recommendation**: Use `openrouter` for unified access to multiple models.
+// Services (advanced)
+import { EmbeddingService, MemoryIndexService } from '@cmdoss/memwal-sdk/services';
+
+// LangChain integration
+import { PDWVectorStore } from '@cmdoss/memwal-sdk/langchain';
+
+// Vercel AI SDK integration
+import { createPDWTool } from '@cmdoss/memwal-sdk/ai-sdk';
+
+// Node.js utilities
+import { rebuildIndexNode, hasExistingIndexNode, clearIndexNode } from '@cmdoss/memwal-sdk';
+```
 
 ## Benchmarks
 
-Tested on localhost with Option A+ (local content retrieval enabled):
+Tested on localhost with encryption enabled:
 
 | Operation | Time | Description |
 |-----------|------|-------------|
-| Vector Search + RAG | ~2.3s | Chat with semantic search |
-| Create Memory | ~2.3s | Classify + embed + upload + index |
-| AI Classification | ~2.3s | Categorize content |
-| Batch Upload (2 items) | ~2.3s | Quilt batching |
-| Blockchain Query | ~2.3s | List memories from Sui |
-
-**Average query time: ~2.3s** (with OpenRouter API latency included)
+| Memory Search | ~0.5s | Semantic search (local HNSW) |
+| Create Memory | ~2.5s | Classify + embed + encrypt + upload + index |
+| AI Classification | ~0.8s | Categorize content |
+| Batch Upload (5 items) | ~3s | Quilt batching with encryption |
+| Blockchain Query | ~0.3s | List memories from Sui |
 
 ## Documentation