@soulcraft/brainy 2.15.0 → 3.0.1

package/README.md

@@ -9,25 +9,36 @@
 [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![TypeScript](https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg)](https://www.typescriptlang.org/)
 
-**🧠 Brainy 2.0 - The Universal Knowledge Protocol™**
+**🧠 Brainy 3.0 - Universal Knowledge Protocol™**
 
-**World's first Triple Intelligence™ database**—unifying vector similarity, graph relationships, and document filtering in one magical API. Model ANY data from ANY domain using 31 standardized noun types × 40 verb types.
+**World's first Triple Intelligence™ database** unifying vector similarity, graph relationships, and document filtering in one magical API. Model ANY data from ANY domain using 31 standardized noun types × 40 verb types.
 
 **Why Brainy Leads**: We're the first to solve the impossible—combining three different database paradigms (vector, graph, document) into one unified query interface. This breakthrough enables us to be the Universal Knowledge Protocol where all tools, augmentations, and AI models speak the same language.
 
-**Build once, integrate everywhere.** O(log n) performance, 3ms search latency, 24MB memory footprint.
+**Build once, integrate everywhere.** O(log n) performance, <10ms search latency, production-ready.
 
-## 🎉 What's New in 2.0
+## 🎉 What's New in 3.0
 
-- **World's First Triple Intelligence™**: Unified vector + graph + document in ONE query
-- **Universal Knowledge Protocol**: 31 nouns × 40 verbs standardize all knowledge
-- **Infinite Expressiveness**: Model ANY data with unlimited metadata
-- **API Consolidation**: 15+ methods → 2 clean APIs (`search()` and `find()`)
-- **Natural Language**: Ask questions in plain English
-- **Zero Configuration**: Works instantly, no setup required
-- **O(log n) Performance**: Binary search on sorted indices
-- **Perfect Interoperability**: All tools and AI models speak the same language
-- **Universal Compatibility**: Node.js, Browser, Edge, Workers
+### 🧠 **Triple Intelligence™ Engine**
+
+- **Vector Search**: HNSW-powered semantic similarity
+- **Graph Relationships**: Navigate connected knowledge
+- **Document Filtering**: MongoDB-style metadata queries
+- **Unified API**: All three in a single query interface
+
+### 🎯 **Clean API Design**
+
+- **Modern Syntax**: `brain.add()`, `brain.find()`, `brain.relate()`
+- **Type Safety**: Full TypeScript integration
+- **Zero Config**: Works out of the box with memory storage
+- **Consistent Parameters**: Clean, predictable API surface
+
+### ⚡ **Performance & Reliability**
+
+- **<10ms Search**: Fast semantic queries
+- **384D Vectors**: Optimized embeddings (all-MiniLM-L6-v2)
+- **Built-in Caching**: Intelligent result caching
+- **Production Ready**: Thoroughly tested core functionality
 
 ## ⚡ Quick Start - Zero Configuration
 
@@ -36,40 +47,54 @@ npm install @soulcraft/brainy
 ```
 
 ### 🎯 **True Zero Configuration**
+
 ```javascript
-import { BrainyData } from '@soulcraft/brainy'
+import {Brainy} from '@soulcraft/brainy'
 
 // Just this - auto-detects everything!
-const brain = new BrainyData()
+const brain = new Brainy()
 await brain.init()
 
-// Add entities (nouns) with automatic embedding
-const jsId = await brain.addNoun("JavaScript is a programming language", 'concept', { 
-  type: "language", 
-  year: 1995,
-  paradigm: "multi-paradigm"
+// Add entities with automatic embedding
+const jsId = await brain.add({
+    data: "JavaScript is a programming language",
+    type: "concept",
+    metadata: {
+        type: "language",
+        year: 1995,
+        paradigm: "multi-paradigm"
+    }
 })
 
-const nodeId = await brain.addNoun("Node.js runtime environment", 'concept', {
-  type: "runtime",
-  year: 2009,
-  platform: "server-side"
+const nodeId = await brain.add({
+    data: "Node.js runtime environment",
+    type: "concept",
+    metadata: {
+        type: "runtime",
+        year: 2009,
+        platform: "server-side"
+    }
 })
 
-// Create relationships (verbs) between entities
-await brain.addVerb(nodeId, jsId, "executes", {
-  since: 2009,
-  performance: "high"
+// Create relationships between entities
+await brain.relate({
+    from: nodeId,
+    to: jsId,
+    type: "executes",
+    metadata: {
+        since: 2009,
+        performance: "high"
+    }
 })
 
 // Natural language search with graph relationships
-const results = await brain.find("programming languages used by server runtimes")
+const results = await brain.find({query: "programming languages used by server runtimes"})
 
 // Triple Intelligence: vector + metadata + relationships
 const filtered = await brain.find({
-  like: "JavaScript",                    // Vector similarity
-  where: { type: "language" },           // Metadata filtering  
-  connected: { from: nodeId, depth: 1 }  // Graph relationships
+    query: "JavaScript",                  // Vector similarity
+    where: {type: "language"},           // Metadata filtering
+    connected: {from: nodeId, depth: 1}  // Graph relationships
 })
 ```
 
@@ -81,14 +106,17 @@ const filtered = await brain.find({
 - ✅ **Node.js 20 LTS** - Compatible (maintenance mode)
 - ❌ **Node.js 24** - Not supported (known ONNX runtime compatibility issues)
 
-> **Important:** Brainy uses ONNX runtime for AI embeddings. Node.js 24 has known compatibility issues that cause crashes during inference operations. We recommend Node.js 22 LTS for maximum stability.
+> **Important:** Brainy uses ONNX runtime for AI embeddings. Node.js 24 has known compatibility issues that cause
+> crashes during inference operations. We recommend Node.js 22 LTS for maximum stability.
 
 If using nvm: `nvm use` (we provide a `.nvmrc` file)
 
 ## 🚀 Key Features
 
 ### World's First Triple Intelligence™ Engine
+
 **The breakthrough that enables the Universal Knowledge Protocol:**
+
 - **Vector Search**: Semantic similarity with HNSW indexing
 - **Graph Relationships**: Navigate connected knowledge like Neo4j
 - **Document Filtering**: MongoDB-style queries with O(log n) performance
@@ -96,7 +124,9 @@ If using nvm: `nvm use` (we provide a `.nvmrc` file)
 - **First to solve this**: Others do vector OR graph OR document—we do ALL
 
 ### Universal Knowledge Protocol with Infinite Expressiveness
+
 **Enabled by Triple Intelligence, standardized for everyone:**
+
 - **24 Noun Types × 40 Verb Types**: 960 base combinations
 - **∞ Expressiveness**: Unlimited metadata = model ANY data
 - **One Language**: All tools, augmentations, AI models speak the same types
@@ -104,6 +134,7 @@ If using nvm: `nvm use` (we provide a `.nvmrc` file)
 - **No Schema Lock-in**: Evolve without migrations
 
 ### Natural Language Understanding
+
 ```javascript
 // Ask questions naturally
 await brain.find("Show me recent React components with tests")
@@ -112,42 +143,41 @@ await brain.find("Documentation about authentication from last month")
 ```
 
 ### 🎯 Zero Configuration Philosophy
-Brainy 2.9+ automatically configures **everything**:
+
+Brainy 3.0 automatically configures **everything**:
 
 ```javascript
-import { BrainyData, PresetName } from '@soulcraft/brainy'
+import {Brainy} from '@soulcraft/brainy'
 
 // 1. Pure zero-config - detects everything
-const brain = new BrainyData()
-
-// 2. Environment presets (strongly typed)
-const devBrain = new BrainyData(PresetName.DEVELOPMENT)     // Memory + verbose
-const prodBrain = new BrainyData(PresetName.PRODUCTION)      // Disk + optimized
-const miniBrain = new BrainyData(PresetName.MINIMAL)         // Q8 + minimal features
-
-// 3. Distributed architecture presets
-const writer = new BrainyData(PresetName.WRITER)             // Write-only instance
-const reader = new BrainyData(PresetName.READER)             // Read-only + caching
-const ingestion = new BrainyData(PresetName.INGESTION_SERVICE) // High-throughput
-const searchAPI = new BrainyData(PresetName.SEARCH_API)      // Low-latency search
-
-// 4. Custom zero-config (type-safe)
-const customBrain = new BrainyData({
-  mode: 'production',
-  model: 'fp32',        // or 'q8', 'auto'
-  storage: 'cloud',     // or 'memory', 'disk', 'auto'
-  features: ['core', 'search', 'cache']
+const brain = new Brainy()
+
+// 2. Custom configuration
+const brain = new Brainy({
+  storage: { type: 'memory' },
+  embeddings: { model: 'all-MiniLM-L6-v2' },
+  cache: { enabled: true, maxSize: 1000 }
+})
+
+// 3. Production configuration
+const customBrain = new Brainy({
+    mode: 'production',
+    model: 'q8',           // Optimized model (99% accuracy, 75% smaller)
+    storage: 'cloud',     // or 'memory', 'disk', 'auto'
+    features: ['core', 'search', 'cache']
 })
 ```
 
 **What's Auto-Detected:**
+
 - **Storage**: S3/GCS/R2 → Filesystem → Memory (priority order)
-- **Models**: FP32 vs Q8 based on memory/performance  
+- **Models**: Always Q8 for optimal balance
 - **Features**: Minimal → Default → Full based on environment
 - **Memory**: Optimal cache sizes and batching
 - **Performance**: Threading, chunking, indexing strategies
 
 ### Production Performance
+
 - **3ms average search** - Lightning fast queries
 - **24MB memory footprint** - Efficient resource usage
 - **Worker-based embeddings** - Non-blocking operations
@@ -158,79 +188,72 @@ const customBrain = new BrainyData({
 Most users **never need this** - zero-config handles everything. For advanced use cases:
 
 ```javascript
-// Model precision control (auto-detected by default)
-const brain = new BrainyData({
-  model: 'fp32'  // Full precision - max compatibility
-})
-const fastBrain = new BrainyData({
-  model: 'q8'    // Quantized - 75% smaller, 99% accuracy  
-})
+// Model is always Q8 for optimal performance
+const brain = new Brainy()  // Uses Q8 automatically
 
 // Storage control (auto-detected by default)
-const memoryBrain = new BrainyData({ storage: 'memory' })     // RAM only
-const diskBrain = new BrainyData({ storage: 'disk' })         // Local filesystem  
-const cloudBrain = new BrainyData({ storage: 'cloud' })       // S3/GCS/R2
+const memoryBrain = new Brainy({storage: 'memory'})     // RAM only
+const diskBrain = new Brainy({storage: 'disk'})         // Local filesystem  
+const cloudBrain = new Brainy({storage: 'cloud'})       // S3/GCS/R2
 
 // Legacy full config (still supported)
-const legacyBrain = new BrainyData({
-  storage: { forceMemoryStorage: true },
-  embeddingOptions: { precision: 'fp32' }
+const legacyBrain = new Brainy({
+    storage: {forceMemoryStorage: true}
 })
 ```
 
-**Model Comparison:**
-- **FP32**: 90MB, 100% accuracy, maximum compatibility
-- **Q8**: 23MB, ~99% accuracy, faster loading, smaller memory
+**Model Details:**
 
-**When to use Q8:**
-- ✅ Memory-constrained environments  
-- ✅ Faster startup required
-- ✅ Mobile or edge deployments
-- ❌ Existing FP32 datasets (incompatible embeddings)
+- **Q8**: 33MB, 99% accuracy, 75% smaller than full precision
+- Fast loading and optimal memory usage
+- Perfect for all environments
 
 **Air-gap deployment:**
+
 ```bash
-npm run download-models        # Both models (recommended)
-npm run download-models:q8     # Q8 only (space-constrained)
-npm run download-models:fp32   # FP32 only (compatibility)
+npm run download-models        # Download Q8 model
+npm run download-models:q8     # Download Q8 model
 ```
 
 ## 📚 Core API
 
 ### `search()` - Vector Similarity
+
 ```javascript
 const results = await brain.search("machine learning", {
-  limit: 10,                    // Number of results
-  metadata: { type: "article" }, // Filter by metadata
-  includeContent: true          // Include full content
+    limit: 10,                    // Number of results
+    metadata: {type: "article"}, // Filter by metadata
+    includeContent: true          // Include full content
 })
 ```
 
 ### `find()` - Natural Language Queries
+
 ```javascript
 // Simple natural language
 const results = await brain.find("recent important documents")
 
 // Structured query with Triple Intelligence
 const results = await brain.find({
-  like: "JavaScript",           // Vector similarity
-  where: {                      // Metadata filters
-    year: { greaterThan: 2020 },
-    important: true
-  },
-  related: { to: "React" }      // Graph relationships
+    like: "JavaScript",           // Vector similarity
+    where: {                      // Metadata filters
+        year: {greaterThan: 2020},
+        important: true
+    },
+    related: {to: "React"}      // Graph relationships
 })
 ```
 
 ### CRUD Operations
+
 ```javascript
 // Create entities (nouns)
 const id = await brain.addNoun(data, nounType, metadata)
 
 // Create relationships (verbs)
 const verbId = await brain.addVerb(sourceId, targetId, "relationType", {
-  strength: 0.9,
-  bidirectional: false
+    strength: 0.9,
+    bidirectional: false
 })
 
 // Read
@@ -247,36 +270,106 @@ await brain.deleteVerb(verbId)
 
 // Bulk operations
 await brain.import(arrayOfData)
-const exported = await brain.export({ format: 'json' })
+const exported = await brain.export({format: 'json'})
 ```
 
+## 🌐 Distributed System (NEW!)
+
+### Zero-Config Distributed Setup
+
+```javascript
+// Single node (default)
+const brain = new Brainy({
+    storage: {type: 's3', options: {bucket: 'my-data'}}
+})
+
+// Distributed cluster - just add one flag!
+const brain = new Brainy({
+    storage: {type: 's3', options: {bucket: 'my-data'}},
+    distributed: true  // That's it! Everything else is automatic
+})
+```
+
+### How It Works
+
+- **Storage-Based Discovery**: Nodes find each other via S3/GCS (no Consul/etcd!)
+- **Automatic Sharding**: Data distributed by content hash
+- **Smart Query Planning**: Queries routed to optimal shards
+- **Live Rebalancing**: Handles node joins/leaves automatically
+- **Zero Downtime**: Streaming shard migration
+
+### Real-World Example: Social Media Firehose
+
+```javascript
+// Ingestion nodes (optimized for writes)
+const ingestionNode = new Brainy({
+    storage: {type: 's3', options: {bucket: 'social-data'}},
+    distributed: true,
+    writeOnly: true  // Optimized for high-throughput writes
+})
+
+// Process Bluesky firehose
+blueskyStream.on('post', async (post) => {
+    await ingestionNode.addNoun(post, 'social-post', {
+        platform: 'bluesky',
+        author: post.author,
+        timestamp: post.createdAt
+    })
+})
+
+// Search nodes (optimized for queries)
+const searchNode = new Brainy({
+    storage: {type: 's3', options: {bucket: 'social-data'}},
+    distributed: true,
+    readOnly: true  // Optimized for fast queries
+})
+
+// Search across ALL data from ALL nodes
+const trending = await searchNode.find('trending AI topics', {
+    where: {timestamp: {greaterThan: Date.now() - 3600000}},
+    limit: 100
+})
+```
+
+### Benefits Over Traditional Systems
+
+| Feature        | Traditional (Pinecone, Weaviate) | Brainy Distributed            |
+|----------------|----------------------------------|-------------------------------|
+| Setup          | Complex (k8s, operators)         | One flag: `distributed: true` |
+| Coordination   | External (etcd, Consul)          | Built-in (via storage)        |
+| Minimum Nodes  | 3-5 for HA                       | 1 (scale as needed)           |
+| Sharding       | Random                           | Domain-aware                  |
+| Query Planning | Basic                            | Triple Intelligence           |
+| Cost           | High (always-on clusters)        | Low (scale to zero)           |
+
 ## 🎯 Use Cases
 
 ### Knowledge Management with Relationships
+
 ```javascript
 // Store documentation with rich relationships
 const apiGuide = await brain.addNoun("REST API Guide", 'document', {
-  title: "API Guide",
-  category: "documentation",
-  version: "2.0"
+    title: "API Guide",
+    category: "documentation",
+    version: "2.0"
 })
 
 const author = await brain.addNoun("Jane Developer", 'person', {
-  type: "person",
-  role: "tech-lead"
+    type: "person",
+    role: "tech-lead"
 })
 
 const project = await brain.addNoun("E-commerce Platform", 'project', {
-  type: "project",
-  status: "active"
+    type: "project",
+    status: "active"
 })
 
 // Create knowledge graph
-await brain.addVerb(author, apiGuide, "authored", { 
-  date: "2024-03-15" 
+await brain.addVerb(author, apiGuide, "authored", {
+    date: "2024-03-15"
 })
-await brain.addVerb(apiGuide, project, "documents", { 
-  coverage: "complete" 
+await brain.addVerb(apiGuide, project, "documents", {
+    coverage: "complete"
 })
 
 // Query the knowledge graph naturally
@@ -284,31 +377,33 @@ const docs = await brain.find("documentation authored by tech leads for active p
 ```
 
 ### Semantic Search
+
 ```javascript
 // Find similar content
 const similar = await brain.search(existingContent, {
-  limit: 5,
-  threshold: 0.8
+    limit: 5,
+    threshold: 0.8
 })
 ```
 
 ### AI Memory Layer with Context
+
 ```javascript
 // Store conversation with relationships
 const userId = await brain.addNoun("User 123", 'user', {
-  type: "user",
-  tier: "premium"
+    type: "user",
+    tier: "premium"
 })
 
 const messageId = await brain.addNoun(userMessage, 'message', {
-  type: "message",
-  timestamp: Date.now(),
-  session: "abc"
+    type: "message",
+    timestamp: Date.now(),
+    session: "abc"
 })
 
 const topicId = await brain.addNoun("Product Support", 'topic', {
-  type: "topic",
-  category: "support"
+    type: "topic",
+    category: "support"
 })
 
 // Link conversation elements
@@ -317,9 +412,9 @@ await brain.addVerb(messageId, topicId, "about")
 
 // Retrieve context with relationships
 const context = await brain.find({
-  where: { type: "message" },
-  connected: { from: userId, type: "sent" },
-  like: "previous product issues"
+    where: {type: "message"},
+    connected: {from: userId, type: "sent"},
+    like: "previous product issues"
 })
 ```
 
@@ -329,30 +424,30 @@ Brainy supports multiple storage backends:
 
 ```javascript
 // Memory (default for testing)
-const brain = new BrainyData({ 
-  storage: { type: 'memory' } 
+const brain = new Brainy({
+    storage: {type: 'memory'}
 })
 
 // FileSystem (Node.js)
-const brain = new BrainyData({ 
-  storage: { 
-    type: 'filesystem',
-    path: './data'
-  } 
+const brain = new Brainy({
+    storage: {
+        type: 'filesystem',
+        path: './data'
+    }
 })
 
 // Browser Storage (OPFS)
-const brain = new BrainyData({ 
-  storage: { type: 'opfs' } 
+const brain = new Brainy({
+    storage: {type: 'opfs'}
 })
 
 // S3 Compatible (Production)
-const brain = new BrainyData({ 
-  storage: { 
-    type: 's3',
-    bucket: 'my-bucket',
-    region: 'us-east-1'
-  } 
+const brain = new Brainy({
+    storage: {
+        type: 's3',
+        bucket: 'my-bucket',
+        region: 'us-east-1'
+    }
 })
 ```
 
@@ -385,6 +480,7 @@ brainy export --format json > backup.json
 Brainy includes a powerful Neural API for advanced semantic analysis:
 
 ### Clustering & Analysis
+
 ```javascript
 // Access via brain.neural
 const neural = brain.neural
@@ -395,9 +491,9 @@ const clusters = await neural.clusters()
 
 // Cluster with options
 const clusters = await neural.clusters({
-  algorithm: 'kmeans',     // or 'hierarchical', 'sample'
-  maxClusters: 5,          // Maximum number of clusters
-  threshold: 0.8           // Similarity threshold
+    algorithm: 'kmeans',     // or 'hierarchical', 'sample'
+    maxClusters: 5,          // Maximum number of clusters
+    threshold: 0.8           // Similarity threshold
 })
 
 // Calculate similarity between any items
@@ -415,19 +511,20 @@ const outliers = await neural.outliers(0.3)
 
 // Generate visualization data for D3/Cytoscape
 const vizData = await neural.visualize({
-  maxNodes: 100,
-  dimensions: 3,
-  algorithm: 'force'
+    maxNodes: 100,
+    dimensions: 3,
+    algorithm: 'force'
 })
 ```
 
 ### Real-World Examples
+
 ```javascript
 // Group customer feedback into themes
 const feedbackClusters = await neural.clusters()
 for (const cluster of feedbackClusters) {
-  console.log(`Theme: ${cluster.label}`)
-  console.log(`Items: ${cluster.members.length}`)
+    console.log(`Theme: ${cluster.label}`)
+    console.log(`Items: ${cluster.members.length}`)
 }
 
 // Find related documents
@@ -460,32 +557,30 @@ brainy cloud setup
 Brainy includes enterprise-grade capabilities at no extra cost. **No premium tiers, no paywalls.**
 
 - **Scales to 10M+ items** with consistent 3ms search latency
-- **Write-Ahead Logging (WAL)** for zero data loss durability
 - **Distributed architecture** with sharding and replication
 - **Read/write separation** for horizontal scaling
 - **Connection pooling** and request deduplication
 - **Built-in monitoring** with metrics and health checks
 - **Production ready** with circuit breakers and backpressure
 
-📖 **[Read the full Enterprise Features guide →](docs/ENTERPRISE-FEATURES.md)**
+📖 **Enterprise features coming in Brainy 3.1** - Stay tuned!
 
 ## 📊 Benchmarks
 
-| Operation | Performance | Memory |
-|-----------|------------|--------|
-| Initialize | 450ms | 24MB |
-| Add Item | 12ms | +0.1MB |
-| Vector Search (1k items) | 3ms | - |
-| Metadata Filter (10k items) | 0.8ms | - |
-| Natural Language Query | 15ms | - |
-| Bulk Import (1000 items) | 2.3s | +8MB |
-| **Production Scale (10M items)** | **5.8ms** | **12GB** |
+| Operation                        | Performance | Memory   |
+|----------------------------------|-------------|----------|
+| Initialize                       | 450ms       | 24MB     |
+| Add Item                         | 12ms        | +0.1MB   |
+| Vector Search (1k items)         | 3ms         | -        |
+| Metadata Filter (10k items)      | 0.8ms       | -        |
+| Natural Language Query           | 15ms        | -        |
+| Bulk Import (1000 items)         | 2.3s        | +8MB     |
+| **Production Scale (10M items)** | **5.8ms**   | **12GB** |
 
-## 🔄 Migration from 1.x
+## 🔄 Migration from 2.x
 
-See [MIGRATION.md](MIGRATION.md) for detailed upgrade instructions.
+Key changes for upgrading to 3.0:
 
-Key changes:
 - Search methods consolidated into `search()` and `find()`
 - Result format now includes full objects with metadata
 - New natural language capabilities
@@ -499,8 +594,9 @@ We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 ### How We Achieved The Impossible
 
 **Triple Intelligence™** makes us the **world's first** to unify three database paradigms:
+
 1. **Vector databases** (Pinecone, Weaviate) - semantic similarity
-2. **Graph databases** (Neo4j, ArangoDB) - relationships  
+2. **Graph databases** (Neo4j, ArangoDB) - relationships
 3. **Document databases** (MongoDB, Elasticsearch) - metadata filtering
 
 **One API to rule them all.** Others make you choose. We unified them.
@@ -519,6 +615,7 @@ We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 ### Why This Changes Everything
 
 **Like HTTP for the web, Brainy for knowledge:**
+
 - All augmentations compose perfectly - same noun-verb language
 - All AI models share knowledge - GPT, Claude, Llama all understand
 - All tools integrate seamlessly - no translation layers