@soulcraft/brainy 3.50.2 → 4.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (57) hide show
  1. package/CHANGELOG.md +201 -0
  2. package/README.md +396 -658
  3. package/dist/api/ConfigAPI.js +56 -19
  4. package/dist/api/DataAPI.js +24 -18
  5. package/dist/augmentations/storageAugmentations.d.ts +24 -0
  6. package/dist/augmentations/storageAugmentations.js +22 -0
  7. package/dist/brainy.js +32 -9
  8. package/dist/cli/commands/core.d.ts +20 -10
  9. package/dist/cli/commands/core.js +384 -82
  10. package/dist/cli/commands/import.d.ts +41 -0
  11. package/dist/cli/commands/import.js +456 -0
  12. package/dist/cli/commands/insights.d.ts +34 -0
  13. package/dist/cli/commands/insights.js +300 -0
  14. package/dist/cli/commands/neural.d.ts +6 -12
  15. package/dist/cli/commands/neural.js +113 -10
  16. package/dist/cli/commands/nlp.d.ts +28 -0
  17. package/dist/cli/commands/nlp.js +246 -0
  18. package/dist/cli/commands/storage.d.ts +64 -0
  19. package/dist/cli/commands/storage.js +730 -0
  20. package/dist/cli/index.js +210 -24
  21. package/dist/coreTypes.d.ts +206 -34
  22. package/dist/distributed/configManager.js +8 -6
  23. package/dist/distributed/shardMigration.js +2 -0
  24. package/dist/distributed/storageDiscovery.js +6 -4
  25. package/dist/embeddings/EmbeddingManager.d.ts +2 -2
  26. package/dist/embeddings/EmbeddingManager.js +5 -1
  27. package/dist/graph/lsm/LSMTree.js +32 -20
  28. package/dist/hnsw/typeAwareHNSWIndex.js +6 -2
  29. package/dist/storage/adapters/azureBlobStorage.d.ts +545 -0
  30. package/dist/storage/adapters/azureBlobStorage.js +1809 -0
  31. package/dist/storage/adapters/baseStorageAdapter.d.ts +16 -13
  32. package/dist/storage/adapters/fileSystemStorage.d.ts +21 -9
  33. package/dist/storage/adapters/fileSystemStorage.js +204 -127
  34. package/dist/storage/adapters/gcsStorage.d.ts +119 -9
  35. package/dist/storage/adapters/gcsStorage.js +317 -62
  36. package/dist/storage/adapters/memoryStorage.d.ts +30 -18
  37. package/dist/storage/adapters/memoryStorage.js +99 -94
  38. package/dist/storage/adapters/opfsStorage.d.ts +48 -10
  39. package/dist/storage/adapters/opfsStorage.js +201 -80
  40. package/dist/storage/adapters/r2Storage.d.ts +12 -5
  41. package/dist/storage/adapters/r2Storage.js +63 -15
  42. package/dist/storage/adapters/s3CompatibleStorage.d.ts +164 -17
  43. package/dist/storage/adapters/s3CompatibleStorage.js +472 -80
  44. package/dist/storage/adapters/typeAwareStorageAdapter.d.ts +38 -6
  45. package/dist/storage/adapters/typeAwareStorageAdapter.js +218 -39
  46. package/dist/storage/baseStorage.d.ts +41 -38
  47. package/dist/storage/baseStorage.js +110 -134
  48. package/dist/storage/storageFactory.d.ts +29 -2
  49. package/dist/storage/storageFactory.js +30 -1
  50. package/dist/utils/entityIdMapper.js +5 -2
  51. package/dist/utils/fieldTypeInference.js +8 -1
  52. package/dist/utils/metadataFilter.d.ts +3 -2
  53. package/dist/utils/metadataFilter.js +1 -0
  54. package/dist/utils/metadataIndex.js +2 -0
  55. package/dist/utils/metadataIndexChunking.js +9 -4
  56. package/dist/utils/periodicCleanup.js +1 -0
  57. package/package.json +3 -1
package/README.md CHANGED
@@ -11,91 +11,126 @@
11
11
 
12
12
  **🧠 Brainy - The Knowledge Operating System**
13
13
 
14
- **The world's first Knowledge Operating System** where every piece of knowledge - files, concepts, entities, ideas - exists as living information that understands itself, evolves over time, and connects to everything related. Built on revolutionary Triple Intelligence™ that unifies vector similarity, graph relationships, and document filtering in one magical API.
14
+ **The world's first Knowledge Operating System** where every piece of knowledge - files, concepts, entities, ideas - exists as living information that understands itself, evolves over time, and connects to everything related.
15
15
 
16
16
  **Why Brainy Changes Everything**: Traditional systems trap knowledge in files or database rows. Brainy liberates it. Your characters exist across stories. Your concepts span projects. Your APIs remember their evolution. Every piece of knowledge - whether it's code, prose, or pure ideas - lives, breathes, and connects in a unified intelligence layer where everything understands its meaning, remembers its history, and relates to everything else.
17
17
 
18
- **Framework-first design.** Built for modern web development with zero configuration and automatic framework compatibility. O(log n) performance, <10ms search latency, production-ready.
18
+ Built on revolutionary **Triple Intelligence™** that unifies vector similarity, graph relationships, and document filtering in one magical API. **Framework-first design.** Zero configuration. O(log n) performance, <10ms search latency. **Production-ready for billion-scale deployments.**
19
19
 
20
- ## 🎉 Key Features
20
+ ---
21
+
22
+ ## 🎉 NEW in v4.0.0 - Enterprise-Scale Cost Optimization
23
+
24
+ **Major Release: Production cost optimization and enterprise-scale features**
25
+
26
+ ### 💰 **Up to 96% Storage Cost Savings**
21
27
 
22
- ### 🚀 **NEW in 3.47.0: Billion-Scale Type-Aware HNSW**
28
+ Automatic cloud storage lifecycle management for **AWS S3**, **Google Cloud Storage**, and **Azure Blob Storage**:
23
29
 
24
- **87% memory reduction for billion-scale deployments with 10x faster queries:**
30
+ - **GCS Autoclass**: Fully automatic tier optimization (94% savings!)
31
+ - **AWS Intelligent-Tiering**: Smart archival with instant retrieval
32
+ - **Azure Lifecycle Policies**: Automatic tier transitions
33
+ - **Cost Impact**: $138,000/year → $5,940/year @ 500TB scale
25
34
 
26
- - **🎯 Type-Aware Vector Index**: Separate HNSW graphs per entity type for massive memory savings
27
- - **Memory @ 1B scale**: 384GB → 50GB (-87% / -334GB)
28
- - **Single-type queries**: 10x faster (search 100M nodes instead of 1B)
29
- - **Multi-type queries**: 5-8x faster (search subset of types)
30
- - **All-types queries**: ~3x faster (31 smaller graphs vs 1 large graph)
35
+ ### **Performance at Billion-Scale**
31
36
 
32
- - **⚡ Optimized Rebuild**: Type-filtered pagination for 31x faster index rebuilding
33
- - **Before**: 31B reads (UNACCEPTABLE)
34
- - **After**: 1B reads with type filtering (CORRECT)
35
- - **Parallel type rebuilds**: 10-20 minutes for all types
36
- - **Lazy loading**: 15 minutes for top 2 types only
37
+ - **1000x faster batch deletions** (533 entities/sec vs 0.5/sec)
38
+ - **60-80% FileSystem compression** with gzip
39
+ - **OPFS quota monitoring** for browser storage
40
+ - **Enhanced CLI** with 47 commands including 9 storage management tools
37
41
 
38
- - **📊 Production-Ready**: Comprehensive testing and zero breaking changes
39
- - 47 new tests (33 unit + 14 integration) - all passing
40
- - Backward compatible - opt-in via configuration
41
- - Works with all storage backends (FileSystem, S3, GCS, R2, Memory, OPFS)
42
+ ### 🛡️ **Zero Breaking Changes**
42
43
 
43
- **[📖 Phase 2 Architecture →](.strategy/PHASE_2_TYPE_AWARE_HNSW_DESIGN.md)**
44
+ **100% backward compatible.** No migration required. All new features are opt-in.
45
+
46
+ **[📖 Read the full v4.0.0 Changelog →](CHANGELOG.md)** | **[Migration Guide →](docs/MIGRATION-V3-TO-V4.md)**
47
+
48
+ ---
44
49
 
45
- ### **NEW in 3.36.0: Production-Scale Memory & Performance**
50
+ ## 🎯 What Makes Brainy Revolutionary?
46
51
 
47
- **Enterprise-grade adaptive sizing and zero-overhead optimizations:**
52
+ ### 🧠 **Triple Intelligence™ - The Impossible Made Possible**
48
53
 
49
- - **🎯 Adaptive Memory Sizing**: Auto-scales from 2GB to 128GB+ based on available system resources
50
- - Container-aware (Docker/K8s cgroups v1/v2 detection)
51
- - Environment-smart (development 25%, container 40%, production 50% allocation)
52
- - Model memory accounting (150MB Q8, 250MB FP32 reserved before cache)
54
+ **The world's first to unify three database paradigms in ONE API:**
53
55
 
54
- - **⚡ Sync Fast Path**: Zero async overhead when vectors are cached
55
- - Intelligent sync/async branching - synchronous when data is in memory
56
- - Falls back to async only when loading from storage
57
- - Massive performance win for hot paths (vector search, distance calculations)
56
+ - **Vector Search** 🔍 Semantic similarity like Pinecone/Weaviate
57
+ - **Graph Relationships** 🕸️ Navigate connections like Neo4j/ArangoDB
58
+ - **Document Filtering** 📊 MongoDB-style queries with O(log n) performance
58
59
 
59
- - **📊 Production Monitoring**: Comprehensive diagnostics
60
- - `getCacheStats()` - UnifiedCache hit rates, fairness metrics, memory pressure
61
- - Actionable recommendations for tuning
62
- - Tracks model memory, cache efficiency, and competition across indexes
60
+ **Others make you choose.** Vector OR graph OR document. **Brainy does ALL THREE together.** This is what enables The Knowledge Operating System.
61
+
62
+ ### 🚀 **Zero Configuration - Just Works™**
63
+
64
+ ```javascript
65
+ import { Brainy } from '@soulcraft/brainy'
66
+
67
+ const brain = new Brainy()
68
+ await brain.init()
69
+
70
+ // That's it! Auto-detects storage, optimizes memory, configures everything.
71
+ ```
72
+
73
+ No configuration files. No environment variables. No complex setup. **It just works.**
74
+
75
+ ### ⚡ **Production Performance at Any Scale**
76
+
77
+ - **<10ms search** across millions of entities
78
+ - **87% memory reduction** @ billion scale (384GB → 50GB)
79
+ - **10x faster queries** with type-aware indexing
80
+ - **99% storage cost savings** with intelligent archival
81
+ - **Container-aware** memory allocation (Docker/K8s)
82
+
83
+ ### 🎯 **31 Noun Types × 40 Verb Types = Infinite Expressiveness**
84
+
85
+ Model **ANY domain** with 1,240 base type combinations + unlimited metadata:
86
+
87
+ - Healthcare: Patient → diagnoses → Condition
88
+ - Finance: Account → transfers → Transaction
89
+ - Manufacturing: Product → assembles → Component
90
+ - Education: Student → completes → Course
91
+ - **Your domain**: Your types + Your relationships = Your knowledge graph
92
+
93
+ [→ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md)
94
+
95
+ ---
63
96
 
64
- - **🛡️ Zero Breaking Changes**: All optimizations are internal - your code stays the same
65
- - Public API unchanged
66
- - Automatic memory detection and allocation
67
- - Progressive enhancement for existing applications
97
+ ## 💡 What Can You Build?
68
98
 
69
- **[📖 Operations Guide →](docs/operations/capacity-planning.md)** | **[🎯 Migration Guide →](docs/guides/migration-3.36.0.md)**
99
+ **Brainy unlocks entirely new categories of applications.** Here's what developers are building:
70
100
 
71
- ### 🚀 **NEW in 3.21.0: Enhanced Import & Neural Processing**
101
+ ### 📚 **Intelligent Documentation Systems**
102
+ - **Living knowledge bases** where docs understand each other and evolve
103
+ - **Smart wikis** that auto-link related concepts and detect outdated information
104
+ - **Research assistants** that remember every paper you've read and find connections
72
105
 
73
- - **📊 Progress Tracking**: Unified progress reporting with automatic time estimation
74
- - **⚡ Entity Caching**: 10-100x speedup on repeated entity extraction
75
- - **🔗 Relationship Confidence**: Multi-factor confidence scoring (0-1 scale)
76
- - **📝 Evidence Tracking**: Understand why relationships were detected
77
- - **🎯 Production Ready**: Fully backward compatible, opt-in features
106
+ ### 🤖 **AI Agents with Perfect Memory**
107
+ - **Conversational AI** that remembers context across months, not just messages
108
+ - **Code assistants** that understand your entire codebase as a knowledge graph
109
+ - **Personal assistants** with unlimited memory that never forgets important details
78
110
 
79
- ### 🧠 **Triple Intelligence™ Engine**
111
+ ### 🎮 **Rich Interactive Experiences**
112
+ - **Game worlds** where NPCs remember every interaction and relationships evolve
113
+ - **Story engines** where characters persist across multiple narratives
114
+ - **Educational platforms** that build personalized knowledge graphs as you learn
80
115
 
81
- - **Vector Search**: HNSW-powered semantic similarity
82
- - **Graph Relationships**: Navigate connected knowledge
83
- - **Document Filtering**: MongoDB-style metadata queries
84
- - **Unified API**: All three in a single query interface
116
+ ### 🔍 **Next-Gen Search & Discovery**
117
+ - **Semantic code search** across millions of repositories
118
+ - **Smart file explorers** that understand code relationships, not just folders
119
+ - **Research platforms** that find papers by meaning, not keywords
85
120
 
86
- ### 🎯 **Clean API Design**
121
+ ### 🏢 **Enterprise Knowledge Management**
122
+ - **Corporate memory systems** where institutional knowledge never gets lost
123
+ - **Customer intelligence** platforms that understand every interaction
124
+ - **Product catalogs** with semantic search and relationship-based recommendations
87
125
 
88
- - **Modern Syntax**: `brain.add()`, `brain.find()`, `brain.relate()`
89
- - **Type Safety**: Full TypeScript integration
90
- - **Zero Config**: Works out of the box with intelligent storage auto-detection
91
- - **Consistent Parameters**: Clean, predictable API surface
126
+ ### 🎨 **Creative Tools & Content Platforms**
127
+ - **Writing assistants** that track characters, plotlines, and themes across stories
128
+ - **Content management** where every asset knows its relationships and history
129
+ - **Media libraries** with intelligent tagging and similarity-based discovery
92
130
 
93
- ### **Performance & Reliability**
131
+ **The Pattern**: If your app needs to **remember**, **understand**, or **connect** information, Brainy makes it trivial.
94
132
 
95
- - **<10ms Search**: Fast semantic queries
96
- - **384D Vectors**: Optimized embeddings (all-MiniLM-L6-v2)
97
- - **Built-in Caching**: Intelligent result caching + new entity extraction cache
98
- - **Production Ready**: Thoroughly tested core functionality
133
+ ---
99
134
 
100
135
  ## ⚡ Quick Start - Zero Configuration
101
136
 
@@ -103,7 +138,7 @@
103
138
  npm install @soulcraft/brainy
104
139
  ```
105
140
 
106
- ### 🎯 **True Zero Configuration**
141
+ ### 🎯 **Your First Knowledge Graph in 30 Seconds**
107
142
 
108
143
  ```javascript
109
144
  import { Brainy, NounType } from '@soulcraft/brainy'
@@ -145,7 +180,9 @@ await brain.relate({
145
180
  })
146
181
 
147
182
  // Natural language search with graph relationships
148
- const results = await brain.find({query: "programming languages used by server runtimes"})
183
+ const results = await brain.find({
184
+ query: "programming languages used by server runtimes"
185
+ })
149
186
 
150
187
  // Triple Intelligence: vector + metadata + relationships
151
188
  const filtered = await brain.find({
@@ -155,525 +192,346 @@ const filtered = await brain.find({
155
192
  })
156
193
  ```
157
194
 
158
- ## 🌐 Framework Integration
159
-
160
- **Brainy is framework-first!** Works seamlessly with any modern JavaScript framework:
161
-
162
- ### ⚛️ **React & Next.js**
163
- ```javascript
164
- import { Brainy } from '@soulcraft/brainy'
165
-
166
- function SearchComponent() {
167
- const [brain] = useState(() => new Brainy())
168
-
169
- useEffect(() => {
170
- brain.init()
171
- }, [])
172
-
173
- const handleSearch = async (query) => {
174
- const results = await brain.find(query)
175
- setResults(results)
176
- }
177
- }
178
- ```
179
-
180
- ### 🟢 **Vue.js & Nuxt.js**
181
- ```javascript
182
- import { Brainy } from '@soulcraft/brainy'
183
-
184
- export default {
185
- async mounted() {
186
- this.brain = new Brainy()
187
- await this.brain.init()
188
- },
189
- methods: {
190
- async search(query) {
191
- return await this.brain.find(query)
192
- }
193
- }
194
- }
195
- ```
196
-
197
- ### 🅰️ **Angular**
198
- ```typescript
199
- import { Injectable } from '@angular/core'
200
- import { Brainy } from '@soulcraft/brainy'
201
-
202
- @Injectable({ providedIn: 'root' })
203
- export class BrainyService {
204
- private brain = new Brainy()
205
-
206
- async init() {
207
- await this.brain.init()
208
- }
209
-
210
- async search(query: string) {
211
- return await this.brain.find(query)
212
- }
213
- }
214
- ```
215
-
216
- ### 🔥 **Other Frameworks**
217
- Brainy works with **any** framework that supports ES6 imports: Svelte, Solid.js, Qwik, Fresh, and more!
218
-
219
- **Framework Compatibility:**
220
- - ✅ All modern bundlers (Webpack, Vite, Rollup, Parcel)
221
- - ✅ SSR/SSG (Next.js, Nuxt, SvelteKit, Astro)
222
- - ✅ Edge runtimes (Vercel Edge, Cloudflare Workers)
223
- - ✅ Browser and Node.js environments
224
-
225
- ## 📋 System Requirements
226
-
227
- **Node.js Version:** 22 LTS or later (recommended)
228
-
229
- - ✅ **Node.js 22 LTS** - Fully supported and recommended for production
230
- - ✅ **Node.js 20 LTS** - Compatible (maintenance mode)
231
- - ❌ **Node.js 24** - Not supported (known ONNX runtime compatibility issues)
232
-
233
- > **Important:** Brainy uses ONNX runtime for AI embeddings. Node.js 24 has known compatibility issues that cause
234
- > crashes during inference operations. We recommend Node.js 22 LTS for maximum stability.
235
-
236
- If using nvm: `nvm use` (we provide a `.nvmrc` file)
237
-
238
- ## 🚀 Key Features
239
-
240
- ### World's First Triple Intelligence™ Engine
195
+ **That's it!** You just created a knowledge graph with semantic search, relationship traversal, and metadata filtering. **No configuration. No complexity. Just works.**
241
196
 
242
- **The breakthrough that enables The Knowledge Operating System:**
243
-
244
- - **Vector Search**: Semantic similarity with HNSW indexing
245
- - **Graph Relationships**: Navigate connected knowledge like Neo4j
246
- - **Document Filtering**: MongoDB-style queries with O(log n) performance
247
- - **Unified in ONE API**: No separate queries, no complex joins
248
- - **First to solve this**: Others do vector OR graph OR document—we do ALL
249
-
250
- ### The Knowledge Operating System with Infinite Expressiveness
251
-
252
- **Enabled by Triple Intelligence, standardized for everyone:**
197
+ ---
253
198
 
254
- - **31 Noun Types × 40 Verb Types**: 1,240 base combinations
255
- - **∞ Expressiveness**: Unlimited metadata = model ANY data
256
- - **One Language**: All tools, augmentations, AI models speak the same types
257
- - **Perfect Interoperability**: Move data between any Brainy instance
258
- - **No Schema Lock-in**: Evolve without migrations
199
+ ## 🌟 Core Features
259
200
 
260
- ### Natural Language Understanding
201
+ ### 🧠 **Natural Language Understanding**
261
202
 
262
203
  ```javascript
263
- // Ask questions naturally
204
+ // Ask questions naturally - Brainy understands
264
205
  await brain.find("Show me recent React components with tests")
265
206
  await brain.find("Popular JavaScript libraries similar to Vue")
266
207
  await brain.find("Documentation about authentication from last month")
267
- ```
268
208
 
269
- ### 🧠🌐 **Virtual Filesystem - Intelligent File Management**
209
+ // Structured queries with Triple Intelligence
210
+ await brain.find({
211
+ like: "React", // Vector similarity
212
+ where: { // Document filtering
213
+ type: "library",
214
+ year: {greaterThan: 2020}
215
+ },
216
+ connected: {to: "JavaScript", depth: 2} // Graph relationships
217
+ })
218
+ ```
270
219
 
271
- **Build file explorers, IDEs, and knowledge systems that never crash from infinite recursion.**
220
+ ### 🌐 **Virtual Filesystem - Intelligent File Management**
272
221
 
273
- - **Tree-Aware Operations**: Safe directory listing prevents recursive loops
274
- - **Semantic Search**: Find files by content, not just filename
275
- - **Production Storage**: Filesystem and cloud storage for real applications
276
- - **Zero-Config**: Works out of the box with intelligent defaults
222
+ Build file explorers, IDEs, and knowledge systems that **never crash**:
277
223
 
278
224
  ```javascript
279
- import { Brainy } from '@soulcraft/brainy'
280
-
281
- // ✅ CORRECT: Use persistent storage for file systems
282
- const brain = new Brainy({
283
- storage: {
284
- type: 'filesystem', // Persisted to disk
285
- path: './brainy-data' // Your file storage
286
- }
287
- })
288
- await brain.init()
289
-
290
225
  const vfs = brain.vfs()
291
226
  await vfs.init()
292
227
 
293
228
  // ✅ Safe file operations
294
229
  await vfs.writeFile('/projects/app/index.js', 'console.log("Hello")')
295
230
  await vfs.mkdir('/docs')
296
- await vfs.writeFile('/docs/README.md', '# My Project')
297
231
 
298
232
  // ✅ NEVER crashes: Tree-aware directory listing
299
233
  const children = await vfs.getDirectChildren('/projects')
300
- // Returns only direct children, never the directory itself
301
234
 
302
235
  // ✅ Build file explorers safely
303
236
  const tree = await vfs.getTreeStructure('/projects', {
304
237
  maxDepth: 3, // Prevent deep recursion
305
- sort: 'name' // Organized results
238
+ sort: 'name'
306
239
  })
307
240
 
308
241
  // ✅ Semantic file search
309
242
  const reactFiles = await vfs.search('React components with hooks')
310
- const docs = await vfs.search('API documentation', {
311
- path: '/docs' // Search within specific directory
312
- })
313
-
314
- // ✅ Connect related files
315
- await vfs.addRelationship('/src/auth.js', '/tests/auth.test.js', 'tested-by')
316
-
317
- // Perfect for: File explorers, IDEs, documentation systems, code analysis
318
243
  ```
319
244
 
320
- **🚨 Prevents Common Mistakes:**
321
- - ❌ No infinite recursion in file trees (like brain-cloud team experienced)
322
- - ❌ No data loss from memory storage
323
- - ❌ No performance issues with large directories
324
- - ❌ No need for complex fallback patterns
245
+ **Prevents infinite recursion** that crashes traditional file systems. Tree-aware operations ensure your file explorer never hangs.
325
246
 
326
247
  **[📖 VFS Quick Start →](docs/vfs/QUICK_START.md)** | **[🎯 Common Patterns →](docs/vfs/COMMON_PATTERNS.md)**
327
248
 
328
- **Your knowledge isn't trapped anymore.** Characters live beyond stories. APIs exist beyond code files. Concepts connect across domains. This is knowledge that happens to support files, not a filesystem that happens to store knowledge.
329
-
330
- ### 🚀 **NEW: Enhanced Directory Import with Caching**
331
-
332
- **Import large projects 10-100x faster with intelligent caching:**
249
+ ### 🚀 **Import Anything - CSV, Excel, PDF, URLs**
333
250
 
334
251
  ```javascript
335
- import { Brainy } from '@soulcraft/brainy'
336
- import { ProgressTracker, formatProgress } from '@soulcraft/brainy/types'
337
- import { detectRelationshipsWithConfidence } from '@soulcraft/brainy/neural'
338
-
339
- const brain = new Brainy()
340
- await brain.init()
341
-
342
- // Progress tracking for long operations
343
- const tracker = ProgressTracker.create(1000)
344
- tracker.start()
345
-
346
- for await (const progress of importer.importStream('./project', {
347
- batchSize: 100,
348
- generateEmbeddings: true
349
- })) {
350
- const p = tracker.update(progress.processed, progress.current)
351
- console.log(formatProgress(p))
352
- // [RUNNING] 45% (450/1000) - 23.5 items/s - 23s remaining
353
- }
354
-
355
- // Entity extraction with intelligent caching
356
- const entities = await brain.neural.extractor.extract(text, {
357
- types: ['person', 'organization', 'technology'],
358
- confidence: 0.7,
359
- cache: {
360
- enabled: true,
361
- ttl: 7 * 24 * 60 * 60 * 1000, // 7 days
362
- invalidateOn: 'mtime' // Re-extract when file changes
363
- }
364
- })
252
+ // Import CSV with auto-detection
253
+ await brain.import('customers.csv')
254
+ // Auto-detects: encoding, delimiter, types, creates entities!
365
255
 
366
- // Relationship detection with confidence scores
367
- const relationships = detectRelationshipsWithConfidence(entities, text, {
368
- minConfidence: 0.7
256
+ // Import Excel workbooks with multi-sheet support
257
+ await brain.import('sales-data.xlsx', {
258
+ excelSheets: ['Q1', 'Q2']
369
259
  })
370
260
 
371
- // Create relationships with evidence tracking
372
- await brain.relate({
373
- from: sourceId,
374
- to: targetId,
375
- type: 'creates',
376
- confidence: 0.85,
377
- evidence: {
378
- sourceText: 'John created the database',
379
- method: 'pattern',
380
- reasoning: 'Matches creation pattern; entities in same sentence'
381
- }
261
+ // Import PDF documents with table extraction
262
+ await brain.import('research-paper.pdf', {
263
+ pdfExtractTables: true
382
264
  })
383
265
 
384
- // Monitor cache performance
385
- const stats = brain.neural.extractor.getCacheStats()
386
- console.log(`Cache hit rate: ${(stats.hitRate * 100).toFixed(1)}%`)
387
- // Cache hit rate: 89.5%
266
+ // Import from URLs (auto-fetched)
267
+ await brain.import('https://api.example.com/data.json')
388
268
  ```
389
269
 
390
- **📚 [See Full Example →](examples/directory-import-with-caching.ts)**
391
-
392
- ### 🎯 Zero Configuration Philosophy
270
+ **[📖 Complete Import Guide →](docs/guides/import-anything.md)**
393
271
 
394
- Brainy automatically configures **everything**:
272
+ ### 🧠 **Neural API - Advanced Semantic Analysis**
395
273
 
396
274
  ```javascript
397
- import { Brainy } from '@soulcraft/brainy'
398
-
399
- // 1. Pure zero-config - detects everything
400
- const brain = new Brainy()
275
+ const neural = brain.neural
401
276
 
402
- // 2. Custom configuration
403
- const brain = new Brainy({
404
- storage: { type: 'filesystem', path: './brainy-data' },
405
- embeddings: { model: 'all-MiniLM-L6-v2' },
406
- cache: { enabled: true, maxSize: 1000 }
277
+ // Automatic semantic clustering
278
+ const clusters = await neural.clusters({
279
+ algorithm: 'kmeans',
280
+ maxClusters: 5,
281
+ threshold: 0.8
407
282
  })
408
283
 
409
- // 3. Production configuration
410
- const customBrain = new Brainy({
411
- mode: 'production',
412
- model: 'q8', // Optimized model (99% accuracy, 75% smaller)
413
- storage: 'cloud', // or 'memory', 'disk', 'auto'
414
- features: ['core', 'search', 'cache']
415
- })
416
- ```
284
+ // Calculate similarity between any items
285
+ const similarity = await neural.similar('item1', 'item2')
417
286
 
418
- **What's Auto-Detected:**
287
+ // Find nearest neighbors
288
+ const neighbors = await neural.neighbors('item-id', 10)
419
289
 
420
- - **Storage**: S3/GCS/R2 → Filesystem (priority order)
421
- - **Models**: Always Q8 for optimal balance
422
- - **Features**: Minimal → Default → Full based on environment
423
- - **Memory**: Optimal cache sizes and batching
424
- - **Performance**: Threading, chunking, indexing strategies
290
+ // Detect outliers
291
+ const outliers = await neural.outliers(0.3)
425
292
 
426
- ### Production Performance
293
+ // Generate visualization data for D3/Cytoscape
294
+ const vizData = await neural.visualize({
295
+ maxNodes: 100,
296
+ dimensions: 3,
297
+ algorithm: 'force'
298
+ })
299
+ ```
427
300
 
428
- - **3ms average search** - Lightning fast queries
429
- - **24MB memory footprint** - Efficient resource usage
430
- - **Worker-based embeddings** - Non-blocking operations
431
- - **Automatic caching** - Intelligent result caching
301
+ ---
432
302
 
433
- ### 🎛️ Advanced Configuration (When Needed)
303
+ ## 🌐 Framework Integration
434
304
 
435
- Most users **never need this** - zero-config handles everything. For advanced use cases:
305
+ **Brainy is framework-first!** Works with **any** modern framework:
436
306
 
307
+ ### ⚛️ React & Next.js
437
308
  ```javascript
438
- // Model is always Q8 for optimal performance
439
- const brain = new Brainy() // Uses Q8 automatically
440
-
441
- // Storage control (auto-detected by default)
442
- const diskBrain = new Brainy({storage: 'disk'}) // Local filesystem
443
- const cloudBrain = new Brainy({storage: 'cloud'}) // S3/GCS/R2
309
+ function SearchComponent() {
310
+ const [brain] = useState(() => new Brainy())
311
+ useEffect(() => { brain.init() }, [])
444
312
 
445
- // Legacy full config (still supported)
446
- const legacyBrain = new Brainy({
447
- storage: {type: 'filesystem', path: './data'}
448
- })
313
+ const handleSearch = async (query) => {
314
+ const results = await brain.find(query)
315
+ setResults(results)
316
+ }
317
+ }
449
318
  ```
450
319
 
451
- **Model Details:**
452
-
453
- - **Q8**: 33MB, 99% accuracy, 75% smaller than full precision
454
- - Fast loading and optimal memory usage
455
- - Perfect for all environments
456
-
457
- **Air-gap deployment:**
320
+ ### 🟢 Vue.js & Nuxt.js
321
+ ```javascript
322
+ export default {
323
+ async mounted() {
324
+ this.brain = new Brainy()
325
+ await this.brain.init()
326
+ },
327
+ methods: {
328
+ async search(query) {
329
+ return await this.brain.find(query)
330
+ }
331
+ }
332
+ }
333
+ ```
458
334
 
459
- ```bash
460
- npm run download-models # Download Q8 model
461
- npm run download-models:q8 # Download Q8 model
335
+ ### 🅰️ Angular
336
+ ```typescript
337
+ @Injectable({ providedIn: 'root' })
338
+ export class BrainyService {
339
+ private brain = new Brainy()
340
+ async search(query: string) {
341
+ return await this.brain.find(query)
342
+ }
343
+ }
462
344
  ```
463
345
 
464
- ## 🚀 Import Anything - Files, Data, URLs
346
+ **Works with:** Svelte, Solid.js, Qwik, Fresh, and more! All bundlers (Webpack, Vite, Rollup). SSR/SSG. Edge runtimes.
465
347
 
466
- Brainy's universal import intelligently handles **any data format**:
348
+ ---
467
349
 
468
- ```javascript
469
- // Import CSV with auto-detection
470
- await brain.import('customers.csv')
471
- // ✨ Auto-detects: encoding, delimiter, types, creates entities!
350
+ ## 💾 Storage - From Development to Production
472
351
 
473
- // Import Excel workbooks with multi-sheet support
474
- await brain.import('sales-data.xlsx', {
475
- excelSheets: ['Q1', 'Q2'] // or 'all' for all sheets
476
- })
477
- // ✨ Processes all sheets, preserves structure, infers types!
352
+ ### 🚀 **Development: Just Works**
353
+ ```javascript
354
+ const brain = new Brainy() // Memory storage, auto-configured
355
+ ```
478
356
 
479
- // Import PDF documents with table extraction
480
- await brain.import('research-paper.pdf', {
481
- pdfExtractTables: true
357
+ ### **Production: FileSystem with Compression**
358
+ ```javascript
359
+ const brain = new Brainy({
360
+ storage: {
361
+ type: 'filesystem',
362
+ path: './brainy-data',
363
+ compression: true // 60-80% space savings!
364
+ }
482
365
  })
483
- // ✨ Extracts text, detects tables, preserves metadata!
484
-
485
- // Import JSON/YAML data
486
- await brain.import([
487
- { name: 'Alice', role: 'Engineer' },
488
- { name: 'Bob', role: 'Designer' }
489
- ])
490
- // ✨ Automatically creates Person entities with relationships!
491
-
492
- // Import from URLs (auto-fetched)
493
- await brain.import('https://api.example.com/data.json')
494
- // ✨ Auto-detects URL, fetches, parses, processes!
495
366
  ```
496
367
 
497
- **📖 [Complete Import Guide →](docs/guides/import-anything.md)** | **[Live Example →](examples/import-excel-pdf-csv.ts)**
498
-
499
- ## 📚 Core API
368
+ ### ☁️ **Production: Cloud Storage** (NEW in v4.0.0)
500
369
 
501
- ### `search()` - Vector Similarity
370
+ Choose your cloud provider - all support **automatic cost optimization:**
502
371
 
372
+ #### **AWS S3 / Cloudflare R2 / DigitalOcean Spaces**
503
373
  ```javascript
504
- const results = await brain.search("machine learning", {
505
- limit: 10, // Number of results
506
- metadata: {type: "article"}, // Filter by metadata
507
- includeContent: true // Include full content
374
+ const brain = new Brainy({
375
+ storage: {
376
+ type: 's3',
377
+ s3Storage: {
378
+ bucketName: 'my-knowledge-base',
379
+ region: 'us-east-1',
380
+ accessKeyId: process.env.AWS_ACCESS_KEY_ID,
381
+ secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
382
+ }
383
+ }
508
384
  })
509
- ```
510
385
 
511
- ### `find()` - Natural Language Queries
386
+ // Enable Intelligent-Tiering for automatic 96% cost savings
387
+ await brain.storage.enableIntelligentTiering('entities/', 'brainy-auto-tier')
388
+ ```
512
389
 
390
+ #### **Google Cloud Storage**
513
391
  ```javascript
514
- // Simple natural language
515
- const results = await brain.find("recent important documents")
516
-
517
- // Structured query with Triple Intelligence
518
- const results = await brain.find({
519
- like: "JavaScript", // Vector similarity
520
- where: { // Metadata filters
521
- year: {greaterThan: 2020},
522
- important: true
523
- },
524
- related: {to: "React"} // Graph relationships
392
+ const brain = new Brainy({
393
+ storage: {
394
+ type: 'gcs',
395
+ gcsStorage: {
396
+ bucketName: 'my-knowledge-base',
397
+ keyFilename: '/path/to/service-account.json'
398
+ }
399
+ }
525
400
  })
526
- ```
527
401
 
528
- ### CRUD Operations
402
+ // Enable Autoclass for automatic 94% cost savings
403
+ await brain.storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
404
+ ```
529
405
 
406
+ #### **Azure Blob Storage**
530
407
  ```javascript
531
- // Create entities (nouns)
532
- const id = await brain.add(data, { nounType: nounType, ...metadata })
533
-
534
- // Create relationships (verbs)
535
- const verbId = await brain.relate(sourceId, targetId, "relationType", {
536
- strength: 0.9,
537
- bidirectional: false
408
+ const brain = new Brainy({
409
+ storage: {
410
+ type: 'azure',
411
+ azureStorage: {
412
+ containerName: 'knowledge-base',
413
+ connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING
414
+ }
415
+ }
538
416
  })
539
417
 
540
- // Read
541
- const item = await brain.getNoun(id)
542
- const verb = await brain.getVerb(verbId)
418
+ // Batch tier changes for 99% cost savings
419
+ await brain.storage.setBlobTierBatch(
420
+ oldBlobs.map(name => ({ blobName: name, tier: 'Archive' }))
421
+ )
422
+ ```
543
423
 
544
- // Update
545
- await brain.updateNoun(id, newData, newMetadata)
546
- await brain.updateVerb(verbId, newMetadata)
424
+ ### 💰 **Cost Optimization Impact**
547
425
 
548
- // Delete
549
- await brain.deleteNoun(id)
550
- await brain.deleteVerb(verbId)
426
+ | Scale | Before | After (Archive) | Savings/Year |
427
+ |-------|--------|-----------------|--------------|
428
+ | 5TB | $1,380/year | $59/year | **$1,321 (96%)** |
429
+ | 50TB | $13,800/year | $594/year | **$13,206 (96%)** |
430
+ | 500TB | $138,000/year | $5,940/year | **$132,060 (96%)** |
551
431
 
552
- // Bulk operations
553
- await brain.import(arrayOfData)
554
- const exported = await brain.export({format: 'json'})
432
+ **[📖 AWS S3 Cost Guide →](docs/operations/cost-optimization-aws-s3.md)** | **[GCS →](docs/operations/cost-optimization-gcs.md)** | **[Azure →](docs/operations/cost-optimization-azure.md)** | **[R2 →](docs/operations/cost-optimization-cloudflare-r2.md)**
555
433
 
556
- // Import from CSV, Excel, PDF files (auto-detected)
557
- await brain.import('customers.csv') // CSV with encoding detection
558
- await brain.import('sales-report.xlsx') // Excel with multi-sheet support
559
- await brain.import('research.pdf') // PDF with table extraction
560
- ```
434
+ ---
561
435
 
562
- ## 🌐 Distributed System (NEW!)
436
+ ## 🚀 Production Scale Features (NEW in v4.0.0)
563
437
 
564
- ### Zero-Config Distributed Setup
438
+ ### 🎯 **Type-Aware HNSW - 87% Memory Reduction**
439
+
440
+ **Billion-scale deployments made affordable:**
441
+
442
+ - **Memory @ 1B entities**: 384GB → 50GB (-87%)
443
+ - **Single-type queries**: 10x faster (search 100M instead of 1B)
444
+ - **Multi-type queries**: 5-8x faster
445
+ - **Optimized rebuilds**: 31x faster with type filtering
565
446
 
566
447
  ```javascript
567
- // Single node (default)
568
448
  const brain = new Brainy({
569
- storage: {
570
- type: 's3',
571
- s3Storage: {
572
- bucketName: 'my-data',
573
- region: 'us-east-1',
574
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
575
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
576
- }
577
- }
449
+ hnsw: {
450
+ typeAware: true // Enable type-aware indexing
451
+ }
578
452
  })
453
+ ```
579
454
 
580
- // Distributed cluster - just add one flag!
581
- const brain = new Brainy({
582
- storage: {
583
- type: 's3',
584
- s3Storage: {
585
- bucketName: 'my-data',
586
- region: 'us-east-1',
587
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
588
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
589
- }
590
- },
591
- distributed: true // That's it! Everything else is automatic
455
+ ### **Production-Ready Storage**
456
+
457
+ **NEW v4.0.0 enterprise features:**
458
+
459
+ - **🗑️ Batch Operations**: Delete thousands of entities with retry logic
460
+ - **📦 Gzip Compression**: 60-80% space savings (FileSystem)
461
+ - **💽 OPFS Quota Monitoring**: Real-time quota tracking (Browser)
462
+ - **🔄 Metadata/Vector Separation**: Billion-entity scalability
463
+ - **🛡️ Enterprise Reliability**: Backpressure, circuit breakers, retries
464
+
465
+ ```javascript
466
+ // Batch delete with retry logic
467
+ await brain.storage.batchDelete(keys, {
468
+ maxRetries: 3,
469
+ continueOnError: true
592
470
  })
471
+
472
+ // Get storage status
473
+ const status = await brain.storage.getStorageStatus()
474
+ console.log(`Used: ${status.used}, Quota: ${status.quota}`)
593
475
  ```
594
476
 
595
- ### How It Works
477
+ ### 📊 **Adaptive Memory Management**
596
478
 
597
- - **Storage-Based Discovery**: Nodes find each other via S3/GCS (no Consul/etcd!)
598
- - **Automatic Sharding**: Data distributed by content hash
599
- - **Smart Query Planning**: Queries routed to optimal shards
600
- - **Live Rebalancing**: Handles node joins/leaves automatically
601
- - **Zero Downtime**: Streaming shard migration
479
+ **Auto-scales from 2GB to 128GB+ based on resources:**
602
480
 
603
- ### Real-World Example: Social Media Firehose
481
+ - Container-aware (Docker/K8s cgroups v1/v2)
482
+ - Environment-smart (25% dev, 40% container, 50% production)
483
+ - Model memory accounting (150MB Q8, 250MB FP32)
484
+ - Built-in cache monitoring with recommendations
604
485
 
605
486
  ```javascript
606
- import { Brainy, NounType } from '@soulcraft/brainy'
487
+ // Get cache statistics and recommendations
488
+ const stats = brain.getCacheStats()
489
+ console.log(`Hit rate: ${stats.hitRate * 100}%`)
490
+ // Actionable tuning recommendations included
491
+ ```
607
492
 
608
- // Ingestion nodes (optimized for writes)
609
- const ingestionNode = new Brainy({
610
- storage: {
611
- type: 's3',
612
- s3Storage: {
613
- bucketName: 'social-data',
614
- region: 'us-east-1',
615
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
616
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
617
- }
618
- },
619
- distributed: true,
620
- writeOnly: true // Optimized for high-throughput writes
621
- })
493
+ **[📖 Capacity Planning Guide →](docs/operations/capacity-planning.md)**
622
494
 
623
- // Process Bluesky firehose
624
- blueskyStream.on('post', async (post) => {
625
- await ingestionNode.add(post, {
626
- nounType: NounType.Message,
627
- platform: 'bluesky',
628
- author: post.author,
629
- timestamp: post.createdAt
630
- })
631
- })
495
+ ---
632
496
 
633
- // Search nodes (optimized for queries)
634
- const searchNode = new Brainy({
635
- storage: {
636
- type: 's3',
637
- s3Storage: {
638
- bucketName: 'social-data',
639
- region: 'us-east-1',
640
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
641
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
642
- }
643
- },
644
- distributed: true,
645
- readOnly: true // Optimized for fast queries
646
- })
497
+ ## 🏢 Enterprise Features - No Paywalls
647
498
 
648
- // Search across ALL data from ALL nodes
649
- const trending = await searchNode.find('trending AI topics', {
650
- where: {timestamp: {greaterThan: Date.now() - 3600000}},
651
- limit: 100
652
- })
653
- ```
499
+ Brainy includes **enterprise-grade capabilities at no extra cost**:
654
500
 
655
- ### Benefits Over Traditional Systems
501
+ - **Scales to billions of entities** with 18ms search latency @ 1B scale
502
+ - ✅ **Distributed architecture** with sharding and replication
503
+ - ✅ **Read/write separation** for horizontal scaling
504
+ - ✅ **Connection pooling** and request deduplication
505
+ - ✅ **Built-in monitoring** with metrics and health checks
506
+ - ✅ **Production ready** with circuit breakers and backpressure
656
507
 
657
- | Feature | Traditional (Pinecone, Weaviate) | Brainy Distributed |
658
- |----------------|----------------------------------|-------------------------------|
659
- | Setup | Complex (k8s, operators) | One flag: `distributed: true` |
660
- | Coordination | External (etcd, Consul) | Built-in (via storage) |
661
- | Minimum Nodes | 3-5 for HA | 1 (scale as needed) |
662
- | Sharding | Random | Domain-aware |
663
- | Query Planning | Basic | Triple Intelligence |
664
- | Cost | High (always-on clusters) | Low (scale to zero) |
508
+ **No premium tiers. No feature gates. Everyone gets the same powerful system.**
665
509
 
666
- ## 🎯 Use Cases
510
+ ---
667
511
 
668
- ### Knowledge Management with Relationships
512
+ ## 📊 Benchmarks
669
513
 
514
+ | Operation | Performance | Memory |
515
+ |----------------------------------|-------------|----------|
516
+ | Initialize | 450ms | 24MB |
517
+ | Add Item | 12ms | +0.1MB |
518
+ | Vector Search (1k items) | 3ms | - |
519
+ | Metadata Filter (10k items) | 0.8ms | - |
520
+ | Natural Language Query | 15ms | - |
521
+ | Bulk Import (1000 items) | 2.3s | +8MB |
522
+ | **Production Scale (10M items)** | **5.8ms** | **12GB** |
523
+ | **Billion Scale (1B items)** | **18ms** | **50GB** |
524
+
525
+ ---
526
+
527
+ ## 🎯 Use Cases
528
+
529
+ ### Knowledge Management
670
530
  ```javascript
671
- // Store documentation with rich relationships
531
+ // Create knowledge graph with relationships
672
532
  const apiGuide = await brain.add("REST API Guide", {
673
533
  nounType: NounType.Document,
674
- title: "API Guide",
675
- category: "documentation",
676
- version: "2.0"
534
+ category: "documentation"
677
535
  })
678
536
 
679
537
  const author = await brain.add("Jane Developer", {
@@ -681,37 +539,17 @@ const author = await brain.add("Jane Developer", {
681
539
  role: "tech-lead"
682
540
  })
683
541
 
684
- const project = await brain.add("E-commerce Platform", {
685
- nounType: NounType.Project,
686
- status: "active"
687
- })
542
+ await brain.relate(author, apiGuide, "authored")
688
543
 
689
- // Create knowledge graph
690
- await brain.relate(author, apiGuide, "authored", {
691
- date: "2024-03-15"
692
- })
693
- await brain.relate(apiGuide, project, "documents", {
694
- coverage: "complete"
695
- })
696
-
697
- // Query the knowledge graph naturally
698
- const docs = await brain.find("documentation authored by tech leads for active projects")
699
- ```
700
-
701
- ### Semantic Search
702
-
703
- ```javascript
704
- // Find similar content
705
- const similar = await brain.search(existingContent, {
706
- limit: 5,
707
- threshold: 0.8
708
- })
544
+ // Query naturally
545
+ const docs = await brain.find(
546
+ "documentation authored by tech leads for active projects"
547
+ )
709
548
  ```
710
549
 
711
- ### AI Memory Layer with Context
712
-
550
+ ### AI Memory Layer
713
551
  ```javascript
714
- // Store messages with relationships
552
+ // Store conversation context with relationships
715
553
  const userId = await brain.add("User 123", {
716
554
  nounType: NounType.User,
717
555
  tier: "premium"
@@ -719,20 +557,12 @@ const userId = await brain.add("User 123", {
719
557
 
720
558
  const messageId = await brain.add(userMessage, {
721
559
  nounType: NounType.Message,
722
- timestamp: Date.now(),
723
- session: "abc"
724
- })
725
-
726
- const topicId = await brain.add("Product Support", {
727
- nounType: NounType.Topic,
728
- category: "support"
560
+ timestamp: Date.now()
729
561
  })
730
562
 
731
- // Link message elements
732
563
  await brain.relate(userId, messageId, "sent")
733
- await brain.relate(messageId, topicId, "about")
734
564
 
735
- // Retrieve context with relationships
565
+ // Retrieve context
736
566
  const context = await brain.find({
737
567
  where: {type: "message"},
738
568
  connected: {from: userId, type: "sent"},
@@ -740,40 +570,20 @@ const context = await brain.find({
740
570
  })
741
571
  ```
742
572
 
743
- ## 💾 Storage Options
744
-
745
- Brainy supports multiple storage backends:
746
-
573
+ ### Semantic Search
747
574
  ```javascript
748
- // FileSystem (Node.js - recommended for development)
749
- const brain = new Brainy({
750
- storage: {
751
- type: 'filesystem',
752
- path: './data'
753
- }
754
- })
755
-
756
- // Browser Storage (OPFS) - Works with frameworks
757
- const brain = new Brainy({
758
- storage: {type: 'opfs'} // Framework handles browser polyfills
759
- })
760
-
761
- // S3 Compatible (Production)
762
- const brain = new Brainy({
763
- storage: {
764
- type: 's3',
765
- bucket: 'my-bucket',
766
- region: 'us-east-1'
767
- }
575
+ // Find similar content
576
+ const similar = await brain.search(existingContent, {
577
+ limit: 5,
578
+ threshold: 0.8
768
579
  })
769
580
  ```
770
581
 
771
- ## 🛠️ CLI
582
+ ---
772
583
 
773
- Brainy includes a powerful CLI for testing and management:
584
+ ## 🛠️ CLI
774
585
 
775
586
  ```bash
776
- # Install globally
777
587
  npm install -g brainy
778
588
 
779
589
  # Add data
@@ -787,137 +597,78 @@ brainy find "awesome programming languages"
787
597
 
788
598
  # Interactive mode
789
599
  brainy chat
790
-
791
- # Export data
792
- brainy export --format json > backup.json
793
- ```
794
-
795
- ## 🧠 Neural API - Advanced AI Features
796
-
797
- Brainy includes a powerful Neural API for advanced semantic analysis:
798
-
799
- ### Clustering & Analysis
800
-
801
- ```javascript
802
- // Access via brain.neural
803
- const neural = brain.neural
804
-
805
- // Automatic semantic clustering
806
- const clusters = await neural.clusters()
807
- // Returns groups of semantically similar items
808
-
809
- // Cluster with options
810
- const clusters = await neural.clusters({
811
- algorithm: 'kmeans', // or 'hierarchical', 'sample'
812
- maxClusters: 5, // Maximum number of clusters
813
- threshold: 0.8 // Similarity threshold
814
- })
815
-
816
- // Calculate similarity between any items
817
- const similarity = await neural.similar('item1', 'item2')
818
- // Returns 0-1 score
819
-
820
- // Find nearest neighbors
821
- const neighbors = await neural.neighbors('item-id', 10)
822
-
823
- // Build semantic hierarchy
824
- const hierarchy = await neural.hierarchy('item-id')
825
-
826
- // Detect outliers
827
- const outliers = await neural.outliers(0.3)
828
-
829
- // Generate visualization data for D3/Cytoscape
830
- const vizData = await neural.visualize({
831
- maxNodes: 100,
832
- dimensions: 3,
833
- algorithm: 'force'
834
- })
835
600
  ```
836
601
 
837
- ### Real-World Examples
838
-
839
- ```javascript
840
- // Group customer feedback into themes
841
- const feedbackClusters = await neural.clusters()
842
- for (const cluster of feedbackClusters) {
843
- console.log(`Theme: ${cluster.label}`)
844
- console.log(`Items: ${cluster.members.length}`)
845
- }
602
+ ---
846
603
 
847
- // Find related documents
848
- const docId = await brain.add("Machine learning guide", { nounType: NounType.Document })
849
- const similar = await neural.neighbors(docId, 5)
850
- // Returns 5 most similar documents
604
+ ## 📖 Documentation
851
605
 
852
- // Detect anomalies in data
853
- const anomalies = await neural.outliers(0.2)
854
- console.log(`Found ${anomalies.length} outliers`)
855
- ```
606
+ ### Getting Started
607
+ - [Getting Started Guide](docs/guides/getting-started.md)
608
+ - [v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md) **← NEW**
609
+ - [AWS S3 Cost Guide](docs/operations/cost-optimization-aws-s3.md) | [GCS](docs/operations/cost-optimization-gcs.md) | [Azure](docs/operations/cost-optimization-azure.md) | [R2](docs/operations/cost-optimization-cloudflare-r2.md) **← NEW**
856
610
 
857
- ## 🔌 Augmentations
611
+ ### Framework Integration
612
+ - [Framework Integration Guide](docs/guides/framework-integration.md)
613
+ - [Next.js Integration](docs/guides/nextjs-integration.md)
614
+ - [Vue.js Integration](docs/guides/vue-integration.md)
858
615
 
859
- Extend Brainy with powerful augmentations:
616
+ ### Virtual Filesystem
617
+ - [VFS Core Documentation](docs/vfs/VFS_CORE.md)
618
+ - [Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md)
619
+ - [Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md)
860
620
 
861
- ```bash
862
- # List available augmentations
863
- brainy augment list
621
+ ### Core Documentation
622
+ - [API Reference](docs/api/README.md)
623
+ - [Architecture Overview](docs/architecture/overview.md)
624
+ - [Data Storage Architecture](docs/architecture/data-storage-architecture.md)
625
+ - [Natural Language Guide](docs/guides/natural-language.md)
626
+ - [Triple Intelligence](docs/architecture/triple-intelligence.md)
627
+ - [Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)
864
628
 
865
- # Install an augmentation
866
- brainy augment install explorer
629
+ ### Operations & Production
630
+ - [Capacity Planning](docs/operations/capacity-planning.md)
631
+ - [Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md) **← NEW**
632
+ - [AWS S3 Cost Guide](docs/operations/cost-optimization-aws-s3.md) | [GCS](docs/operations/cost-optimization-gcs.md) | [Azure](docs/operations/cost-optimization-azure.md) | [R2](docs/operations/cost-optimization-cloudflare-r2.md) **← NEW**
867
633
 
868
- # Connect to Brain Cloud
869
- brainy cloud setup
870
- ```
871
-
872
- ## 🏢 Enterprise Features - Included for Everyone
634
+ ---
873
635
 
874
- Brainy includes enterprise-grade capabilities at no extra cost. **No premium tiers, no paywalls.**
636
+ ## 💖 Support Brainy
875
637
 
876
- - **Scales to 10M+ items** with consistent 3ms search latency
877
- - **Distributed architecture** with sharding and replication
878
- - **Read/write separation** for horizontal scaling
879
- - **Connection pooling** and request deduplication
880
- - **Built-in monitoring** with metrics and health checks
881
- - **Production ready** with circuit breakers and backpressure
638
+ Brainy is **free and open source** - no paywalls, no premium tiers, no feature gates. If Brainy helps your project, consider supporting development:
882
639
 
883
- 📖 **More enterprise features coming soon** - Stay tuned!
640
+ - **Star us on [GitHub](https://github.com/soulcraftlabs/brainy)**
641
+ - 💝 **Sponsor via [GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)**
642
+ - 🐛 **Report issues** and contribute code
643
+ - 📣 **Share** with your team and community
884
644
 
885
- ## 📊 Benchmarks
645
+ Your support keeps Brainy free for everyone and enables continued development of enterprise features at no cost.
886
646
 
887
- | Operation | Performance | Memory |
888
- |----------------------------------|-------------|----------|
889
- | Initialize | 450ms | 24MB |
890
- | Add Item | 12ms | +0.1MB |
891
- | Vector Search (1k items) | 3ms | - |
892
- | Metadata Filter (10k items) | 0.8ms | - |
893
- | Natural Language Query | 15ms | - |
894
- | Bulk Import (1000 items) | 2.3s | +8MB |
895
- | **Production Scale (10M items)** | **5.8ms** | **12GB** |
647
+ ---
896
648
 
897
- ## 🔄 Migration from Previous Versions
649
+ ## 📋 System Requirements
898
650
 
899
- Key changes in the latest version:
651
+ **Node.js Version:** 22 LTS (recommended)
900
652
 
901
- - Search methods consolidated into `search()` and `find()`
902
- - Result format now includes full objects with metadata
903
- - Enhanced natural language capabilities
904
- - Distributed architecture support
653
+ - **Node.js 22 LTS** - Fully supported (recommended for production)
654
+ - **Node.js 20 LTS** - Compatible (maintenance mode)
655
+ - **Node.js 24** - Not supported (ONNX compatibility issues)
905
656
 
906
- ## 🤝 Contributing
657
+ If using nvm: `nvm use` (we provide a `.nvmrc` file)
907
658
 
908
- We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
659
+ ---
909
660
 
910
661
  ## 🧠 The Knowledge Operating System Explained
911
662
 
912
663
  ### How We Achieved The Impossible
913
664
 
914
- **Triple Intelligence™** makes us the **world's first** to unify three database paradigms:
665
+ **Triple Intelligence™** unifies three database paradigms that were previously incompatible:
915
666
 
916
667
  1. **Vector databases** (Pinecone, Weaviate) - semantic similarity
917
668
  2. **Graph databases** (Neo4j, ArangoDB) - relationships
918
669
  3. **Document databases** (MongoDB, Elasticsearch) - metadata filtering
919
670
 
920
- **One API to rule them all.** Others make you choose. We unified them.
671
+ **Others make you choose. Brainy does all three together.**
921
672
 
922
673
  ### The Math of Infinite Expressiveness
923
674
 
@@ -945,40 +696,26 @@ We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
945
696
 
946
697
  [→ See the Mathematical Proof & Full Taxonomy](docs/architecture/noun-verb-taxonomy.md)
947
698
 
948
- ## 📖 Documentation
949
-
950
- ### Framework Integration
951
- - [Framework Integration Guide](docs/guides/framework-integration.md) - Complete framework setup guide
952
- - [Next.js Integration](docs/guides/nextjs-integration.md) - React and Next.js examples
953
- - [Vue.js Integration](docs/guides/vue-integration.md) - Vue and Nuxt examples
954
-
955
- ### Virtual Filesystem (Semantic VFS) 🧠📁
956
- - [VFS Core Documentation](docs/vfs/VFS_CORE.md) - Complete filesystem architecture and API
957
- - [Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md) - Multi-dimensional file access (6 semantic dimensions)
958
- - [Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md) - AI-powered concept and entity extraction
959
- - [Examples & Scenarios](docs/vfs/VFS_EXAMPLES_SCENARIOS.md) - Real-world use cases and code
960
- - [VFS API Guide](docs/vfs/VFS_API_GUIDE.md) - Complete API reference
961
-
962
- ### Core Documentation
963
- - [Getting Started Guide](docs/guides/getting-started.md)
964
- - [API Reference](docs/api/README.md)
965
- - [Architecture Overview](docs/architecture/overview.md)
966
- - [Data Storage Architecture](docs/architecture/data-storage-architecture.md) - Deep dive into storage, indexing, and sharding
967
- - [Natural Language Guide](docs/guides/natural-language.md)
968
- - [Triple Intelligence](docs/architecture/triple-intelligence.md)
969
- - [Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)
699
+ ---
970
700
 
971
701
  ## 🏢 Enterprise & Cloud
972
702
 
973
703
  **Brain Cloud** - Managed Brainy with team sync, persistent memory, and enterprise connectors.
974
704
 
975
705
  ```bash
976
- # Get started with free trial
977
706
  brainy cloud setup
978
707
  ```
979
708
 
980
709
  Visit [soulcraft.com](https://soulcraft.com) for more information.
981
710
 
711
+ ---
712
+
713
+ ## 🤝 Contributing
714
+
715
+ We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
716
+
717
+ ---
718
+
982
719
  ## 📄 License
983
720
 
984
721
  MIT © Brainy Contributors
@@ -987,5 +724,6 @@ MIT © Brainy Contributors
987
724
 
988
725
  <p align="center">
989
726
  <strong>Built with ❤️ by the Brainy community</strong><br>
990
- <em>Zero-Configuration AI Database with Triple Intelligence™</em>
991
- </p>
727
+ <em>Zero-Configuration AI Database with Triple Intelligence™</em><br>
728
+ <em>v4.0.0 - Production-Scale Storage with 99% Cost Savings</em>
729
+ </p>