@soulcraft/brainy 3.50.2 → 4.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +201 -0
- package/README.md +396 -658
- package/dist/api/ConfigAPI.js +56 -19
- package/dist/api/DataAPI.js +24 -18
- package/dist/augmentations/storageAugmentations.d.ts +24 -0
- package/dist/augmentations/storageAugmentations.js +22 -0
- package/dist/brainy.js +32 -9
- package/dist/cli/commands/core.d.ts +20 -10
- package/dist/cli/commands/core.js +384 -82
- package/dist/cli/commands/import.d.ts +41 -0
- package/dist/cli/commands/import.js +456 -0
- package/dist/cli/commands/insights.d.ts +34 -0
- package/dist/cli/commands/insights.js +300 -0
- package/dist/cli/commands/neural.d.ts +6 -12
- package/dist/cli/commands/neural.js +113 -10
- package/dist/cli/commands/nlp.d.ts +28 -0
- package/dist/cli/commands/nlp.js +246 -0
- package/dist/cli/commands/storage.d.ts +64 -0
- package/dist/cli/commands/storage.js +730 -0
- package/dist/cli/index.js +210 -24
- package/dist/coreTypes.d.ts +206 -34
- package/dist/distributed/configManager.js +8 -6
- package/dist/distributed/shardMigration.js +2 -0
- package/dist/distributed/storageDiscovery.js +6 -4
- package/dist/embeddings/EmbeddingManager.d.ts +2 -2
- package/dist/embeddings/EmbeddingManager.js +5 -1
- package/dist/graph/lsm/LSMTree.js +32 -20
- package/dist/hnsw/typeAwareHNSWIndex.js +6 -2
- package/dist/storage/adapters/azureBlobStorage.d.ts +545 -0
- package/dist/storage/adapters/azureBlobStorage.js +1809 -0
- package/dist/storage/adapters/baseStorageAdapter.d.ts +16 -13
- package/dist/storage/adapters/fileSystemStorage.d.ts +21 -9
- package/dist/storage/adapters/fileSystemStorage.js +204 -127
- package/dist/storage/adapters/gcsStorage.d.ts +119 -9
- package/dist/storage/adapters/gcsStorage.js +317 -62
- package/dist/storage/adapters/memoryStorage.d.ts +30 -18
- package/dist/storage/adapters/memoryStorage.js +99 -94
- package/dist/storage/adapters/opfsStorage.d.ts +48 -10
- package/dist/storage/adapters/opfsStorage.js +201 -80
- package/dist/storage/adapters/r2Storage.d.ts +12 -5
- package/dist/storage/adapters/r2Storage.js +63 -15
- package/dist/storage/adapters/s3CompatibleStorage.d.ts +164 -17
- package/dist/storage/adapters/s3CompatibleStorage.js +472 -80
- package/dist/storage/adapters/typeAwareStorageAdapter.d.ts +38 -6
- package/dist/storage/adapters/typeAwareStorageAdapter.js +218 -39
- package/dist/storage/baseStorage.d.ts +41 -38
- package/dist/storage/baseStorage.js +110 -134
- package/dist/storage/storageFactory.d.ts +29 -2
- package/dist/storage/storageFactory.js +30 -1
- package/dist/utils/entityIdMapper.js +5 -2
- package/dist/utils/fieldTypeInference.js +8 -1
- package/dist/utils/metadataFilter.d.ts +3 -2
- package/dist/utils/metadataFilter.js +1 -0
- package/dist/utils/metadataIndex.js +2 -0
- package/dist/utils/metadataIndexChunking.js +9 -4
- package/dist/utils/periodicCleanup.js +1 -0
- package/package.json +3 -1
package/README.md
CHANGED
|
@@ -11,91 +11,126 @@
|
|
|
11
11
|
|
|
12
12
|
**🧠 Brainy - The Knowledge Operating System**
|
|
13
13
|
|
|
14
|
-
**The world's first Knowledge Operating System** where every piece of knowledge - files, concepts, entities, ideas - exists as living information that understands itself, evolves over time, and connects to everything related.
|
|
14
|
+
**The world's first Knowledge Operating System** where every piece of knowledge - files, concepts, entities, ideas - exists as living information that understands itself, evolves over time, and connects to everything related.
|
|
15
15
|
|
|
16
16
|
**Why Brainy Changes Everything**: Traditional systems trap knowledge in files or database rows. Brainy liberates it. Your characters exist across stories. Your concepts span projects. Your APIs remember their evolution. Every piece of knowledge - whether it's code, prose, or pure ideas - lives, breathes, and connects in a unified intelligence layer where everything understands its meaning, remembers its history, and relates to everything else.
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
Built on revolutionary **Triple Intelligence™** that unifies vector similarity, graph relationships, and document filtering in one magical API. **Framework-first design.** Zero configuration. O(log n) performance, <10ms search latency. **Production-ready for billion-scale deployments.**
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## 🎉 NEW in v4.0.0 - Enterprise-Scale Cost Optimization
|
|
23
|
+
|
|
24
|
+
**Major Release: Production cost optimization and enterprise-scale features**
|
|
25
|
+
|
|
26
|
+
### 💰 **Up to 96% Storage Cost Savings**
|
|
21
27
|
|
|
22
|
-
|
|
28
|
+
Automatic cloud storage lifecycle management for **AWS S3**, **Google Cloud Storage**, and **Azure Blob Storage**:
|
|
23
29
|
|
|
24
|
-
**
|
|
30
|
+
- **GCS Autoclass**: Fully automatic tier optimization (94% savings!)
|
|
31
|
+
- **AWS Intelligent-Tiering**: Smart archival with instant retrieval
|
|
32
|
+
- **Azure Lifecycle Policies**: Automatic tier transitions
|
|
33
|
+
- **Cost Impact**: $138,000/year → $5,940/year @ 500TB scale
|
|
25
34
|
|
|
26
|
-
|
|
27
|
-
- **Memory @ 1B scale**: 384GB → 50GB (-87% / -334GB)
|
|
28
|
-
- **Single-type queries**: 10x faster (search 100M nodes instead of 1B)
|
|
29
|
-
- **Multi-type queries**: 5-8x faster (search subset of types)
|
|
30
|
-
- **All-types queries**: ~3x faster (31 smaller graphs vs 1 large graph)
|
|
35
|
+
### ⚡ **Performance at Billion-Scale**
|
|
31
36
|
|
|
32
|
-
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
- **Lazy loading**: 15 minutes for top 2 types only
|
|
37
|
+
- **1000x faster batch deletions** (533 entities/sec vs 0.5/sec)
|
|
38
|
+
- **60-80% FileSystem compression** with gzip
|
|
39
|
+
- **OPFS quota monitoring** for browser storage
|
|
40
|
+
- **Enhanced CLI** with 47 commands including 9 storage management tools
|
|
37
41
|
|
|
38
|
-
|
|
39
|
-
- 47 new tests (33 unit + 14 integration) - all passing
|
|
40
|
-
- Backward compatible - opt-in via configuration
|
|
41
|
-
- Works with all storage backends (FileSystem, S3, GCS, R2, Memory, OPFS)
|
|
42
|
+
### 🛡️ **Zero Breaking Changes**
|
|
42
43
|
|
|
43
|
-
**
|
|
44
|
+
**100% backward compatible.** No migration required. All new features are opt-in.
|
|
45
|
+
|
|
46
|
+
**[📖 Read the full v4.0.0 Changelog →](CHANGELOG.md)** | **[Migration Guide →](docs/MIGRATION-V3-TO-V4.md)**
|
|
47
|
+
|
|
48
|
+
---
|
|
44
49
|
|
|
45
|
-
|
|
50
|
+
## 🎯 What Makes Brainy Revolutionary?
|
|
46
51
|
|
|
47
|
-
**
|
|
52
|
+
### 🧠 **Triple Intelligence™ - The Impossible Made Possible**
|
|
48
53
|
|
|
49
|
-
|
|
50
|
-
- Container-aware (Docker/K8s cgroups v1/v2 detection)
|
|
51
|
-
- Environment-smart (development 25%, container 40%, production 50% allocation)
|
|
52
|
-
- Model memory accounting (150MB Q8, 250MB FP32 reserved before cache)
|
|
54
|
+
**The world's first to unify three database paradigms in ONE API:**
|
|
53
55
|
|
|
54
|
-
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
- Massive performance win for hot paths (vector search, distance calculations)
|
|
56
|
+
- **Vector Search** 🔍 Semantic similarity like Pinecone/Weaviate
|
|
57
|
+
- **Graph Relationships** 🕸️ Navigate connections like Neo4j/ArangoDB
|
|
58
|
+
- **Document Filtering** 📊 MongoDB-style queries with O(log n) performance
|
|
58
59
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
60
|
+
**Others make you choose.** Vector OR graph OR document. **Brainy does ALL THREE together.** This is what enables The Knowledge Operating System.
|
|
61
|
+
|
|
62
|
+
### 🚀 **Zero Configuration - Just Works™**
|
|
63
|
+
|
|
64
|
+
```javascript
|
|
65
|
+
import { Brainy } from '@soulcraft/brainy'
|
|
66
|
+
|
|
67
|
+
const brain = new Brainy()
|
|
68
|
+
await brain.init()
|
|
69
|
+
|
|
70
|
+
// That's it! Auto-detects storage, optimizes memory, configures everything.
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
No configuration files. No environment variables. No complex setup. **It just works.**
|
|
74
|
+
|
|
75
|
+
### ⚡ **Production Performance at Any Scale**
|
|
76
|
+
|
|
77
|
+
- **<10ms search** across millions of entities
|
|
78
|
+
- **87% memory reduction** @ billion scale (384GB → 50GB)
|
|
79
|
+
- **10x faster queries** with type-aware indexing
|
|
80
|
+
- **99% storage cost savings** with intelligent archival
|
|
81
|
+
- **Container-aware** memory allocation (Docker/K8s)
|
|
82
|
+
|
|
83
|
+
### 🎯 **31 Noun Types × 40 Verb Types = Infinite Expressiveness**
|
|
84
|
+
|
|
85
|
+
Model **ANY domain** with 1,240 base type combinations + unlimited metadata:
|
|
86
|
+
|
|
87
|
+
- Healthcare: Patient → diagnoses → Condition
|
|
88
|
+
- Finance: Account → transfers → Transaction
|
|
89
|
+
- Manufacturing: Product → assembles → Component
|
|
90
|
+
- Education: Student → completes → Course
|
|
91
|
+
- **Your domain**: Your types + Your relationships = Your knowledge graph
|
|
92
|
+
|
|
93
|
+
[→ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md)
|
|
94
|
+
|
|
95
|
+
---
|
|
63
96
|
|
|
64
|
-
|
|
65
|
-
- Public API unchanged
|
|
66
|
-
- Automatic memory detection and allocation
|
|
67
|
-
- Progressive enhancement for existing applications
|
|
97
|
+
## 💡 What Can You Build?
|
|
68
98
|
|
|
69
|
-
**
|
|
99
|
+
**Brainy unlocks entirely new categories of applications.** Here's what developers are building:
|
|
70
100
|
|
|
71
|
-
###
|
|
101
|
+
### 📚 **Intelligent Documentation Systems**
|
|
102
|
+
- **Living knowledge bases** where docs understand each other and evolve
|
|
103
|
+
- **Smart wikis** that auto-link related concepts and detect outdated information
|
|
104
|
+
- **Research assistants** that remember every paper you've read and find connections
|
|
72
105
|
|
|
73
|
-
|
|
74
|
-
-
|
|
75
|
-
-
|
|
76
|
-
-
|
|
77
|
-
- **🎯 Production Ready**: Fully backward compatible, opt-in features
|
|
106
|
+
### 🤖 **AI Agents with Perfect Memory**
|
|
107
|
+
- **Conversational AI** that remembers context across months, not just messages
|
|
108
|
+
- **Code assistants** that understand your entire codebase as a knowledge graph
|
|
109
|
+
- **Personal assistants** with unlimited memory that never forgets important details
|
|
78
110
|
|
|
79
|
-
###
|
|
111
|
+
### 🎮 **Rich Interactive Experiences**
|
|
112
|
+
- **Game worlds** where NPCs remember every interaction and relationships evolve
|
|
113
|
+
- **Story engines** where characters persist across multiple narratives
|
|
114
|
+
- **Educational platforms** that build personalized knowledge graphs as you learn
|
|
80
115
|
|
|
81
|
-
|
|
82
|
-
- **
|
|
83
|
-
- **
|
|
84
|
-
- **
|
|
116
|
+
### 🔍 **Next-Gen Search & Discovery**
|
|
117
|
+
- **Semantic code search** across millions of repositories
|
|
118
|
+
- **Smart file explorers** that understand code relationships, not just folders
|
|
119
|
+
- **Research platforms** that find papers by meaning, not keywords
|
|
85
120
|
|
|
86
|
-
###
|
|
121
|
+
### 🏢 **Enterprise Knowledge Management**
|
|
122
|
+
- **Corporate memory systems** where institutional knowledge never gets lost
|
|
123
|
+
- **Customer intelligence** platforms that understand every interaction
|
|
124
|
+
- **Product catalogs** with semantic search and relationship-based recommendations
|
|
87
125
|
|
|
88
|
-
|
|
89
|
-
- **
|
|
90
|
-
- **
|
|
91
|
-
- **
|
|
126
|
+
### 🎨 **Creative Tools & Content Platforms**
|
|
127
|
+
- **Writing assistants** that track characters, plotlines, and themes across stories
|
|
128
|
+
- **Content management** where every asset knows its relationships and history
|
|
129
|
+
- **Media libraries** with intelligent tagging and similarity-based discovery
|
|
92
130
|
|
|
93
|
-
|
|
131
|
+
**The Pattern**: If your app needs to **remember**, **understand**, or **connect** information, Brainy makes it trivial.
|
|
94
132
|
|
|
95
|
-
|
|
96
|
-
- **384D Vectors**: Optimized embeddings (all-MiniLM-L6-v2)
|
|
97
|
-
- **Built-in Caching**: Intelligent result caching + new entity extraction cache
|
|
98
|
-
- **Production Ready**: Thoroughly tested core functionality
|
|
133
|
+
---
|
|
99
134
|
|
|
100
135
|
## ⚡ Quick Start - Zero Configuration
|
|
101
136
|
|
|
@@ -103,7 +138,7 @@
|
|
|
103
138
|
npm install @soulcraft/brainy
|
|
104
139
|
```
|
|
105
140
|
|
|
106
|
-
### 🎯 **
|
|
141
|
+
### 🎯 **Your First Knowledge Graph in 30 Seconds**
|
|
107
142
|
|
|
108
143
|
```javascript
|
|
109
144
|
import { Brainy, NounType } from '@soulcraft/brainy'
|
|
@@ -145,7 +180,9 @@ await brain.relate({
|
|
|
145
180
|
})
|
|
146
181
|
|
|
147
182
|
// Natural language search with graph relationships
|
|
148
|
-
const results = await brain.find({
|
|
183
|
+
const results = await brain.find({
|
|
184
|
+
query: "programming languages used by server runtimes"
|
|
185
|
+
})
|
|
149
186
|
|
|
150
187
|
// Triple Intelligence: vector + metadata + relationships
|
|
151
188
|
const filtered = await brain.find({
|
|
@@ -155,525 +192,346 @@ const filtered = await brain.find({
|
|
|
155
192
|
})
|
|
156
193
|
```
|
|
157
194
|
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
**Brainy is framework-first!** Works seamlessly with any modern JavaScript framework:
|
|
161
|
-
|
|
162
|
-
### ⚛️ **React & Next.js**
|
|
163
|
-
```javascript
|
|
164
|
-
import { Brainy } from '@soulcraft/brainy'
|
|
165
|
-
|
|
166
|
-
function SearchComponent() {
|
|
167
|
-
const [brain] = useState(() => new Brainy())
|
|
168
|
-
|
|
169
|
-
useEffect(() => {
|
|
170
|
-
brain.init()
|
|
171
|
-
}, [])
|
|
172
|
-
|
|
173
|
-
const handleSearch = async (query) => {
|
|
174
|
-
const results = await brain.find(query)
|
|
175
|
-
setResults(results)
|
|
176
|
-
}
|
|
177
|
-
}
|
|
178
|
-
```
|
|
179
|
-
|
|
180
|
-
### 🟢 **Vue.js & Nuxt.js**
|
|
181
|
-
```javascript
|
|
182
|
-
import { Brainy } from '@soulcraft/brainy'
|
|
183
|
-
|
|
184
|
-
export default {
|
|
185
|
-
async mounted() {
|
|
186
|
-
this.brain = new Brainy()
|
|
187
|
-
await this.brain.init()
|
|
188
|
-
},
|
|
189
|
-
methods: {
|
|
190
|
-
async search(query) {
|
|
191
|
-
return await this.brain.find(query)
|
|
192
|
-
}
|
|
193
|
-
}
|
|
194
|
-
}
|
|
195
|
-
```
|
|
196
|
-
|
|
197
|
-
### 🅰️ **Angular**
|
|
198
|
-
```typescript
|
|
199
|
-
import { Injectable } from '@angular/core'
|
|
200
|
-
import { Brainy } from '@soulcraft/brainy'
|
|
201
|
-
|
|
202
|
-
@Injectable({ providedIn: 'root' })
|
|
203
|
-
export class BrainyService {
|
|
204
|
-
private brain = new Brainy()
|
|
205
|
-
|
|
206
|
-
async init() {
|
|
207
|
-
await this.brain.init()
|
|
208
|
-
}
|
|
209
|
-
|
|
210
|
-
async search(query: string) {
|
|
211
|
-
return await this.brain.find(query)
|
|
212
|
-
}
|
|
213
|
-
}
|
|
214
|
-
```
|
|
215
|
-
|
|
216
|
-
### 🔥 **Other Frameworks**
|
|
217
|
-
Brainy works with **any** framework that supports ES6 imports: Svelte, Solid.js, Qwik, Fresh, and more!
|
|
218
|
-
|
|
219
|
-
**Framework Compatibility:**
|
|
220
|
-
- ✅ All modern bundlers (Webpack, Vite, Rollup, Parcel)
|
|
221
|
-
- ✅ SSR/SSG (Next.js, Nuxt, SvelteKit, Astro)
|
|
222
|
-
- ✅ Edge runtimes (Vercel Edge, Cloudflare Workers)
|
|
223
|
-
- ✅ Browser and Node.js environments
|
|
224
|
-
|
|
225
|
-
## 📋 System Requirements
|
|
226
|
-
|
|
227
|
-
**Node.js Version:** 22 LTS or later (recommended)
|
|
228
|
-
|
|
229
|
-
- ✅ **Node.js 22 LTS** - Fully supported and recommended for production
|
|
230
|
-
- ✅ **Node.js 20 LTS** - Compatible (maintenance mode)
|
|
231
|
-
- ❌ **Node.js 24** - Not supported (known ONNX runtime compatibility issues)
|
|
232
|
-
|
|
233
|
-
> **Important:** Brainy uses ONNX runtime for AI embeddings. Node.js 24 has known compatibility issues that cause
|
|
234
|
-
> crashes during inference operations. We recommend Node.js 22 LTS for maximum stability.
|
|
235
|
-
|
|
236
|
-
If using nvm: `nvm use` (we provide a `.nvmrc` file)
|
|
237
|
-
|
|
238
|
-
## 🚀 Key Features
|
|
239
|
-
|
|
240
|
-
### World's First Triple Intelligence™ Engine
|
|
195
|
+
**That's it!** You just created a knowledge graph with semantic search, relationship traversal, and metadata filtering. **No configuration. No complexity. Just works.**
|
|
241
196
|
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
- **Vector Search**: Semantic similarity with HNSW indexing
|
|
245
|
-
- **Graph Relationships**: Navigate connected knowledge like Neo4j
|
|
246
|
-
- **Document Filtering**: MongoDB-style queries with O(log n) performance
|
|
247
|
-
- **Unified in ONE API**: No separate queries, no complex joins
|
|
248
|
-
- **First to solve this**: Others do vector OR graph OR document—we do ALL
|
|
249
|
-
|
|
250
|
-
### The Knowledge Operating System with Infinite Expressiveness
|
|
251
|
-
|
|
252
|
-
**Enabled by Triple Intelligence, standardized for everyone:**
|
|
197
|
+
---
|
|
253
198
|
|
|
254
|
-
|
|
255
|
-
- **∞ Expressiveness**: Unlimited metadata = model ANY data
|
|
256
|
-
- **One Language**: All tools, augmentations, AI models speak the same types
|
|
257
|
-
- **Perfect Interoperability**: Move data between any Brainy instance
|
|
258
|
-
- **No Schema Lock-in**: Evolve without migrations
|
|
199
|
+
## 🌟 Core Features
|
|
259
200
|
|
|
260
|
-
### Natural Language Understanding
|
|
201
|
+
### 🧠 **Natural Language Understanding**
|
|
261
202
|
|
|
262
203
|
```javascript
|
|
263
|
-
// Ask questions naturally
|
|
204
|
+
// Ask questions naturally - Brainy understands
|
|
264
205
|
await brain.find("Show me recent React components with tests")
|
|
265
206
|
await brain.find("Popular JavaScript libraries similar to Vue")
|
|
266
207
|
await brain.find("Documentation about authentication from last month")
|
|
267
|
-
```
|
|
268
208
|
|
|
269
|
-
|
|
209
|
+
// Structured queries with Triple Intelligence
|
|
210
|
+
await brain.find({
|
|
211
|
+
like: "React", // Vector similarity
|
|
212
|
+
where: { // Document filtering
|
|
213
|
+
type: "library",
|
|
214
|
+
year: {greaterThan: 2020}
|
|
215
|
+
},
|
|
216
|
+
connected: {to: "JavaScript", depth: 2} // Graph relationships
|
|
217
|
+
})
|
|
218
|
+
```
|
|
270
219
|
|
|
271
|
-
|
|
220
|
+
### 🌐 **Virtual Filesystem - Intelligent File Management**
|
|
272
221
|
|
|
273
|
-
|
|
274
|
-
- **Semantic Search**: Find files by content, not just filename
|
|
275
|
-
- **Production Storage**: Filesystem and cloud storage for real applications
|
|
276
|
-
- **Zero-Config**: Works out of the box with intelligent defaults
|
|
222
|
+
Build file explorers, IDEs, and knowledge systems that **never crash**:
|
|
277
223
|
|
|
278
224
|
```javascript
|
|
279
|
-
import { Brainy } from '@soulcraft/brainy'
|
|
280
|
-
|
|
281
|
-
// ✅ CORRECT: Use persistent storage for file systems
|
|
282
|
-
const brain = new Brainy({
|
|
283
|
-
storage: {
|
|
284
|
-
type: 'filesystem', // Persisted to disk
|
|
285
|
-
path: './brainy-data' // Your file storage
|
|
286
|
-
}
|
|
287
|
-
})
|
|
288
|
-
await brain.init()
|
|
289
|
-
|
|
290
225
|
const vfs = brain.vfs()
|
|
291
226
|
await vfs.init()
|
|
292
227
|
|
|
293
228
|
// ✅ Safe file operations
|
|
294
229
|
await vfs.writeFile('/projects/app/index.js', 'console.log("Hello")')
|
|
295
230
|
await vfs.mkdir('/docs')
|
|
296
|
-
await vfs.writeFile('/docs/README.md', '# My Project')
|
|
297
231
|
|
|
298
232
|
// ✅ NEVER crashes: Tree-aware directory listing
|
|
299
233
|
const children = await vfs.getDirectChildren('/projects')
|
|
300
|
-
// Returns only direct children, never the directory itself
|
|
301
234
|
|
|
302
235
|
// ✅ Build file explorers safely
|
|
303
236
|
const tree = await vfs.getTreeStructure('/projects', {
|
|
304
237
|
maxDepth: 3, // Prevent deep recursion
|
|
305
|
-
sort: 'name'
|
|
238
|
+
sort: 'name'
|
|
306
239
|
})
|
|
307
240
|
|
|
308
241
|
// ✅ Semantic file search
|
|
309
242
|
const reactFiles = await vfs.search('React components with hooks')
|
|
310
|
-
const docs = await vfs.search('API documentation', {
|
|
311
|
-
path: '/docs' // Search within specific directory
|
|
312
|
-
})
|
|
313
|
-
|
|
314
|
-
// ✅ Connect related files
|
|
315
|
-
await vfs.addRelationship('/src/auth.js', '/tests/auth.test.js', 'tested-by')
|
|
316
|
-
|
|
317
|
-
// Perfect for: File explorers, IDEs, documentation systems, code analysis
|
|
318
243
|
```
|
|
319
244
|
|
|
320
|
-
|
|
321
|
-
- ❌ No infinite recursion in file trees (like brain-cloud team experienced)
|
|
322
|
-
- ❌ No data loss from memory storage
|
|
323
|
-
- ❌ No performance issues with large directories
|
|
324
|
-
- ❌ No need for complex fallback patterns
|
|
245
|
+
**Prevents infinite recursion** that crashes traditional file systems. Tree-aware operations ensure your file explorer never hangs.
|
|
325
246
|
|
|
326
247
|
**[📖 VFS Quick Start →](docs/vfs/QUICK_START.md)** | **[🎯 Common Patterns →](docs/vfs/COMMON_PATTERNS.md)**
|
|
327
248
|
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
### 🚀 **NEW: Enhanced Directory Import with Caching**
|
|
331
|
-
|
|
332
|
-
**Import large projects 10-100x faster with intelligent caching:**
|
|
249
|
+
### 🚀 **Import Anything - CSV, Excel, PDF, URLs**
|
|
333
250
|
|
|
334
251
|
```javascript
|
|
335
|
-
|
|
336
|
-
import
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
const brain = new Brainy()
|
|
340
|
-
await brain.init()
|
|
341
|
-
|
|
342
|
-
// Progress tracking for long operations
|
|
343
|
-
const tracker = ProgressTracker.create(1000)
|
|
344
|
-
tracker.start()
|
|
345
|
-
|
|
346
|
-
for await (const progress of importer.importStream('./project', {
|
|
347
|
-
batchSize: 100,
|
|
348
|
-
generateEmbeddings: true
|
|
349
|
-
})) {
|
|
350
|
-
const p = tracker.update(progress.processed, progress.current)
|
|
351
|
-
console.log(formatProgress(p))
|
|
352
|
-
// [RUNNING] 45% (450/1000) - 23.5 items/s - 23s remaining
|
|
353
|
-
}
|
|
354
|
-
|
|
355
|
-
// Entity extraction with intelligent caching
|
|
356
|
-
const entities = await brain.neural.extractor.extract(text, {
|
|
357
|
-
types: ['person', 'organization', 'technology'],
|
|
358
|
-
confidence: 0.7,
|
|
359
|
-
cache: {
|
|
360
|
-
enabled: true,
|
|
361
|
-
ttl: 7 * 24 * 60 * 60 * 1000, // 7 days
|
|
362
|
-
invalidateOn: 'mtime' // Re-extract when file changes
|
|
363
|
-
}
|
|
364
|
-
})
|
|
252
|
+
// Import CSV with auto-detection
|
|
253
|
+
await brain.import('customers.csv')
|
|
254
|
+
// ✨ Auto-detects: encoding, delimiter, types, creates entities!
|
|
365
255
|
|
|
366
|
-
//
|
|
367
|
-
|
|
368
|
-
|
|
256
|
+
// Import Excel workbooks with multi-sheet support
|
|
257
|
+
await brain.import('sales-data.xlsx', {
|
|
258
|
+
excelSheets: ['Q1', 'Q2']
|
|
369
259
|
})
|
|
370
260
|
|
|
371
|
-
//
|
|
372
|
-
await brain.
|
|
373
|
-
|
|
374
|
-
to: targetId,
|
|
375
|
-
type: 'creates',
|
|
376
|
-
confidence: 0.85,
|
|
377
|
-
evidence: {
|
|
378
|
-
sourceText: 'John created the database',
|
|
379
|
-
method: 'pattern',
|
|
380
|
-
reasoning: 'Matches creation pattern; entities in same sentence'
|
|
381
|
-
}
|
|
261
|
+
// Import PDF documents with table extraction
|
|
262
|
+
await brain.import('research-paper.pdf', {
|
|
263
|
+
pdfExtractTables: true
|
|
382
264
|
})
|
|
383
265
|
|
|
384
|
-
//
|
|
385
|
-
|
|
386
|
-
console.log(`Cache hit rate: ${(stats.hitRate * 100).toFixed(1)}%`)
|
|
387
|
-
// Cache hit rate: 89.5%
|
|
266
|
+
// Import from URLs (auto-fetched)
|
|
267
|
+
await brain.import('https://api.example.com/data.json')
|
|
388
268
|
```
|
|
389
269
|
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
### 🎯 Zero Configuration Philosophy
|
|
270
|
+
**[📖 Complete Import Guide →](docs/guides/import-anything.md)**
|
|
393
271
|
|
|
394
|
-
|
|
272
|
+
### 🧠 **Neural API - Advanced Semantic Analysis**
|
|
395
273
|
|
|
396
274
|
```javascript
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
// 1. Pure zero-config - detects everything
|
|
400
|
-
const brain = new Brainy()
|
|
275
|
+
const neural = brain.neural
|
|
401
276
|
|
|
402
|
-
//
|
|
403
|
-
const
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
277
|
+
// Automatic semantic clustering
|
|
278
|
+
const clusters = await neural.clusters({
|
|
279
|
+
algorithm: 'kmeans',
|
|
280
|
+
maxClusters: 5,
|
|
281
|
+
threshold: 0.8
|
|
407
282
|
})
|
|
408
283
|
|
|
409
|
-
//
|
|
410
|
-
const
|
|
411
|
-
mode: 'production',
|
|
412
|
-
model: 'q8', // Optimized model (99% accuracy, 75% smaller)
|
|
413
|
-
storage: 'cloud', // or 'memory', 'disk', 'auto'
|
|
414
|
-
features: ['core', 'search', 'cache']
|
|
415
|
-
})
|
|
416
|
-
```
|
|
284
|
+
// Calculate similarity between any items
|
|
285
|
+
const similarity = await neural.similar('item1', 'item2')
|
|
417
286
|
|
|
418
|
-
|
|
287
|
+
// Find nearest neighbors
|
|
288
|
+
const neighbors = await neural.neighbors('item-id', 10)
|
|
419
289
|
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
- **Features**: Minimal → Default → Full based on environment
|
|
423
|
-
- **Memory**: Optimal cache sizes and batching
|
|
424
|
-
- **Performance**: Threading, chunking, indexing strategies
|
|
290
|
+
// Detect outliers
|
|
291
|
+
const outliers = await neural.outliers(0.3)
|
|
425
292
|
|
|
426
|
-
|
|
293
|
+
// Generate visualization data for D3/Cytoscape
|
|
294
|
+
const vizData = await neural.visualize({
|
|
295
|
+
maxNodes: 100,
|
|
296
|
+
dimensions: 3,
|
|
297
|
+
algorithm: 'force'
|
|
298
|
+
})
|
|
299
|
+
```
|
|
427
300
|
|
|
428
|
-
|
|
429
|
-
- **24MB memory footprint** - Efficient resource usage
|
|
430
|
-
- **Worker-based embeddings** - Non-blocking operations
|
|
431
|
-
- **Automatic caching** - Intelligent result caching
|
|
301
|
+
---
|
|
432
302
|
|
|
433
|
-
|
|
303
|
+
## 🌐 Framework Integration
|
|
434
304
|
|
|
435
|
-
|
|
305
|
+
**Brainy is framework-first!** Works with **any** modern framework:
|
|
436
306
|
|
|
307
|
+
### ⚛️ React & Next.js
|
|
437
308
|
```javascript
|
|
438
|
-
|
|
439
|
-
const brain = new Brainy()
|
|
440
|
-
|
|
441
|
-
// Storage control (auto-detected by default)
|
|
442
|
-
const diskBrain = new Brainy({storage: 'disk'}) // Local filesystem
|
|
443
|
-
const cloudBrain = new Brainy({storage: 'cloud'}) // S3/GCS/R2
|
|
309
|
+
function SearchComponent() {
|
|
310
|
+
const [brain] = useState(() => new Brainy())
|
|
311
|
+
useEffect(() => { brain.init() }, [])
|
|
444
312
|
|
|
445
|
-
|
|
446
|
-
const
|
|
447
|
-
|
|
448
|
-
}
|
|
313
|
+
const handleSearch = async (query) => {
|
|
314
|
+
const results = await brain.find(query)
|
|
315
|
+
setResults(results)
|
|
316
|
+
}
|
|
317
|
+
}
|
|
449
318
|
```
|
|
450
319
|
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
320
|
+
### 🟢 Vue.js & Nuxt.js
|
|
321
|
+
```javascript
|
|
322
|
+
export default {
|
|
323
|
+
async mounted() {
|
|
324
|
+
this.brain = new Brainy()
|
|
325
|
+
await this.brain.init()
|
|
326
|
+
},
|
|
327
|
+
methods: {
|
|
328
|
+
async search(query) {
|
|
329
|
+
return await this.brain.find(query)
|
|
330
|
+
}
|
|
331
|
+
}
|
|
332
|
+
}
|
|
333
|
+
```
|
|
458
334
|
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
335
|
+
### 🅰️ Angular
|
|
336
|
+
```typescript
|
|
337
|
+
@Injectable({ providedIn: 'root' })
|
|
338
|
+
export class BrainyService {
|
|
339
|
+
private brain = new Brainy()
|
|
340
|
+
async search(query: string) {
|
|
341
|
+
return await this.brain.find(query)
|
|
342
|
+
}
|
|
343
|
+
}
|
|
462
344
|
```
|
|
463
345
|
|
|
464
|
-
|
|
346
|
+
**Works with:** Svelte, Solid.js, Qwik, Fresh, and more! All bundlers (Webpack, Vite, Rollup). SSR/SSG. Edge runtimes.
|
|
465
347
|
|
|
466
|
-
|
|
348
|
+
---
|
|
467
349
|
|
|
468
|
-
|
|
469
|
-
// Import CSV with auto-detection
|
|
470
|
-
await brain.import('customers.csv')
|
|
471
|
-
// ✨ Auto-detects: encoding, delimiter, types, creates entities!
|
|
350
|
+
## 💾 Storage - From Development to Production
|
|
472
351
|
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
// ✨ Processes all sheets, preserves structure, infers types!
|
|
352
|
+
### 🚀 **Development: Just Works**
|
|
353
|
+
```javascript
|
|
354
|
+
const brain = new Brainy() // Memory storage, auto-configured
|
|
355
|
+
```
|
|
478
356
|
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
357
|
+
### ⚡ **Production: FileSystem with Compression**
|
|
358
|
+
```javascript
|
|
359
|
+
const brain = new Brainy({
|
|
360
|
+
storage: {
|
|
361
|
+
type: 'filesystem',
|
|
362
|
+
path: './brainy-data',
|
|
363
|
+
compression: true // 60-80% space savings!
|
|
364
|
+
}
|
|
482
365
|
})
|
|
483
|
-
// ✨ Extracts text, detects tables, preserves metadata!
|
|
484
|
-
|
|
485
|
-
// Import JSON/YAML data
|
|
486
|
-
await brain.import([
|
|
487
|
-
{ name: 'Alice', role: 'Engineer' },
|
|
488
|
-
{ name: 'Bob', role: 'Designer' }
|
|
489
|
-
])
|
|
490
|
-
// ✨ Automatically creates Person entities with relationships!
|
|
491
|
-
|
|
492
|
-
// Import from URLs (auto-fetched)
|
|
493
|
-
await brain.import('https://api.example.com/data.json')
|
|
494
|
-
// ✨ Auto-detects URL, fetches, parses, processes!
|
|
495
366
|
```
|
|
496
367
|
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
## 📚 Core API
|
|
368
|
+
### ☁️ **Production: Cloud Storage** (NEW in v4.0.0)
|
|
500
369
|
|
|
501
|
-
|
|
370
|
+
Choose your cloud provider - all support **automatic cost optimization:**
|
|
502
371
|
|
|
372
|
+
#### **AWS S3 / Cloudflare R2 / DigitalOcean Spaces**
|
|
503
373
|
```javascript
|
|
504
|
-
const
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
374
|
+
const brain = new Brainy({
|
|
375
|
+
storage: {
|
|
376
|
+
type: 's3',
|
|
377
|
+
s3Storage: {
|
|
378
|
+
bucketName: 'my-knowledge-base',
|
|
379
|
+
region: 'us-east-1',
|
|
380
|
+
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
|
381
|
+
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
|
382
|
+
}
|
|
383
|
+
}
|
|
508
384
|
})
|
|
509
|
-
```
|
|
510
385
|
|
|
511
|
-
|
|
386
|
+
// Enable Intelligent-Tiering for automatic 96% cost savings
|
|
387
|
+
await brain.storage.enableIntelligentTiering('entities/', 'brainy-auto-tier')
|
|
388
|
+
```
|
|
512
389
|
|
|
390
|
+
#### **Google Cloud Storage**
|
|
513
391
|
```javascript
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
important: true
|
|
523
|
-
},
|
|
524
|
-
related: {to: "React"} // Graph relationships
|
|
392
|
+
const brain = new Brainy({
|
|
393
|
+
storage: {
|
|
394
|
+
type: 'gcs',
|
|
395
|
+
gcsStorage: {
|
|
396
|
+
bucketName: 'my-knowledge-base',
|
|
397
|
+
keyFilename: '/path/to/service-account.json'
|
|
398
|
+
}
|
|
399
|
+
}
|
|
525
400
|
})
|
|
526
|
-
```
|
|
527
401
|
|
|
528
|
-
|
|
402
|
+
// Enable Autoclass for automatic 94% cost savings
|
|
403
|
+
await brain.storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
|
|
404
|
+
```
|
|
529
405
|
|
|
406
|
+
#### **Azure Blob Storage**
|
|
530
407
|
```javascript
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
408
|
+
const brain = new Brainy({
|
|
409
|
+
storage: {
|
|
410
|
+
type: 'azure',
|
|
411
|
+
azureStorage: {
|
|
412
|
+
containerName: 'knowledge-base',
|
|
413
|
+
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING
|
|
414
|
+
}
|
|
415
|
+
}
|
|
538
416
|
})
|
|
539
417
|
|
|
540
|
-
//
|
|
541
|
-
|
|
542
|
-
|
|
418
|
+
// Batch tier changes for 99% cost savings
|
|
419
|
+
await brain.storage.setBlobTierBatch(
|
|
420
|
+
oldBlobs.map(name => ({ blobName: name, tier: 'Archive' }))
|
|
421
|
+
)
|
|
422
|
+
```
|
|
543
423
|
|
|
544
|
-
|
|
545
|
-
await brain.updateNoun(id, newData, newMetadata)
|
|
546
|
-
await brain.updateVerb(verbId, newMetadata)
|
|
424
|
+
### 💰 **Cost Optimization Impact**
|
|
547
425
|
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
426
|
+
| Scale | Before | After (Archive) | Savings/Year |
|
|
427
|
+
|-------|--------|-----------------|--------------|
|
|
428
|
+
| 5TB | $1,380/year | $59/year | **$1,321 (96%)** |
|
|
429
|
+
| 50TB | $13,800/year | $594/year | **$13,206 (96%)** |
|
|
430
|
+
| 500TB | $138,000/year | $5,940/year | **$132,060 (96%)** |
|
|
551
431
|
|
|
552
|
-
|
|
553
|
-
await brain.import(arrayOfData)
|
|
554
|
-
const exported = await brain.export({format: 'json'})
|
|
432
|
+
**[📖 AWS S3 Cost Guide →](docs/operations/cost-optimization-aws-s3.md)** | **[GCS →](docs/operations/cost-optimization-gcs.md)** | **[Azure →](docs/operations/cost-optimization-azure.md)** | **[R2 →](docs/operations/cost-optimization-cloudflare-r2.md)**
|
|
555
433
|
|
|
556
|
-
|
|
557
|
-
await brain.import('customers.csv') // CSV with encoding detection
|
|
558
|
-
await brain.import('sales-report.xlsx') // Excel with multi-sheet support
|
|
559
|
-
await brain.import('research.pdf') // PDF with table extraction
|
|
560
|
-
```
|
|
434
|
+
---
|
|
561
435
|
|
|
562
|
-
##
|
|
436
|
+
## 🚀 Production Scale Features (NEW in v4.0.0)
|
|
563
437
|
|
|
564
|
-
###
|
|
438
|
+
### 🎯 **Type-Aware HNSW - 87% Memory Reduction**
|
|
439
|
+
|
|
440
|
+
**Billion-scale deployments made affordable:**
|
|
441
|
+
|
|
442
|
+
- **Memory @ 1B entities**: 384GB → 50GB (-87%)
|
|
443
|
+
- **Single-type queries**: 10x faster (search 100M instead of 1B)
|
|
444
|
+
- **Multi-type queries**: 5-8x faster
|
|
445
|
+
- **Optimized rebuilds**: 31x faster with type filtering
|
|
565
446
|
|
|
566
447
|
```javascript
|
|
567
|
-
// Single node (default)
|
|
568
448
|
const brain = new Brainy({
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
bucketName: 'my-data',
|
|
573
|
-
region: 'us-east-1',
|
|
574
|
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
|
575
|
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
|
576
|
-
}
|
|
577
|
-
}
|
|
449
|
+
hnsw: {
|
|
450
|
+
typeAware: true // Enable type-aware indexing
|
|
451
|
+
}
|
|
578
452
|
})
|
|
453
|
+
```
|
|
579
454
|
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
455
|
+
### ⚡ **Production-Ready Storage**
|
|
456
|
+
|
|
457
|
+
**NEW v4.0.0 enterprise features:**
|
|
458
|
+
|
|
459
|
+
- **🗑️ Batch Operations**: Delete thousands of entities with retry logic
|
|
460
|
+
- **📦 Gzip Compression**: 60-80% space savings (FileSystem)
|
|
461
|
+
- **💽 OPFS Quota Monitoring**: Real-time quota tracking (Browser)
|
|
462
|
+
- **🔄 Metadata/Vector Separation**: Billion-entity scalability
|
|
463
|
+
- **🛡️ Enterprise Reliability**: Backpressure, circuit breakers, retries
|
|
464
|
+
|
|
465
|
+
```javascript
|
|
466
|
+
// Batch delete with retry logic
|
|
467
|
+
await brain.storage.batchDelete(keys, {
|
|
468
|
+
maxRetries: 3,
|
|
469
|
+
continueOnError: true
|
|
592
470
|
})
|
|
471
|
+
|
|
472
|
+
// Get storage status
|
|
473
|
+
const status = await brain.storage.getStorageStatus()
|
|
474
|
+
console.log(`Used: ${status.used}, Quota: ${status.quota}`)
|
|
593
475
|
```
|
|
594
476
|
|
|
595
|
-
###
|
|
477
|
+
### 📊 **Adaptive Memory Management**
|
|
596
478
|
|
|
597
|
-
|
|
598
|
-
- **Automatic Sharding**: Data distributed by content hash
|
|
599
|
-
- **Smart Query Planning**: Queries routed to optimal shards
|
|
600
|
-
- **Live Rebalancing**: Handles node joins/leaves automatically
|
|
601
|
-
- **Zero Downtime**: Streaming shard migration
|
|
479
|
+
**Auto-scales from 2GB to 128GB+ based on resources:**
|
|
602
480
|
|
|
603
|
-
|
|
481
|
+
- Container-aware (Docker/K8s cgroups v1/v2)
|
|
482
|
+
- Environment-smart (25% dev, 40% container, 50% production)
|
|
483
|
+
- Model memory accounting (150MB Q8, 250MB FP32)
|
|
484
|
+
- Built-in cache monitoring with recommendations
|
|
604
485
|
|
|
605
486
|
```javascript
|
|
606
|
-
|
|
487
|
+
// Get cache statistics and recommendations
|
|
488
|
+
const stats = brain.getCacheStats()
|
|
489
|
+
console.log(`Hit rate: ${stats.hitRate * 100}%`)
|
|
490
|
+
// Actionable tuning recommendations included
|
|
491
|
+
```
|
|
607
492
|
|
|
608
|
-
|
|
609
|
-
const ingestionNode = new Brainy({
|
|
610
|
-
storage: {
|
|
611
|
-
type: 's3',
|
|
612
|
-
s3Storage: {
|
|
613
|
-
bucketName: 'social-data',
|
|
614
|
-
region: 'us-east-1',
|
|
615
|
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
|
616
|
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
|
617
|
-
}
|
|
618
|
-
},
|
|
619
|
-
distributed: true,
|
|
620
|
-
writeOnly: true // Optimized for high-throughput writes
|
|
621
|
-
})
|
|
493
|
+
**[📖 Capacity Planning Guide →](docs/operations/capacity-planning.md)**
|
|
622
494
|
|
|
623
|
-
|
|
624
|
-
blueskyStream.on('post', async (post) => {
|
|
625
|
-
await ingestionNode.add(post, {
|
|
626
|
-
nounType: NounType.Message,
|
|
627
|
-
platform: 'bluesky',
|
|
628
|
-
author: post.author,
|
|
629
|
-
timestamp: post.createdAt
|
|
630
|
-
})
|
|
631
|
-
})
|
|
495
|
+
---
|
|
632
496
|
|
|
633
|
-
|
|
634
|
-
const searchNode = new Brainy({
|
|
635
|
-
storage: {
|
|
636
|
-
type: 's3',
|
|
637
|
-
s3Storage: {
|
|
638
|
-
bucketName: 'social-data',
|
|
639
|
-
region: 'us-east-1',
|
|
640
|
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
|
641
|
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
|
642
|
-
}
|
|
643
|
-
},
|
|
644
|
-
distributed: true,
|
|
645
|
-
readOnly: true // Optimized for fast queries
|
|
646
|
-
})
|
|
497
|
+
## 🏢 Enterprise Features - No Paywalls
|
|
647
498
|
|
|
648
|
-
|
|
649
|
-
const trending = await searchNode.find('trending AI topics', {
|
|
650
|
-
where: {timestamp: {greaterThan: Date.now() - 3600000}},
|
|
651
|
-
limit: 100
|
|
652
|
-
})
|
|
653
|
-
```
|
|
499
|
+
Brainy includes **enterprise-grade capabilities at no extra cost**:
|
|
654
500
|
|
|
655
|
-
|
|
501
|
+
- ✅ **Scales to billions of entities** with 18ms search latency @ 1B scale
|
|
502
|
+
- ✅ **Distributed architecture** with sharding and replication
|
|
503
|
+
- ✅ **Read/write separation** for horizontal scaling
|
|
504
|
+
- ✅ **Connection pooling** and request deduplication
|
|
505
|
+
- ✅ **Built-in monitoring** with metrics and health checks
|
|
506
|
+
- ✅ **Production ready** with circuit breakers and backpressure
|
|
656
507
|
|
|
657
|
-
|
|
658
|
-
|----------------|----------------------------------|-------------------------------|
|
|
659
|
-
| Setup | Complex (k8s, operators) | One flag: `distributed: true` |
|
|
660
|
-
| Coordination | External (etcd, Consul) | Built-in (via storage) |
|
|
661
|
-
| Minimum Nodes | 3-5 for HA | 1 (scale as needed) |
|
|
662
|
-
| Sharding | Random | Domain-aware |
|
|
663
|
-
| Query Planning | Basic | Triple Intelligence |
|
|
664
|
-
| Cost | High (always-on clusters) | Low (scale to zero) |
|
|
508
|
+
**No premium tiers. No feature gates. Everyone gets the same powerful system.**
|
|
665
509
|
|
|
666
|
-
|
|
510
|
+
---
|
|
667
511
|
|
|
668
|
-
|
|
512
|
+
## 📊 Benchmarks
|
|
669
513
|
|
|
514
|
+
| Operation | Performance | Memory |
|
|
515
|
+
|----------------------------------|-------------|----------|
|
|
516
|
+
| Initialize | 450ms | 24MB |
|
|
517
|
+
| Add Item | 12ms | +0.1MB |
|
|
518
|
+
| Vector Search (1k items) | 3ms | - |
|
|
519
|
+
| Metadata Filter (10k items) | 0.8ms | - |
|
|
520
|
+
| Natural Language Query | 15ms | - |
|
|
521
|
+
| Bulk Import (1000 items) | 2.3s | +8MB |
|
|
522
|
+
| **Production Scale (10M items)** | **5.8ms** | **12GB** |
|
|
523
|
+
| **Billion Scale (1B items)** | **18ms** | **50GB** |
|
|
524
|
+
|
|
525
|
+
---
|
|
526
|
+
|
|
527
|
+
## 🎯 Use Cases
|
|
528
|
+
|
|
529
|
+
### Knowledge Management
|
|
670
530
|
```javascript
|
|
671
|
-
//
|
|
531
|
+
// Create knowledge graph with relationships
|
|
672
532
|
const apiGuide = await brain.add("REST API Guide", {
|
|
673
533
|
nounType: NounType.Document,
|
|
674
|
-
|
|
675
|
-
category: "documentation",
|
|
676
|
-
version: "2.0"
|
|
534
|
+
category: "documentation"
|
|
677
535
|
})
|
|
678
536
|
|
|
679
537
|
const author = await brain.add("Jane Developer", {
|
|
@@ -681,37 +539,17 @@ const author = await brain.add("Jane Developer", {
|
|
|
681
539
|
role: "tech-lead"
|
|
682
540
|
})
|
|
683
541
|
|
|
684
|
-
|
|
685
|
-
nounType: NounType.Project,
|
|
686
|
-
status: "active"
|
|
687
|
-
})
|
|
542
|
+
await brain.relate(author, apiGuide, "authored")
|
|
688
543
|
|
|
689
|
-
//
|
|
690
|
-
await brain.
|
|
691
|
-
|
|
692
|
-
|
|
693
|
-
await brain.relate(apiGuide, project, "documents", {
|
|
694
|
-
coverage: "complete"
|
|
695
|
-
})
|
|
696
|
-
|
|
697
|
-
// Query the knowledge graph naturally
|
|
698
|
-
const docs = await brain.find("documentation authored by tech leads for active projects")
|
|
699
|
-
```
|
|
700
|
-
|
|
701
|
-
### Semantic Search
|
|
702
|
-
|
|
703
|
-
```javascript
|
|
704
|
-
// Find similar content
|
|
705
|
-
const similar = await brain.search(existingContent, {
|
|
706
|
-
limit: 5,
|
|
707
|
-
threshold: 0.8
|
|
708
|
-
})
|
|
544
|
+
// Query naturally
|
|
545
|
+
const docs = await brain.find(
|
|
546
|
+
"documentation authored by tech leads for active projects"
|
|
547
|
+
)
|
|
709
548
|
```
|
|
710
549
|
|
|
711
|
-
### AI Memory Layer
|
|
712
|
-
|
|
550
|
+
### AI Memory Layer
|
|
713
551
|
```javascript
|
|
714
|
-
// Store
|
|
552
|
+
// Store conversation context with relationships
|
|
715
553
|
const userId = await brain.add("User 123", {
|
|
716
554
|
nounType: NounType.User,
|
|
717
555
|
tier: "premium"
|
|
@@ -719,20 +557,12 @@ const userId = await brain.add("User 123", {
|
|
|
719
557
|
|
|
720
558
|
const messageId = await brain.add(userMessage, {
|
|
721
559
|
nounType: NounType.Message,
|
|
722
|
-
timestamp: Date.now()
|
|
723
|
-
session: "abc"
|
|
724
|
-
})
|
|
725
|
-
|
|
726
|
-
const topicId = await brain.add("Product Support", {
|
|
727
|
-
nounType: NounType.Topic,
|
|
728
|
-
category: "support"
|
|
560
|
+
timestamp: Date.now()
|
|
729
561
|
})
|
|
730
562
|
|
|
731
|
-
// Link message elements
|
|
732
563
|
await brain.relate(userId, messageId, "sent")
|
|
733
|
-
await brain.relate(messageId, topicId, "about")
|
|
734
564
|
|
|
735
|
-
// Retrieve context
|
|
565
|
+
// Retrieve context
|
|
736
566
|
const context = await brain.find({
|
|
737
567
|
where: {type: "message"},
|
|
738
568
|
connected: {from: userId, type: "sent"},
|
|
@@ -740,40 +570,20 @@ const context = await brain.find({
|
|
|
740
570
|
})
|
|
741
571
|
```
|
|
742
572
|
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
Brainy supports multiple storage backends:
|
|
746
|
-
|
|
573
|
+
### Semantic Search
|
|
747
574
|
```javascript
|
|
748
|
-
//
|
|
749
|
-
const
|
|
750
|
-
|
|
751
|
-
|
|
752
|
-
path: './data'
|
|
753
|
-
}
|
|
754
|
-
})
|
|
755
|
-
|
|
756
|
-
// Browser Storage (OPFS) - Works with frameworks
|
|
757
|
-
const brain = new Brainy({
|
|
758
|
-
storage: {type: 'opfs'} // Framework handles browser polyfills
|
|
759
|
-
})
|
|
760
|
-
|
|
761
|
-
// S3 Compatible (Production)
|
|
762
|
-
const brain = new Brainy({
|
|
763
|
-
storage: {
|
|
764
|
-
type: 's3',
|
|
765
|
-
bucket: 'my-bucket',
|
|
766
|
-
region: 'us-east-1'
|
|
767
|
-
}
|
|
575
|
+
// Find similar content
|
|
576
|
+
const similar = await brain.search(existingContent, {
|
|
577
|
+
limit: 5,
|
|
578
|
+
threshold: 0.8
|
|
768
579
|
})
|
|
769
580
|
```
|
|
770
581
|
|
|
771
|
-
|
|
582
|
+
---
|
|
772
583
|
|
|
773
|
-
|
|
584
|
+
## 🛠️ CLI
|
|
774
585
|
|
|
775
586
|
```bash
|
|
776
|
-
# Install globally
|
|
777
587
|
npm install -g brainy
|
|
778
588
|
|
|
779
589
|
# Add data
|
|
@@ -787,137 +597,78 @@ brainy find "awesome programming languages"
|
|
|
787
597
|
|
|
788
598
|
# Interactive mode
|
|
789
599
|
brainy chat
|
|
790
|
-
|
|
791
|
-
# Export data
|
|
792
|
-
brainy export --format json > backup.json
|
|
793
|
-
```
|
|
794
|
-
|
|
795
|
-
## 🧠 Neural API - Advanced AI Features
|
|
796
|
-
|
|
797
|
-
Brainy includes a powerful Neural API for advanced semantic analysis:
|
|
798
|
-
|
|
799
|
-
### Clustering & Analysis
|
|
800
|
-
|
|
801
|
-
```javascript
|
|
802
|
-
// Access via brain.neural
|
|
803
|
-
const neural = brain.neural
|
|
804
|
-
|
|
805
|
-
// Automatic semantic clustering
|
|
806
|
-
const clusters = await neural.clusters()
|
|
807
|
-
// Returns groups of semantically similar items
|
|
808
|
-
|
|
809
|
-
// Cluster with options
|
|
810
|
-
const clusters = await neural.clusters({
|
|
811
|
-
algorithm: 'kmeans', // or 'hierarchical', 'sample'
|
|
812
|
-
maxClusters: 5, // Maximum number of clusters
|
|
813
|
-
threshold: 0.8 // Similarity threshold
|
|
814
|
-
})
|
|
815
|
-
|
|
816
|
-
// Calculate similarity between any items
|
|
817
|
-
const similarity = await neural.similar('item1', 'item2')
|
|
818
|
-
// Returns 0-1 score
|
|
819
|
-
|
|
820
|
-
// Find nearest neighbors
|
|
821
|
-
const neighbors = await neural.neighbors('item-id', 10)
|
|
822
|
-
|
|
823
|
-
// Build semantic hierarchy
|
|
824
|
-
const hierarchy = await neural.hierarchy('item-id')
|
|
825
|
-
|
|
826
|
-
// Detect outliers
|
|
827
|
-
const outliers = await neural.outliers(0.3)
|
|
828
|
-
|
|
829
|
-
// Generate visualization data for D3/Cytoscape
|
|
830
|
-
const vizData = await neural.visualize({
|
|
831
|
-
maxNodes: 100,
|
|
832
|
-
dimensions: 3,
|
|
833
|
-
algorithm: 'force'
|
|
834
|
-
})
|
|
835
600
|
```
|
|
836
601
|
|
|
837
|
-
|
|
838
|
-
|
|
839
|
-
```javascript
|
|
840
|
-
// Group customer feedback into themes
|
|
841
|
-
const feedbackClusters = await neural.clusters()
|
|
842
|
-
for (const cluster of feedbackClusters) {
|
|
843
|
-
console.log(`Theme: ${cluster.label}`)
|
|
844
|
-
console.log(`Items: ${cluster.members.length}`)
|
|
845
|
-
}
|
|
602
|
+
---
|
|
846
603
|
|
|
847
|
-
|
|
848
|
-
const docId = await brain.add("Machine learning guide", { nounType: NounType.Document })
|
|
849
|
-
const similar = await neural.neighbors(docId, 5)
|
|
850
|
-
// Returns 5 most similar documents
|
|
604
|
+
## 📖 Documentation
|
|
851
605
|
|
|
852
|
-
|
|
853
|
-
|
|
854
|
-
|
|
855
|
-
|
|
606
|
+
### Getting Started
|
|
607
|
+
- [Getting Started Guide](docs/guides/getting-started.md)
|
|
608
|
+
- [v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md) **← NEW**
|
|
609
|
+
- [AWS S3 Cost Guide](docs/operations/cost-optimization-aws-s3.md) | [GCS](docs/operations/cost-optimization-gcs.md) | [Azure](docs/operations/cost-optimization-azure.md) | [R2](docs/operations/cost-optimization-cloudflare-r2.md) **← NEW**
|
|
856
610
|
|
|
857
|
-
|
|
611
|
+
### Framework Integration
|
|
612
|
+
- [Framework Integration Guide](docs/guides/framework-integration.md)
|
|
613
|
+
- [Next.js Integration](docs/guides/nextjs-integration.md)
|
|
614
|
+
- [Vue.js Integration](docs/guides/vue-integration.md)
|
|
858
615
|
|
|
859
|
-
|
|
616
|
+
### Virtual Filesystem
|
|
617
|
+
- [VFS Core Documentation](docs/vfs/VFS_CORE.md)
|
|
618
|
+
- [Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md)
|
|
619
|
+
- [Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md)
|
|
860
620
|
|
|
861
|
-
|
|
862
|
-
|
|
863
|
-
|
|
621
|
+
### Core Documentation
|
|
622
|
+
- [API Reference](docs/api/README.md)
|
|
623
|
+
- [Architecture Overview](docs/architecture/overview.md)
|
|
624
|
+
- [Data Storage Architecture](docs/architecture/data-storage-architecture.md)
|
|
625
|
+
- [Natural Language Guide](docs/guides/natural-language.md)
|
|
626
|
+
- [Triple Intelligence](docs/architecture/triple-intelligence.md)
|
|
627
|
+
- [Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)
|
|
864
628
|
|
|
865
|
-
|
|
866
|
-
|
|
629
|
+
### Operations & Production
|
|
630
|
+
- [Capacity Planning](docs/operations/capacity-planning.md)
|
|
631
|
+
- [Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md) **← NEW**
|
|
632
|
+
- [AWS S3 Cost Guide](docs/operations/cost-optimization-aws-s3.md) | [GCS](docs/operations/cost-optimization-gcs.md) | [Azure](docs/operations/cost-optimization-azure.md) | [R2](docs/operations/cost-optimization-cloudflare-r2.md) **← NEW**
|
|
867
633
|
|
|
868
|
-
|
|
869
|
-
brainy cloud setup
|
|
870
|
-
```
|
|
871
|
-
|
|
872
|
-
## 🏢 Enterprise Features - Included for Everyone
|
|
634
|
+
---
|
|
873
635
|
|
|
874
|
-
|
|
636
|
+
## 💖 Support Brainy
|
|
875
637
|
|
|
876
|
-
|
|
877
|
-
- **Distributed architecture** with sharding and replication
|
|
878
|
-
- **Read/write separation** for horizontal scaling
|
|
879
|
-
- **Connection pooling** and request deduplication
|
|
880
|
-
- **Built-in monitoring** with metrics and health checks
|
|
881
|
-
- **Production ready** with circuit breakers and backpressure
|
|
638
|
+
Brainy is **free and open source** - no paywalls, no premium tiers, no feature gates. If Brainy helps your project, consider supporting development:
|
|
882
639
|
|
|
883
|
-
|
|
640
|
+
- ⭐ **Star us on [GitHub](https://github.com/soulcraftlabs/brainy)**
|
|
641
|
+
- 💝 **Sponsor via [GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)**
|
|
642
|
+
- 🐛 **Report issues** and contribute code
|
|
643
|
+
- 📣 **Share** with your team and community
|
|
884
644
|
|
|
885
|
-
|
|
645
|
+
Your support keeps Brainy free for everyone and enables continued development of enterprise features at no cost.
|
|
886
646
|
|
|
887
|
-
|
|
888
|
-
|----------------------------------|-------------|----------|
|
|
889
|
-
| Initialize | 450ms | 24MB |
|
|
890
|
-
| Add Item | 12ms | +0.1MB |
|
|
891
|
-
| Vector Search (1k items) | 3ms | - |
|
|
892
|
-
| Metadata Filter (10k items) | 0.8ms | - |
|
|
893
|
-
| Natural Language Query | 15ms | - |
|
|
894
|
-
| Bulk Import (1000 items) | 2.3s | +8MB |
|
|
895
|
-
| **Production Scale (10M items)** | **5.8ms** | **12GB** |
|
|
647
|
+
---
|
|
896
648
|
|
|
897
|
-
##
|
|
649
|
+
## 📋 System Requirements
|
|
898
650
|
|
|
899
|
-
|
|
651
|
+
**Node.js Version:** 22 LTS (recommended)
|
|
900
652
|
|
|
901
|
-
-
|
|
902
|
-
-
|
|
903
|
-
-
|
|
904
|
-
- Distributed architecture support
|
|
653
|
+
- ✅ **Node.js 22 LTS** - Fully supported (recommended for production)
|
|
654
|
+
- ✅ **Node.js 20 LTS** - Compatible (maintenance mode)
|
|
655
|
+
- ❌ **Node.js 24** - Not supported (ONNX compatibility issues)
|
|
905
656
|
|
|
906
|
-
|
|
657
|
+
If using nvm: `nvm use` (we provide a `.nvmrc` file)
|
|
907
658
|
|
|
908
|
-
|
|
659
|
+
---
|
|
909
660
|
|
|
910
661
|
## 🧠 The Knowledge Operating System Explained
|
|
911
662
|
|
|
912
663
|
### How We Achieved The Impossible
|
|
913
664
|
|
|
914
|
-
**Triple Intelligence™**
|
|
665
|
+
**Triple Intelligence™** unifies three database paradigms that were previously incompatible:
|
|
915
666
|
|
|
916
667
|
1. **Vector databases** (Pinecone, Weaviate) - semantic similarity
|
|
917
668
|
2. **Graph databases** (Neo4j, ArangoDB) - relationships
|
|
918
669
|
3. **Document databases** (MongoDB, Elasticsearch) - metadata filtering
|
|
919
670
|
|
|
920
|
-
**
|
|
671
|
+
**Others make you choose. Brainy does all three together.**
|
|
921
672
|
|
|
922
673
|
### The Math of Infinite Expressiveness
|
|
923
674
|
|
|
@@ -945,40 +696,26 @@ We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
|
|
945
696
|
|
|
946
697
|
[→ See the Mathematical Proof & Full Taxonomy](docs/architecture/noun-verb-taxonomy.md)
|
|
947
698
|
|
|
948
|
-
|
|
949
|
-
|
|
950
|
-
### Framework Integration
|
|
951
|
-
- [Framework Integration Guide](docs/guides/framework-integration.md) - Complete framework setup guide
|
|
952
|
-
- [Next.js Integration](docs/guides/nextjs-integration.md) - React and Next.js examples
|
|
953
|
-
- [Vue.js Integration](docs/guides/vue-integration.md) - Vue and Nuxt examples
|
|
954
|
-
|
|
955
|
-
### Virtual Filesystem (Semantic VFS) 🧠📁
|
|
956
|
-
- [VFS Core Documentation](docs/vfs/VFS_CORE.md) - Complete filesystem architecture and API
|
|
957
|
-
- [Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md) - Multi-dimensional file access (6 semantic dimensions)
|
|
958
|
-
- [Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md) - AI-powered concept and entity extraction
|
|
959
|
-
- [Examples & Scenarios](docs/vfs/VFS_EXAMPLES_SCENARIOS.md) - Real-world use cases and code
|
|
960
|
-
- [VFS API Guide](docs/vfs/VFS_API_GUIDE.md) - Complete API reference
|
|
961
|
-
|
|
962
|
-
### Core Documentation
|
|
963
|
-
- [Getting Started Guide](docs/guides/getting-started.md)
|
|
964
|
-
- [API Reference](docs/api/README.md)
|
|
965
|
-
- [Architecture Overview](docs/architecture/overview.md)
|
|
966
|
-
- [Data Storage Architecture](docs/architecture/data-storage-architecture.md) - Deep dive into storage, indexing, and sharding
|
|
967
|
-
- [Natural Language Guide](docs/guides/natural-language.md)
|
|
968
|
-
- [Triple Intelligence](docs/architecture/triple-intelligence.md)
|
|
969
|
-
- [Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)
|
|
699
|
+
---
|
|
970
700
|
|
|
971
701
|
## 🏢 Enterprise & Cloud
|
|
972
702
|
|
|
973
703
|
**Brain Cloud** - Managed Brainy with team sync, persistent memory, and enterprise connectors.
|
|
974
704
|
|
|
975
705
|
```bash
|
|
976
|
-
# Get started with free trial
|
|
977
706
|
brainy cloud setup
|
|
978
707
|
```
|
|
979
708
|
|
|
980
709
|
Visit [soulcraft.com](https://soulcraft.com) for more information.
|
|
981
710
|
|
|
711
|
+
---
|
|
712
|
+
|
|
713
|
+
## 🤝 Contributing
|
|
714
|
+
|
|
715
|
+
We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
|
716
|
+
|
|
717
|
+
---
|
|
718
|
+
|
|
982
719
|
## 📄 License
|
|
983
720
|
|
|
984
721
|
MIT © Brainy Contributors
|
|
@@ -987,5 +724,6 @@ MIT © Brainy Contributors
|
|
|
987
724
|
|
|
988
725
|
<p align="center">
|
|
989
726
|
<strong>Built with ❤️ by the Brainy community</strong><br>
|
|
990
|
-
<em>Zero-Configuration AI Database with Triple Intelligence™</em>
|
|
991
|
-
</
|
|
727
|
+
<em>Zero-Configuration AI Database with Triple Intelligence™</em><br>
|
|
728
|
+
<em>v4.0.0 - Production-Scale Storage with 99% Cost Savings</em>
|
|
729
|
+
</p>
|