@soulcraft/cortex 1.3.1 → 1.4.1

package/README.md

@@ -1,97 +1,121 @@
-# @soulcraft/cortex
+# Cortex
 
-Native Rust acceleration for [Brainy](https://github.com/soulcraftlabs/brainy). Drop-in performance upgrade — zero configuration required.
+<p align="center">
+  <img src="https://raw.githubusercontent.com/soulcraftlabs/brainy/main/brainy.png" alt="Brainy Logo" width="200">
+</p>
 
-## Pricing
+<p align="center">
+  <strong>Native Rust acceleration for <a href="https://github.com/soulcraftlabs/brainy">Brainy</a></strong>
+</p>
 
-Cortex is a commercial add-on for Brainy. A license key is required after the 14-day trial.
+<p align="center">
+  <a href="https://www.npmjs.com/package/@soulcraft/cortex"><img src="https://badge.fury.io/js/%40soulcraft%2Fcortex.svg" alt="npm version"></a>
+  <a href="https://soulcraft.com/product"><img src="https://img.shields.io/badge/info-soulcraft.com-blue.svg" alt="Product Info"></a>
+</p>
 
-| Plan | Price | For |
-|------|-------|-----|
-| **Standard** | $199/year | Individuals and small teams |
-| **Enterprise** | $999/year | Companies, priority support |
+---
 
-**[Purchase a license at soulcraft.com/pricing](https://soulcraft.com/pricing?focus=pro)**
+## What is Cortex?
 
-**14-day free trial** — starts automatically on first use, no signup required.
+Cortex replaces Brainy's JavaScript internals with native Rust implementations. Same API, same data, dramatically faster.
 
-## Installation
+Install it, and Brainy automatically uses Rust for distance calculations, embeddings, metadata queries, graph operations, and more. No code changes needed.
 
 ```bash
 npm install @soulcraft/cortex
 ```
 
-That's it. Brainy auto-detects cortex during `init()` and uses native implementations where available.
+```typescript
+import Brainy from '@soulcraft/brainy'
 
-## License Setup
+const brain = new Brainy()
+await brain.init()
+// Cortex is auto-detected. That's it.
 
-After purchasing, set your license key via environment variable or file:
+// Verify what's accelerated
+const diag = brain.diagnostics()
+console.log(diag.providers)
+// { distance: { source: 'plugin' }, embeddings: { source: 'plugin' }, ... }
+```
 
-```bash
-# Option 1: Environment variable
-export SOULCRAFT_LICENSE=sc_cortex_<your-key>
+## Why Cortex?
 
-# Option 2: License file
-echo "sc_cortex_<your-key>" > ~/.soulcraft/license
-```
+Brainy works great on its own with pure JavaScript and WASM. Cortex is for when you need more:
+
+- **SIMD distance calculations** — every vector comparison gets faster
+- **Native ML inference** — Candle engine replaces WASM embeddings, with CPU, CUDA, and Metal support
+- **Rust metadata engine** — queries and mutations run in Rust with CRoaring bitmaps
+- **Native graph index** — 4 LSM-trees with verb tracking, all in Rust
+- **Batch embeddings** — single Rust forward pass for bulk operations
+- **Zero-copy storage** — mmap-backed SSTables for disk-heavy workloads
 
-License validation is offline-only — no network calls, no telemetry.
+Brainy's plugin system means cortex acceleration flows through every operation — `add()`, `find()`, `update()`, `delete()`, `similar()`, VFS path resolution, neural APIs, and everything in between.
 
 ## What Gets Accelerated
 
-| Subsystem | Native Implementation | Impact |
-|-----------|----------------------|--------|
-| Distance functions | SIMD cosine/euclidean/manhattan/dot product | Every vector search uses native distance |
-| Vector quantization | SIMD SQ8/SQ4 distance on compressed vectors | 4-8x memory reduction |
-| Product quantization | Native PQ codebook training and ADC lookup | 16-32x compression for large datasets |
-| Graph compression | Delta-varint encoded connection lists | 2-3x smaller graph storage |
-| Metadata index | Full Rust query/mutation engine with bitmap operations | Faster metadata filtering and mutations |
-| Graph adjacency | 4 LSM-trees with verb tracking in Rust | Faster relationship queries |
-| Embeddings | Candle ML (CPU, CUDA, Metal) | Native inference, no WASM compilation |
-| Roaring bitmaps | CRoaring bindings (binary-compatible) | Faster set operations on entity IDs |
-| Msgpack | Native encode/decode | Faster serialization |
-| Storage | MmapFileSystemStorage with zero-copy SSTables | Faster disk I/O for graph data |
+| Subsystem | What Cortex Provides | Impact |
+|-----------|---------------------|--------|
+| **Distance** | SIMD cosine, euclidean, manhattan, dot product | Every vector comparison |
+| **Embeddings** | Candle ML engine (CPU/CUDA/Metal) | No WASM compilation, GPU support |
+| **Batch embeddings** | Single forward pass for multiple texts | Bulk import and reindex |
+| **Metadata index** | Rust query/mutation engine with CRoaring | Faster filtering and type queries |
+| **Graph adjacency** | 4 LSM-trees with verb tracking | Faster relationship operations |
+| **Entity ID mapper** | Rust O(1) HashMap for UUID-integer mapping | Bitmap index operations |
+| **Vector quantization** | SIMD SQ8/SQ4 distance on compressed vectors | 4-8x memory reduction |
+| **Product quantization** | Native PQ codebook training and ADC | 16-32x compression |
+| **Graph compression** | Delta-varint encoded connections | 2-3x smaller graph storage |
+| **Roaring bitmaps** | CRoaring bindings | Faster set operations |
+| **Msgpack** | Native encode/decode | Faster serialization |
+| **Storage** | MmapFileSystemStorage with zero-copy SSTables | Faster disk I/O |
+
+## Getting Started
+
+### 1. Install
 
-## Platform Support
+```bash
+npm install @soulcraft/brainy @soulcraft/cortex
+```
 
-| Platform | Architecture | Status |
-|----------|-------------|--------|
-| Linux | x64 (glibc) | Supported |
-| Linux | arm64 (glibc) | Supported |
-| macOS | arm64 (Apple Silicon) | Supported |
-| macOS | x64 (Intel) | Supported |
-| Windows | x64 | Supported |
+### 2. Set up your license
+
+```bash
+# Environment variable
+export SOULCRAFT_LICENSE=sc_cortex_<your-key>
 
-## Usage
+# Or license file
+echo "sc_cortex_<your-key>" > ~/.soulcraft/license
+```
+
+License validation is offline — no network calls, no telemetry.
+
+### 3. Use Brainy as normal
 
 ```typescript
 import Brainy from '@soulcraft/brainy'
 
-// cortex is auto-detected — no import or config needed
 const brain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './data' } })
 await brain.init()
-
-// Check if native acceleration is active
-console.log(brain.getActivePlugins())
-// → ['@soulcraft/cortex']
+// [brainy] Plugin activated: @soulcraft/cortex
+// [brainy] Providers: 10/10 native (@soulcraft/cortex)
 ```
 
-### Manual Registration
+### 4. Verify in production
 
-If auto-detection doesn't suit your setup, register the plugin manually:
+Use `requireProviders()` to fail fast if cortex isn't providing expected acceleration:
 
 ```typescript
-import Brainy from '@soulcraft/brainy'
-import cortexPlugin from '@soulcraft/cortex'
+brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex'])
+```
 
-const brain = new Brainy({ storage: { type: 'memory' } })
-brain.use(cortexPlugin)
-await brain.init()
+Or inspect the full picture with diagnostics:
+
+```typescript
+console.log(brain.diagnostics())
 ```
 
-### MmapFileSystemStorage
+## MmapFileSystemStorage
 
-When cortex is installed, a new storage adapter becomes available:
+Cortex includes a storage adapter with mmap-backed binary blob support:
 
 ```typescript
 const brain = new Brainy({
@@ -99,17 +123,26 @@ const brain = new Brainy({
 })
 ```
 
-This extends the standard FileSystemStorage with binary blob support for zero-copy mmap access to LSM-tree SSTables.
+Zero-copy access to LSM-tree SSTables — useful for large graph datasets where disk I/O is the bottleneck.
+
+## Platform Support
+
+| Platform | Architecture | Status |
+|----------|-------------|--------|
+| Linux | x64 (glibc) | Supported |
+| Linux | arm64 (glibc) | Supported |
+| macOS | arm64 (Apple Silicon) | Supported |
+| macOS | x64 (Intel) | Supported |
+| Windows | x64 | Supported |
 
 ## Requirements
 
 - `@soulcraft/brainy` >= 7.0.0
 - Node.js >= 22 or Bun >= 1.3.0
-- Platform: Linux/macOS/Windows (see table above)
 
-## Direct Access
+## Direct Native Access
 
-For advanced use cases, you can access native bindings directly:
+For advanced use cases, access the Rust bindings directly:
 
 ```typescript
 import { loadNativeModule, isNativeAvailable } from '@soulcraft/cortex'
@@ -120,6 +153,13 @@ if (isNativeAvailable()) {
 }
 ```
 
+## Learn More
+
+- **[Product info and licensing](https://soulcraft.com/product)**
+- **[Brainy documentation](https://soulcraft.com/docs)**
+- **[Brainy on GitHub](https://github.com/soulcraftlabs/brainy)**
+- **[Plugin development guide](https://github.com/soulcraftlabs/brainy/blob/main/docs/PLUGINS.md)**
+
 ## License
 
 Commercial. See [LICENSE](./LICENSE) for details.

package/dist/hnsw/NativeHNSWWrapper.d.ts

@@ -0,0 +1,115 @@
+/**
+ * NativeHNSWWrapper — Thin TS wrapper around the Rust HNSW graph engine
+ *
+ * Bridges brainy's HNSWIndex API to the native Rust NativeHNSWIndex.
+ * The Rust engine handles all graph operations (insert, search, remove) in-memory
+ * while this wrapper handles async storage I/O for persistence.
+ *
+ * Key design:
+ * - Uses addItemFull() for single-FFI-call inserts (returns node + neighbors + system)
+ * - Persistence modes: 'immediate' (safe) or 'deferred' (30-50x faster for cloud)
+ * - COW via native fork() (Arc-based copy-on-write in Rust)
+ * - rebuild() restores pre-computed graph from storage (O(N) vs O(N log N) rebuild)
+ */
+import type { HNSWConfig, HNSWNoun, Vector, VectorDocument, DistanceFunction, StorageAdapter } from '@soulcraft/brainy';
+export declare class NativeHNSWWrapper {
+    private native;
+    private config;
+    private distanceFunction;
+    private storage;
+    private persistMode;
+    private dirtyNodes;
+    private dirtySystem;
+    private useParallelization;
+    private unifiedCache;
+    private cowEnabled;
+    constructor(config: (Partial<HNSWConfig> & {
+        distanceFunction?: DistanceFunction;
+    }) | undefined, distanceFunction: DistanceFunction, options?: {
+        storage?: StorageAdapter | null;
+        persistMode?: 'immediate' | 'deferred';
+    });
+    addItem(item: VectorDocument): Promise<string>;
+    removeItem(id: string): Promise<boolean>;
+    search(queryVector: Vector, k?: number, filter?: (id: string) => Promise<boolean>, options?: {
+        rerank?: {
+            multiplier: number;
+        };
+        candidateIds?: string[];
+    }): Promise<Array<[string, number]>>;
+    flush(): Promise<number>;
+    getDirtyNodeCount(): number;
+    getPersistMode(): 'immediate' | 'deferred';
+    rebuild(options?: {
+        lazy?: boolean;
+        batchSize?: number;
+        onProgress?: (loaded: number, total: number) => void;
+    }): Promise<void>;
+    enableCOW(parent: NativeHNSWWrapper): void;
+    setUseParallelization(useParallelization: boolean): void;
+    getUseParallelization(): boolean;
+    size(): number;
+    clear(): void;
+    getEntryPointId(): string | null;
+    getMaxLevel(): number;
+    getDimension(): number | null;
+    getConfig(): HNSWConfig;
+    getDistanceFunction(): DistanceFunction;
+    /**
+     * Get all nouns — builds HNSWNoun objects from native data.
+     * @deprecated Use getNounsPaginated() instead
+     */
+    getNouns(): Map<string, HNSWNoun>;
+    getNounsPaginated(options?: {
+        offset?: number;
+        limit?: number;
+        filter?: (noun: HNSWNoun) => boolean;
+    }): {
+        items: Map<string, HNSWNoun>;
+        totalCount: number;
+        hasMore: boolean;
+    };
+    getNodesAtLevel(level: number): HNSWNoun[];
+    getLevelStats(): Array<{
+        level: number;
+        nodeCount: number;
+        avgConnections: number;
+    }>;
+    getIndexHealth(): {
+        averageConnections: number;
+        layerDistribution: number[];
+        maxLayer: number;
+        totalNodes: number;
+    };
+    getCacheStats(): {
+        cachingStrategy: 'preloaded' | 'on-demand';
+        autoDetection: {
+            entityCount: number;
+            estimatedVectorMemoryMB: number;
+            availableCacheMB: number;
+            threshold: number;
+            rationale: string;
+        };
+        unifiedCache: {
+            totalSize: number;
+            maxSize: number;
+            utilizationPercent: number;
+            itemCount: number;
+            hitRatePercent: number;
+            totalAccessCount: number;
+        };
+        hnswCache: {
+            vectorsInCache: number;
+            cacheKeyPrefix: string;
+            estimatedMemoryMB: number;
+        };
+        fairness: {
+            hnswAccessCount: number;
+            hnswAccessPercent: number;
+            totalAccessCount: number;
+            fairnessViolation: boolean;
+        };
+        recommendations: string[];
+    };
+}
+//# sourceMappingURL=NativeHNSWWrapper.d.ts.map