@soulcraft/cor 3.0.0-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +16 -0
- package/README.md +328 -0
- package/dist/aggregation/NativeAggregationEngine.d.ts +118 -0
- package/dist/aggregation/NativeAggregationEngine.js +186 -0
- package/dist/cli.d.ts +12 -0
- package/dist/cli.js +236 -0
- package/dist/graph/GraphAccelerationAdapter.d.ts +153 -0
- package/dist/graph/GraphAccelerationAdapter.js +326 -0
- package/dist/graph/NativeGraphAdjacencyIndex.d.ts +286 -0
- package/dist/graph/NativeGraphAdjacencyIndex.js +944 -0
- package/dist/hnsw/AdaptiveDiskAnnModeSelector.d.ts +187 -0
- package/dist/hnsw/AdaptiveDiskAnnModeSelector.js +313 -0
- package/dist/hnsw/NativeDiskAnnWrapper.d.ts +489 -0
- package/dist/hnsw/NativeDiskAnnWrapper.js +1456 -0
- package/dist/index.d.ts +28 -0
- package/dist/index.js +29 -0
- package/dist/legacyLayoutGuard.d.ts +93 -0
- package/dist/legacyLayoutGuard.js +237 -0
- package/dist/license/constants.d.ts +30 -0
- package/dist/license/constants.js +30 -0
- package/dist/license/onlineValidator.d.ts +43 -0
- package/dist/license/onlineValidator.js +225 -0
- package/dist/license/types.d.ts +39 -0
- package/dist/license/types.js +5 -0
- package/dist/license.d.ts +64 -0
- package/dist/license.js +165 -0
- package/dist/migration/MigrationCoordinator.d.ts +171 -0
- package/dist/migration/MigrationCoordinator.js +249 -0
- package/dist/migration/indexEpochGuard.d.ts +163 -0
- package/dist/migration/indexEpochGuard.js +202 -0
- package/dist/native/NativeEmbeddingEngine.d.ts +79 -0
- package/dist/native/NativeEmbeddingEngine.js +318 -0
- package/dist/native/NativeRoaringBitmap32.d.ts +114 -0
- package/dist/native/NativeRoaringBitmap32.js +221 -0
- package/dist/native/ffi.d.ts +20 -0
- package/dist/native/ffi.js +48 -0
- package/dist/native/idMapperTestSupport.d.ts +67 -0
- package/dist/native/idMapperTestSupport.js +112 -0
- package/dist/native/index.d.ts +49 -0
- package/dist/native/index.js +87 -0
- package/dist/native/napi.d.ts +21 -0
- package/dist/native/napi.js +88 -0
- package/dist/native/types.d.ts +1298 -0
- package/dist/native/types.js +16 -0
- package/dist/plugin.d.ts +50 -0
- package/dist/plugin.js +388 -0
- package/dist/providerContracts.d.ts +277 -0
- package/dist/providerContracts.js +38 -0
- package/dist/resource/OsMemoryProbe.d.ts +175 -0
- package/dist/resource/OsMemoryProbe.js +206 -0
- package/dist/resource/ResourceManager.d.ts +491 -0
- package/dist/resource/ResourceManager.js +960 -0
- package/dist/utils/ColumnManifest.d.ts +97 -0
- package/dist/utils/ColumnManifest.js +129 -0
- package/dist/utils/NativeColumnStore.d.ts +284 -0
- package/dist/utils/NativeColumnStore.js +685 -0
- package/dist/utils/NativeMetadataIndex.d.ts +882 -0
- package/dist/utils/NativeMetadataIndex.js +2631 -0
- package/dist/utils/NativeUnifiedCache.d.ts +87 -0
- package/dist/utils/NativeUnifiedCache.js +273 -0
- package/dist/utils/binaryIdMapperFactory.d.ts +59 -0
- package/dist/utils/binaryIdMapperFactory.js +94 -0
- package/dist/utils/collation.d.ts +30 -0
- package/dist/utils/collation.js +40 -0
- package/dist/utils/columnStoreTypes.d.ts +161 -0
- package/dist/utils/columnStoreTypes.js +29 -0
- package/dist/utils/nativeBinaryEntityIdMapper.d.ts +211 -0
- package/dist/utils/nativeBinaryEntityIdMapper.js +381 -0
- package/dist/utils/nativeEntityIdMapper.d.ts +111 -0
- package/dist/utils/nativeEntityIdMapper.js +170 -0
- package/docs/ADR-002-diskann-100-percent-rust.md +294 -0
- package/docs/ADR-003-semantic-time-travel.md +100 -0
- package/docs/aggregation.md +251 -0
- package/docs/comparison.md +162 -0
- package/docs/deployment-limits.md +87 -0
- package/docs/diskann.md +184 -0
- package/docs/migration-3.0.md +396 -0
- package/docs/performance-budget.md +274 -0
- package/docs/performance.md +117 -0
- package/docs/scaling.md +439 -0
- package/docs/snapshot-safety.md +246 -0
- package/docs/u64-id-space.md +214 -0
- package/docs/verification-report.md +670 -0
- package/native/brainy-native.node +0 -0
- package/native/index.d.ts +3916 -0
- package/package.json +92 -0
- package/scripts/migrate-cortex-2x-to-3x.mjs +970 -0
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: How Cor compares
|
|
3
|
+
slug: cor/comparison
|
|
4
|
+
public: true
|
|
5
|
+
category: cor
|
|
6
|
+
template: guide
|
|
7
|
+
order: 2
|
|
8
|
+
description: Side-by-side. Cor 3.0 is the only system that composes metadata + graph + vector with time-travel in one query; index-level latency runs 13–18× pgvector at small-to-medium scale on the reference host (recall-matched, real-data library A/B in progress).
|
|
9
|
+
next:
|
|
10
|
+
- cor/diskann
|
|
11
|
+
- cor/performance
|
|
12
|
+
- guides/plugins
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# How Cor compares
|
|
16
|
+
|
|
17
|
+
> **vs pgvector HNSW** — index-level p50/p99 **latency**, same host
|
|
18
|
+
> (bxl9000: Ryzen 9 7950X3D / NVMe), same dataset + query workload:
|
|
19
|
+
> - 10 K: cor p50 0.048 ms vs pgvector p50 0.865 ms → **18.02×**
|
|
20
|
+
> - 100 K: cor p50 0.041 ms vs pgvector p50 0.536 ms → **13.07×**
|
|
21
|
+
> - 1 M: cor p50 0.148 ms vs pgvector p50 1.081 ms → **7.30×**
|
|
22
|
+
>
|
|
23
|
+
> Cor stays sub-millisecond at p99 even at 1 M (0.309 ms p99 cor vs 1.604 ms p99 pgvector); pgvector crosses 1 ms p99 at this scale.
|
|
24
|
+
>
|
|
25
|
+
> **Read this as latency, not a blanket "faster":** a latency ratio is only
|
|
26
|
+
> meaningful when both engines sit at the same recall. The recall-matched
|
|
27
|
+
> comparison on real-manifold data (SIFT/BIGANN) is being measured in the
|
|
28
|
+
> library A/B; treat these as cor's index-level latency headroom on this
|
|
29
|
+
> reference host, not an end-to-end product claim.
|
|
30
|
+
>
|
|
31
|
+
> Source: `scripts/verify-vs-pgvector.mjs` · result JSON in `docs/verification/`
|
|
32
|
+
|
|
33
|
+
> **Cor's acceleration of Brainy (isolated-operation micro-benchmarks):**
|
|
34
|
+
> 5.2× geometric mean across 15 *isolated* operations on bxl9000
|
|
35
|
+
> (Embedding 2.8× · Infrastructure 7.8× · Search 25.2× · Data ops 7.3×).
|
|
36
|
+
> This is per-operation acceleration measured in isolation — **not** an
|
|
37
|
+
> end-to-end `find()` speedup. The end-to-end library A/B (brainy-alone vs
|
|
38
|
+
> brainy+cor through the public `find()` path, recall@10 on SIFT) is in
|
|
39
|
+
> progress; that table will be the headline product number.
|
|
40
|
+
|
|
41
|
+
## Where cor genuinely leads
|
|
42
|
+
|
|
43
|
+
Most vector databases can do vector ANN. A few can filter by metadata. Almost
|
|
44
|
+
none can hop graph edges. **None** can do all three in a single sub-
|
|
45
|
+
millisecond query AND support point-in-time reads. Cor is the only
|
|
46
|
+
system in that category:
|
|
47
|
+
|
|
48
|
+
| | Vector ANN | Metadata filter | Graph hop | Time-travel | Triple find() composition |
|
|
49
|
+
|---|:---:|:---:|:---:|:---:|:---:|
|
|
50
|
+
| **Cor 3.0** | ✅ | ✅ | ✅ | ✅ | ✅ |
|
|
51
|
+
| pgvector | ✅ | ✅ (SQL) | partial (SQL JOIN) | ❌ | partial |
|
|
52
|
+
| Qdrant | ✅ | ✅ | ❌ | ❌ | ❌ |
|
|
53
|
+
| Milvus | ✅ | ✅ | ❌ | ❌ | ❌ |
|
|
54
|
+
| Weaviate | ✅ | ✅ | weak | ❌ | ❌ |
|
|
55
|
+
| Pinecone | ✅ | ✅ | ❌ | ❌ | ❌ |
|
|
56
|
+
| FAISS | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
57
|
+
|
|
58
|
+
**Triple-find composition at 1 M — per-stage substrate latency:** LSM filter
|
|
59
|
+
16 µs + graph BFS 16 µs + DiskANN 194 µs + JS-side intersection, composing to
|
|
60
|
+
**p50 ≈ 0.230 ms** on bxl9000. Source: `scripts/verify-triple-find.mjs`.
|
|
61
|
+
|
|
62
|
+
> **Honesty note (being re-validated):** this composition was measured on a
|
|
63
|
+
> *synthetic* corpus, where random vectors are near-orthogonal under cosine
|
|
64
|
+
> (concentration of measure) so the vector leg's recall is ~0 and the composed
|
|
65
|
+
> result set is near-empty. The **per-stage latencies are valid** (each
|
|
66
|
+
> substrate did its work), but the *end-to-end* triple-find latency **and
|
|
67
|
+
> recall** are being re-measured on real-manifold data (SIFT/BIGANN). Treat
|
|
68
|
+
> 0.230 ms as the substrate-composition ceiling, not the validated end-to-end
|
|
69
|
+
> number. Brainy 8.0 also pushes the intersection into native primitives.
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
## At a Glance
|
|
74
|
+
|
|
75
|
+
| Category | Avg Speedup | What it covers |
|
|
76
|
+
|:---------|:------------|:---------------|
|
|
77
|
+
| **Embedding** | **2.8x** | Real ML model inference (all-MiniLM-L6-v2) |
|
|
78
|
+
| **Data Operations** | **7.3x** | Adding, reading, and deleting entities |
|
|
79
|
+
| **Search** | **25.2x** | Finding entities by meaning, filters, and similarity |
|
|
80
|
+
| **Graph** | **1.3x** | Querying relationships between entities |
|
|
81
|
+
| **Neural** | **1.9x** | AI-powered neighbors and clustering |
|
|
82
|
+
|
|
83
|
+
## Scale Ceiling
|
|
84
|
+
|
|
85
|
+
The numbers above measure speedup at the 200-entity tier — the "make my queries faster" story. Cor also moves the **scale ceiling itself**: with DiskANN engaged, a single Brainy instance reaches workloads that were previously single-machine impossible.
|
|
86
|
+
|
|
87
|
+
> **PROJECTED — design targets, not measured.** The table below is extrapolated
|
|
88
|
+
> from algorithm math (Vamana / PQ memory + latency models). Measured numbers
|
|
89
|
+
> at 100M and 1B on real hardware ship in `docs/verification-report.md` as
|
|
90
|
+
> part of Piece 9 of the cor 3.0 release. Per `CLAUDE.md`, perf claims
|
|
91
|
+
> without a `MEASURED` citation must carry a `PROJECTED` label until verified.
|
|
92
|
+
|
|
93
|
+
| Workload | Brainy alone | Brainy + Cor (HNSW) | Brainy + Cor (DiskANN) |
|
|
94
|
+
|:---|:---|:---|:---|
|
|
95
|
+
| 1 M vectors @ 384-dim | OK, ~1 GB RAM | OK, ~1 GB RAM, 25× faster search | OK, ~0.5 GB RAM, sub-ms search |
|
|
96
|
+
| 10 M vectors | OK, ~15 GB RAM, slow build | OK, ~10 GB RAM | OK, ~3 GB RAM, 1–3 ms search |
|
|
97
|
+
| 100 M vectors | Out of practical reach | OK if you have 100+ GB RAM | OK on a 32 GB box, 2–5 ms search |
|
|
98
|
+
| **1 B vectors** | **Not possible** (1.5 TB RAM) | **Not possible** (1.5+ TB RAM) | **OK on a 64 GB box, 5–10 ms search** |
|
|
99
|
+
|
|
100
|
+
DiskANN auto-engages when storage is local and a stable idMapper is available — see [docs/diskann.md](./diskann.md). Cloud-storage adapters stay on HNSW transparently.
|
|
101
|
+
|
|
102
|
+
> Embedding consistency: max element-wise diff = 1.43e-7, avg = 2.64e-8 (WASM and native produce equivalent vectors)
|
|
103
|
+
|
|
104
|
+
> **Same results, just faster.** Cor accelerates Brainy without changing any
|
|
105
|
+
> answer: native output matches the JavaScript baseline byte-for-byte, enforced by
|
|
106
|
+
> a 104-test cross-language parity suite (tokenization, value normalization,
|
|
107
|
+
> string collation, SQ8 distance, top-K ranking, roaring/msgpack). See
|
|
108
|
+
> [Cor Performance → Cross-Language Consistency](./performance.md).
|
|
109
|
+
|
|
110
|
+
## Detailed Results
|
|
111
|
+
|
|
112
|
+
### Embedding
|
|
113
|
+
|
|
114
|
+
| What it does | WASM | Native Rust | Speedup |
|
|
115
|
+
|:-------------|-----:|------------:|:--------|
|
|
116
|
+
| Engine initialization | 192 ms | 115 ms | **1.7x** |
|
|
117
|
+
| Embed one sentence | 116 ms | 47 ms | **2.4x** |
|
|
118
|
+
| Embed 1 sentence (batch path) | 116 ms | 55 ms | **2.1x** |
|
|
119
|
+
| Embed 10 sentences | 1173 ms | 330 ms | **3.6x** |
|
|
120
|
+
| Embed 20 sentences | 2371 ms | 611 ms | **3.9x** |
|
|
121
|
+
| Embed 50 sentences | 6050 ms | 1577 ms | **3.8x** |
|
|
122
|
+
|
|
123
|
+
### Data Operations
|
|
124
|
+
|
|
125
|
+
| What it does | Brainy | + Cor | Speedup |
|
|
126
|
+
|:-------------|-------:|---------:|:--------|
|
|
127
|
+
| Store one entity | 21 ms | 3.0 ms | **7.0x** |
|
|
128
|
+
| Store 20 entities at once | 1384 ms | 110 ms | **12.6x** |
|
|
129
|
+
| Retrieve one entity | 0.001 ms | 0.001 ms | **1.3x** |
|
|
130
|
+
| Remove and re-insert | 113 ms | 4.5 ms | **25.0x** |
|
|
131
|
+
|
|
132
|
+
### Search
|
|
133
|
+
|
|
134
|
+
| What it does | Brainy | + Cor | Speedup |
|
|
135
|
+
|:-------------|-------:|---------:|:--------|
|
|
136
|
+
| Search by meaning | 21 ms | 0.4 ms | **50.8x** |
|
|
137
|
+
| Search by fields | 1.9 ms | 0.018 ms | **108.3x** |
|
|
138
|
+
| Find similar items | 0.3 ms | 0.1 ms | **2.9x** |
|
|
139
|
+
|
|
140
|
+
### Graph
|
|
141
|
+
|
|
142
|
+
| What it does | Brainy | + Cor | Speedup |
|
|
143
|
+
|:-------------|-------:|---------:|:--------|
|
|
144
|
+
| Query relationships | 0.011 ms | 0.008 ms | **1.3x** |
|
|
145
|
+
|
|
146
|
+
### Neural
|
|
147
|
+
|
|
148
|
+
| What it does | Brainy | + Cor | Speedup |
|
|
149
|
+
|:-------------|-------:|---------:|:--------|
|
|
150
|
+
| Nearest neighbors | 0.001 ms | 0.001 ms | **1.9x** |
|
|
151
|
+
| Auto-cluster items | — | — | skipped |
|
|
152
|
+
|
|
153
|
+
### Embedding Throughput (batch of 50)
|
|
154
|
+
|
|
155
|
+
| Engine | Texts/sec | ms/text |
|
|
156
|
+
|:-------|----------:|--------:|
|
|
157
|
+
| WASM | 8 | 121.0 |
|
|
158
|
+
| Native Rust | 32 | 31.5 |
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
*Methodology: Infrastructure operations measured 20 times after 3 warmup runs; embedding operations measured 10 times after 2 warmup runs. Reported value is the median. Infrastructure benchmarks use a lightweight hash-based embedding to isolate acceleration from model differences. Embedding benchmarks use the real all-MiniLM-L6-v2 model (384-dim). "Speedup" is the geometric mean of per-operation ratios within each category.*
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Deployment — OS limits for multi-tenant pools
|
|
3
|
+
slug: cor/deployment-limits
|
|
4
|
+
public: true
|
|
5
|
+
category: cor
|
|
6
|
+
template: guide
|
|
7
|
+
order: 6
|
|
8
|
+
description: The two OS limits a per-tenant brain pool needs (open files + mmap regions), what cor sets automatically, what the deployment must set, and copy-paste Docker / Compose / Kubernetes / host config.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Deployment — OS limits for multi-tenant pools
|
|
12
|
+
|
|
13
|
+
Running **one brain per user** (e.g. Workshop) puts many isolated brains in one process. Each is lean — a few open files + a few mmap regions — but they add up. Two OS limits can bottleneck a pool **before RAM does**:
|
|
14
|
+
|
|
15
|
+
| Limit | What it caps | Default (Linux) | Who sets it |
|
|
16
|
+
|---|---|---|---|
|
|
17
|
+
| `RLIMIT_NOFILE` (`ulimit -n`) | open files / process | soft **1024**, hard higher | **soft: cor (auto)** · hard: deployment |
|
|
18
|
+
| `vm.max_map_count` | mmap regions / process (Linux-only) | **65530** | **deployment only** (global sysctl) |
|
|
19
|
+
|
|
20
|
+
A single big brain (e.g. Venue) doesn't need these raised. They matter at **pool scale** (hundreds–thousands of brains in one process).
|
|
21
|
+
|
|
22
|
+
## What cor does automatically
|
|
23
|
+
|
|
24
|
+
- **Raises its own soft `nofile` limit toward the hard cap on startup** — no privilege needed, safe, zero-config (`native/src/resource_limits.rs`). So you only ever need to manage the **hard** cap.
|
|
25
|
+
- **Warns once at pool scale** if the hard cap or `vm.max_map_count` is below the projected need, with the exact fix command (so a missing setting is never silent). Single-brain deployments stay quiet.
|
|
26
|
+
- **`getOpenFileLimitInfo()`** exposes the live numbers for health checks / dashboards.
|
|
27
|
+
|
|
28
|
+
A process **cannot** raise its hard `nofile` cap or `vm.max_map_count` — those are the two things the deployment must set.
|
|
29
|
+
|
|
30
|
+
## Recommended values
|
|
31
|
+
|
|
32
|
+
Per brain, post-lazy-shard: ~10–16 open files + ~24 mmap regions (with margin). For headroom to thousands of pooled brains:
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
ulimit -n = 65536 (hard; cor raises soft to match)
|
|
36
|
+
vm.max_map_count = 262144
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## Copy-paste config
|
|
40
|
+
|
|
41
|
+
**Docker (Workshop runs on Docker/GCE):**
|
|
42
|
+
```bash
|
|
43
|
+
docker run \
|
|
44
|
+
--ulimit nofile=65536:65536 \
|
|
45
|
+
--sysctl vm.max_map_count=262144 \
|
|
46
|
+
…your-image
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
**Docker Compose:**
|
|
50
|
+
```yaml
|
|
51
|
+
services:
|
|
52
|
+
workshop:
|
|
53
|
+
ulimits:
|
|
54
|
+
nofile: { soft: 65536, hard: 65536 }
|
|
55
|
+
sysctls:
|
|
56
|
+
vm.max_map_count: 262144
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
**Kubernetes** (`vm.max_map_count` is a *non-namespaced* sysctl on most clusters → set it on the node, e.g. via a privileged initContainer or node tuning; `nofile` via the runtime):
|
|
60
|
+
```yaml
|
|
61
|
+
# Node-level (initContainer or DaemonSet):
|
|
62
|
+
# sysctl -w vm.max_map_count=262144
|
|
63
|
+
# Pod:
|
|
64
|
+
spec:
|
|
65
|
+
containers:
|
|
66
|
+
- name: workshop
|
|
67
|
+
# container runtime nofile via securityContext / runtime config
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
**Bare GCE / host (persistent):**
|
|
71
|
+
```bash
|
|
72
|
+
# /etc/sysctl.d/99-cor.conf
|
|
73
|
+
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-cor.conf
|
|
74
|
+
sudo sysctl --system
|
|
75
|
+
# /etc/security/limits.d/99-cor.conf
|
|
76
|
+
echo '* hard nofile 65536' | sudo tee /etc/security/limits.d/99-cor.conf
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## Verify
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
ulimit -Hn # hard open-file cap
|
|
83
|
+
cat /proc/sys/vm/max_map_count # mmap region cap
|
|
84
|
+
```
|
|
85
|
+
Cor logs both at pool scale if they're low. If you see the warning, apply the config above and restart.
|
|
86
|
+
|
|
87
|
+
> **Note:** `vm.max_map_count` is Linux-only. On a macOS dev laptop it doesn't exist (and rarely matters at dev scale); `ulimit -n` applies on both. Production (Linux/Docker/GCE) needs both.
|
package/docs/diskann.md
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Adaptive DiskANN — From Laptop to Billion-Scale
|
|
3
|
+
slug: cor/diskann
|
|
4
|
+
public: true
|
|
5
|
+
category: cor
|
|
6
|
+
template: guide
|
|
7
|
+
order: 3
|
|
8
|
+
description: Cor's 100% pure-Rust Adaptive DiskANN. Vamana α-pruned graph + Product Quantization. MEASURED at 1 M (p50 0.85 ms, p99 5.23 ms) on bxl9000; PROJECTED to 1 B at 5–10 ms via mode-adaptive residency. Auto-engages on local storage; full migration path from existing HNSW.
|
|
9
|
+
next:
|
|
10
|
+
- cor/scaling
|
|
11
|
+
- cor/performance
|
|
12
|
+
- cor/comparison
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Adaptive DiskANN — From Laptop to Billion-Scale
|
|
16
|
+
|
|
17
|
+
Cor ships a **100% pure-Rust Adaptive DiskANN** implementation that takes a single Brainy instance from the ~10 M-vector HNSW comfort zone to **1 B+ vectors on commodity hardware**. MEASURED at 1 M: **p50 0.85 ms / p99 5.23 ms** on bxl9000 (mode=auto, in-memory). 1 B target is PROJECTED at ~5–10 ms via Mode 3 (on-disk) — the DiskANN paper's published range. It's an algorithmic alternative to HNSW, not a competitor service — it lives in the same plugin, behind the same Brainy APIs.
|
|
18
|
+
|
|
19
|
+
**The "Adaptive" part:** the same on-disk index file silently changes residency mode based on dataset size + available RAM:
|
|
20
|
+
|
|
21
|
+
| Mode | When auto-selects | What changes |
|
|
22
|
+
|---|---|---|
|
|
23
|
+
| **Mode 1 — in-memory** | Vectors × dim × 4 fits in RAM (typical up through 10 M at 128-dim, 1 M at 1536-dim) | Zero PQ compression. Full Vamana graph + vectors mmap'd into RAM. **Fastest** — no disk I/O on hot path. |
|
|
24
|
+
| **Mode 2 — hybrid** | RAM-constrained but PQ centroids fit (100 M tier on commodity boxes) | PQ-compressed centroids in RAM (16–32× smaller); full vectors mmap'd from disk with page-cache promotion. ~2-3× slower than Mode 1. |
|
|
25
|
+
| **Mode 3 — on-disk** | Billion-scale where even uncompressed vectors don't fit on commodity hardware | Maximally PQ-compressed; search walks disk via OS page cache + DiskANN's neighbour-prefetch pattern. Published 5 ms range from the paper. |
|
|
26
|
+
|
|
27
|
+
You build the index once with cor 3.0 defaults; as your dataset grows over time, the same file changes its residency mode under the hood. No re-index, no API change, no perf cliff at boundaries.
|
|
28
|
+
|
|
29
|
+
## The problem DiskANN solves
|
|
30
|
+
|
|
31
|
+
HNSW is excellent up to roughly 10 M vectors per machine. Beyond that, two costs compound:
|
|
32
|
+
|
|
33
|
+
1. **Memory pressure.** At 1 B vectors of 384-dim float32, the vectors alone are ~1.5 TB and the HNSW graph metadata adds another ~2 TB. Even with the cortex 2.4.0 mmap vector backend, the graph traversal pattern faults pages with no spatial locality.
|
|
34
|
+
2. **No locality.** HNSW's insertion order has no correlation with traversal order on disk, so every search hop on a cold cache is a fresh ~10 μs page fault.
|
|
35
|
+
|
|
36
|
+
DiskANN (Subramanya et al., NeurIPS 2019) was designed for exactly this regime:
|
|
37
|
+
|
|
38
|
+
- **Vamana α-pruned graph** picks neighbours so that nodes visited together during search end up adjacent on disk — disk locality emerges from construction.
|
|
39
|
+
- **Product Quantization** compresses each vector to M ≤ 16 bytes resident in RAM. At 1 B vectors that's ~16 GB of PQ codes instead of 1.5 TB of full vectors.
|
|
40
|
+
- **Full vectors on disk** are only touched to re-rank the top candidate set, not during the graph walk.
|
|
41
|
+
|
|
42
|
+
## What cor delivers
|
|
43
|
+
|
|
44
|
+
- **One contiguous file** (header + codebook + PQ codes + Vamana graph + full vectors), mmap-mappable so an SSD-resident billion-vector dataset never has to be copied into RAM.
|
|
45
|
+
- **Zero-copy section accessors** (`vectors_f32()` returns `&[f32]` directly into the mapped region).
|
|
46
|
+
- **Parallel build** with rayon. The graph and PQ encoding both parallelize.
|
|
47
|
+
- **File-backed build adjacency** for billion-scale construction (the in-RAM concurrent adjacency would consume ~64 GB of bookkeeping at 1 B nodes; the mmap variant uses atomic-u32 slots with sharded write locks).
|
|
48
|
+
- **Connectivity-repair pass** that guarantees every node is reachable from the entry point. Sequential Vamana provides this via insertion order; parallel Vamana doesn't, so cor closes the gap explicitly.
|
|
49
|
+
- **PQ-walk + full-vector re-rank** search. Greedy walk uses ADC distance over RAM-resident PQ codes (M-byte table lookups, ~50 ns per hop); re-rank then scores the **full walked candidate list** (the ~`l_search` visited set) with the exact full-precision distance — the mmap-resident vectors make that near-free, and it recovers true neighbours that PQ ranked below `k`. `l_search` is the recall dial; an optional `paddingFactor` can cap the re-rank to `ceil(k × paddingFactor)` to bound I/O on cold-NVMe billion-scale indexes.
|
|
50
|
+
- **HNSW-shaped TS wrapper** (`NativeDiskAnnWrapper`) so Brainy's higher layers don't know which engine is underneath.
|
|
51
|
+
|
|
52
|
+
## Scale envelope
|
|
53
|
+
|
|
54
|
+
> **MEASURED at 1 M on bxl9000** (Ryzen 9 7950X3D / 184 GB / NVMe), reproducible via `scripts/verify-diskann.mjs`. SIFT1M canonical: **p50 0.86 ms / p99 1.22 ms, recall 0.9942** on the BIGANN reference dataset. 10 M / 100 M numbers below are partial-measured (SIFT10M canonical at p50 0.78 ms / p99 1.13 ms, hybrid mode). 1 B numbers are PROJECTED from the SIFT1B reference run + Subramanya et al. algorithm model.
|
|
55
|
+
|
|
56
|
+
| Vectors | RAM with DiskANN | RAM with HNSW | Mode | DiskANN search latency |
|
|
57
|
+
|--------:|---:|---:|---:|---:|
|
|
58
|
+
| 1 M | 0.5–2 GB | 0.5–2 GB | Mode 1 | **p50 0.85 ms / p99 5.23 ms (MEASURED, mode=auto)** |
|
|
59
|
+
| 10 M | 1–5 GB | 8–20 GB | Mode 1 | **p50 0.78 ms / p99 1.13 ms (MEASURED on SIFT10M, hybrid)** |
|
|
60
|
+
| 100 M | 5–20 GB | 80–200 GB (impractical on single machine) | Mode 2 | 2–5 ms (PROJECTED from SIFT trends) |
|
|
61
|
+
| 1 B | 20–70 GB | 1.5+ TB (single-machine impossible) | Mode 3 | 5–10 ms (PROJECTED from SIFT1B trends) |
|
|
62
|
+
|
|
63
|
+
**End-to-end query latency at 1 B includes filesystem hydration of the returned entities.** Design target is ~100–500 ms total (dominated by the FileSystemStorage random reads + metadata lookup, PROJECTED). The roadmap to bring end-to-end query latency down to match search latency is in [`docs/scaling.md`](./scaling.md#operational-ceiling-at-1-b) — all fixes ship inside cor, no external storage.
|
|
64
|
+
|
|
65
|
+
## How it engages
|
|
66
|
+
|
|
67
|
+
Cor registers a `'diskann'` provider; Brainy's `createIndex()` consults it at init:
|
|
68
|
+
|
|
69
|
+
1. **Explicit opt-in** via `config.index.type: 'diskann'` — required if the engagement conditions aren't satisfied, otherwise throws.
|
|
70
|
+
2. **Auto-engagement** when all of:
|
|
71
|
+
- The cor DiskANN provider is registered (you've loaded the plugin).
|
|
72
|
+
- The storage adapter exposes a local filesystem path (`getBinaryBlobPath('_diskann/main')`). Cloud-storage adapters return `null` here and stay on HNSW.
|
|
73
|
+
- The metadata index has a stable `idMapper` (cortex 2.4.0's stable EntityIdMapper).
|
|
74
|
+
3. **Explicit opt-out** via `config.index.type: 'hnsw'` keeps the historical in-memory index for the rare workload where you want it.
|
|
75
|
+
|
|
76
|
+
```typescript
|
|
77
|
+
import { BrainyData } from '@soulcraft/brainy'
|
|
78
|
+
import { register as registerCor } from '@soulcraft/cor'
|
|
79
|
+
|
|
80
|
+
const brain = new BrainyData({
|
|
81
|
+
storage: { type: 'filesystem', path: '/data/idx' }
|
|
82
|
+
})
|
|
83
|
+
await registerCor(brain)
|
|
84
|
+
await brain.init()
|
|
85
|
+
// → [brainy] DiskANN engaged (path=/data/idx/_diskann/main.bin, dim=384)
|
|
86
|
+
|
|
87
|
+
const hits = await brain.search(queryVector, 10)
|
|
88
|
+
// MEASURED at 1 M: p50 0.85 ms / p99 5.23 ms (mode=auto on bxl9000)
|
|
89
|
+
// PROJECTED at 1 B: ~5–10 ms depending on cache state and SSD
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
All Brainy APIs — `add`, `search`, `relate`, `searchSimilarVerbs`, `find` — work unchanged.
|
|
93
|
+
|
|
94
|
+
## Migrating an existing index
|
|
95
|
+
|
|
96
|
+
Existing HNSW-backed Brainy installs **do not auto-migrate** on upgrade. They keep working as-is. To convert:
|
|
97
|
+
|
|
98
|
+
```typescript
|
|
99
|
+
const result = await brain.migrateToDiskAnn({
|
|
100
|
+
recallTarget: 0.95, // require ≥95% recall vs old index before swapping
|
|
101
|
+
paddingFactor: 1.2, // search-time over-fetch for re-rank
|
|
102
|
+
verifySampleSize: 100 // sample queries for the recall check
|
|
103
|
+
})
|
|
104
|
+
// 1. Builds new index in parallel (old HNSW keeps serving)
|
|
105
|
+
// 2. Samples queries — compares top-k results between old and new
|
|
106
|
+
// 3. Aborts if recall < target; old index stays in place
|
|
107
|
+
// 4. Atomically swaps if recall passes
|
|
108
|
+
|
|
109
|
+
// Reversible:
|
|
110
|
+
await brain.migrateToHnsw()
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Reversibility is a contract — production rollbacks are always available.
|
|
114
|
+
|
|
115
|
+
## Tuning
|
|
116
|
+
|
|
117
|
+
Defaults match the published DiskANN paper and work well for sentence/image embedding workloads (dim 128–1024). Tune via `config.index.diskann`:
|
|
118
|
+
|
|
119
|
+
```typescript
|
|
120
|
+
config.index.diskann = {
|
|
121
|
+
pqM: 16, // PQ subspaces; dim must be divisible by m
|
|
122
|
+
pqKsub: 256, // centroids per subspace (8-bit codes — standard)
|
|
123
|
+
maxDegree: 64, // Vamana R (out-degree per node)
|
|
124
|
+
searchListSize: 100, // Vamana L (build-time candidate set)
|
|
125
|
+
alpha: 1.2, // α-pruning density factor
|
|
126
|
+
useMmapAdjacency: true, // file-backed build adjacency — REQUIRED at >100M nodes
|
|
127
|
+
mmapAdjacencyPath: '/data/scratch/diskann-build.adj'
|
|
128
|
+
}
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
**At 1 B nodes you must set `useMmapAdjacency: true`.** The in-RAM concurrent adjacency would consume ~64 GB of bookkeeping at that scale — the mmap variant uses atomic-u32 slots with sharded write locks and bounded RAM (a few hundred mutexes for the shard table).
|
|
132
|
+
|
|
133
|
+
## Dynamic writes
|
|
134
|
+
|
|
135
|
+
DiskANN graphs are build-once by design. The cor wrapper handles dynamic writes via a delta buffer pattern that mirrors FreshDiskANN (Singh et al., 2021):
|
|
136
|
+
|
|
137
|
+
- `addItem` → appends to an in-memory delta map.
|
|
138
|
+
- `search` → queries the main index AND brute-forces the delta, merges, returns top-k.
|
|
139
|
+
- `removeItem` → tombstone bitmap, filtered out at search time.
|
|
140
|
+
- `rebuild()` → folds the delta into a new main index, swaps atomically.
|
|
141
|
+
|
|
142
|
+
Operationally: insertions are O(1), reads stay sub-ms while delta fits in cache, and you schedule `rebuild()` during off-peak windows. The delta brute-force scales linearly in delta size; you keep it small.
|
|
143
|
+
|
|
144
|
+
## On-disk format
|
|
145
|
+
|
|
146
|
+
Single contiguous `main.bin`, all little-endian:
|
|
147
|
+
|
|
148
|
+
```
|
|
149
|
+
+--------------------------------------------------------------+
|
|
150
|
+
| Header (4 KB, page-aligned) |
|
|
151
|
+
| magic="DKAN" · version · dim · node_count |
|
|
152
|
+
| pq_m · pq_ksub · pq_dsub · max_degree · entry_point |
|
|
153
|
+
+--------------------------------------------------------------+
|
|
154
|
+
| PQ codebook (m × ksub × dsub × f32) |
|
|
155
|
+
+--------------------------------------------------------------+
|
|
156
|
+
| PQ codes (node_count × m bytes) |
|
|
157
|
+
+--------------------------------------------------------------+
|
|
158
|
+
| Vamana graph (node_count × max_degree × u32) |
|
|
159
|
+
| Fixed-degree CSR. Sentinel `u32::MAX` marks unused slots. |
|
|
160
|
+
+--------------------------------------------------------------+
|
|
161
|
+
| Full vectors (node_count × dim × f32) |
|
|
162
|
+
+--------------------------------------------------------------+
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
Fixed-degree adjacency means neighbour-offset math is O(1) — at search time `graph[node]` is a single seek to `graph_offset + node * max_degree * 4`. The mmap base is page-aligned by the OS and section offsets are 4-byte aligned by construction, so `bytemuck::cast_slice` reinterprets section bytes as `&[f32]` / `&[u32]` without copying.
|
|
166
|
+
|
|
167
|
+
The same layout is what the build writes and the searcher mmaps — no separate serialization step.
|
|
168
|
+
|
|
169
|
+
## Why 100% Rust, no C++ FFI
|
|
170
|
+
|
|
171
|
+
Cor re-implements Vamana from the published paper rather than wrapping Microsoft's C++ reference. Reasons:
|
|
172
|
+
|
|
173
|
+
1. **Cross-platform builds** for Node native modules become operationally expensive with C++ (Linux/macOS/Windows × x64/arm64 binaries, headers, link-time gotchas). napi-rs gives mature cross-platform binary distribution.
|
|
174
|
+
2. **License posture** stays clean — pure Rust port from a published algorithm + permissive Rust deps (`memmap2`, `bytemuck`, `rayon`, `rand`, `thiserror`). No patent grant ambiguity.
|
|
175
|
+
3. **Full control over the on-disk format** + napi bindings + future cor-specific optimizations.
|
|
176
|
+
|
|
177
|
+
The pure-Rust DiskANN crate (`native/diskann/`) compiles and tests independently of napi, so it's separately benchmarkable and fuzz-target-ready.
|
|
178
|
+
|
|
179
|
+
## See also
|
|
180
|
+
|
|
181
|
+
- [ADR-002](./ADR-002-diskann.md) — the architectural decision record with full design rationale
|
|
182
|
+
- [Scaling & Resource Management](./scaling.md) — multi-tenant density + the 1 B operational ceiling
|
|
183
|
+
- [Performance benchmarks](./performance.md) — measured numbers from the included benchmark suite
|
|
184
|
+
- [Brainy vs Brainy + Cor comparison](./comparison.md) — side-by-side feature and speed comparison
|