@soulcraft/cor 3.0.0-rc.1

package/docs/performance-budget.md

@@ -0,0 +1,274 @@
+---
+title: Performance Budget
+slug: cor/performance-budget
+public: true
+category: cor
+template: concept
+order: 11
+description: The latency budget cor commits to for find() queries at 1M, 10M, and 100M scale, enforced by CI regression gates and grounded in the triple-intelligence workload that defines the brainy + cor stack.
+next:
+  - cor/snapshot-safety
+  - cor/verification-report
+---
+
+# Performance Budget
+
+This document is a contract. The latency targets below are
+CI-enforced regression gates, not aspirations. Code that fails a
+gate does not ship.
+
+## Why a budget matters
+
+Brainy + cor is the only database that answers a single query
+combining:
+
+- **Metadata filter** — structured `where` over indexed fields
+  (LSM metadata index with Roaring bitmap evaluation)
+- **Graph traversal** — multi-hop relationship walks with verb-type
+  selectivity (graph adjacency LSM with verb endpoint lookups)
+- **Vector similarity** — semantic search over the entire corpus
+  (Adaptive DiskANN: Mode 1 for in-memory recall, hybrid for
+  PQ-compressed billion-scale)
+
+No pure vector database (Pinecone, Weaviate, Qdrant), pure graph
+database (Neo4j, Memgraph), or document database (MongoDB,
+Elasticsearch) does this. PostgreSQL with pgvector gets two of
+three with a multi-second join cost. The combined query is the
+differentiator and it is only differentiating if it is sub-
+millisecond at billion scale.
+
+The budget locks in that property as a release criterion.
+
+## The budget
+
+All measurements are p50 / p99 over 100 queries on a single 32-
+core box (workstation tier: i9-14900HX / 64 GB / NVMe Gen 4;
+billion-tier: bxl9000 / Ryzen 9 7950X3D / 184 GB / NVMe). Queries
+are drawn from a representative workload mix described in
+*Methodology* below.
+
+### Read path budget
+
+| Query shape | 1 M entities | 10 M | 100 M |
+|---|---:|---:|---:|
+| Filter-only `find({where})` p50 | ≤ 0.5 ms | ≤ 1 ms | ≤ 3 ms |
+| Filter-only p99 | ≤ 2 ms | ≤ 5 ms | ≤ 15 ms |
+| Vector-only `find({vector})` p50 | ≤ 1 ms | ≤ 2 ms | ≤ 5 ms |
+| Vector-only p99 | ≤ 3 ms | ≤ 6 ms | ≤ 15 ms |
+| Hybrid `find({where, vector})` p50 | ≤ 2 ms | ≤ 4 ms | ≤ 10 ms |
+| Hybrid p99 | ≤ 8 ms | ≤ 15 ms | ≤ 30 ms |
+| 1-hop `find({where, hop})` p50 | ≤ 1.5 ms | ≤ 3 ms | ≤ 8 ms |
+| 2-hop `find({where, hop, hop})` p50 | ≤ 3 ms | ≤ 6 ms | ≤ 15 ms |
+| 2-hop p99 | ≤ 12 ms | ≤ 25 ms | ≤ 50 ms |
+| Hybrid + 2-hop p50 | ≤ 5 ms | ≤ 10 ms | ≤ 25 ms |
+
+### Cross-subsystem ID resolution budget
+
+`find()` returns UUID strings to JS, so every result row pays the
+int→UUID resolution. Batched APIs are mandatory; per-call APIs are
+not part of the hot path.
+
+| Operation | Budget |
+|---|---:|
+| `getOrAssignBatch(uuids, len=1K)` | ≤ 100 μs |
+| `intsToUuidsBatch(ints, len=1K)` | ≤ 80 μs |
+| `intsToUuidsBatch(ints, len=10K)` | ≤ 500 μs |
+| `getNeighborsBatch(int_ids, len=1K)` | ≤ 150 μs |
+| `getVerbsBatch(verb_ids, len=1K)` | ≤ 120 μs |
+
+### Write path budget
+
+| Operation | 1 M | 10 M | 100 M |
+|---|---:|---:|---:|
+| Single `add(noun)` p50 | ≤ 0.3 ms | ≤ 0.3 ms | ≤ 0.5 ms |
+| Single `add(verb)` p50 | ≤ 0.2 ms | ≤ 0.2 ms | ≤ 0.3 ms |
+| Batched `add` (1K) p50 | ≤ 20 ms | ≤ 20 ms | ≤ 30 ms |
+| Sustained insert throughput (32-core box) | ≥ 50 K/s | ≥ 30 K/s | ≥ 20 K/s |
+
+### Recall floor
+
+Recall is a correctness property, but it has a budget too: a query
+that returns the wrong answer in 0.5 ms is worse than one that
+takes 1 ms and returns the right one.
+
+| Workload | Mode | Recall@10 floor |
+|---|---|---:|
+| Production embeddings (dim ≥ 384, clustered) | Mode 1 in-memory | ≥ 0.97 |
+| Production embeddings | Mode 2 hybrid (PQ16) | ≥ 0.95 |
+| SIFT canonical (dim 128, BIGANN ground truth) | Mode 1 in-memory | ≥ 0.98 |
+| SIFT canonical | Mode 2 hybrid | ≥ 0.55 (PQ16 is adversarial here — operators tune via `recall` preset) |
+
+## How the budget is enforced
+
+Every gate in the table corresponds to a regression test in
+`src/native/findPerfGates.test.ts` (to be added; tracked under
+the Cor 3.0 release plan). The test pattern follows the
+existing `lsmPerfGates.test.ts`:
+
+1. Build a corpus at the target scale with a representative
+   field distribution (50 fields, mixed cardinality, ~10 verbs
+   per noun, dim 384 vectors with clustered structure).
+2. Warm the page cache with one pass over the query set.
+3. Run 100 queries per shape, record p50 and p99.
+4. Assert p50 ≤ budget × 1.0 and p99 ≤ budget × 1.2 (20 %
+   headroom on p99 absorbs CI variance).
+
+A red gate is a release blocker. The test prints the measured
+distribution so the regression is bisectable to a commit.
+
+## Methodology
+
+### Workload mix
+
+The representative workload is a 70 / 20 / 10 split:
+
+- 70 % filter-driven queries (`where` with 1-3 predicates,
+  selectivity from 0.1 % to 10 %)
+- 20 % vector-driven queries (vector search with optional
+  metadata pre-filter)
+- 10 % graph-driven queries (1- or 2-hop walks with verb-type
+  selectivity)
+
+The mix tracks the access pattern Soulcraft Memory observes in
+production: most queries are structured browse / filter; vector
+search is the discovery path; graph hops are the relationship
+expansion.
+
+### Hardware floor
+
+The budget is anchored on the cor 3.0 design floor — a 32-
+thread Intel Core i9-14900HX with 64 GB DDR5 and NVMe Gen 4 SSD.
+That's roughly a $2K commodity workstation; production targets
+sit on top of it. The bxl9000 measurements (Ryzen 9 7950X3D, 184
+GB) are the billion-tier ceiling. Numbers from cloud VMs are
+typically 1.3–1.8× slower for the same work because of memory
+bandwidth and NVMe queue depth.
+
+### What "p50" and "p99" mean here
+
+- **p50** is the median measurement over 100 query repetitions
+  with a warm page cache. It represents the steady-state cost.
+- **p99** is the 99th percentile and represents the worst typical
+  query — page-cache miss, garbage collection pause, or a
+  background msync.
+- We do not publish a p99.9 budget. p99.9 measurements are
+  dominated by external noise (kernel scheduling, NVMe firmware
+  latency spikes) and are not reproducible across runs.
+
+## The hot paths inside `find()`
+
+A unified `find({where, vector, hop, orderBy, limit, offset})`
+query executes against three cor subsystems in sequence. Each
+subsystem has a sub-budget that rolls into the totals above.
+
+1. **Filter pushdown** — `metadataIndex.getIdsForFilter(field,
+   value)` returns a Roaring bitmap of matching entity ints. AND
+   / OR / NOT combinators are bitmap operations. Sub-budget:
+   ≤ 0.3 ms for a 3-predicate AND at 100 M.
+2. **Vector search** — `NativeDiskAnnWrapper.searchWithFilter`
+   walks the Vamana graph, prefilters candidates against the
+   metadata bitmap, scores by distance. Sub-budget: ≤ 1 ms at
+   1 M (Mode 1), ≤ 4 ms at 100 M (hybrid).
+3. **Graph hop expansion** — `graphIndex.getNeighbors(int_id,
+   verb_type)` returns a Roaring bitmap of neighbor ints. Sub-
+   budget: ≤ 0.3 ms per hop at 100 M.
+4. **Cross-subsystem ID resolution** — `intsToUuidsBatch` over
+   the final result set. Sub-budget: ≤ 0.1 ms per 1 K results.
+
+The sub-budgets sum to within the top-level budget when the
+query plan is correct. They blow the budget when the plan is
+wrong (e.g. vector search runs over the whole corpus when a
+1 %-selective filter could prune it first). The query planner
+described in *Investments* below is what keeps the plan correct.
+
+## The cor 3.0 performance investments
+
+The shadow-page rework (G4.1, G4.4, G4.2 redo per the snapshot
+safety contract) sets the substrate. The following investments
+build on it to reach the budget at billion scale:
+
+1. **Cross-subsystem query planner.** Today `find()` runs filter
+   → vector → graph in a fixed order. The planner uses
+   `metadataIndex.getCountForCriteria` for cardinality estimates
+   and picks the lowest-cost order per query (e.g. a 0.01 %-
+   selective filter goes first, prunes 99.99 % of vector work).
+   PostgreSQL does this. Pure vector DBs don't.
+
+2. **Batched cross-subsystem APIs.** Every napi boundary call
+   pays ~5 μs of overhead. Resolving 10 K UUIDs one-at-a-time
+   costs 50 ms in napi alone, before any real work. Batched
+   variants (`intsToUuidsBatch`, `getNeighborsBatch`,
+   `getVerbsBatchCached`) amortise that to ~0.5 μs per item.
+   The hot path uses batches exclusively.
+
+3. **Lock-free delta cache reads.** The LSM delta cache (added
+   in G4.3, applied in G4.1 + G4.4) currently sits behind a
+   `Mutex`. A sharded `DashMap` or `ArcSwap<Arc<HashMap>>` MVCC
+   pattern lifts the read-side lock entirely, lets concurrent
+   `find()` calls scale linearly with cores.
+
+4. **SIMD bitmap operations.** CRoaring uses portable
+   intersect/union; modern AVX-512 popcnt and `vpternlogq`
+   kernels are ~3–5× faster on bitmap-heavy filter combinations.
+
+5. **DiskANN predicate pushdown.** The Vamana beam search
+   currently scores every neighbor of every visited node. With a
+   metadata-bitmap predicate passed into the search engine,
+   pruned nodes are skipped during the walk, not after. Massive
+   win for high-selectivity hybrid queries.
+
+6. **Pipelined result resolution.** DiskANN currently returns all
+   top-K candidates before resolution starts. Pipelining the
+   resolution (`intsToUuidsBatch` over the first 10 candidates
+   while the walk continues) hides the resolution latency
+   entirely.
+
+7. **Zero-allocation hot path.** `find()` allocates intermediate
+   `Vec<u64>` and `HashSet<String>` per query. A bumpalo-style
+   arena allocator (reset per query, no per-element free)
+   removes the allocation pressure and the GC pause.
+
+Each investment ships with the gates above as the proof point.
+
+## Forbidden patterns
+
+The performance contract has anti-patterns the way the snapshot
+safety contract does:
+
+1. **Per-row napi calls on the hot path.** Anything in `find()`
+   that does `for (id of ids) { resolve(id) }` over napi is
+   incorrect. Use a batch.
+
+2. **Lock acquisition on every read.** A `Mutex` on the read
+   path costs at minimum a CAS on every call. At 100 K QPS with
+   contention, this is a bottleneck before the first real
+   operation runs. Use `ArcSwap` or per-shard locking.
+
+3. **Synchronous compactions on the hot write path.** Compaction
+   is a background operation. A write that blocks for the
+   duration of a compaction is a p99 cliff; we currently run
+   compaction synchronously in G4.3's `ShadowPageEndpointsStore`
+   (acceptable at 3.0 launch scale) but the pattern is
+   explicitly recognised as a future fix.
+
+4. **String allocation in the inner loop.** UUIDs are converted
+   to strings for the JS boundary, but the conversion has to
+   happen once per result row, not per intermediate set. Carry
+   integer ids through the planner; convert at the egress.
+
+5. **Quadratic scans where indexed lookups exist.** Filter
+   evaluation that walks all entities checking each against a
+   predicate, instead of consulting the metadata index, is the
+   hidden cliff at scale. Every `where` clause is index-
+   accelerated or it's a release blocker.
+
+## See also
+
+- `docs/snapshot-safety.md` — the snapshot safety contract this
+  performance work is co-designed with.
+- `docs/verification-report.md` — measured baseline numbers
+  (SIFT1M / SIFT10M / 1M synthetic, AVX-512 baseline) that the
+  budget is anchored on.
+- `docs/comparison.md` — comparison of brainy + cor against
+  pure vector / pure graph / pure document databases on the
+  triple-intelligence query.

package/docs/performance.md

@@ -0,0 +1,117 @@
+---
+title: Cor Performance
+slug: cor/performance
+public: true
+category: cor
+template: guide
+order: 1
+description: Measured benchmarks for SIMD distance calculations, vector quantization, aggregation, and column-store orderBy sort, plus the cross-language consistency guarantee. Run yourself with npm run bench.
+next:
+  - cor/comparison
+  - cor/aggregation
+---
+
+# Cor Performance
+
+All benchmarks measured using `vitest bench` on the included benchmark suite. Run yourself with:
+
+```bash
+npm run bench
+```
+
+## Distance Calculations
+
+384-dimensional vectors (all-MiniLM-L6-v2 embedding size).
+
+| Operation | Throughput | Mean Latency | P99 |
+|-----------|-----------|-------------|-----|
+| `cosineDistance` (single pair) | 45,965 ops/s | 21.8 μs | 34.1 μs |
+| `euclideanDistance` (single pair) | 46,320 ops/s | 21.6 μs | — |
+| `cosineDistanceSq8` (quantized pair) | 3,715,855 ops/s | 0.27 μs | 0.3 μs |
+| `cosineDistanceSq8Batch` (1K vectors) | 2,142 ops/s | 467 μs | 1.1 ms |
+| `cosineDistanceBatch` (1K vectors) | 77 ops/s | 13.0 ms | — |
+
+**Key insight**: SQ8 quantized distance is **81x faster** than full-precision for single-pair comparisons. For batch operations, SQ8 batch is **28x faster** than full-precision batch.
+
+### Quantization
+
+| Operation | Throughput | Mean Latency |
+|-----------|-----------|-------------|
+| `quantizeSq8` (384-dim) | 74,020 ops/s | 13.5 μs |
+| `dequantizeSq8` (384-dim) | 53,440 ops/s | 18.7 μs |
+
+## Aggregation Engine
+
+| Operation | Throughput | Mean Latency |
+|-----------|-----------|-------------|
+| `incrementalUpdate` (1K entities, 3 metrics) | 809 ops/s | 1.2 ms |
+| `rebuildAggregate` (10K entities, 5 groups) | 475 ops/s | 2.1 ms |
+| `rebuildAggregate` (100K entities, Rayon) | 66 ops/s | 15.2 ms |
+| `queryAggregate` (1K groups, sort + paginate) | 986 ops/s | 1.0 ms |
+| `computeGroupKey` (10K entities, time bucketing) | 146 ops/s | 6.8 ms |
+
+Rebuild scales linearly with entity count. Rayon parallelism activates above 1,000 entities.
+
+## Column Store — orderBy Sort
+
+The native column store backs Brainy's `find({ orderBy, limit })`. Its `sortTopK`
+(O(N log K) k-way merge over sorted segments) scales near-linearly with entity
+count.
+
+| Operation | Entities | Mean Latency |
+|-----------|---------:|-------------|
+| `sortTopK` (k=100) | 1,000 | ~0.028 ms |
+| `sortTopK` (k=100) | 10,000 | ~0.21 ms |
+| `sortTopK` (k=100) | 100,000 | ~3.3 ms |
+
+MEASURED on AMD Ryzen 9 7950X3D, Node 22, release native (`src/benchmarks/scaling.bench.test.ts`).
+Growth from 1k→100k is ~125x against a 100x linear ideal — emphatically not the
+~10,000x of an O(N²) sort. `scaling.bench.test.ts` runs in CI and fails the build
+if the growth ratio ever turns super-linear.
+
+## Cross-Language Consistency
+
+Cor is a pure accelerator: every native result is byte-for-byte equal to the
+Brainy JavaScript baseline it replaces, so query results are identical with or
+without the plugin installed. This is enforced by a 104-test cross-language parity
+suite (`src/native/crossLanguageParity.test.ts`) covering tokenization, value
+normalization (the UTF-16/ASCII boundary), code-point string collation, SQ8
+quantized distance, top-K ranking, and roaring/msgpack round-trips against
+Brainy's golden outputs.
+
+## Serialization
+
+| Operation | Throughput | Mean Latency |
+|-----------|-----------|-------------|
+| `msgpackEncode` (1K entities) | 235 ops/s | 4.3 ms |
+| `msgpackDecode` (1K entities) | 355 ops/s | 2.8 ms |
+| `msgpackEncodeBatch` (1K entities) | 290 ops/s | 3.4 ms |
+| `msgpackDecodeBatch` (1K entities) | 356 ops/s | 2.8 ms |
+| `RoaringBitmap32.create` (10K elements) | 3,067 ops/s | 326 μs |
+| `RoaringBitmap32.serialize` (10K elements) | 361,258 ops/s | 2.8 μs |
+| `RoaringBitmap32.deserialize` (10K elements) | 548,714 ops/s | 1.8 μs |
+| `RoaringBitmap32.and` (two 10K bitmaps) | 1,379 ops/s | 725 μs |
+| `RoaringBitmap32.or` (two 10K bitmaps) | 1,219 ops/s | 820 μs |
+| `encodeConnections` (1K lists) | 913 ops/s | 1.1 ms |
+| `decodeConnections` (1K lists) | 1,078 ops/s | 928 μs |
+
+## Hardware Recommendations
+
+- **CPU**: Multi-core for Rayon parallel rebuild. x86_64 for SIMD distance.
+- **Memory**: Cor native allocations are tracked and reported via the cache subsystem.
+- **Storage**: SSD (NVMe ideal) for the memory-mapped HNSW index and graph-adjacency SSTables — the kernel keeps the hot working set in RAM and pages the rest from disk. See [Scaling & Resource Management](./scaling.md).
+
+## Running Benchmarks
+
+```bash
+# Run all benchmarks
+npm run bench
+
+# Run specific benchmark
+npx vitest bench src/benchmarks/distance.bench.ts
+
+# Run with verbose output
+npx vitest bench --reporter=verbose
+```
+
+Benchmarks use vitest bench mode and run multiple iterations to produce stable statistics. Results include Hz (ops/sec), min, max, mean, P75, P99, P99.5, P99.9, and relative margin of error (RME).