@soulcraft/cor 3.0.0-rc.1

package/docs/ADR-002-diskann-100-percent-rust.md

@@ -0,0 +1,294 @@
+---
+title: ADR-002 — DiskANN as cor's billion-scale index option
+slug: cor/adr-002-diskann
+public: true
+category: cor
+template: concept
+order: 2
+description: Architectural decision record for cor's planned DiskANN integration. 100% pure Rust, filesystem-only, auto-engages when conditions are met. The billion-scale upgrade path that pairs with brainy's existing TS HNSWIndex.
+---
+
+# ADR-002 — DiskANN as cor's billion-scale index option
+
+**Status:** Decided 2026-05-28. Implementation queued across three coordinated sessions for cor 3.0.0 + brainy 8.0.0.
+
+**Supersedes:** Original DiskANN spike task (#35), retired 2026-05-28 in favour of the three-session plan captured here.
+
+**Related:**
+- [ADR-001](./ADR-001-column-store-string-support.md) — native column store, shipped 2.3.0
+- brainy 7.28.0 SQ4 (4-bit) quantization — paves the PQ path for DiskANN's compressed in-RAM distance
+- Cortex 2.4.0 storage foundations (#23–#26) — stable IDs, mmap vector layer, graph compression — all transfer directly to DiskANN
+
+## Context
+
+Brainy ships a TypeScript HNSW index that works excellently up to roughly 10M vectors per node on commodity hardware. Cortex 2.3.0 added a Rust-native HNSW variant via the `hnsw` provider hook — same algorithm, ~3–10× the throughput on hot paths thanks to SIMD distance and a tighter graph layout. The 2.4.0 storage foundations (vector mmap store, graph link compression, stable entity IDs) push HNSW further into the disk-resident regime.
+
+But HNSW's design assumes the graph fits comfortably in memory. Past ~10M vectors, two costs compound:
+
+1. **Memory pressure** — the graph alone (`M × node_count` neighbour pointers + level metadata) plus the vector store (`dim × 4 × node_count` bytes for float32) blows past the RAM budget of normal nodes. At 100M vectors of 384-dim embeddings: ~150 GB of vectors + ~13 GB of graph = ~163 GB RAM minimum. At 1B vectors: ~1.6 TB RAM — out of reach on single boxes.
+2. **Disk-locality on cold caches** — even with vectors offloaded to mmap, HNSW's traversal order has no correlation with insertion order on disk. Each search hop typically faults a new page, costing ~10 μs per hop on NVMe SSD. A 100-hop search burns ~1 ms of disk wait that proper locality would have served from a single 10 μs read.
+
+[DiskANN](https://github.com/microsoft/DiskANN) (Microsoft, 2019) was designed for exactly this regime. Its Vamana graph construction uses α-pruning to choose neighbours that produce **disk-locality natively**: nodes visited together during search end up adjacent on disk. Combined with **product quantization (PQ)** in RAM for approximate distance, and full vectors on disk for re-ranking the final candidate set, DiskANN holds single-machine billion-scale search at a fraction of HNSW's RAM cost.
+
+**For cor specifically, DiskANN is the natural billion-scale upgrade path** because:
+
+- The 2.4.0 foundations (stable IDs, mmap vector layer, graph link compression) transfer to it without rework
+- The 2.5.0 #30 SQ4 quantization work primes the PQ codec path
+- Cor's positioning has always been "billion-scale via Rust acceleration" — DiskANN fits the message
+- We control the rest of the stack (storage adapters, idMapper, HNSW provider), so the integration is in friendly territory
+
+We considered **ScaNN** (Google, Apache 2.0) as an alternative. It posts SOTA recall/QPS numbers at moderate scale with anisotropic vector quantization. We declined: ScaNN is IVF-based (inverted file with partition centroids), which doesn't align with brainy's graph-native architecture. Switching to IVF would mean losing the structural symmetry between brainy's verb graph and its vector index, plus introducing periodic clustering retraining (an operational concern brainy doesn't currently have).
+
+## Decisions
+
+### Decision 1 — DiskANN as the billion-scale upgrade path (not a replacement for HNSW)
+
+HNSW stays the brainy default forever. Every existing user, including those without cor, continues to get the TS `HNSWIndex` they ship with today. DiskANN is added as an **alternative provider that engages when its constraints are met**.
+
+This preserves three properties we don't want to give up:
+- Zero-friction onboarding for new brainy users (no cor required, no config tuning to pick an algorithm).
+- Backward compatibility for every existing brainy install (no surprise migrations on upgrade).
+- The "cor makes brainy faster" story (not "cor makes brainy different").
+
+### Decision 2 — 100% pure Rust, no C++ FFI
+
+We will port DiskANN's Vamana algorithm to Rust from the published paper (Subramanya et al., NeurIPS 2019; Singh et al., 2021) rather than wrap Microsoft's C++ reference implementation via FFI. The Vamana algorithm is straightforward: greedy graph construction with an α-pruning step that controls graph density. The published pseudocode plus the reference implementation's behaviour give us everything we need to validate correctness.
+
+PQ codec: we will either compose a battle-tested Rust crate (e.g., `qdrant-quantization`, Apache 2.0) or implement PQ training + encode/decode in cor directly, depending on parity test outcomes. Either way, no C++.
+
+**Why not FFI:** cross-platform C++ builds for Node native modules are operationally expensive (Linux/macOS/Windows × x64/arm64 binaries, headers, link-time gotchas), Microsoft's reference impl has its own build dependencies that would propagate, and we'd inherit any patent grant ambiguities at the binary level. Pure Rust gives us napi-rs's mature cross-platform binary distribution and a license posture we fully control.
+
+**Why not adopt an existing Rust crate wholesale:** no mature Rust port of Vamana exists at our knowledge cutoff. We will track this and pivot if a high-quality one emerges; for now we're building it.
+
+### Decision 3 — Filesystem-only deployment in the first release
+
+DiskANN is local-SSD-by-design. The whole point of the architecture is that disk reads are cheap (NVMe-cheap, ~10 μs) and predictable, so the search algorithm can lean on the OS page cache + the on-disk layout's locality.
+
+Cloud object storage (S3, R2, GCS) breaks that assumption: range reads of large objects cost ~100 ms of round-trip latency, and the locality model has to account for HTTP/2 framing instead of OS pages. Supporting cloud storage for DiskANN would require either:
+
+- A persistent "DiskANN file lives on a local cache disk that we sync from cloud" model (operationally heavy), or
+- A fundamentally different search algorithm with batched range reads (no longer DiskANN, really).
+
+**For the first DiskANN release, the activation conditions explicitly require `storage.adapter === 'filesystem'`.** Cloud-storage users continue to use HNSW. We may revisit cloud support if there's demand and an approach that doesn't compromise the algorithm's strengths.
+
+### Decision 4 — Auto-engagement, zero configuration
+
+When all of these conditions hold at brainy init, DiskANN replaces HNSW as the active index without any user config:
+
+1. Cor is loaded as a plugin (the `index:diskann` provider is registered)
+2. The storage adapter is `FileSystemStorage` (local SSD)
+3. The metadata index exposes a stable `idMapper` (the 2.4.0 #23 foundation)
+
+This mirrors the existing `MmapVectorBackend` wiring pattern from 2.4.0 #24: the heavy machinery activates when its preconditions are met, and otherwise silently falls back. Users who don't want it can opt out via `config.index.type = 'hnsw'`.
+
+**Why auto-engage instead of opt-in by config:**
+
+- Matches cor's "loading cor makes brainy faster" value proposition (no extra knob to turn)
+- The constraints (cor + filesystem) are exactly the deployment shape DiskANN targets, so the conditions ARE the signal
+- Opt-in-only would leave most filesystem-using cor installs on HNSW out of caution — defeating the point
+
+**Why not unconditional default:**
+
+- Cloud-storage users have no DiskANN-compatible path; we can't break their existing HNSW workflows
+- Cor-less users (the brainy-only crowd) never see DiskANN regardless — preserves the "brainy works the same with or without cor" property
+
+### Decision 5 — Explicit migration API for existing installs
+
+Existing brainy installs with an HNSW index on disk **do not auto-migrate to DiskANN on upgrade**. The on-disk HNSW state is detected at init; if `config.index.type` is unset, brainy logs:
+
+> `[brainy] Existing HNSW index detected at <path>. The new cor default for filesystem storage is DiskANN. Continue using HNSW (set config.index.type='hnsw' to silence this message) or run brain.migrateToDiskAnn() to convert.`
+
+The migration API:
+
+```typescript
+// Convert an existing HNSW index to DiskANN.
+// Builds the DiskANN index in parallel (separate files), verifies recall
+// parity at the configured threshold, then atomically swaps the active
+// index. Reversible via brain.migrateToHnsw().
+await brain.migrateToDiskAnn({
+  recallTarget?: number,    // default 0.95 — verification target before swap
+  paddingFactor?: number,   // default 1.2 — slack for re-ranking candidate set
+  parallel?: boolean        // default true — build new index alongside live old
+})
+```
+
+Reversibility (`brain.migrateToHnsw()`) is a contract, not a courtesy. Users need to be able to roll back if recall regression or any other issue surfaces in production.
+
+## Architecture
+
+### Brainy provider contract
+
+Cor registers two new providers (mirrors the existing `hnsw` provider shape):
+
+```typescript
+// brainy: src/plugin.ts
+export interface DiskAnnProvider {
+  create(config: DiskAnnConfig, distance: DistanceFunction, options: DiskAnnOptions): DiskAnnInstance
+  openExisting(path: string, distance: DistanceFunction): DiskAnnInstance
+}
+
+export interface DiskAnnInstance extends HnswProvider {
+  // Implements the same interface HNSWIndex/HnswProvider exposes, so the rest
+  // of brainy doesn't care which index is active. Adds one DiskANN-specific
+  // method for the migration API:
+  rebuildPQCodebook(): Promise<void>  // Re-trains PQ from current vectors
+}
+```
+
+The instance implementing `HnswProvider` is the load-bearing decision. brainy's search/find/get code paths call into the provider through this surface; an `HNSWIndex` and a `NativeDiskANN` are interchangeable from brainy's POV. No control-flow plumbing changes in brainy beyond the choice of which provider to instantiate.
+
+### Cor Rust modules
+
+```
+cor/native/src/diskann/
+├── mod.rs        — napi exports + the NativeDiskANN class
+├── vamana.rs     — α-pruning greedy graph construction (~500 LOC)
+├── pq.rs         — Product Quantization codebook training + encode/decode
+├── format.rs     — On-disk file format (header + PQ codebook + graph + vectors)
+└── search.rs     — Greedy graph search with PQ-approximate distance + re-rank
+```
+
+### On-disk file format
+
+Single contiguous file `<dataDir>/_diskann/main.bin` (path mirrors `_vectors/main.bin` from #24):
+
+```
++--------------------------------------------------------------+
+| Header (4 KB, aligned)                                       |
+|   magic: u32         "DKAN"                                  |
+|   version: u32       layout revision                         |
+|   dim: u32           vector dimensionality                   |
+|   node_count: u32    total vectors                           |
+|   pq_subspaces: u8   PQ M parameter (typically 8 or 16)      |
+|   pq_bits: u8        bits per subspace (typically 8)         |
+|   max_degree: u8     Vamana R parameter (typically 64-96)    |
+|   entry_point: u32   slot id of the entry node               |
+|   ... reserved bytes for forward compatibility ...           |
++--------------------------------------------------------------+
+| PQ codebook (M × 256 × subvec_dim × f32)                     |
++--------------------------------------------------------------+
+| PQ codes (node_count × M bytes)                              |
+| — one PQ code per node, M bytes each, in slot order          |
++--------------------------------------------------------------+
+| Vamana graph (node_count × max_degree × u32)                 |
+| — flat CSR-like array of neighbour slot ids                  |
+| — fixed degree per node for predictable offset math          |
++--------------------------------------------------------------+
+| Full vectors (node_count × dim × f32)                        |
+| — only touched for re-ranking the final candidate set        |
++--------------------------------------------------------------+
+```
+
+The fixed-degree Vamana graph trades a small density loss for O(1) neighbour-offset arithmetic. PQ codes pack tightly in RAM (M bytes per vector — at M=16 that's 16 bytes/vector regardless of dim, so 1B vectors fit in ~16 GB RAM for the PQ-resident layer).
+
+### Search algorithm
+
+```
+async function search(query: Vector, k: number): Promise<Result[]> {
+  // 1. PQ-encode the query into M sub-vector codes
+  const queryPq = pqEncode(query, codebook)
+
+  // 2. Greedy graph walk using PQ-approximate distance
+  const visited = new Set<u32>()
+  const candidates = new BoundedHeap(maxLen = k * paddingFactor)
+  let current = entryPoint
+
+  while (improving(candidates)) {
+    const neighbours = graph[current]
+    for (const n of neighbours) {
+      if (visited.has(n)) continue
+      visited.add(n)
+      const approxDist = pqDistance(queryPq, codes[n])
+      candidates.insert(n, approxDist)
+    }
+    current = candidates.bestUnvisited()
+  }
+
+  // 3. Re-rank the top-(k * paddingFactor) candidates with full vectors
+  const topCandidates = candidates.topN(k * paddingFactor)
+  return topCandidates
+    .map(n => ({ id: idMapper.getUuid(n), distance: trueDistance(query, vectors[n]) }))
+    .sort()
+    .slice(0, k)
+}
+```
+
+The `paddingFactor` (default 1.2 = 20% over-fetch) controls the recall/cost tradeoff. PQ approximate distance is fast but lossy; re-ranking on the over-fetched candidate set with full-precision vectors recovers recall at a small cost (typically a few hundred extra full-vector reads per query, which is fine on SSD).
+
+## Implementation plan
+
+### Session 35a — Vamana + PQ in pure Rust (cor)
+
+**Scope (~3–5 hrs focused):**
+
+- `cor/native/src/diskann/vamana.rs` — Vamana graph construction with α-pruning, ~500 LOC. Inputs: vector buffer, dim, R (max degree), α (density parameter, typically 1.2–1.4). Output: CSR adjacency.
+- `cor/native/src/diskann/pq.rs` — PQ codebook training (k-means on subvector partitions) + encode/decode. M subspaces × 256 centroids each, configurable.
+- `cor/native/src/diskann/format.rs` — On-disk file format struct + read/write primitives.
+- Rust unit tests: graph connectivity invariants, PQ recall on small synthetic dataset, format round-trip.
+
+**Exit criteria:** Vamana graph build over 10k random vectors produces a connected graph with degree ≤ R, search recall ≥ 95% at k=10 on synthetic dataset.
+
+### Session 35b — Search + napi bindings (cor)
+
+**Scope (~3–5 hrs focused):**
+
+- `cor/native/src/diskann/search.rs` — Greedy search with PQ-approximate distance and full-vector re-ranking on the candidate set.
+- `cor/native/src/diskann/mod.rs` — `#[napi]` exports of `NativeDiskANN` class with `create` / `openExisting` / `addItem` / `search` / `rebuildPQCodebook` methods.
+- `cor/native/index.d.ts` regeneration.
+- Recall validation against published DiskANN benchmark numbers (sanity check, not full BIGANN — that's a separate effort).
+
+**Exit criteria:** Search recall ≥ 95% at k=10 over a 100k-vector dataset matches the published DiskANN paper's numbers within 2 percentage points.
+
+### Session 35c — Brainy hookup + cor 3.0.0 + brainy 8.0.0 release
+
+**Scope (~3–5 hrs focused):**
+
+- `brainy/src/hnsw/diskAnnIndex.ts` — TS wrapper class implementing brainy's `HnswProvider` contract over `NativeDiskANN`. Same surface as `HNSWIndex` so the rest of brainy is agnostic.
+- `brainy/src/brainy.ts` — `wireDiskAnn()` private method that runs after `wireMmapVectorBackend()` during init. Auto-engagement conditions; opt-out via `config.index.type = 'hnsw'`.
+- `brainy/src/plugin.ts` — `DiskAnnProvider` and `DiskAnnInstance` interfaces (mirrors the `VectorStoreMmapProvider` pattern from 2.4.0 #24).
+- `brain.migrateToDiskAnn()` and `brain.migrateToHnsw()` explicit migration APIs.
+- Tests: provider hookup, auto-engagement conditions, opt-out, recall parity at 10k–100k vectors, migration round-trip integrity.
+- Coordinated release: `cor 3.0.0` + `brainy 8.0.0`. Major bumps because the default index type changes for filesystem+cor users (semver discipline matters).
+
+**Exit criteria:** Recall parity between brainy 8.0.0 + cor 3.0.0 DiskANN path and brainy 7.x + cortex 2.x HNSW path is within 1% at standard k values (1, 10, 50). Migration round-trip preserves index integrity.
+
+## Consequences
+
+### Positive
+
+- **Single-machine billion-scale becomes a supported workload.** At 100M to 1B vectors, RAM cost drops by ~16–20× compared to HNSW. NVMe disk locality replaces RAM pressure as the bottleneck.
+- **Cor's "billion-scale via Rust acceleration" positioning becomes literal**, not aspirational.
+- **Zero impact to non-cor users.** brainy keeps shipping its TS HNSWIndex; no API change, no behaviour change for them.
+- **Foundations carry forward.** The 2.4.0 storage work (stable IDs, mmap layer, graph compression) and 2.5.0 #30 (SQ4 quantization, the PQ precursor) all transfer.
+- **License posture is clean.** Pure Rust port from a published algorithm + permissive (MIT/Apache 2.0) Rust deps. No C++ FFI license entanglement.
+- **Future-utility carry.** The cor Rust compaction primitives (`compute_bfs_order`, `compute_hnsw_traversal_order`) stay in the codebase; if HNSW's disk locality ever becomes interesting again, the math is already there.
+
+### Negative / Tradeoffs
+
+- **Build cost.** DiskANN graph construction is slower than HNSW because Vamana's α-pruning requires examining more candidate neighbours per node. On 100M vectors this is hours, not minutes. Acceptable for once-per-deployment cost.
+- **PQ recall ceiling.** Product Quantization is lossy. Recall maxes out around 95–98% on typical embedding workloads; HNSW with full precision can reach 99%+. The re-ranking step recovers most of the gap. Users with extreme recall requirements (e.g., legal-discovery search) may want to stay on HNSW.
+- **Filesystem-only constraint.** Cloud-storage users get no benefit from DiskANN in the first release. We've accepted this; cloud DiskANN is a future investigation, not a commitment.
+- **Major version bump.** Auto-engagement changing the default index type for filesystem+cor users is a semver-major event. brainy 8.0.0 and cor 3.0.0 must coordinate. Some communication overhead at release time.
+
+### Risks
+
+- **Correctness drift from the reference implementation.** Vamana has subtle algorithmic choices (the α-pruning order, the entry-point selection strategy) that affect recall by small but real amounts. Mitigation: explicit recall validation against the published numbers + reference implementations in 35a and 35b's exit criteria.
+- **Brainy provider contract surface mismatch.** The `HnswProvider` interface was designed for HNSW; DiskANN may surface operations (codebook retraining, segment-level compaction) that don't fit cleanly. Mitigation: keep `DiskAnnInstance` as an extension of `HnswProvider` plus DiskANN-specific methods; never narrow the parent interface.
+- **Migration API regressions.** `migrateToDiskAnn` runs over potentially billions of vectors. A bug here could mean hours of wasted compute or, worse, an inconsistent index. Mitigation: parallel build (the old HNSW stays serving until the new DiskANN is validated), explicit recall verification before the atomic swap, fully reversible via `migrateToHnsw`.
+- **Long-running PQ codebook drift.** As vectors are added over time, the original PQ codebook can drift away from the data distribution, eroding recall. Mitigation: expose `rebuildPQCodebook()` for explicit retrains; document the operational guideline (retrain after the dataset doubles, or after a measurable recall regression).
+
+## Open questions
+
+1. **PQ codebook strategy at scale.** Do we train PQ once on a sample of the data, or use online/streaming PQ updates? Tradeoff: simpler vs. better recall over time. Lean toward sample-once-with-explicit-retrain to keep the operational model simple.
+2. **Vamana parameters as runtime config vs. baked into the file format.** R (max degree), α (density), the search candidate set padding factor — how much do we expose to users? Lean toward fixed-good-defaults in 3.0.0, expose later if a workload demands it.
+3. **Filtered search support.** brainy's `find({ where, ... })` interacts with HNSW via a filter callback. DiskANN's PQ-distance loop needs different filter integration. Plan to defer — initial release supports unfiltered top-K search; filtered search is a follow-up.
+4. **Multi-shard / single-node-of-cluster deployments.** Cor isn't a cluster engine, but some users run multiple cor+brainy nodes behind a load balancer. Does each node need its own DiskANN file, or can they share one? Plan to defer — start with per-node files.
+
+## References
+
+- Subramanya et al., *DiskANN: Fast Accurate Billion-Point Nearest Neighbor Search on a Single Node*, NeurIPS 2019. [arXiv:1907.07574](https://arxiv.org/abs/1907.07574)
+- Singh et al., *FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search*, 2021. [arXiv:2105.09613](https://arxiv.org/abs/2105.09613)
+- Microsoft DiskANN open-source reference implementation: [github.com/microsoft/DiskANN](https://github.com/microsoft/DiskANN) (MIT licensed)
+- ADR-001 — Native column store with raw mmap segments (the same architectural pattern of "cor registers a provider, brainy consumes when present")
+- Brainy 7.28.0 SQ4 quantization (the PQ precursor — scalar quantization scoped to a single vector; PQ extends the same idea to subvector partitions with learned codebooks)
+- Cortex 2.4.0 storage foundations: stable EntityIdMapper (#23), mmap vector backend (#24), graph link compression (#25), column-store interchange (#26)

package/docs/ADR-003-semantic-time-travel.md

@@ -0,0 +1,100 @@
+---
+title: ADR-003 — Semantic (vector) time travel
+slug: cor/adr-003-semantic-time-travel
+public: true
+category: cor
+template: concept
+order: 3
+description: How db.asOf(g).find() returns semantically-correct results as of a past generation — across metadata, graph, AND the vector index. A tiered strategy that exploits triple-intelligence pre-filtering, reuses the shadow-page generation substrate, and handles embedding-model drift.
+---
+
+# ADR-003 — Semantic (vector) time travel
+
+**Status:** Proposed 2026-06-15. Phase 0 (verification) targeted for the 8.0.0/3.0.0 lockstep release; Phases 1–2 for the immediate follow-up (8.1/3.1); Phase 3 is a research track.
+
+**Related:**
+- [ADR-002](./ADR-002-diskann-100-percent-rust.md) — DiskANN vector index
+- `docs/snapshot-safety.md` — the shadow-page generation substrate this builds on
+- Graph time-travel generation-threading (cor 3.0 / brainy 8.0 `getVerbEndpointsAtGeneration`)
+
+## Context
+
+Brainy 8.0's `db.asOf(g)` returns the database as it existed at generation `g`. For a triple-intelligence `find()` (metadata filter ∩ graph hop ∩ vector similarity), *every* substrate must answer "what did you look like at `g`."
+
+Two of the three are solved by the **shadow-page generation substrate**: the metadata LSM and the graph endpoints store record brainy's generation per write and resolve point-in-time reads from a version chain. The id-mappers are monotonic. So the **existence dimension** — which entities and edges were present at `g` — is correct.
+
+The **vector/semantic dimension is the hard one**, and it decomposes into three sub-problems:
+
+1. **Candidate set at `g`** — which entities existed. *Already solved* (the versioned substrates above).
+2. **Vector *values* at `g`** — an entity re-embedded after an edit has a different vector now than at `g`. The historical value must be recoverable.
+3. **Index *structure* at `g`** — HNSW/DiskANN build a navigation graph. The current graph differs from the one at `g`.
+
+Plus a fourth, cross-cutting limit:
+
+4. **Embedding-model drift** — the query is embedded with the *current* model. If the embedding model changed between `g` and now, comparing a current-model query against old-model data vectors is a comparison across *different vector spaces* — silently wrong unless detected.
+
+Versioning a million-node ANN graph per generation is prohibitive, which is why no standalone vector database offers semantic time travel. This ADR records a strategy that does, without paying that cost in the common case.
+
+## The key insight
+
+In a triple-intelligence query the vector leg rarely runs alone — it runs over the **intersection of the metadata and graph filters, which are already time-travel-correct and usually small**. Over a candidate set of hundreds (not millions), an **exact rerank** over the entities' gen-`g` vectors gives perfect recall in sub-millisecond time, with **no historical ANN index at all**. The other two intelligences pre-pay the cost of vector time travel.
+
+The hard residue is only the **unfiltered** case (pure semantic search over everything, as of `g`), which is where the index itself must time-travel.
+
+## Decision
+
+A **tiered strategy** on the existing generation substrate, selected automatically by the `AdaptiveDiskAnnModeSelector` on `(filter selectivity × history depth)`:
+
+| Regime | Mechanism | Recall | Cost |
+|---|---|---|---|
+| **Filtered, any depth** (common) | **Exact rerank** over the pre-filtered candidate set's gen-`g` vectors | Exact | Sub-ms (small set); no new index |
+| **Unfiltered, recent** | **Differential overlay**: current index + visibility bitmap + sparse per-node vector version chain (only re-embedded nodes); navigate current graph, mask future nodes, evaluate gen-`g` vectors | Near-exact | One index; near-zero overhead |
+| **Unfiltered, deep** | **LSM-tiered shadow-page ANN**: immutable index segments at generation checkpoints + deltas; search the segment(s) covering `g`; lazily materialized + cached | Exact (segment) | Bounded per-segment; pay-per-use storage |
+| **Cross-cutting** | **Embedding-model versioning**: stamp each vector with its model; auto-project the query into the historical space, or hard-error — never silently wrong | — | Detection is O(1) |
+
+### Options considered (and why not, alone)
+
+1. **Exact rerank over the pre-filtered window** — chosen for the filtered case; insufficient alone (unfiltered queries fall through).
+2. **LSM-tiered / shadow-page ANN** — chosen for the unfiltered-deep case; reuses the proven base+delta+compaction primitive so the vector index joins one unified MVCC substrate. Cost: search amplification across segments.
+3. **Generation-tagged Vamana + delta-replay** — exact historical *structure*, but Vamana's robust-prune rewires many neighbors per insert → fat delta logs; folded into (2) as the segment-internal representation rather than a standalone mode.
+4. **Persistent / copy-on-write immutable ANN** (structural sharing, à la Datomic) — elegant and instant `asOf`, but ANN's high fan-out + rewiring limits sharing and CoW pointer-chasing fights the cache locality ANN depends on. Research track (Phase 3).
+5. **Differential overlay** — chosen for the unfiltered-recent case; structure is current (approximate recall for deep history), which is why it is scoped to recent generations.
+6. **Time as a first-class ANN filter** (`[birth_gen, death_gen]` predicate) — composes with our metadata/graph filters and is how the candidate set is computed; suffers known recall cliffs at extreme selectivity, so it feeds the rerank/overlay rather than standing alone.
+7. **Embedding-model re-projection** — chosen as the cross-cutting correctness layer; the alignment map (Phase 3) is research-grade, but the *detection* of model drift (Phase 1) is cheap and ships early.
+
+## Open-core behavior (brainy alone vs brainy + cor)
+
+The API and correctness are identical; only speed and scale differ — the open-core contract.
+
+- **brainy alone (MIT):** `db.asOf(g).find()` is correct via brainy's canonical-record materialization — the historical candidate set is rebuilt from versioned records and reranked in JS. Slower (brute-force, no native acceleration), bounded to the JS-feasible scale (~10⁵–10⁶), but **semantically correct**.
+- **brainy + cor:** the candidate set comes from cor's versioned native substrates, the rerank is native, the unfiltered-deep path uses the tiered shadow-page ANN, and scale extends to 10⁸–10¹⁰. Same `asOf(g).find()` call; faster and larger.
+
+This means semantic time travel is a **brainy feature** that cor accelerates — not a cor-only capability.
+
+## Phasing
+
+- **Phase 0 — Verify + document (this release, 8.0/3.0). ✅ VERIFIED — see the guarantee + limits below.** A cross-layer test (`src/native/semanticAsOf.test.ts`) builds entities, re-embeds one across generations, then asserts `asOf(g).find({vector, where})` returns only the g-valid set, ranked by the *at-g* embedding, with the `where` filter seeing the at-g metadata. It passes (incl. a leak-detector that fails if the vector leg used the live index). No new critical-path code.
+
+### Phase 0 — verified guarantee + its limits (be honest in GA messaging)
+
+**Guarantee (verified, 8.0.0-rc.2 + cor 3.0):** `db.asOf(g).find({ vector, where })` is set-correct, rank-correct (at-g embedding), and filter-correct (at-g metadata) — **for mutations committed via `brain.transact([...])`**.
+
+**Limits — all four are real; do not claim past them:**
+1. **`transact()`-only in the shipped brainy rc.2.** Single-op `brain.add()`/`brain.remove()` writes bump the generation counter but do **not** retain a before-image or register the generation (`generationStore.js:178 noteSingleOpWrite`), so `asOf(g)` silently resolves to **current** state for them. Time-travel-correct history requires `transact()`. Brainy's rc.3 Model-B is slated to retain single-op writes; the guard test (`semanticAsOf.test.ts`) flips red when it lands. (Handoff: IMMUTABILITY-TIMETRAVEL thread.)
+2. **The historical leg is brainy-JS-served this release.** `asOf(g)` builds an ephemeral reader brain and rebuilds a JS HNSW from the at-g vectors; cor's native vector index (`NativeDiskAnnWrapper`) is **not** on the historical path. cor accelerates only the live (`now`) leg in 3.0. (Open-core: brainy-alone is correct via JS rerank.)
+3. **O(n-at-g) rebuild per `asOf` Db — small/medium scale only.** One `addItem` per at-g vector into a fresh JS HNSW; this OOMs at billion-scale. Billion-scale historical semantic find is **Phase 1–2** (cor native rerank / overlay / tiered ANN; #35). Phase 0 is correctness, not scale.
+4. **No embedding-model-drift guard yet.** Comparing a query embedded with model vN against stored vectors from model vM is silently wrong across a model upgrade — detection lands in Phase 1, alignment in Phase 3.
+- **Phase 1 — Native-accelerate + drift detection (8.1/3.1).** Cor native exact-rerank path; embedding-model version stamping + hard-error on cross-model comparison (never silently wrong).
+- **Phase 2 — Unfiltered paths (8.1/3.1+).** Differential overlay (recent) + LSM-tiered shadow-page ANN (deep), lazily materialized + cached.
+- **Phase 3 — Research track (lands when ready).** Persistent/CoW ANN; learned cross-model re-projection (semantic time travel that survives embedding-model upgrades).
+
+## Consequences
+
+- Semantic time travel ships **correct-first** (Phase 0 verifies the realistic query), then **fast** (Phase 1+), rather than waiting on a versioned ANN index that may never be worth its cost.
+- The vector index becomes a first-class citizen of the same generation substrate as metadata and graph — one MVCC clock across all three intelligences.
+- The honest limit (embedding-model drift) is made explicit and, in Phase 3, solvable — rather than ignored.
+
+## Division of labor
+
+- **brainy:** `asOf` routing (when to materialize vs serve native); canonical-record retention of per-generation vectors; embedding-model version stamping in the record; the open-core JS rerank path.
+- **cor:** native exact-rerank; the differential overlay; the LSM-tiered shadow-page ANN; mode selection by selectivity × depth; the model-drift detector + (Phase 3) re-projection.

package/docs/aggregation.md

@@ -0,0 +1,251 @@
+---
+title: Cor Aggregation Engine
+slug: cor/aggregation
+public: true
+category: cor
+template: concept
+order: 3
+description: Rust-accelerated aggregation for real-time analytics — incremental updates, exact percentile and distinctCount, HAVING filters, array-unnest grouping, Welford online statistics, parallel rebuilds, and time-window bucketing.
+next:
+  - cor/performance
+  - cor/scaling
+---
+
+# Cor Aggregation Engine
+
+Cor provides a Rust-accelerated aggregation engine for Brainy, enabling real-time analytics on entity data with incremental updates, parallel rebuilds, and precise statistical operations.
+
+## Architecture
+
+Brainy owns the storage and lifecycle. Cor owns the compute.
+
+```
+┌─────────────────────────────────────────┐
+│              Brainy                      │
+│  ┌─────────────────────────────────┐    │
+│  │   AggregationIndex              │    │
+│  │   ├── defineAggregate()         │    │
+│  │   ├── removeAggregate()         │    │
+│  │   ├── onEntityAdd/Update/Delete │    │
+│  │   └── query()                   │    │
+│  └──────────────┬──────────────────┘    │
+│                 │ provider interface     │
+│  ┌──────────────▼──────────────────┐    │
+│  │   Cor AggregationProvider    │    │
+│  │   ├── NativeAggregationEngine   │───── Rust (NAPI)
+│  │   ├── value multiset (min/max,  │    │
+│  │   │   percentile, distinctCount)│    │
+│  │   ├── HAVING + array-unnest      │    │
+│  │   ├── Rayon parallel rebuild    │    │
+│  │   └── Welford's online stddev   │    │
+│  └─────────────────────────────────┘    │
+└─────────────────────────────────────────┘
+```
+
+When Cor is installed as a Brainy plugin, the aggregation provider automatically registers. All aggregation computation runs in Rust through NAPI bindings while Brainy handles storage, persistence, and the public API.
+
+## Operations
+
+Cor supports all 9 aggregation operations:
+
+| Operation | Description | Precision |
+|-----------|-------------|-----------|
+| `sum` | Running total of a numeric field | Exact (f64) |
+| `count` | Number of matching entities | Exact (u64) |
+| `avg` | Running average (sum/count) | Exact (f64) |
+| `min` | Minimum value across all entities | Exact (value multiset) |
+| `max` | Maximum value across all entities | Exact (value multiset) |
+| `stddev` | Sample standard deviation | Online (Welford's) |
+| `variance` | Sample variance | Online (Welford's) |
+| `percentile` | Value at fraction `p` (e.g. p50, p95) | Exact (value multiset, linear interpolation) |
+| `distinctCount` | Number of distinct values | Exact (value multiset) |
+
+`percentile` requires a `p` property in `[0, 1]` on the metric definition
+(`{ op: 'percentile', field: 'latency', p: 0.95 }`). Both `percentile` and
+`distinctCount` are **exact** — they read the same per-group value multiset that
+backs precise MIN/MAX, so they are never approximate and never stale after
+deletes. Percentile uses standard linear interpolation between ranks, verified
+against numpy's percentiles in `src/aggregation/aggregation.test.ts`.
+
+### Exact value multiset (MIN/MAX, percentile, distinctCount)
+
+Unlike simpler implementations that become stale after deletes, Cor keeps a per-group Rust `BTreeMap<OrderedFloat<f64>, u32>` (a sorted value multiset) tracking the exact frequency of every value:
+
+- **Add**: Insert or increment the count for the value
+- **Delete**: Decrement the count; remove the key if count reaches zero
+- **MIN / MAX**: first / last key of the multiset
+- **percentile**: walk the sorted multiset to the target rank, linear-interpolating between neighbours
+- **distinctCount**: number of keys in the multiset
+
+This single structure guarantees exact MIN, MAX, percentile, and distinct-count after any sequence of add/update/delete operations without requiring a full rescan.
+
+### Welford's Online Algorithm
+
+Standard deviation and variance use Welford's numerically-stable online algorithm with `M2` tracking (sum of squared differences from the running mean). This computes incrementally without storing all values:
+
+```
+On add(x):
+  n += 1
+  delta = x - mean
+  mean += delta / n
+  delta2 = x - mean
+  M2 += delta * delta2
+
+Sample variance = M2 / (n - 1)
+Sample stddev = sqrt(variance)
+```
+
+## Rayon Parallel Rebuild
+
+When rebuilding an aggregate from scratch (e.g., after definition change or cold start), Cor uses Rayon's parallel iterators to process entities across all CPU cores:
+
+- Entities below 1,000 are processed sequentially (overhead not worth it)
+- Above 1,000, Rayon splits the work across threads
+- Each thread computes partial aggregation state
+- Results are merged with thread-safe combining
+
+For 100K entities with 20 groups, rebuild completes in ~15ms.
+
+## Time Window Bucketing
+
+GroupBy dimensions can specify time windows for temporal aggregation. The native engine performs integer-based timestamp bucketing without allocating `Date` objects:
+
+| Window | Format | Example |
+|--------|--------|---------|
+| `hour` | `YYYY-MM-DDTHH` | `2024-01-15T14` |
+| `day` | `YYYY-MM-DD` | `2024-01-15` |
+| `week` | `YYYY-WNN` (ISO week) | `2024-W03` |
+| `month` | `YYYY-MM` | `2024-01` |
+| `quarter` | `YYYY-QN` | `2024-Q1` |
+| `year` | `YYYY` | `2024` |
+| custom (N seconds) | ISO 8601 UTC, floored | `2024-01-15T10:30:00.000Z` |
+
+Keys are computed with pure integer arithmetic (no `Date` allocation) and are byte-for-byte identical to Brainy's `bucketTimestamp()` (`native/src/aggregation/timestamp.rs`).
+
+Combined groupBy dimensions (plain field + time window) produce composite group keys:
+
+```typescript
+brain.defineAggregate({
+  name: 'monthly_sales',
+  source: { type: 'Event' },
+  groupBy: ['region', { field: 'date', window: 'month' }],
+  metrics: {
+    revenue: { op: 'sum', field: 'amount' },
+    count: { op: 'count' }
+  }
+})
+```
+
+This produces groups like `{ region: 'US', date: '2024-01' }`.
+
+## Array-Unnest GroupBy
+
+A groupBy dimension can `unnest` an array field, so an entity contributes once per
+distinct array element. The classic use is tag/label frequency:
+
+```typescript
+brain.defineAggregate({
+  name: 'tag_frequency',
+  source: { type: 'Post' },
+  groupBy: [{ field: 'tags', unnest: true }],
+  metrics: { count: { op: 'count' } }
+})
+```
+
+A post with `tags: ['rust', 'db', 'rust']` contributes once to `rust` and once to
+`db` (duplicates within one entity are de-duplicated). An entity with a missing or
+empty array contributes to no group.
+
+## HAVING
+
+`queryAggregate` can filter groups *after* metrics are computed via a `having`
+clause, using the same Brainy field-operator syntax as `where` (`greaterThan`,
+`lessThan`, `between`, `anyOf`/`allOf`/`not`, …). It applies to computed metric
+values and to `count`:
+
+```typescript
+brain.queryAggregate('monthly_sales', {
+  having: { revenue: { greaterThan: 10000 } }
+})
+```
+
+Filtering happens post-aggregation in Rust (O(groups)), so a `having` clause never
+re-scans entities.
+
+## State Serialization
+
+The engine serializes all internal state (definitions, group states, the per-group
+value multiset, Welford's M2 values) to JSON for persistence:
+
+```typescript
+// Brainy handles this automatically via the provider interface
+const state = engine.serializeState()   // JSON string
+engine.restoreState(state)              // Restore on restart
+```
+
+State includes:
+- All registered definitions
+- Per-aggregate, per-group metric state
+- The value multiset backing MIN/MAX, percentile, and distinctCount
+- Welford's `mean`, `M2`, and `count` for stddev/variance
+
+## Source Filtering
+
+Aggregate definitions can specify source filters to only aggregate entities of a specific type:
+
+```json
+{
+  "name": "event_stats",
+  "source": { "type": "Event" },
+  "groupBy": ["category"],
+  "metrics": { "count": { "op": "count" } }
+}
+```
+
+During incremental updates, entities that don't match the source filter are skipped. During rebuild, the filter is compiled and applied before aggregation.
+
+Aggregate entities themselves (entities with `service: 'brainy:aggregation'` or `metadata.__aggregate: true`) are always skipped to prevent infinite feedback loops.
+
+## Incremental Update Flow
+
+When Brainy calls `incrementalUpdate()`:
+
+1. **Source filter check** — skip if entity doesn't match
+2. **Aggregate entity check** — skip if entity is itself an aggregate
+3. **Group key computation** — extract groupBy fields, apply time bucketing
+4. **Metric update** — for each metric in the definition:
+   - `add`: Increment sum/count/mean/M2, insert into BTreeMap
+   - `delete`: Decrement sum/count/mean/M2, remove from BTreeMap
+   - `update`: Delete old values, add new values (handles group changes)
+
+## Performance
+
+Run the included benchmarks: `npm run bench`
+
+| Operation | Throughput | Latency |
+|-----------|-----------|---------|
+| `incrementalUpdate` (1K entities) | 809 ops/s | 1.2 ms |
+| `rebuildAggregate` (10K entities) | 475 ops/s | 2.1 ms |
+| `rebuildAggregate` (100K entities) | 66 ops/s | 15.2 ms |
+| `queryAggregate` (1K groups, sort + paginate) | 986 ops/s | 1.0 ms |
+| `computeGroupKey` (10K entities) | 146 ops/s | 6.8 ms |
+
+## Troubleshooting
+
+### Aggregation not using native engine
+
+Verify Cor is loaded and the aggregation provider is registered:
+
+```typescript
+const diag = brain.diagnostics()
+console.log(diag.providers.aggregation)
+// Should show { source: 'plugin' }
+```
+
+### Stale MIN/MAX after deletes
+
+This should not happen with Cor — the BTreeMap guarantees precision. If you see stale values, verify you're running Cor (not the JS fallback) and that the delete operation includes the correct entity metadata.
+
+### Rebuild performance
+
+For datasets over 100K entities, rebuild uses Rayon parallelism automatically. Ensure your system has multiple CPU cores available. Single-core environments still work but won't benefit from parallel rebuild.