@hviana/sema 0.1.4 → 0.1.6

package/AGENTS.md

@@ -44,7 +44,7 @@ store.ts             AbstractStore: ALL domain logic of the DAG store
 store-sqlite.ts      the one concrete backend (thin SQL wrappers)
 geometry.ts + vec/alphabet/sema
                      vectors, the fold, and every derived threshold
-derive/ · alu/ · rabitq-hnsw/
+derive/ · alu/ · rabitq-ivf/
                      firewalled sublibraries with their own READMEs and tests
 ```
 
@@ -80,7 +80,8 @@ D, the perception window W, or the corpus size N, defined once in
 `src/geometry.ts` (`mergeThreshold`, `identityBar`, `reachThreshold`,
 `significanceBar`, `estimatorNoise`, `conceptThreshold`, `consensusFloor`,
 `dominates`, …). `src/config.ts` holds **capacities and budgets only** (cache
-byte budgets, batch sizes, HNSW parameters, query k, ALU precision, the seed).
+byte budgets, batch sizes, vector-index parameters, query k, ALU precision, the
+seed).
 
 _Follow it:_ if you are about to add a tunable cutoff to config, derive it in
 `geometry.ts` instead. A threshold knob is a design bug here — one was already
@@ -360,7 +361,7 @@ story of the fix.
 | Post-grounding (reason, fuse, articulate)     | `src/mind/reasoning.ts`, `src/mind/articulation.ts`                    |
 | Rationale / trace                             | `src/mind/rationale.ts`, `src/mind/trace.ts`                           |
 | Extension host types                          | `src/extension.ts`                                                     |
-| Sublibraries (own READMEs, own tests)         | `src/derive/`, `src/alu/`, `src/rabitq-hnsw/`                          |
+| Sublibraries (own READMEs, own tests)         | `src/derive/`, `src/alu/`, `src/rabitq-ivf/`                           |
 
 Mind functions are **free functions over `MindContext`** (`mind/types.ts`), not
 methods — `mind.ts` is a thin assembly that implements the context and
@@ -453,9 +454,9 @@ against the built `dist/` (`npm test`; one suite:
   silence). A "simplification" that fails an existing test is wrong until you
   can argue the _test_ is wrong — several guards exist precisely because a
   plausible simplification once failed a dozen suites.
-- Sublibraries test themselves (`src/{alu,derive,rabitq-hnsw}/test/`) with zero
+- Sublibraries test themselves (`src/{alu,derive,rabitq-ivf}/test/`) with zero
   Sema dependency. Keep it so.
-- Performance claims are tested (the rabitq-hnsw benchmark asserts sub-linear
+- Performance claims are tested (the rabitq-ivf benchmark asserts sub-linear
   scaling and compression). Changing index behaviour means running it.
 
 ---

package/CITATION.cff

@@ -0,0 +1,49 @@
+cff-version: 1.2.0
+message: >-
+  If you use Sema in academic work, please cite the accompanying preprint.
+  For software-specific references, you may also cite this repository.
+title: "Sema"
+abstract: >-
+  Sema is a non-parametric, instance-based reasoning system built as a
+  Vector Symbolic Architecture over content-addressable memory, with
+  inference performed by weighted automated deduction.
+type: software
+authors:
+  - family-names: "Viana"
+    given-names: "Henrique Emanoel"
+    orcid: "https://orcid.org/0000-0002-7119-5867"
+    affiliation: "NONE"
+repository-code: "https://github.com/hviana/sema"
+url: "https://github.com/hviana/sema"
+license: "PolyForm-Noncommercial-1.0.0"
+version: "0.1.0"
+date-released: "2026-07-16"
+keywords:
+  - artificial intelligence
+  - cognitive architecture
+  - non-parametric reasoning
+  - instance-based reasoning
+  - vector symbolic architecture
+  - hyperdimensional computing
+  - content-addressable memory
+  - weighted automated deduction
+  - symbolic reasoning
+  - explainable artificial intelligence
+  - TypeScript
+
+preferred-citation:
+  type: article
+  title: "Sema: Reasoning by Example over a Vector-Symbolic Memory"
+  authors:
+    - family-names: "Viana"
+      given-names: "Henrique Emanoel"
+      orcid: "https://orcid.org/0000-0002-7119-5867"
+      affiliation: "NONE"
+  year: 2026
+  month: 7
+  day: 16
+  version: "0.1.0"
+  doi: "10.5281/zenodo.21385723"
+  url: "https://doi.org/10.5281/zenodo.21385723"
+  publisher:
+    name: "Zenodo"

package/HOW_IT_WORKS.md

@@ -575,11 +575,13 @@ controlled amount of recall for sub-linear query time.
 
 Sema uses two established techniques in combination:
 
-- **HNSW** — Hierarchical Navigable Small World graphs (Malkov & Yashunin 2018).
-  Vectors are connected into a multi-layer proximity graph; a query descends
-  greedily from a sparse top layer to a dense bottom layer, examining only a
-  logarithmic-ish neighbourhood of the collection. Empirically the work grows
-  around N^0.3 in Sema's configuration — decisively sub-linear.
+- **IVF (inverted-file partitioning)** — the collection is split into clusters,
+  each with a binary pivot code; a query ranks the pivots and scans only the few
+  nearest clusters. Cluster size is bounded (an oversized cluster
+  deterministically splits in two), so the work per query is set by the number
+  of probes, not by the collection — decisively sub-linear. Inserting is
+  route-and-append: one RAM scan of the pivot table, no graph maintenance, so
+  ingestion cost stays flat as the collection grows.
 - **RaBitQ 1-bit quantization** (Gao & Long 2024). Each stored vector is
   randomly rotated and reduced to one _sign bit_ per dimension — a 32×
   compression — with an unbiased, theoretically-grounded estimator of the
@@ -663,7 +665,7 @@ RELATIONS             │  continuation edges (what follows what)  │
 (distributional, §4)  │  halos (what company each node keeps)    │
                       ├─────────────────────────────────────────┤
 INDEXES               │  gist index + halo index                 │
-(ANN, §6)             │  (HNSW over 1-bit RaBitQ codes)          │
+(ANN, §6)             │  (IVF over 1-bit RaBitQ codes)           │
                       └──────────────────┬──────────────────────┘
                                          │ axioms & rule candidates
                                          ▼
@@ -1696,7 +1698,7 @@ requirements:
 
 - **ANN index: 1-bit RaBitQ, irreversible.** The same halo vector, projected
   through a random rotation and reduced to one sign bit per dimension (32×
-  compression), serves as a search code in the HNSW index. This code is _never_
+  compression), serves as a search code in the IVF index. This code is _never_
   decoded back — it only answers "which halos are near this query?" The
   estimator is unbiased (expected cosine is recoverable from the bit count), so
   ranking quality is preserved despite the loss of reversibility.
@@ -4124,8 +4126,8 @@ terms of art borrowed from the literature.
   §4.
 - **Hash-consing** — constructing structures modulo equality so equal
   substructures are shared. §3.2.
-- **HNSW** — Hierarchical Navigable Small World graph; sub-linear ANN search
-  (Malkov & Yashunin 2018). §6.1.
+- **IVF** — inverted-file partitioned ANN index; bounded-probe sub-linear search
+  over clustered codes. §6.1.
 - **Hilbert curve** — the locality-preserving space-filling curve used to
   linearize grids. §6.3.
 - **Hyperdimensional computing** — Kanerva's (2009) umbrella term for computing
@@ -4193,9 +4195,6 @@ Foundations cited in this document, in alphabetical order:
   Processing Letters 6(1), 1–5.
 - Kolodner, J. L. (1992). _An Introduction to Case-Based Reasoning._ Artificial
   Intelligence Review 6, 3–34.
-- Malkov, Y. A. & Yashunin, D. A. (2018). _Efficient and Robust Approximate
-  Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs._ IEEE
-  TPAMI 42(4), 824–836.
 - Merkle, R. C. (1987). _A Digital Signature Based on a Conventional Encryption
   Function._ Proc. CRYPTO.
 - Plate, T. A. (1995). _Holographic Reduced Representations._ IEEE Transactions

package/README.md

@@ -1,3 +1,5 @@
+[![npm version](https://img.shields.io/npm/v/@hviana/sema.svg)](https://www.npmjs.com/package/@hviana/sema)
+
 <div align="center">
 
 ```
@@ -223,11 +225,11 @@ Pablo Picasso co-founded the Cubist movement
 Sema is composed of three self-contained, independently documented engines. They
 are fully decoupled — Sema reaches them only through interfaces.
 
-| Engine            | What it solves                                                                             | Result                                                                                                                       |
-| :---------------- | :----------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
-| **`rabitq-hnsw`** | _"Given a vector, find the nodes whose gist resonates with it — fast, at scale, on disk."_ | HNSW over 1-bit RaBitQ codes · ~32× compression · bounded RAM · sub-linear queries                                           |
-| **`derive`**      | _"Explore a huge implicit space of derivations and return the single lightest one."_       | adapted A\*LD (adapted A\* Lightest Derivation) over a weighted deduction hypergraph — Sema's thinking _is_ one call to this |
-| **`alu`**         | _"Compute, exactly and symbolically, the things that are rules, not facts."_               | A tiny irreducible kernel from which arithmetic, logic, and n-dimensional computation are derived                            |
+| Engine           | What it solves                                                                             | Result                                                                                                                       |
+| :--------------- | :----------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
+| **`rabitq-ivf`** | _"Given a vector, find the nodes whose gist resonates with it — fast, at scale, on disk."_ | Partitioned (IVF) index over 1-bit RaBitQ codes · ~32× compression · bounded RAM · flat inserts, bounded queries             |
+| **`derive`**     | _"Explore a huge implicit space of derivations and return the single lightest one."_       | adapted A\*LD (adapted A\* Lightest Derivation) over a weighted deduction hypergraph — Sema's thinking _is_ one call to this |
+| **`alu`**        | _"Compute, exactly and symbolically, the things that are rules, not facts."_               | A tiny irreducible kernel from which arithmetic, logic, and n-dimensional computation are derived                            |
 
 ---
 

package/dist/example/train_base.js

@@ -1298,20 +1298,23 @@ async function saveProgress(store, p) {
 // §10  Main
 // ═══════════════════════════════════════════════════════════════════════
 async function main() {
-    // The vector indices' single memory knob (MiB) — sizes both the SQLite page
-    // cache and the decoded-code LRU of each index.  Training is HNSW-insert
-    // bound at scale (profiled at >80% of ingest wall-clock on a 5.5M-vector
-    // store, nearly all of it cache-missing code reads), and inserts walk the
-    // graph, so a cache that covers the indexed codes cuts wall-clock directly
-    // (measured ~1.6× at 1 GB vs 64 MB on a 2M-node store).  1 GB caches ~5.8M
-    // D=1024 codes — the whole content index of a 350 MB-corpus store — and is
-    // the default; override with VECTOR_CACHE_MB (e.g. 256 on a small-RAM
-    // machine, 64 for the library default).
-    const VECTOR_CACHE_MB = Math.max(0, Number(env("VECTOR_CACHE_MB", "1024")));
+    // The vector indices' memory knob (MiB) — each index's SQLite page cache.
+    // The IVF index routes inserts through a RAM-resident pivot table and
+    // appends to chunk blobs, so this cache mostly serves query-time cluster
+    // scans; 256 MiB comfortably covers the probed working set of a trained
+    // store.  Override with VECTOR_CACHE_MB (64 is the library default).
+    const VECTOR_CACHE_MB = Math.max(0, Number(env("VECTOR_CACHE_MB", "256")));
+    // Page cache for the MAIN DAG database (node/kid/edge/contain tables).
+    // Training issues millions of content-addressed point probes per session
+    // against a GB-scale file; the library default (64 MiB) is sized for a
+    // small machine — a training box affords more.  Override with
+    // SQLITE_CACHE_MB.
+    const SQLITE_CACHE_MB = Math.max(0, Number(env("SQLITE_CACHE_MB", "256")));
     const store = new SQliteStore({
         path: DB_PATH,
         D,
         vectorCacheMb: VECTOR_CACHE_MB,
+        sqliteCacheMb: SQLITE_CACHE_MB,
     });
     // The store IS the model: memories, progress, and metadata all persist in
     // it, so a resumed run just reopens the same store and continues. Guard

package/dist/src/alu/src/index.js

@@ -1,7 +1,7 @@
 // alu — the ALU sub-library: a tiny irreducible kernel from which arithmetic,
 // logic, and numerical computation are all DERIVED.
 //
-// Self-contained in the spirit of derive/ and rabitq-hnsw/: it imports only the
+// Self-contained in the spirit of derive/ and rabitq-ivf/: it imports only the
 // pure byte helpers (../../bytes.js) and nothing else from SEMA.  The host
 // reaches meaning (operator synonymy, the symbolic inverse) through the injected
 // AluResonance, pre-resolved into a synchronous snapshot before the search runs.

package/dist/src/config.d.ts

@@ -26,18 +26,9 @@ export interface AlphabetConfig {
 }
 export interface StoreConfig {
     minHaloMass: number;
-    m: number;
-    efConstruction: number;
-    /** Construction budget for REACH-ONLY interior gists — the ~90% of content
-     *  index inserts that exist so a partial query can resonate a sub-region
-     *  and climb, never as dedup targets.  Deposit roots and halo-bearing
-     *  targets always build with the full `efConstruction`.  A smaller budget
-     *  here is the one deliberate speed-for-quality trade in ingestion (an
-     *  interior's layer-0 wiring is built from a narrower candidate beam);
-     *  the recall suite (partial recall, multi-topic attention, counterfactual
-     *  anchoring) is the gate for its value.  Set equal to `efConstruction`
-     *  to disable the trade. */
-    efConstructionInterior: number;
+    /** Query breadth of the IVF vector indices: clusters probed per query =
+     *  ceil(efSearch / 4).  Inserts have no quality knob — the partitioned
+     *  index routes and appends, so ingestion cost is flat by construction. */
     efSearch: number;
     /** Compact the in-memory vector indices after this many vectors are written.
      *  Compaction rebuilds an index from its live codes to reclaim the slots left
@@ -45,7 +36,7 @@ export interface StoreConfig {
      *  (not on a flush count that goes quiet during repeat-heavy training) keeps
      *  the index dense and query cost bounded. */
     compactEveryNWrites: number;
-    /** Over-fetch factor for HNSW queries (ANN recall cushion). */
+    /** Over-fetch factor for vector-index queries (ANN recall cushion). */
     overfetch: number;
     /** Combined buffered-write ceiling before a flush of both vector indices
      *  (content + halo). Higher ⇒ fewer, larger flushes into the in-memory
@@ -78,14 +69,20 @@ export interface StoreConfig {
      *  round-trips through the quantizer. An eviction or a reopen reads the
      *  2-bit row — the fidelity every cross-session consumer already gets. */
     haloCacheBytes: number;
-    /** Size, in MiB, of each `rabitq-hnsw` `VectorDatabase`'s memory budget
-     *  (forwarded as its `cacheSizeMb`).  It is the index's SINGLE memory knob: it
-     *  sizes both SQLite's page cache and the derived immutable-code LRU.  A PURE
-     *  latency optimisation — the index reads codes from SQLite on demand, so its
-     *  correctness and its asymptotic per-operation storage-read count are identical
-     *  with the budget at 0.  Exposed so a scaling test can set it to 0 and measure
+    /** Size, in MiB, of each `rabitq-ivf` `VectorDatabase`'s memory budget
+     *  (forwarded as its `cacheSizeMb` — its SQLite page cache).  A PURE latency
+     *  optimisation — the index reads chunk blobs from SQLite on demand, so its
+     *  correctness and its per-operation storage-read count are identical with
+     *  the budget at 0.  Exposed so a scaling test can set it to 0 and measure
      *  the honest, cache-independent cost. */
     vectorCacheMb: number;
+    /** Size, in MiB, of the MAIN DAG database's SQLite page cache.  The node /
+     *  kid / edge / contain tables serve millions of point probes per training
+     *  session (content-addressed findLeaf/findBranch, parent probes, contain
+     *  appends); SQLite's default cache (~2 MiB) thrashes once the DB outgrows
+     *  it, so every probe pays a file read.  A PURE latency knob — correctness
+     *  and result identical at any value. */
+    sqliteCacheMb: number;
     /** Max entries in the skipped-interior LRU set.  Interiors that
      *  {@link Store.indexSubtree} has already visited (indexed or skipped) are
      *  remembered here so subsequent calls prune their subtrees.  Session-local

package/dist/src/config.js

@@ -22,9 +22,6 @@ export const DEFAULT_CONFIG = {
     },
     store: {
         minHaloMass: 1,
-        m: 8,
-        efConstruction: 64,
-        efConstructionInterior: 16,
         efSearch: 64,
         compactEveryNWrites: 50_000,
         overfetch: 4,
@@ -36,6 +33,7 @@ export const DEFAULT_CONFIG = {
         pendingGistBytes: 16_000_000,
         haloCacheBytes: 16_000_000,
         vectorCacheMb: 64,
+        sqliteCacheMb: 64,
         coveredIdsMax: 100_000,
         chainCacheBytes: 16_000_000,
     },
@@ -63,11 +61,6 @@ export function resolveConfig(opts = {}) {
         },
         store: {
             minHaloMass: opts.store?.minHaloMass ?? DEFAULT_CONFIG.store.minHaloMass,
-            m: opts.store?.m ?? DEFAULT_CONFIG.store.m,
-            efConstruction: opts.store?.efConstruction ??
-                DEFAULT_CONFIG.store.efConstruction,
-            efConstructionInterior: opts.store?.efConstructionInterior ??
-                DEFAULT_CONFIG.store.efConstructionInterior,
             efSearch: opts.store?.efSearch ?? DEFAULT_CONFIG.store.efSearch,
             compactEveryNWrites: opts.store?.compactEveryNWrites ??
                 DEFAULT_CONFIG.store.compactEveryNWrites,
@@ -87,6 +80,8 @@ export function resolveConfig(opts = {}) {
                 DEFAULT_CONFIG.store.haloCacheBytes,
             vectorCacheMb: opts.store?.vectorCacheMb ??
                 DEFAULT_CONFIG.store.vectorCacheMb,
+            sqliteCacheMb: opts.store?.sqliteCacheMb ??
+                DEFAULT_CONFIG.store.sqliteCacheMb,
             coveredIdsMax: opts.store?.coveredIdsMax ??
                 DEFAULT_CONFIG.store.coveredIdsMax,
             chainCacheBytes: opts.store?.chainCacheBytes ??

package/dist/src/index.d.ts

@@ -10,6 +10,6 @@ export * from "./store-sqlite.js";
 export * from "./config.js";
 export * from "./extension.js";
 export * from "./ingest-cache.js";
-export { Heap, HnswIndex, Prng, RaBitQuantizer, VectorDatabase, } from "./rabitq-hnsw/src/index.js";
-export type { DatabaseOptions, ExternalId, HnswParams, KnnHit, QueryContext, QueryResult, RaBitQOptions, StorageStats, } from "./rabitq-hnsw/src/index.js";
+export { IvfIndex, Prng, RaBitQuantizer, VectorDatabase, } from "./rabitq-ivf/src/index.js";
+export type { DatabaseOptions, ExternalId, IvfConfig, IvfHit, QueryContext, QueryResult, RaBitQOptions, StorageStats, } from "./rabitq-ivf/src/index.js";
 export * from "./derive/src/index.js";

package/dist/src/index.js

@@ -12,10 +12,6 @@ export * from "./store-sqlite.js";
 export * from "./config.js";
 export * from "./extension.js";
 export * from "./ingest-cache.js";
-// rabitq-hnsw is re-exported selectively: its `Store`, `NodeRec`, and
-// `StoreConfig` are internal names that collide with sema's own top-level
-// `store.js` / `config.js` exports, and sema code that needs the rabitq ones
-// imports them from the subpath directly. Re-export the public vector-DB
-// surface under the sema root, omitting the three colliding names.
-export { Heap, HnswIndex, Prng, RaBitQuantizer, VectorDatabase, } from "./rabitq-hnsw/src/index.js";
+// rabitq-ivf: the partitioned (IVF) vector index over 1-bit RaBitQ codes.
+export { IvfIndex, Prng, RaBitQuantizer, VectorDatabase, } from "./rabitq-ivf/src/index.js";
 export * from "./derive/src/index.js";

package/dist/src/rabitq-ivf/src/database.d.ts

@@ -0,0 +1,113 @@
+/** External ids are integers only. */
+export type ExternalId = number;
+export interface DatabaseOptions {
+    /** Path to the SQLite database file (use ":memory:" for a transient store). */
+    dbPath: string;
+    /** Vector dimensionality. Required for a new database; ignored when reopening. */
+    dim?: number;
+    /** Query breadth: clusters probed per query = ceil(efSearch / 4).  Default
+     *  64 (16 probes). Tunable at runtime and at reopen. */
+    efSearch?: number;
+    /** Bits used to scalar-quantise the query for the RaBitQ estimator.
+     *  Query-side only (stored codes are independent of it), so it is tunable
+     *  at reopen, like efSearch.  Default 8. */
+    queryBits?: number;
+    /** Number of sign-flip + Hadamard rounds in the random rotation. Default 3. */
+    rotationRounds?: number;
+    /** Seed for the rotation. Default fixed. */
+    seed?: number;
+    /** Optional centroid the vectors are centered by before quantisation. */
+    centroid?: ArrayLike<number>;
+    /** SQLite page-cache budget in MiB.  Purely a latency knob: correctness and
+     *  the per-operation storage-read count are identical with it at 0. */
+    cacheSizeMb?: number;
+}
+export interface QueryResult {
+    id: ExternalId;
+    /** Estimated cosine distance (1 - cosine). */
+    distance: number;
+}
+export interface StorageStats {
+    float32BytesPerVector: number;
+    codeBytesPerVector: number;
+    bytesPerVector: number;
+    compressionRatio: number;
+}
+/**
+ * A persistent vector database: an adaptive PARTITIONED (IVF) index over
+ * 1-bit RaBitQ codes (cosine), stored entirely in SQLite at `dbPath`.  See
+ * ivf.ts for the index design.  The original float vectors are never
+ * retained — only the sign codes — so a 1024-d vector costs 128 bytes.
+ *
+ * Inserting is route-and-append: cost is essentially FLAT in collection size
+ * (one RAM scan of the pivot table + one chunk append — no per-insert graph
+ * walk).  Query cost is bounded by nprobe × cluster size — constant in N
+ * once the collection is past its first splits.
+ */
+export declare class VectorDatabase {
+    private readonly index;
+    constructor(options: DatabaseOptions);
+    get dim(): number;
+    /** Number of live (non-deleted) vectors. */
+    get size(): number;
+    /** Physical slot count including tombstones from deletes/updates. */
+    get physicalSize(): number;
+    /** Number of clusters (partitions) currently in the index. */
+    get clusterCount(): number;
+    get efSearch(): number;
+    set efSearch(value: number);
+    /** Distance computations performed during the most recent query. */
+    get lastQueryDistanceComputations(): number;
+    /** Storage row reads issued by the most recent query — the honest,
+     *  cache-independent scalability metric. */
+    get lastQueryStorageReads(): number;
+    get storage(): StorageStats;
+    private checkId;
+    /** Convert a value to code bytes, selecting by length: `codeWords`
+     *  elements → an existing 1-bit code; otherwise a raw `dim`-vector. */
+    private toCodeBytes;
+    has(id: ExternalId): boolean;
+    /** Stream every live external id (bounded memory). */
+    keys(): IterableIterator<ExternalId>;
+    /** Stream live entries whose INTERNAL id is > `after` — a durable
+     *  incremental watermark (internal ids are monotone at insert and preserved
+     *  by update and compact). */
+    keysSince(after: number): IterableIterator<{
+        ext: ExternalId;
+        internal: number;
+    }>;
+    /** Read the stored 1-bit code for an id (a copy as a Uint32Array), or null. */
+    get(id: ExternalId): Uint32Array | null;
+    insert(id: ExternalId, value: ArrayLike<number>): void;
+    update(id: ExternalId, value: ArrayLike<number>): void;
+    upsert(id: ExternalId, value: ArrayLike<number>): void;
+    /**
+     * Upsert many vectors under ONE transaction — one WAL commit for the whole
+     * batch instead of one per vector, with chunk appends to the same cluster
+     * coalesced in the index's write-back buffer.
+     */
+    upsertMany(entries: Array<{
+        id: ExternalId;
+        vector: ArrayLike<number>;
+    }>): void;
+    /** Delete many ids under ONE transaction. Absent ids are skipped.
+     *  Returns the number of vectors actually removed. */
+    deleteMany(ids: ExternalId[]): number;
+    /** Delete the vector bound to an id. Returns false if absent. */
+    delete(id: ExternalId): boolean;
+    /** Heat the SQLite page cache with sequential scans (latency only). */
+    warmCache(): number;
+    /** Reclaim tombstoned slots by rewriting the clusters that carry any, then
+     *  VACUUM.  Internal ids and cluster assignment are preserved. */
+    compact(): void;
+    /**
+     * k-NN search. The argument's length selects the mode:
+     *  - `dim` elements       → a raw vector (accurate estimator)
+     *  - `codeWords` elements → a 1-bit code, by sign-bit Hamming
+     */
+    query(query: ArrayLike<number>, k?: number, opts?: {
+        ef?: number;
+    }): QueryResult[];
+    /** Close the underlying database. The instance must not be used afterwards. */
+    close(): void;
+}

package/dist/src/rabitq-ivf/src/database.js

@@ -0,0 +1,201 @@
+import { IvfIndex } from "./ivf.js";
+/**
+ * A persistent vector database: an adaptive PARTITIONED (IVF) index over
+ * 1-bit RaBitQ codes (cosine), stored entirely in SQLite at `dbPath`.  See
+ * ivf.ts for the index design.  The original float vectors are never
+ * retained — only the sign codes — so a 1024-d vector costs 128 bytes.
+ *
+ * Inserting is route-and-append: cost is essentially FLAT in collection size
+ * (one RAM scan of the pivot table + one chunk append — no per-insert graph
+ * walk).  Query cost is bounded by nprobe × cluster size — constant in N
+ * once the collection is past its first splits.
+ */
+export class VectorDatabase {
+    index;
+    constructor(options) {
+        if (!options || typeof options.dbPath !== "string" ||
+            options.dbPath.length === 0) {
+            throw new Error("DatabaseOptions.dbPath (string) is required");
+        }
+        this.index = new IvfIndex(options.dbPath, options);
+    }
+    get dim() {
+        return this.index.dim;
+    }
+    /** Number of live (non-deleted) vectors. */
+    get size() {
+        return this.index.size;
+    }
+    /** Physical slot count including tombstones from deletes/updates. */
+    get physicalSize() {
+        return this.index.physicalSize;
+    }
+    /** Number of clusters (partitions) currently in the index. */
+    get clusterCount() {
+        return this.index.clusterCount;
+    }
+    get efSearch() {
+        return this.index.efSearch;
+    }
+    set efSearch(value) {
+        this.index.efSearch = value;
+    }
+    /** Distance computations performed during the most recent query. */
+    get lastQueryDistanceComputations() {
+        return this.index.lastQueryDistComps;
+    }
+    /** Storage row reads issued by the most recent query — the honest,
+     *  cache-independent scalability metric. */
+    get lastQueryStorageReads() {
+        return this.index.lastQueryStorageReads;
+    }
+    get storage() {
+        const f32 = this.dim * 4;
+        const bpv = this.index.bytesPerVector;
+        return {
+            float32BytesPerVector: f32,
+            codeBytesPerVector: bpv,
+            bytesPerVector: bpv,
+            compressionRatio: f32 / bpv,
+        };
+    }
+    checkId(id) {
+        // 32-bit signed range: chunk blobs store exts as int32 — a wider id
+        // would silently truncate, so it is rejected at the door instead.
+        if (!Number.isInteger(id) || id > 0x7fffffff || id < -0x80000000) {
+            throw new Error(`External id must be a 32-bit integer, got ${id}`);
+        }
+        return id;
+    }
+    /** Convert a value to code bytes, selecting by length: `codeWords`
+     *  elements → an existing 1-bit code; otherwise a raw `dim`-vector. */
+    toCodeBytes(value) {
+        if (value.length === this.index.codeWords) {
+            return this.index.codeToBytes(value);
+        }
+        if (value.length !== this.dim) {
+            throw new Error(`Vector dimension mismatch: expected ${this.dim}, got ${value.length}`);
+        }
+        return this.index.encodeToBytes(value);
+    }
+    has(id) {
+        return this.index.has(this.checkId(id));
+    }
+    /** Stream every live external id (bounded memory). */
+    *keys() {
+        yield* this.index.keys();
+    }
+    /** Stream live entries whose INTERNAL id is > `after` — a durable
+     *  incremental watermark (internal ids are monotone at insert and preserved
+     *  by update and compact). */
+    *keysSince(after) {
+        yield* this.index.keysSince(after);
+    }
+    /** Read the stored 1-bit code for an id (a copy as a Uint32Array), or null. */
+    get(id) {
+        const bytes = this.index.codeOf(this.checkId(id));
+        return bytes ? this.index.bytesToCode(bytes) : null;
+    }
+    insert(id, value) {
+        this.index.begin();
+        try {
+            this.index.insert(this.checkId(id), this.toCodeBytes(value));
+            this.index.commit();
+        }
+        catch (e) {
+            this.index.rollback();
+            throw e;
+        }
+    }
+    update(id, value) {
+        this.index.begin();
+        try {
+            this.index.update(this.checkId(id), this.toCodeBytes(value));
+            this.index.commit();
+        }
+        catch (e) {
+            this.index.rollback();
+            throw e;
+        }
+    }
+    upsert(id, value) {
+        this.upsertMany([{ id, vector: value }]);
+    }
+    /**
+     * Upsert many vectors under ONE transaction — one WAL commit for the whole
+     * batch instead of one per vector, with chunk appends to the same cluster
+     * coalesced in the index's write-back buffer.
+     */
+    upsertMany(entries) {
+        if (entries.length === 0)
+            return;
+        this.index.begin();
+        try {
+            for (const e of entries) {
+                // One shared vmap probe decides insert vs update — the per-row point
+                // query is the dominant bulk-load cost, so it is never duplicated.
+                this.index.upsert(this.checkId(e.id), this.toCodeBytes(e.vector));
+            }
+            this.index.commit();
+        }
+        catch (e) {
+            this.index.rollback();
+            throw e;
+        }
+    }
+    /** Delete many ids under ONE transaction. Absent ids are skipped.
+     *  Returns the number of vectors actually removed. */
+    deleteMany(ids) {
+        if (ids.length === 0)
+            return 0;
+        let removed = 0;
+        this.index.begin();
+        try {
+            for (const id of ids) {
+                if (this.index.remove(this.checkId(id)))
+                    removed++;
+            }
+            this.index.commit();
+        }
+        catch (e) {
+            this.index.rollback();
+            throw e;
+        }
+        return removed;
+    }
+    /** Delete the vector bound to an id. Returns false if absent. */
+    delete(id) {
+        return this.deleteMany([id]) === 1;
+    }
+    /** Heat the SQLite page cache with sequential scans (latency only). */
+    warmCache() {
+        return this.index.warmCache();
+    }
+    /** Reclaim tombstoned slots by rewriting the clusters that carry any, then
+     *  VACUUM.  Internal ids and cluster assignment are preserved. */
+    compact() {
+        this.index.compact();
+    }
+    /**
+     * k-NN search. The argument's length selects the mode:
+     *  - `dim` elements       → a raw vector (accurate estimator)
+     *  - `codeWords` elements → a 1-bit code, by sign-bit Hamming
+     */
+    query(query, k = 10, opts) {
+        let hits;
+        if (query.length === this.index.codeWords) {
+            hits = this.index.searchKnnByCode(this.index.codeToBytes(query), k, opts?.ef);
+        }
+        else {
+            if (query.length !== this.dim) {
+                throw new Error(`Vector dimension mismatch: expected ${this.dim}, got ${query.length}`);
+            }
+            hits = this.index.searchKnn(query, k, opts?.ef);
+        }
+        return hits;
+    }
+    /** Close the underlying database. The instance must not be used afterwards. */
+    close() {
+        this.index.close();
+    }
+}