eigen-db 4.0.2 → 4.2.0

package/CHANGELOG.md

@@ -1,3 +1,11 @@
+# v4.2.0
+
+Added: import/export in binary format
+
+# v4.1.0
+
+Added: performance optimization in WASM
+
 # v4.0.2
 
 Fixed: `query()` overload type hint issue

package/README.md

@@ -50,11 +50,11 @@ Notes:
 ```ts
 const queryVector = embeddingQuery;
 
-// Returns a plain array of { key, score } sorted by similarity
+// Returns a plain array of { key, similarity } sorted by descending similarity
 const results = db.query(queryVector, { topK: 10 });
 
-for (const { key, score } of results) {
-  console.log(key, score);
+for (const { key, similarity } of results) {
+  console.log(key, similarity);
 }
 ```
 
@@ -64,15 +64,25 @@ For lazy iteration (useful for pagination or early stopping):
 const results = db.query(queryVector, { topK: 100, iterable: true });
 
 // Iterate and break early — keys are resolved on demand
-for (const { key, score } of results) {
-  if (score < 0.5) break;
-  console.log(key, score);
+for (const { key, similarity } of results) {
+  if (similarity < 0.5) break;
+  console.log(key, similarity);
 }
 
 // Or spread into an array when you need all results
 const all = [...results];
 ```
 
+Use `minSimilarity` to automatically cut off results below a threshold:
+
+```ts
+// Only return results with similarity ≥ 0.7 (inclusive)
+const results = db.query(queryVector, { minSimilarity: 0.7 });
+
+// Works with iterable mode too — iteration stops early at the threshold
+const results = db.query(queryVector, { minSimilarity: 0.7, iterable: true });
+```
+
 ### 4) Persist and lifecycle
 
 ```ts
@@ -86,6 +96,55 @@ To delete all vectors and storage:
 await db.clear();
 ```
 
+### 5) Export and import
+
+Export the entire database as a streaming binary file:
+
+```ts
+const stream = await db.export(); // ReadableStream<Uint8Array>
+
+// In a browser — download as a file
+const response = new Response(stream);
+const blob = await response.blob();
+const url = URL.createObjectURL(blob);
+const a = document.createElement("a");
+a.href = url;
+a.download = "database.bin";
+a.click();
+```
+
+Import from a stream, replacing all existing data:
+
+```ts
+// From a File (e.g., <input type="file">)
+await db.import(file.stream());
+
+// From a fetch response
+const res = await fetch("/path/to/database.bin");
+await db.import(res.body!);
+```
+
+Notes:
+
+- `import()` replaces all existing data in the target database.
+- A dimension check is performed on import: the stream must contain data exported from a database with the same `dimensions` setting.
+- Both methods use the Web Streams API to avoid large heap allocations — vectors are streamed in 64KB chunks.
+
+## Similarity metric
+
+Similarity is the dot product of the query and stored vectors.
+
+- **With normalization enabled** (the default): vectors are L2-normalized before storage and query, so the dot product equals cosine similarity. Similarity ranges from **1** (identical) to **-1** (opposite), with **0** indicating orthogonal vectors.
+- **With normalization disabled** (`normalize: false`): the dot product is computed on raw vectors. The range depends on the magnitude of your vectors. Use this mode when your vectors are already normalized or when you want raw dot-product semantics.
+
+**When to normalize:**
+
+| Scenario | Normalize? | Notes |
+| --- | --- | --- |
+| Using embeddings from OpenAI, Cohere, etc. | `true` (default) | Embeddings may not be unit-length; normalization ensures cosine similarity. |
+| Vectors are already unit-length | Either | Setting `false` avoids redundant work. |
+| You need raw dot-product semantics | `false` | Similarity will be the raw dot product; range depends on vector magnitudes. |
+
 ## Full API Reference
 
 ## Exports
@@ -94,7 +153,7 @@ await db.clear();
 export { DB };
 export type { ResultItem };
 export { VectorCapacityExceededError };
-export type { OpenOptions, OpenOptionsInternal, SetOptions, QueryOptions, IterableQueryOptions, VectorInput };
+export type { OpenOptions, OpenOptionsInternal, SetOptions, QueryOptions, VectorInput };
 export { InMemoryStorageProvider, OPFSStorageProvider };
 export type { StorageProvider };
 ```
@@ -126,9 +185,9 @@ Opens (or creates) a database instance and loads persisted data.
 - `getMany(keys: string[]): (number[] | undefined)[]`
   - Batch lookup.
 - `query(value: VectorInput, options?: QueryOptions): ResultItem[]`
-  - Returns similarity-ranked results as a plain array.
+  - Returns results sorted by descending similarity as a plain array.
   - Throws on dimension mismatch.
-- `query(value: VectorInput, options: IterableQueryOptions): Iterable<ResultItem>`
+- `query(value: VectorInput, options: QueryOptions & { iterable: true }): Iterable<ResultItem>`
   - With `{ iterable: true }`, returns a lazy iterable. Keys are resolved
     only as each item is consumed, enabling early stopping and pagination.
   - Throws on dimension mismatch.
@@ -139,16 +198,23 @@ Opens (or creates) a database instance and loads persisted data.
   - Subsequent operations throw.
 - `clear(): Promise<void>`
   - Clears in-memory state and destroys storage for this DB.
+- `export(): Promise<ReadableStream<Uint8Array>>`
+  - Exports the entire database as a streaming binary. Vectors are streamed in 64KB chunks.
+- `import(stream: ReadableStream<Uint8Array>): Promise<void>`
+  - Imports data from a stream, replacing all existing data.
+  - Throws on dimension mismatch between the stream data and the database.
 
 ### `ResultItem`
 
 ```ts
 interface ResultItem {
   key: string;
-  score: number;
+  similarity: number;
 }
 ```
 
+- `similarity` — The dot product of query and stored vectors. With normalization (default), this is cosine similarity: 1 = identical, -1 = opposite.
+
 ### Option types
 
 #### `OpenOptions`
@@ -190,16 +256,10 @@ interface SetOptions {
 
 ```ts
 interface QueryOptions {
-  topK?: number; // default: all vectors
+  topK?: number; // default: Infinity (all results)
+  minSimilarity?: number; // inclusive lower bound on similarity; results below this are excluded
   normalize?: boolean;
-}
-```
-
-#### `IterableQueryOptions`
-
-```ts
-interface IterableQueryOptions extends QueryOptions {
-  iterable: true; // returns Iterable<ResultItem> instead of ResultItem[]
+  iterable?: boolean; // when true, returns Iterable<ResultItem> instead of ResultItem[]
 }
 ```
 
@@ -238,8 +298,41 @@ new InMemoryStorageProvider();
 
 Thrown when memory growth would exceed WASM 32-bit memory limits for the configured dimension size.
 
+## Benchmark results
+
+WASM SIMD vs pure JavaScript performance on 1536-dimensional vectors (OpenAI embedding size), measured with `vitest bench` (Node.js):
+
+| Operation | JS (ops/s) | WASM SIMD (ops/s) | Speedup |
+| --- | --- | --- | --- |
+| normalize (1536 dims) | 223,117 | 2,226,734 | **~10×** |
+| searchAll (100 vectors × 1536 dims) | 3,429 | 77,130 | **~22×** |
+| searchAll (1,000 vectors × 1536 dims) | 344 | 8,009 | **~23×** |
+| searchAll (10,000 vectors × 1536 dims) | 34 | 398 | **~12×** |
+
+The WASM SIMD layer uses 2-vector outer loop unrolling (halving query memory reads) and 4× inner loop unrolling with multiple independent accumulators.
+
+### Running benchmarks
+
+**Node.js** (via vitest):
+
+```bash
+npm run bench
+```
+
+**Browser**: start the dev server and navigate to the benchmark page:
+
+```bash
+npm run dev
+# Open http://localhost:5173/bench.html
+```
+
 ## Practical notes
 
-- Similarity is dot product; with normalization enabled (default), this behaves like cosine similarity.
+- Similarity is the dot product of query and stored vectors; with normalization enabled (default), this behaves like cosine similarity (1 = identical, -1 = opposite).
+- `topK` defaults to `Infinity`, returning all stored vectors sorted by similarity. Use `minSimilarity` to limit results by proximity.
 - Querying an empty database returns an empty array (`[]`).
 - `flush()` writes deduplicated state, and reopen preserves key-to-slot mapping.
+
+## Related
+
+- Just need cosine similarity? Try [fast-theta](https://github.com/chuanqisun/fast-theta).