eigen-db 4.1.0 → 4.3.0

package/CHANGELOG.md

@@ -1,3 +1,11 @@
+# v4.3.0
+
+Added: option to choose between in memory or OPFS storage backends
+
+# v4.2.0
+
+Added: import/export in binary format
+
 # v4.1.0
 
 Added: performance optimization in WASM

package/README.md

@@ -4,7 +4,7 @@ High-performance vector database for the web.
 
 `eigen-db` stores and queries embedding vectors in-browser, using:
 
-- OPFS (Origin Private File System) for persistence
+- Pluggable storage backends (in-memory by default, OPFS for browser persistence)
 - WASM SIMD for fast compute when available
 - JavaScript fallback when WASM SIMD is unavailable
 
@@ -21,13 +21,24 @@ npm install eigen-db
 ```ts
 import { DB } from "eigen-db";
 
+// In-memory (default) — no persistence, great for ephemeral sessions
 const db = await DB.open({
-  name: "my-index", // optional, defaults to "default"
   dimensions: 1536, // required
   normalize: true, // optional, defaults to true
 });
 ```
 
+For browser persistence, mount an OPFS storage backend:
+
+```ts
+import { DB, OPFSStorageProvider } from "eigen-db";
+
+const db = await DB.open({
+  dimensions: 1536,
+  storage: new OPFSStorageProvider("my-index"), // persistent OPFS directory
+});
+```
+
 ### 2) Insert vectors
 
 ```ts
@@ -50,11 +61,11 @@ Notes:
 ```ts
 const queryVector = embeddingQuery;
 
-// Returns a plain array of { key, distance } sorted by ascending distance
+// Returns a plain array of { key, similarity } sorted by descending similarity
 const results = db.query(queryVector, { topK: 10 });
 
-for (const { key, distance } of results) {
-  console.log(key, distance);
+for (const { key, similarity } of results) {
+  console.log(key, similarity);
 }
 ```
 
@@ -64,23 +75,23 @@ For lazy iteration (useful for pagination or early stopping):
 const results = db.query(queryVector, { topK: 100, iterable: true });
 
 // Iterate and break early — keys are resolved on demand
-for (const { key, distance } of results) {
-  if (distance > 0.5) break;
-  console.log(key, distance);
+for (const { key, similarity } of results) {
+  if (similarity < 0.5) break;
+  console.log(key, similarity);
 }
 
 // Or spread into an array when you need all results
 const all = [...results];
 ```
 
-Use `maxDistance` to automatically cut off results beyond a threshold:
+Use `minSimilarity` to automatically cut off results below a threshold:
 
 ```ts
-// Only return results within distance 0.3 (inclusive)
-const results = db.query(queryVector, { maxDistance: 0.3 });
+// Only return results with similarity ≥ 0.7 (inclusive)
+const results = db.query(queryVector, { minSimilarity: 0.7 });
 
 // Works with iterable mode too — iteration stops early at the threshold
-const results = db.query(queryVector, { maxDistance: 0.3, iterable: true });
+const results = db.query(queryVector, { minSimilarity: 0.7, iterable: true });
 ```
 
 ### 4) Persist and lifecycle
@@ -96,20 +107,54 @@ To delete all vectors and storage:
 await db.clear();
 ```
 
-## Distance metric
+### 5) Export and import
+
+Export the entire database as a streaming binary file:
+
+```ts
+const stream = await db.export(); // ReadableStream<Uint8Array>
+
+// In a browser — download as a file
+const response = new Response(stream);
+const blob = await response.blob();
+const url = URL.createObjectURL(blob);
+const a = document.createElement("a");
+a.href = url;
+a.download = "database.bin";
+a.click();
+```
+
+Import from a stream, replacing all existing data:
+
+```ts
+// From a File (e.g., <input type="file">)
+await db.import(file.stream());
+
+// From a fetch response
+const res = await fetch("/path/to/database.bin");
+await db.import(res.body!);
+```
 
-Distance is defined as `1 - dotProduct(query, stored)`.
+Notes:
 
-- **With normalization enabled** (the default): vectors are L2-normalized before storage and query, so the dot product equals cosine similarity. Distance then equals **cosine distance**, ranging from **0** (identical) to **2** (opposite).
-- **With normalization disabled** (`normalize: false`): the dot product is computed on raw vectors. Distance is `1 - dotProduct`, which is not a standard metric and its range depends on the magnitude of your vectors. Use this mode when your vectors are already normalized or when you want raw dot-product semantics.
+- `import()` replaces all existing data in the target database.
+- A dimension check is performed on import: the stream must contain data exported from a database with the same `dimensions` setting.
+- Both methods use the Web Streams API to avoid large heap allocations — vectors are streamed in 64KB chunks.
+
+## Similarity metric
+
+Similarity is the dot product of the query and stored vectors.
+
+- **With normalization enabled** (the default): vectors are L2-normalized before storage and query, so the dot product equals cosine similarity. Similarity ranges from **1** (identical) to **-1** (opposite), with **0** indicating orthogonal vectors.
+- **With normalization disabled** (`normalize: false`): the dot product is computed on raw vectors. The range depends on the magnitude of your vectors. Use this mode when your vectors are already normalized or when you want raw dot-product semantics.
 
 **When to normalize:**
 
 | Scenario | Normalize? | Notes |
 | --- | --- | --- |
-| Using embeddings from OpenAI, Cohere, etc. | `true` (default) | Embeddings may not be unit-length; normalization ensures cosine distance. |
+| Using embeddings from OpenAI, Cohere, etc. | `true` (default) | Embeddings may not be unit-length; normalization ensures cosine similarity. |
 | Vectors are already unit-length | Either | Setting `false` avoids redundant work. |
-| You need raw dot-product semantics | `false` | Distance will be `1 - dotProduct`; range depends on vector magnitudes. |
+| You need raw dot-product semantics | `false` | Similarity will be the raw dot product; range depends on vector magnitudes. |
 
 ## Full API Reference
 
@@ -151,7 +196,7 @@ Opens (or creates) a database instance and loads persisted data.
 - `getMany(keys: string[]): (number[] | undefined)[]`
   - Batch lookup.
 - `query(value: VectorInput, options?: QueryOptions): ResultItem[]`
-  - Returns results sorted by ascending distance as a plain array.
+  - Returns results sorted by descending similarity as a plain array.
   - Throws on dimension mismatch.
 - `query(value: VectorInput, options: QueryOptions & { iterable: true }): Iterable<ResultItem>`
   - With `{ iterable: true }`, returns a lazy iterable. Keys are resolved
@@ -164,17 +209,22 @@ Opens (or creates) a database instance and loads persisted data.
   - Subsequent operations throw.
 - `clear(): Promise<void>`
   - Clears in-memory state and destroys storage for this DB.
+- `export(): Promise<ReadableStream<Uint8Array>>`
+  - Exports the entire database as a streaming binary. Vectors are streamed in 64KB chunks.
+- `import(stream: ReadableStream<Uint8Array>): Promise<void>`
+  - Imports data from a stream, replacing all existing data.
+  - Throws on dimension mismatch between the stream data and the database.
 
 ### `ResultItem`
 
 ```ts
 interface ResultItem {
   key: string;
-  distance: number;
+  similarity: number;
 }
 ```
 
-- `distance` — Defined as `1 - dotProduct`. With normalization (default), this is cosine distance: 0 = identical, 2 = opposite.
+- `similarity` — The dot product of query and stored vectors. With normalization (default), this is cosine similarity: 1 = identical, -1 = opposite.
 
 ### Option types
 
@@ -182,9 +232,9 @@ interface ResultItem {
 
 ```ts
 interface OpenOptions {
-  name?: string; // OPFS directory name, default: "default"
   dimensions: number; // vector size
   normalize?: boolean; // default: true
+  storage?: StorageProvider; // default: InMemoryStorageProvider
 }
 ```
 
@@ -194,12 +244,10 @@ Advanced/testing override options.
 
 ```ts
 interface OpenOptionsInternal extends OpenOptions {
-  storage?: StorageProvider;
   wasmBinary?: Uint8Array | null;
 }
 ```
 
-- `storage`: provide custom storage implementation (for example, tests)
 - `wasmBinary`:
   - `Uint8Array`: use provided precompiled WASM
   - `null`: force JavaScript-only compute
@@ -218,7 +266,7 @@ interface SetOptions {
 ```ts
 interface QueryOptions {
   topK?: number; // default: Infinity (all results)
-  maxDistance?: number; // inclusive upper bound on distance; results beyond this are excluded
+  minSimilarity?: number; // inclusive lower bound on similarity; results below this are excluded
   normalize?: boolean;
   iterable?: boolean; // when true, returns Iterable<ResultItem> instead of ResultItem[]
 }
@@ -289,7 +337,11 @@ npm run dev
 
 ## Practical notes
 
-- Distance is `1 - dotProduct`; with normalization enabled (default), this behaves like cosine distance (0 = identical, 2 = opposite).
-- `topK` defaults to `Infinity`, returning all stored vectors sorted by distance. Use `maxDistance` to limit results by proximity.
+- Similarity is the dot product of query and stored vectors; with normalization enabled (default), this behaves like cosine similarity (1 = identical, -1 = opposite).
+- `topK` defaults to `Infinity`, returning all stored vectors sorted by similarity. Use `minSimilarity` to limit results by proximity.
 - Querying an empty database returns an empty array (`[]`).
 - `flush()` writes deduplicated state, and reopen preserves key-to-slot mapping.
+
+## Related
+
+- Just need cosine similarity? Try [fast-theta](https://github.com/chuanqisun/fast-theta).