verso-db 0.1.1

package/CHANGELOG.md

@@ -0,0 +1,46 @@
+## [0.1.1](https://github.com/briansunter/verso/compare/v0.1.0...v0.1.1) (2025-12-29)
+
+
+### Bug Fixes
+
+* update package for fresh publish ([62f7bb9](https://github.com/briansunter/verso/commit/62f7bb9d13bd38fea3ae5c01053bdbc59222f58b))
+
+# [0.1.0](https://github.com/briansunter/verso/compare/v0.0.1...v0.1.0) (2025-12-29)
+
+
+### Features
+
+* initial release of verso-db ([2afb903](https://github.com/briansunter/verso/commit/2afb903df981e940489909510fbb5e65c3f29ab1)), closes [Hi#performance](https://github.com/Hi/issues/performance)
+
+## [0.0.1](https://github.com/briansunter/verso/compare/v0.0.0...v0.0.1) (2025-12-27)
+
+
+### Bug Fixes
+
+* prepare for initial release ([6740745](https://github.com/briansunter/verso/commit/67407459b261e3d40aee108800ca151c94a46b4f))
+* trigger release with clean tag history ([d57f9a9](https://github.com/briansunter/verso/commit/d57f9a90d3734992cc698cc208cfa58fd7f7c07d))
+
+# [1.1.0](https://github.com/briansunter/verso/compare/v1.0.1...v1.1.0) (2025-12-27)
+
+
+### Bug Fixes
+
+* initial release of verso-db v0.0.1 ([2de2e36](https://github.com/briansunter/verso/commit/2de2e36b043d450ac13fe71f3bd21d404efffd53)), closes [Hi#performance](https://github.com/Hi/issues/performance)
+* prepare for initial release ([6740745](https://github.com/briansunter/verso/commit/67407459b261e3d40aee108800ca151c94a46b4f))
+
+
+### Features
+
+* initial release of verso-db v0.0.1 ([cbcbf70](https://github.com/briansunter/verso/commit/cbcbf70c412e8fff9170d9385d761e3b85bf9914)), closes [Hi#performance](https://github.com/Hi/issues/performance)
+
+# [1.1.0](https://github.com/briansunter/verso/compare/v1.0.1...v1.1.0) (2025-12-27)
+
+
+### Bug Fixes
+
+* initial release of verso-db v0.0.1 ([2de2e36](https://github.com/briansunter/verso/commit/2de2e36b043d450ac13fe71f3bd21d404efffd53)), closes [Hi#performance](https://github.com/Hi/issues/performance)
+
+
+### Features
+
+* initial release of verso-db v0.0.1 ([cbcbf70](https://github.com/briansunter/verso/commit/cbcbf70c412e8fff9170d9385d761e3b85bf9914)), closes [Hi#performance](https://github.com/Hi/issues/performance)

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Brian Sunter
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,252 @@
+# Verso
+
+High-performance vector search with HNSW indexing for Bun and Browser.
+
+[![npm version](https://img.shields.io/npm/v/verso.svg)](https://www.npmjs.com/package/verso)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+## Performance
+
+| Metric | Value |
+|--------|-------|
+| Recall@10 | **100%** on 768D Wikipedia embeddings |
+| Query Performance | **95.5%** improvement from baseline |
+| Memory Reduction | **4x** with Int8 quantization |
+
+## Features
+
+- **HNSW Algorithm** - Hierarchical Navigable Small World for fast approximate nearest neighbor search
+- **Multiple Distance Metrics** - Cosine similarity, Euclidean, dot product
+- **Int8 Quantization** - 4x memory reduction with minimal recall loss
+- **Multi-Platform** - Bun (file system) and Browser (OPFS)
+- **Parameter Presets** - Pre-tuned configurations for different use cases
+- **Batch Queries** - Efficient batch processing for throughput
+- **Metadata Filtering** - MongoDB-style query operators
+
+## Installation
+
+```bash
+# Bun
+bun add verso
+
+# npm
+npm install verso
+```
+
+## Quick Start
+
+```typescript
+import { VectorDB, getRecommendedPreset } from 'verso';
+
+// Create database
+const db = new VectorDB();
+
+// Get recommended parameters for your vector dimensions
+const preset = getRecommendedPreset(768);
+
+// Create collection
+const collection = await db.createCollection('documents', {
+  dimension: 768,
+  metric: 'cosine',
+  M: preset.M,
+  efConstruction: preset.efConstruction
+});
+
+// Add vectors
+await collection.add({
+  ids: ['doc1', 'doc2', 'doc3'],
+  vectors: [
+    new Float32Array(768).fill(0.1),
+    new Float32Array(768).fill(0.2),
+    new Float32Array(768).fill(0.3)
+  ],
+  metadata: [
+    { title: 'Document 1', category: 'tech' },
+    { title: 'Document 2', category: 'science' },
+    { title: 'Document 3', category: 'tech' }
+  ]
+});
+
+// Query
+const results = await collection.query({
+  queryVector: new Float32Array(768).fill(0.15),
+  k: 10,
+  efSearch: preset.efSearch
+});
+
+console.log(results);
+// [{ id: 'doc1', score: 0.99, metadata: { title: 'Document 1', ... } }, ...]
+```
+
+## API Reference
+
+### VectorDB
+
+Main database class for managing collections.
+
+```typescript
+const db = new VectorDB();
+
+// Create collection
+const collection = await db.createCollection('name', {
+  dimension: 768,
+  metric: 'cosine',  // 'cosine' | 'euclidean' | 'dot_product'
+  M: 16,             // Max connections per node
+  efConstruction: 200 // Build-time search depth
+});
+
+// Get existing collection
+const collection = await db.getCollection('name');
+
+// List collections
+const names = await db.listCollections();
+
+// Delete collection
+await db.deleteCollection('name');
+```
+
+### Collection
+
+```typescript
+// Add vectors
+await collection.add({
+  ids: string[],
+  vectors: Float32Array[],
+  metadata?: object[]
+});
+
+// Query
+const results = await collection.query({
+  queryVector: Float32Array,
+  k: number,
+  efSearch?: number,
+  filter?: FilterExpression
+});
+
+// Batch query
+const batchResults = await collection.queryBatch({
+  queryVectors: Float32Array[],
+  k: number,
+  efSearch?: number
+});
+
+// Delete vectors
+await collection.delete(ids: string[]);
+
+// Get collection info
+const info = await collection.info();
+```
+
+### Metadata Filtering
+
+```typescript
+const results = await collection.query({
+  queryVector: queryVec,
+  k: 10,
+  filter: {
+    category: 'tech',           // Exact match
+    score: { $gt: 0.5 },        // Greater than
+    tags: { $in: ['ai', 'ml'] } // In array
+  }
+});
+```
+
+Supported operators: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`
+
+### Parameter Presets
+
+```typescript
+import { getRecommendedPreset, getRAGPreset, PRESETS } from 'verso';
+
+// Automatic preset based on dimensions
+const preset = getRecommendedPreset(768);
+
+// RAG-optimized preset (high recall)
+const ragPreset = getRAGPreset(768);
+
+// Available presets
+PRESETS.LOW_DIM        // <= 128 dimensions
+PRESETS.MEDIUM_DIM     // 256-512 dimensions
+PRESETS.HIGH_DIM       // 768+ dimensions
+PRESETS.VERY_HIGH_DIM  // 1024+ dimensions
+PRESETS.SMALL_DATASET  // < 10k vectors
+PRESETS.LARGE_DATASET  // 100k+ vectors
+PRESETS.MAX_RECALL     // Prioritize accuracy
+PRESETS.LOW_LATENCY    // Prioritize speed
+```
+
+### Int8 Quantization
+
+Reduce memory usage by 4x with minimal recall loss:
+
+```typescript
+import { ScalarQuantizer, QuantizedVectorStore } from 'verso';
+
+// Create quantizer
+const quantizer = new ScalarQuantizer(768);
+
+// Train on sample vectors
+quantizer.train(sampleVectors);
+
+// Quantize vectors
+const quantized = quantizer.quantize(vector);
+
+// Use with QuantizedVectorStore for integrated search
+const store = new QuantizedVectorStore(768);
+store.add(ids, vectors);
+const results = store.search(queryVector, k);
+```
+
+## Storage Backends
+
+Verso automatically selects the appropriate storage backend:
+
+| Environment | Backend | Storage |
+|-------------|---------|---------|
+| Bun | `BunStorageBackend` | File system |
+| Browser | `OPFSBackend` | Origin Private File System |
+| Fallback | `MemoryBackend` | In-memory (no persistence) |
+
+```typescript
+import { createStorageBackend, getRecommendedStorageType } from 'verso';
+
+// Automatic detection
+const backend = await createStorageBackend({
+  basePath: './vectors'
+});
+
+// Check available types
+const type = getRecommendedStorageType(); // 'bun' | 'opfs' | 'memory'
+```
+
+## Benchmarks
+
+See [docs/BENCHMARKS.md](docs/BENCHMARKS.md) for detailed performance analysis.
+
+**Quick Summary (768D vectors, Cohere Wikipedia dataset):**
+- 100% Recall@10 with optimized parameters
+- 95.5% query performance improvement through optimizations
+- 4x memory reduction with Int8 quantization
+
+## Development
+
+```bash
+# Install dependencies
+bun install
+
+# Run tests
+bun test
+
+# Run browser tests
+bun run test:browser
+
+# Build
+bun run build
+
+# Run benchmarks
+bun run bench
+```
+
+## License
+
+MIT

package/dist/BinaryHeap.d.ts

@@ -0,0 +1,25 @@
+export declare class BinaryHeap {
+    private ids;
+    private dists;
+    private _size;
+    private capacity;
+    private lastPoppedValue;
+    constructor(capacity: number);
+    /**
+     * Push an element onto the heap.
+     * When at capacity, simply returns without adding (caller manages bounds).
+     * This is the correct behavior for HNSW candidate exploration.
+     */
+    push(id: number, dist: number): void;
+    pop(): number;
+    peek(): number;
+    peekValue(): number;
+    getLastPoppedValue(): number;
+    size(): number;
+    clear(): void;
+    isEmpty(): boolean;
+    getCapacity(): number;
+    private heapifyUp;
+    private heapifyDown;
+}
+//# sourceMappingURL=BinaryHeap.d.ts.map

package/dist/BinaryHeap.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"BinaryHeap.d.ts","sourceRoot":"","sources":["../src/BinaryHeap.ts"],"names":[],"mappings":"AAAA,qBAAa,UAAU;IACrB,OAAO,CAAC,GAAG,CAAc;IACzB,OAAO,CAAC,KAAK,CAAe;IAC5B,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,QAAQ,CAAS;IACzB,OAAO,CAAC,eAAe,CAAa;gBAExB,QAAQ,EAAE,MAAM;IAO5B;;;;OAIG;IACH,IAAI,CAAC,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,IAAI;IAapC,GAAG,IAAI,MAAM;IAcb,IAAI,IAAI,MAAM;IAId,SAAS,IAAI,MAAM;IAInB,kBAAkB,IAAI,MAAM;IAI5B,IAAI,IAAI,MAAM;IAId,KAAK,IAAI,IAAI;IAIb,OAAO,IAAI,OAAO;IAIlB,WAAW,IAAI,MAAM;IAIrB,OAAO,CAAC,SAAS;IAuBjB,OAAO,CAAC,WAAW;CAiCpB"}

package/dist/Collection.d.ts

@@ -0,0 +1,156 @@
+import { DistanceMetric } from './HNSWIndex';
+/**
+ * Configuration for adding vectors to a collection.
+ */
+export interface AddConfig {
+    /** Unique string identifiers for each vector */
+    ids: string[];
+    /** Array of vectors to add (must match collection dimension) */
+    vectors: number[][];
+    /** Optional metadata for each vector (same length as ids/vectors) */
+    metadata?: Array<Record<string, any>>;
+}
+/**
+ * Configuration for querying a collection.
+ *
+ * @example
+ * ```typescript
+ * // Simple query
+ * const results = await collection.query({
+ *   queryVector: [0.1, 0.2, 0.3],
+ *   k: 10
+ * });
+ *
+ * // Query with metadata filter
+ * const results = await collection.query({
+ *   queryVector: [0.1, 0.2, 0.3],
+ *   k: 10,
+ *   filter: {
+ *     category: 'science',           // Exact match
+ *     year: { $gte: 2020 },          // Greater than or equal
+ *     status: { $in: ['active', 'pending'] }  // In array
+ *   }
+ * });
+ * ```
+ */
+export interface QueryConfig {
+    /** The query vector (must match collection dimension) */
+    queryVector: number[];
+    /** Number of nearest neighbors to return */
+    k: number;
+    /**
+     * Optional metadata filter. Supports MongoDB-style operators:
+     * - Simple equality: `{ field: value }`
+     * - `$gt`: Greater than `{ field: { $gt: 5 } }`
+     * - `$gte`: Greater than or equal `{ field: { $gte: 5 } }`
+     * - `$lt`: Less than `{ field: { $lt: 10 } }`
+     * - `$lte`: Less than or equal `{ field: { $lte: 10 } }`
+     * - `$ne`: Not equal `{ field: { $ne: 'excluded' } }`
+     * - `$in`: In array `{ field: { $in: ['a', 'b', 'c'] } }`
+     * - `$nin`: Not in array `{ field: { $nin: ['x', 'y'] } }`
+     */
+    filter?: Record<string, any>;
+    /** Search effort parameter (higher = better recall, slower). Default: max(k*2, 50) */
+    efSearch?: number;
+}
+export interface QueryResult {
+    ids: string[];
+    distances: number[];
+    metadata: Array<Record<string, any>>;
+}
+export declare class Collection {
+    private name;
+    private dimension;
+    private metric;
+    private M;
+    private efConstruction;
+    private indexPath;
+    private metaPath;
+    private deletedPath;
+    private hnsw;
+    private idMap;
+    private idReverseMap;
+    private metadata;
+    private deletedIds;
+    constructor(name: string, config: {
+        dimension: number;
+        metric?: DistanceMetric;
+        M?: number;
+        efConstruction?: number;
+    }, collectionPath: string);
+    init(): Promise<void>;
+    private loadFromDisk;
+    add(config: AddConfig): Promise<void>;
+    query(config: QueryConfig): Promise<QueryResult>;
+    /**
+     * Batch query for multiple vectors at once.
+     * More efficient than calling query() multiple times.
+     *
+     * @param configs Array of query configurations
+     * @returns Array of query results, one per query
+     */
+    queryBatch(configs: QueryConfig[]): Promise<QueryResult[]>;
+    /**
+     * Brute-force KNN search for validation and correctness checking
+     * This checks all vectors and returns the true k nearest neighbors
+     */
+    queryBruteForce(config: QueryConfig): Promise<QueryResult>;
+    /**
+     * Upsert vectors (insert or update).
+     *
+     * NOTE: HNSW indices don't support efficient in-place updates.
+     * Duplicate IDs will throw an error. For updates, rebuild the index
+     * without the old vectors and add the new ones.
+     *
+     * @throws Error if any ID already exists
+     */
+    upsert(config: AddConfig): Promise<void>;
+    /**
+     * Returns the number of active (non-deleted) vectors in the collection.
+     */
+    count(): number;
+    /**
+     * Returns the total number of vectors including deleted (tombstoned) ones.
+     * Use this to determine when compaction might be beneficial.
+     */
+    countWithDeleted(): number;
+    /**
+     * Returns the number of deleted (tombstoned) vectors.
+     */
+    deletedCount(): number;
+    /**
+     * Mark a vector as deleted (tombstone deletion).
+     * The vector remains in the index but is excluded from search results.
+     * Use compact() to permanently remove deleted vectors and reclaim space.
+     *
+     * @param id The string ID of the vector to delete
+     * @returns true if the vector was deleted, false if it didn't exist or was already deleted
+     */
+    delete(id: string): boolean;
+    /**
+     * Mark multiple vectors as deleted (tombstone deletion).
+     *
+     * @param ids Array of string IDs to delete
+     * @returns Number of vectors that were successfully deleted
+     */
+    deleteBatch(ids: string[]): number;
+    /**
+     * Check if a vector exists and is not deleted.
+     */
+    has(id: string): boolean;
+    /**
+     * Check if a vector was deleted (tombstoned).
+     */
+    isDeleted(id: string): boolean;
+    saveToDisk(): Promise<void>;
+    private matchesFilter;
+    /**
+     * Compact the collection by rebuilding the index without deleted vectors.
+     * This permanently removes tombstoned vectors and reclaims space.
+     *
+     * @returns Number of vectors removed during compaction
+     */
+    compact(): Promise<number>;
+    destroy(): Promise<void>;
+}
+//# sourceMappingURL=Collection.d.ts.map

package/dist/Collection.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"Collection.d.ts","sourceRoot":"","sources":["../src/Collection.ts"],"names":[],"mappings":"AAAA,OAAO,EAAa,cAAc,EAAE,MAAM,aAAa,CAAC;AAIxD;;GAEG;AACH,MAAM,WAAW,SAAS;IACxB,gDAAgD;IAChD,GAAG,EAAE,MAAM,EAAE,CAAC;IACd,gEAAgE;IAChE,OAAO,EAAE,MAAM,EAAE,EAAE,CAAC;IACpB,qEAAqE;IACrE,QAAQ,CAAC,EAAE,KAAK,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,CAAC;CACvC;AAED;;;;;;;;;;;;;;;;;;;;;;GAsBG;AACH,MAAM,WAAW,WAAW;IAC1B,yDAAyD;IACzD,WAAW,EAAE,MAAM,EAAE,CAAC;IACtB,4CAA4C;IAC5C,CAAC,EAAE,MAAM,CAAC;IACV;;;;;;;;;;OAUG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IAC7B,sFAAsF;IACtF,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,WAAW;IAC1B,GAAG,EAAE,MAAM,EAAE,CAAC;IACd,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,QAAQ,EAAE,KAAK,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,CAAC;CACtC;AAED,qBAAa,UAAU;IACrB,OAAO,CAAC,IAAI,CAAS;IACrB,OAAO,CAAC,SAAS,CAAS;IAC1B,OAAO,CAAC,MAAM,CAAiB;IAC/B,OAAO,CAAC,CAAC,CAAS;IAClB,OAAO,CAAC,cAAc,CAAS;IAC/B,OAAO,CAAC,SAAS,CAAS;IAC1B,OAAO,CAAC,QAAQ,CAAS;IACzB,OAAO,CAAC,WAAW,CAAS;IAC5B,OAAO,CAAC,IAAI,CAAY;IACxB,OAAO,CAAC,KAAK,CAAsB;IACnC,OAAO,CAAC,YAAY,CAAsB;IAC1C,OAAO,CAAC,QAAQ,CAAmC;IACnD,OAAO,CAAC,UAAU,CAAc;gBAEpB,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,MAAM,CAAC,EAAE,cAAc,CAAC;QAAC,CAAC,CAAC,EAAE,MAAM,CAAC;QAAC,cAAc,CAAC,EAAE,MAAM,CAAA;KAAE,EAAE,cAAc,EAAE,MAAM;IAwBvI,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;YAKb,YAAY;IA8DpB,GAAG,CAAC,MAAM,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAyDrC,KAAK,CAAC,MAAM,EAAE,WAAW,GAAG,OAAO,CAAC,WAAW,CAAC;IAuEtD;;;;;;OAMG;IACG,UAAU,CAAC,OAAO,EAAE,WAAW,EAAE,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC;IA2FhE;;;OAGG;IACG,eAAe,CAAC,MAAM,EAAE,WAAW,GAAG,OAAO,CAAC,WAAW,CAAC;IA0DhE;;;;;;;;OAQG;IACG,MAAM,CAAC,MAAM,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAqB9C;;OAEG;IACH,KAAK,IAAI,MAAM;IAIf;;;OAGG;IACH,gBAAgB,IAAI,MAAM;IAI1B;;OAEG;IACH,YAAY,IAAI,MAAM;IAItB;;;;;;;OAOG;IACH,MAAM,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO;IAS3B;;;;;OAKG;IACH,WAAW,CAAC,GAAG,EAAE,MAAM,EAAE,GAAG,MAAM;IAQlC;;OAEG;IACH,GAAG,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO;IAMxB;;OAEG;IACH,SAAS,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO;IAMxB,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IA6BjC,OAAO,CAAC,aAAa;IAsBrB;;;;;OAKG;IACG,OAAO,IAAI,OAAO,CAAC,MAAM,CAAC;IA6D1B,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;CAe/B"}