@ruvector/node 0.1.17 → 0.1.22

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md DELETED
@@ -1,1085 +0,0 @@
1
- # Ruvector Node.js
2
-
3
- [![npm version](https://img.shields.io/npm/v/ruvector.svg)](https://www.npmjs.com/package/ruvector)
4
- [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
5
- [![Node.js](https://img.shields.io/badge/Node.js-18%2B-green.svg)](https://nodejs.org)
6
- [![TypeScript](https://img.shields.io/badge/TypeScript-Ready-blue.svg)](https://www.typescriptlang.org)
7
- [![Performance](https://img.shields.io/badge/latency-<0.5ms-green.svg)](../../docs/TECHNICAL_PLAN.md)
8
-
9
- **Native Rust performance for Node.js vector databases via NAPI-RS**
10
-
11
- Bring the power of Ruvector's blazing-fast vector search to your Node.js and TypeScript applications. Built with NAPI-RS for zero-overhead native bindings, async/await support, and complete type safety.
12
-
13
- > Part of the [Ruvector](https://github.com/ruvnet/ruvector) ecosystem - next-generation vector database built in Rust.
14
-
15
- ## 🌟 Why Ruvector Node.js?
16
-
17
- In the age of AI, Node.js applications need **fast, efficient vector search** for RAG systems, semantic search, and recommendation engines. But existing JavaScript solutions are slow, memory-intensive, or lack critical features.
18
-
19
- **Ruvector Node.js eliminates these limitations.**
20
-
21
- ### Key Advantages
22
-
23
- - ⚡ **Native Performance**: <0.5ms search latency with Rust-powered HNSW indexing
24
- - 🚀 **10-100x Faster**: Outperforms pure JavaScript vector databases by orders of magnitude
25
- - 💾 **Memory Efficient**: 4-32x compression with product quantization
26
- - 🔒 **Zero-Copy Buffers**: Direct Float32Array memory sharing (no serialization overhead)
27
- - ⚡ **Async/Await**: Full Promise-based API with TypeScript async/await support
28
- - 📘 **Type Safety**: Complete TypeScript definitions auto-generated from Rust
29
- - 🌐 **Universal**: CommonJS and ESM support for all Node.js environments
30
- - 🎯 **Production Ready**: Battle-tested algorithms with comprehensive error handling
31
-
32
- ## 📊 Performance Comparison
33
-
34
- ### vs Pure JavaScript Alternatives
35
-
36
- ```
37
- Operation Ruvector Pure JS Speedup
38
- ────────────────────────────────────────────────────────
39
- Insert 1M vectors 2.1s 45s 21x
40
- Search (k=10) 0.4ms 50ms 125x
41
- Memory (1M vectors) 800MB 3GB 3.75x
42
- HNSW Build 1.8s N/A Native only
43
- Product Quantization Yes No 32x compression
44
- SIMD Acceleration Yes No 4-16x faster
45
- ```
46
-
47
- ### Local Performance (Single Instance)
48
-
49
- ```
50
- Metric Value Details
51
- ──────────────────────────────────────────────────────────
52
- Query Latency (p50) <0.5ms HNSW + SIMD optimizations
53
- Throughput (QPS) 50K+ Single-threaded Node.js
54
- Memory (1M vectors) ~800MB With scalar quantization
55
- Recall @ k=10 95%+ HNSW configuration
56
- Browser Support Via WASM Use ruvector-wasm package
57
- Offline Capable ✅ Embedded database
58
- ```
59
-
60
- ## 🚀 Installation
61
-
62
- ```bash
63
- npm install ruvector
64
- ```
65
-
66
- **Requirements:**
67
- - Node.js 18.0 or higher
68
- - Supported platforms: Linux (x64, arm64), macOS (x64, arm64), Windows (x64)
69
- - No additional dependencies required (native binary included)
70
-
71
- **Optional: Verify installation**
72
-
73
- ```bash
74
- node -e "const {version} = require('ruvector'); console.log('Ruvector v' + version())"
75
- ```
76
-
77
- ## ⚡ Quick Start
78
-
79
- ### JavaScript (CommonJS)
80
-
81
- ```javascript
82
- const { VectorDB } = require('ruvector');
83
-
84
- async function main() {
85
- // Create database with 384 dimensions (e.g., for sentence-transformers)
86
- const db = VectorDB.withDimensions(384);
87
-
88
- // Insert vectors with metadata
89
- const id1 = await db.insert({
90
- vector: new Float32Array(384).fill(0.1),
91
- metadata: { text: 'Hello world', category: 'greeting' }
92
- });
93
-
94
- const id2 = await db.insert({
95
- vector: new Float32Array(384).fill(0.2),
96
- metadata: { text: 'Goodbye world', category: 'farewell' }
97
- });
98
-
99
- console.log(`Inserted: ${id1}, ${id2}`);
100
-
101
- // Search for similar vectors
102
- const results = await db.search({
103
- vector: new Float32Array(384).fill(0.15),
104
- k: 10,
105
- filter: { category: 'greeting' }
106
- });
107
-
108
- results.forEach(result => {
109
- console.log(`ID: ${result.id}, Score: ${result.score}`);
110
- console.log(`Metadata:`, result.metadata);
111
- });
112
-
113
- // Get database stats
114
- console.log(`Total vectors: ${await db.len()}`);
115
- }
116
-
117
- main().catch(console.error);
118
- ```
119
-
120
- ### TypeScript (ESM)
121
-
122
- ```typescript
123
- import { VectorDB, JsDbOptions, JsSearchQuery } from 'ruvector';
124
-
125
- interface DocumentMetadata {
126
- text: string;
127
- category: string;
128
- timestamp: number;
129
- }
130
-
131
- async function semanticSearch() {
132
- // Advanced configuration
133
- const options: JsDbOptions = {
134
- dimensions: 768,
135
- distanceMetric: 'Cosine',
136
- storagePath: './my-vectors.db',
137
- hnswConfig: {
138
- m: 32, // Connections per layer
139
- efConstruction: 200, // Build quality
140
- efSearch: 100 // Search quality
141
- },
142
- quantization: {
143
- type: 'product',
144
- subspaces: 16,
145
- k: 256
146
- }
147
- };
148
-
149
- const db = new VectorDB(options);
150
-
151
- // Batch insert for better performance
152
- const embeddings = await getEmbeddings(['doc1', 'doc2', 'doc3']);
153
- const ids = await db.insertBatch(
154
- embeddings.map((vec, i) => ({
155
- vector: new Float32Array(vec),
156
- metadata: {
157
- text: `Document ${i}`,
158
- category: 'article',
159
- timestamp: Date.now()
160
- }
161
- }))
162
- );
163
-
164
- console.log(`Inserted ${ids.length} vectors`);
165
-
166
- // Semantic search with filters
167
- const query: JsSearchQuery = {
168
- vector: new Float32Array(await getEmbedding('search query')),
169
- k: 10,
170
- filter: { category: 'article' },
171
- efSearch: 150 // Higher = more accurate but slower
172
- };
173
-
174
- const results = await db.search(query);
175
-
176
- return results.map(r => ({
177
- id: r.id,
178
- similarity: 1 - r.score, // Convert distance to similarity
179
- metadata: r.metadata as DocumentMetadata
180
- }));
181
- }
182
-
183
- // Helper function (replace with your embedding model)
184
- async function getEmbedding(text: string): Promise<number[]> {
185
- // Use OpenAI, Cohere, or local model like sentence-transformers
186
- return new Array(768).fill(0);
187
- }
188
-
189
- async function getEmbeddings(texts: string[]): Promise<number[][]> {
190
- return Promise.all(texts.map(getEmbedding));
191
- }
192
- ```
193
-
194
- ## 📖 API Reference
195
-
196
- ### VectorDB Class
197
-
198
- #### Constructor
199
-
200
- ```typescript
201
- // Option 1: Full configuration
202
- const db = new VectorDB({
203
- dimensions: 384, // Required: Vector dimensions
204
- distanceMetric?: 'Euclidean' | 'Cosine' | 'DotProduct' | 'Manhattan',
205
- storagePath?: string, // Default: './ruvector.db'
206
- hnswConfig?: {
207
- m?: number, // Default: 32 (16-64 recommended)
208
- efConstruction?: number, // Default: 200 (100-500)
209
- efSearch?: number, // Default: 100 (50-500)
210
- maxElements?: number // Default: 10,000,000
211
- },
212
- quantization?: {
213
- type: 'none' | 'scalar' | 'product' | 'binary',
214
- subspaces?: number, // For product quantization (Default: 16)
215
- k?: number // Codebook size (Default: 256)
216
- }
217
- });
218
-
219
- // Option 2: Simple factory method (uses defaults)
220
- const db = VectorDB.withDimensions(384);
221
- ```
222
-
223
- **Configuration Guide:**
224
-
225
- - **dimensions**: Must match your embedding model output (e.g., 384 for all-MiniLM-L6-v2, 768 for BERT, 1536 for OpenAI text-embedding-3-small)
226
- - **distanceMetric**:
227
- - `Cosine`: Best for normalized vectors (text embeddings, most ML models) - Default
228
- - `Euclidean`: Best for absolute distances (images, spatial data)
229
- - `DotProduct`: Best for positive vectors with magnitude info
230
- - `Manhattan`: Best for sparse vectors (L1 norm)
231
- - **storagePath**: Path to persistent storage file
232
- - **hnswConfig**: Controls search quality and speed tradeoff
233
- - **quantization**: Enables memory compression (4-32x reduction)
234
-
235
- #### Methods
236
-
237
- ##### `insert(entry): Promise<string>`
238
-
239
- Insert a single vector and return its ID.
240
-
241
- ```typescript
242
- const id = await db.insert({
243
- id?: string, // Optional (auto-generated UUID if not provided)
244
- vector: Float32Array, // Required: Vector data
245
- metadata?: Record<string, any> // Optional: JSON object
246
- });
247
- ```
248
-
249
- **Example:**
250
-
251
- ```javascript
252
- const id = await db.insert({
253
- vector: new Float32Array([0.1, 0.2, 0.3, ...]),
254
- metadata: {
255
- text: 'example document',
256
- category: 'research',
257
- timestamp: Date.now()
258
- }
259
- });
260
- console.log(`Inserted with ID: ${id}`);
261
- ```
262
-
263
- ##### `insertBatch(entries): Promise<string[]>`
264
-
265
- Insert multiple vectors efficiently in a batch (10-50x faster than sequential inserts).
266
-
267
- ```typescript
268
- const ids = await db.insertBatch([
269
- { vector: new Float32Array([...]) },
270
- { vector: new Float32Array([...]), metadata: { text: 'example' } }
271
- ]);
272
- ```
273
-
274
- **Example:**
275
-
276
- ```javascript
277
- // Bad: Sequential inserts (slow)
278
- for (const vector of vectors) {
279
- await db.insert({ vector }); // Don't do this!
280
- }
281
-
282
- // Good: Batch insert (10-50x faster)
283
- await db.insertBatch(
284
- vectors.map(v => ({ vector: new Float32Array(v) }))
285
- );
286
- ```
287
-
288
- ##### `search(query): Promise<SearchResult[]>`
289
-
290
- Search for similar vectors using HNSW approximate nearest neighbor search.
291
-
292
- ```typescript
293
- const results = await db.search({
294
- vector: Float32Array, // Required: Query vector
295
- k: number, // Required: Number of results
296
- filter?: Record<string, any>, // Optional: Metadata filters
297
- efSearch?: number // Optional: HNSW search parameter (higher = more accurate)
298
- });
299
-
300
- // Result format:
301
- interface SearchResult {
302
- id: string; // Vector ID
303
- score: number; // Distance (lower is better for most metrics)
304
- vector?: number[]; // Original vector (optional, for debugging)
305
- metadata?: any; // Metadata object
306
- }
307
- ```
308
-
309
- **Example:**
310
-
311
- ```javascript
312
- const results = await db.search({
313
- vector: queryEmbedding,
314
- k: 10,
315
- filter: { category: 'research', year: 2024 },
316
- efSearch: 150 // Higher = better recall, slower search
317
- });
318
-
319
- results.forEach(result => {
320
- const similarity = 1 - result.score; // Convert distance to similarity
321
- console.log(`${result.metadata.text}: ${similarity.toFixed(3)}`);
322
- });
323
- ```
324
-
325
- ##### `get(id): Promise<VectorEntry | null>`
326
-
327
- Retrieve a vector by ID.
328
-
329
- ```typescript
330
- const entry = await db.get('vector-id');
331
- if (entry) {
332
- console.log(entry.vector, entry.metadata);
333
- }
334
- ```
335
-
336
- ##### `delete(id): Promise<boolean>`
337
-
338
- Delete a vector by ID. Returns `true` if deleted, `false` if not found.
339
-
340
- ```typescript
341
- const deleted = await db.delete('vector-id');
342
- console.log(deleted ? 'Deleted' : 'Not found');
343
- ```
344
-
345
- ##### `len(): Promise<number>`
346
-
347
- Get the total number of vectors in the database.
348
-
349
- ```typescript
350
- const count = await db.len();
351
- console.log(`Database contains ${count} vectors`);
352
- ```
353
-
354
- ##### `isEmpty(): Promise<boolean>`
355
-
356
- Check if the database is empty.
357
-
358
- ```typescript
359
- if (await db.isEmpty()) {
360
- console.log('No vectors yet');
361
- }
362
- ```
363
-
364
- ### Utility Functions
365
-
366
- ##### `version(): string`
367
-
368
- Get the Ruvector library version.
369
-
370
- ```typescript
371
- import { version } from 'ruvector';
372
- console.log(`Ruvector v${version()}`);
373
- ```
374
-
375
- ## 🎯 Common Use Cases
376
-
377
- ### 1. RAG (Retrieval-Augmented Generation)
378
-
379
- Build production-ready RAG systems with fast vector retrieval for LLMs.
380
-
381
- ```javascript
382
- const { VectorDB } = require('ruvector');
383
- const { OpenAI } = require('openai');
384
-
385
- class RAGSystem {
386
- constructor() {
387
- this.db = VectorDB.withDimensions(1536); // OpenAI embedding size
388
- this.openai = new OpenAI();
389
- }
390
-
391
- async indexDocument(text, metadata) {
392
- // Split into chunks (use better chunking in production)
393
- const chunks = this.chunkText(text, 512);
394
-
395
- // Get embeddings from OpenAI
396
- const embeddings = await this.openai.embeddings.create({
397
- model: 'text-embedding-3-small',
398
- input: chunks
399
- });
400
-
401
- // Insert into vector DB
402
- const ids = await this.db.insertBatch(
403
- embeddings.data.map((emb, i) => ({
404
- vector: new Float32Array(emb.embedding),
405
- metadata: { ...metadata, chunk: i, text: chunks[i] }
406
- }))
407
- );
408
-
409
- return ids;
410
- }
411
-
412
- async query(question, k = 5) {
413
- // Embed the question
414
- const response = await this.openai.embeddings.create({
415
- model: 'text-embedding-3-small',
416
- input: [question]
417
- });
418
-
419
- // Search for relevant chunks
420
- const results = await this.db.search({
421
- vector: new Float32Array(response.data[0].embedding),
422
- k
423
- });
424
-
425
- // Extract context
426
- const context = results
427
- .map(r => r.metadata.text)
428
- .join('\n\n');
429
-
430
- // Generate answer with LLM
431
- const completion = await this.openai.chat.completions.create({
432
- model: 'gpt-4',
433
- messages: [
434
- { role: 'system', content: 'Answer based on the context provided.' },
435
- { role: 'user', content: `Context:\n${context}\n\nQuestion: ${question}` }
436
- ]
437
- });
438
-
439
- return {
440
- answer: completion.choices[0].message.content,
441
- sources: results.map(r => r.metadata)
442
- };
443
- }
444
-
445
- chunkText(text, maxLength) {
446
- // Simple word-based chunking
447
- const words = text.split(' ');
448
- const chunks = [];
449
- let current = [];
450
-
451
- for (const word of words) {
452
- current.push(word);
453
- if (current.join(' ').length > maxLength) {
454
- chunks.push(current.join(' '));
455
- current = [];
456
- }
457
- }
458
-
459
- if (current.length > 0) {
460
- chunks.push(current.join(' '));
461
- }
462
-
463
- return chunks;
464
- }
465
- }
466
-
467
- // Usage
468
- const rag = new RAGSystem();
469
- await rag.indexDocument('Long document text...', { source: 'doc.pdf' });
470
- const result = await rag.query('What is the main topic?');
471
- console.log(result.answer);
472
- console.log('Sources:', result.sources);
473
- ```
474
-
475
- ### 2. Semantic Code Search
476
-
477
- Find similar code patterns across your codebase.
478
-
479
- ```typescript
480
- import { VectorDB } from 'ruvector';
481
- import { pipeline } from '@xenova/transformers';
482
-
483
- // Use a code-specific embedding model
484
- const embedder = await pipeline(
485
- 'feature-extraction',
486
- 'Xenova/codebert-base'
487
- );
488
-
489
- const db = VectorDB.withDimensions(768);
490
-
491
- // Index code snippets
492
- async function indexCodebase(codeFiles: Array<{ path: string, code: string }>) {
493
- for (const file of codeFiles) {
494
- const embedding = await embedder(file.code, {
495
- pooling: 'mean',
496
- normalize: true
497
- });
498
-
499
- await db.insert({
500
- vector: new Float32Array(embedding.data),
501
- metadata: {
502
- path: file.path,
503
- code: file.code,
504
- language: file.path.split('.').pop()
505
- }
506
- });
507
- }
508
- }
509
-
510
- // Search for similar code
511
- async function findSimilarCode(query: string, k = 10) {
512
- const embedding = await embedder(query, {
513
- pooling: 'mean',
514
- normalize: true
515
- });
516
-
517
- const results = await db.search({
518
- vector: new Float32Array(embedding.data),
519
- k
520
- });
521
-
522
- return results.map(r => ({
523
- path: r.metadata.path,
524
- code: r.metadata.code,
525
- similarity: 1 - r.score
526
- }));
527
- }
528
-
529
- // Example usage
530
- await indexCodebase([
531
- { path: 'utils.ts', code: 'function parseJSON(str) { ... }' },
532
- { path: 'api.ts', code: 'async function fetchData(url) { ... }' }
533
- ]);
534
-
535
- const similar = await findSimilarCode('parse JSON string');
536
- console.log(similar);
537
- ```
538
-
539
- ### 3. Recommendation System
540
-
541
- Build personalized recommendation engines.
542
-
543
- ```javascript
544
- const { VectorDB } = require('ruvector');
545
-
546
- class RecommendationEngine {
547
- constructor() {
548
- // User/item embeddings from collaborative filtering or content features
549
- this.db = VectorDB.withDimensions(128);
550
- }
551
-
552
- async addItem(itemId, features, metadata) {
553
- await this.db.insert({
554
- id: itemId,
555
- vector: new Float32Array(features),
556
- metadata: {
557
- ...metadata,
558
- addedAt: Date.now()
559
- }
560
- });
561
- }
562
-
563
- async recommendSimilar(itemId, k = 10, filters = {}) {
564
- // Get the item's embedding
565
- const item = await this.db.get(itemId);
566
- if (!item) return [];
567
-
568
- // Find similar items
569
- const results = await this.db.search({
570
- vector: item.vector,
571
- k: k + 1, // +1 because it will include itself
572
- filter: filters
573
- });
574
-
575
- // Remove the item itself from results
576
- return results
577
- .filter(r => r.id !== itemId)
578
- .slice(0, k)
579
- .map(r => ({
580
- id: r.id,
581
- similarity: 1 - r.score,
582
- ...r.metadata
583
- }));
584
- }
585
-
586
- async recommendForUser(userVector, k = 10, category = null) {
587
- const filter = category ? { category } : undefined;
588
-
589
- const results = await this.db.search({
590
- vector: new Float32Array(userVector),
591
- k,
592
- filter
593
- });
594
-
595
- return results.map(r => ({
596
- id: r.id,
597
- score: 1 - r.score,
598
- ...r.metadata
599
- }));
600
- }
601
- }
602
-
603
- // Usage
604
- const engine = new RecommendationEngine();
605
-
606
- // Add products
607
- await engine.addItem('prod-1', new Array(128).fill(0.1), {
608
- name: 'Laptop',
609
- category: 'electronics',
610
- price: 999
611
- });
612
-
613
- // Get similar products
614
- const similar = await engine.recommendSimilar('prod-1', 5, {
615
- category: 'electronics'
616
- });
617
-
618
- // Get recommendations for user
619
- const userPreferences = new Array(128).fill(0.15); // From ML model
620
- const recommended = await engine.recommendForUser(userPreferences, 10);
621
- ```
622
-
623
- ### 4. Duplicate Detection
624
-
625
- Find and deduplicate similar documents or records.
626
-
627
- ```typescript
628
- import { VectorDB } from 'ruvector';
629
-
630
- class DuplicateDetector {
631
- private db: VectorDB;
632
- private threshold: number;
633
-
634
- constructor(dimensions: number, threshold = 0.95) {
635
- this.db = VectorDB.withDimensions(dimensions);
636
- this.threshold = threshold; // Similarity threshold
637
- }
638
-
639
- async addDocument(id: string, embedding: Float32Array, metadata: any) {
640
- // Check for duplicates before adding
641
- const duplicates = await this.findDuplicates(embedding, 1);
642
-
643
- if (duplicates.length > 0) {
644
- return {
645
- added: false,
646
- duplicate: duplicates[0]
647
- };
648
- }
649
-
650
- await this.db.insert({ id, vector: embedding, metadata });
651
- return { added: true };
652
- }
653
-
654
- async findDuplicates(embedding: Float32Array, k = 5) {
655
- const results = await this.db.search({
656
- vector: embedding,
657
- k
658
- });
659
-
660
- return results
661
- .filter(r => (1 - r.score) >= this.threshold)
662
- .map(r => ({
663
- id: r.id,
664
- similarity: 1 - r.score,
665
- metadata: r.metadata
666
- }));
667
- }
668
- }
669
-
670
- // Usage
671
- const detector = new DuplicateDetector(384, 0.95);
672
- const result = await detector.addDocument(
673
- 'doc-1',
674
- documentEmbedding,
675
- { text: 'Example document' }
676
- );
677
-
678
- if (!result.added) {
679
- console.log('Duplicate found:', result.duplicate);
680
- }
681
- ```
682
-
683
- ## 🔧 Performance Tuning
684
-
685
- ### HNSW Parameters
686
-
687
- Tune the HNSW index for your specific use case:
688
-
689
- ```javascript
690
- // High-recall configuration (research, critical applications)
691
- const highRecallDb = new VectorDB({
692
- dimensions: 384,
693
- hnswConfig: {
694
- m: 64, // More connections = better recall
695
- efConstruction: 400, // Higher quality index build
696
- efSearch: 200 // More thorough search
697
- }
698
- });
699
-
700
- // Balanced configuration (most applications) - DEFAULT
701
- const balancedDb = new VectorDB({
702
- dimensions: 384,
703
- hnswConfig: {
704
- m: 32,
705
- efConstruction: 200,
706
- efSearch: 100
707
- }
708
- });
709
-
710
- // Speed-optimized configuration (real-time applications)
711
- const fastDb = new VectorDB({
712
- dimensions: 384,
713
- hnswConfig: {
714
- m: 16, // Fewer connections = faster
715
- efConstruction: 100,
716
- efSearch: 50 // Faster search, slightly lower recall
717
- }
718
- });
719
- ```
720
-
721
- **Parameter Guide:**
722
-
723
- - **m** (16-64): Number of connections per node
724
- - Higher = better recall, more memory, slower inserts
725
- - Lower = faster inserts, less memory, slightly lower recall
726
-
727
- - **efConstruction** (100-500): Quality of index construction
728
- - Higher = better index quality, slower builds
729
- - Lower = faster builds, slightly lower recall
730
-
731
- - **efSearch** (50-500): Search quality parameter
732
- - Higher = better recall, slower searches
733
- - Lower = faster searches, slightly lower recall
734
- - Can be overridden per query
735
-
736
- ### Quantization for Large Datasets
737
-
738
- Reduce memory usage by 4-32x with quantization:
739
-
740
- ```javascript
741
- // Product Quantization: 8-32x memory compression
742
- const pqDb = new VectorDB({
743
- dimensions: 768,
744
- quantization: {
745
- type: 'product',
746
- subspaces: 16, // 768 / 16 = 48 dims per subspace
747
- k: 256 // 256 centroids (8-bit quantization)
748
- }
749
- });
750
-
751
- // Binary Quantization: 32x compression, very fast (best for Cosine)
752
- const binaryDb = new VectorDB({
753
- dimensions: 384,
754
- quantization: { type: 'binary' }
755
- });
756
-
757
- // Scalar Quantization: 4x compression, minimal accuracy loss
758
- const scalarDb = new VectorDB({
759
- dimensions: 384,
760
- quantization: { type: 'scalar' }
761
- });
762
-
763
- // No Quantization: Maximum accuracy, more memory
764
- const fullDb = new VectorDB({
765
- dimensions: 384,
766
- quantization: { type: 'none' }
767
- });
768
- ```
769
-
770
- **Quantization Guide:**
771
-
772
- - **Product**: Best for large datasets (>100K vectors), 8-32x compression
773
- - **Binary**: Fastest search, 32x compression, works best with Cosine metric
774
- - **Scalar**: Good balance (4x compression, <1% accuracy loss)
775
- - **None**: Maximum accuracy, no compression
776
-
777
- ### Batch Operations
778
-
779
- Always use batch operations for better performance:
780
-
781
- ```javascript
782
- // ❌ Bad: Sequential inserts (slow)
783
- for (const vector of vectors) {
784
- await db.insert({ vector });
785
- }
786
-
787
- // ✅ Good: Batch insert (10-50x faster)
788
- await db.insertBatch(
789
- vectors.map(v => ({ vector: new Float32Array(v) }))
790
- );
791
- ```
792
-
793
- ### Distance Metrics
794
-
795
- Choose the right metric for your embeddings:
796
-
797
- ```javascript
798
- // Cosine: Best for normalized vectors (text embeddings, most ML models)
799
- const cosineDb = new VectorDB({
800
- dimensions: 384,
801
- distanceMetric: 'Cosine' // Range: [0, 2], lower is better
802
- });
803
-
804
- // Euclidean: Best for absolute distances (images, spatial data)
805
- const euclideanDb = new VectorDB({
806
- dimensions: 384,
807
- distanceMetric: 'Euclidean' // Range: [0, ∞], lower is better
808
- });
809
-
810
- // DotProduct: Best for positive vectors with magnitude info
811
- const dotProductDb = new VectorDB({
812
- dimensions: 384,
813
- distanceMetric: 'DotProduct' // Range: (-∞, ∞), higher is better
814
- });
815
-
816
- // Manhattan: Best for sparse vectors (L1 norm)
817
- const manhattanDb = new VectorDB({
818
- dimensions: 384,
819
- distanceMetric: 'Manhattan' // Range: [0, ∞], lower is better
820
- });
821
- ```
822
-
823
- ## 🛠️ Building from Source
824
-
825
- ### Prerequisites
826
-
827
- - **Rust**: 1.77 or higher
828
- - **Node.js**: 18.0 or higher
829
- - **Build tools**:
830
- - Linux: `gcc`, `g++`
831
- - macOS: Xcode Command Line Tools
832
- - Windows: Visual Studio Build Tools
833
-
834
- ### Build Steps
835
-
836
- ```bash
837
- # Clone the repository
838
- git clone https://github.com/ruvnet/ruvector.git
839
- cd ruvector/crates/ruvector-node
840
-
841
- # Install dependencies
842
- npm install
843
-
844
- # Build native addon (development)
845
- npm run build
846
-
847
- # Run tests
848
- npm test
849
-
850
- # Build for production (optimized)
851
- npm run build:release
852
-
853
- # Link for local development
854
- npm link
855
- cd /path/to/your/project
856
- npm link ruvector
857
- ```
858
-
859
- ### Cross-Compilation
860
-
861
- Build for different platforms:
862
-
863
- ```bash
864
- # Install cross-compilation tools
865
- npm install -g @napi-rs/cli
866
-
867
- # Build for specific platforms
868
- npx napi build --platform --release
869
-
870
- # Available platforms:
871
- # - linux-x64-gnu
872
- # - linux-arm64-gnu
873
- # - darwin-x64 (macOS Intel)
874
- # - darwin-arm64 (macOS Apple Silicon)
875
- # - win32-x64-msvc (Windows)
876
- ```
877
-
878
- ### Development Workflow
879
-
880
- ```bash
881
- # Format code
882
- cargo fmt
883
-
884
- # Lint code
885
- cargo clippy
886
-
887
- # Run Rust tests
888
- cargo test
889
-
890
- # Build and test Node.js bindings
891
- npm run build && npm test
892
-
893
- # Benchmark performance
894
- cargo bench
895
- ```
896
-
897
- ## 📊 Benchmarks
898
-
899
- ### Local Performance
900
-
901
- **10,000 vectors (128D):**
902
- - Insert: ~1,000 vectors/sec
903
- - Search (k=10): ~1ms average latency
904
- - QPS: ~1,000 queries/sec (single-threaded)
905
-
906
- **1,000,000 vectors (128D):**
907
- - Insert: ~500-1,000 vectors/sec
908
- - Search (k=10): ~5ms average latency
909
- - QPS: ~200-500 queries/sec
910
- - Memory: ~800MB (with scalar quantization)
911
-
912
- **10,000,000 vectors (128D):**
913
- - Search (k=10): ~10ms average latency
914
- - Memory: ~8GB (with product quantization)
915
- - Recall: 95%+ with optimized HNSW parameters
916
-
917
- ## 📚 Examples
918
-
919
- See the [examples](./examples) directory for complete working examples:
920
-
921
- - **simple.mjs**: Basic insert and search operations
922
- - **advanced.mjs**: HNSW configuration and batch operations
923
- - **semantic-search.mjs**: Text similarity search with embeddings
924
-
925
- Run examples:
926
-
927
- ```bash
928
- npm run build
929
- node examples/simple.mjs
930
- node examples/advanced.mjs
931
- node examples/semantic-search.mjs
932
- ```
933
-
934
- ## 🔍 Comparison with Alternatives
935
-
936
- | Feature | Ruvector | Pure JS | Python (Faiss) | Pinecone |
937
- |---------|----------|---------|----------------|----------|
938
- | **Language** | Rust (NAPI) | JavaScript | Python | Cloud API |
939
- | **Local Latency** | <0.5ms | 10-100ms | 1-5ms | 20-50ms+ |
940
- | **Throughput** | 50K+ QPS | 100-1K | 10K+ | 10K+ |
941
- | **Memory (1M)** | 800MB | 3GB | 1.5GB | N/A |
942
- | **HNSW Index** | ✅ Native | ❌ or slow | ✅ | ✅ |
943
- | **Quantization** | ✅ 4-32x | ❌ | ✅ | ✅ |
944
- | **SIMD** | ✅ Hardware | ❌ | ✅ | ✅ |
945
- | **TypeScript** | ✅ Auto-gen | Varies | ❌ | ✅ |
946
- | **Async/Await** | ✅ Native | ✅ | ✅ | ✅ |
947
- | **Offline** | ✅ | ✅ | ✅ | ❌ |
948
- | **Cost** | Free | Free | Free | $$$ |
949
- | **Bundle Size** | ~2MB | 100KB-1MB | N/A | N/A |
950
-
951
- ## 🐛 Troubleshooting
952
-
953
- ### Installation fails
954
-
955
- **Error**: `Cannot find module 'ruvector'`
956
-
957
- Make sure you have Rust installed:
958
-
959
- ```bash
960
- curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
961
- source $HOME/.cargo/env
962
- ```
963
-
964
- ### Build errors
965
-
966
- **Error**: `error: linking with 'cc' failed`
967
-
968
- Install build tools:
969
-
970
- ```bash
971
- # Linux (Ubuntu/Debian)
972
- sudo apt-get install build-essential
973
-
974
- # macOS
975
- xcode-select --install
976
-
977
- # Windows
978
- # Install Visual Studio Build Tools
979
- ```
980
-
981
- Update NAPI-RS CLI:
982
-
983
- ```bash
984
- npm install -g @napi-rs/cli
985
- ```
986
-
987
- ### Performance issues
988
-
989
- - ✅ Use HNSW indexing for datasets >10K vectors
990
- - ✅ Enable quantization for large datasets
991
- - ✅ Adjust `efSearch` for speed/accuracy tradeoff
992
- - ✅ Use `insertBatch` instead of individual `insert` calls
993
- - ✅ Use appropriate distance metric for your embeddings
994
- - ✅ Consider product quantization for >100K vectors
995
-
996
- ### Memory issues
997
-
998
- - ✅ Enable product quantization (8-32x compression)
999
- - ✅ Reduce `m` parameter in HNSW config
1000
- - ✅ Use binary quantization for maximum compression
1001
- - ✅ Batch operations to reduce memory overhead
1002
-
1003
- ## 🤝 Contributing
1004
-
1005
- We welcome contributions! See the main [Contributing Guide](../../docs/development/CONTRIBUTING.md).
1006
-
1007
- ### Development Workflow
1008
-
1009
- ```bash
1010
- # Format Rust code
1011
- cargo fmt --all
1012
-
1013
- # Lint Rust code
1014
- cargo clippy --workspace -- -D warnings
1015
-
1016
- # Run Rust tests
1017
- cargo test -p ruvector-node
1018
-
1019
- # Build Node.js bindings
1020
- npm run build
1021
-
1022
- # Run Node.js tests
1023
- npm test
1024
-
1025
- # Benchmark performance
1026
- cargo bench -p ruvector-bench
1027
- ```
1028
-
1029
- ## 📖 Documentation
1030
-
1031
- - **[Main Documentation](../../docs/README.md)** - Complete Ruvector documentation
1032
- - **[Node.js API Reference](../../docs/api/NODEJS_API.md)** - Detailed API documentation
1033
- - **[Rust API Reference](../../docs/api/RUST_API.md)** - Core Rust API
1034
- - **[Performance Guide](../../docs/optimization/PERFORMANCE_TUNING_GUIDE.md)** - Optimization tips
1035
- - **[Getting Started](../../docs/guide/GETTING_STARTED.md)** - Quick start guide
1036
- - **[Examples](./examples)** - Code examples
1037
-
1038
- ## 🌐 Support
1039
-
1040
- - **GitHub Issues**: [Report bugs or request features](https://github.com/ruvnet/ruvector/issues)
1041
- - **Discussions**: [Ask questions and share ideas](https://github.com/ruvnet/ruvector/discussions)
1042
- - **Discord**: [Join our community](https://discord.gg/ruvnet)
1043
- - **Twitter**: [@ruvnet](https://twitter.com/ruvnet)
1044
- - **Enterprise**: [enterprise@ruv.io](mailto:enterprise@ruv.io)
1045
-
1046
- ## 📜 License
1047
-
1048
- **MIT License** - see [LICENSE](../../LICENSE) for details.
1049
-
1050
- Free to use for commercial and personal projects.
1051
-
1052
- ## 🙏 Acknowledgments
1053
-
1054
- Built with battle-tested technologies:
1055
-
1056
- - **[NAPI-RS](https://napi.rs)** - Native Node.js bindings for Rust
1057
- - **[hnsw_rs](https://github.com/jean-pierreBoth/hnswlib-rs)** - HNSW implementation
1058
- - **[SimSIMD](https://github.com/ashvardanian/simsimd)** - SIMD distance metrics
1059
- - **[redb](https://github.com/cberner/redb)** - Embedded database
1060
- - **[Tokio](https://tokio.rs)** - Async runtime for Rust
1061
-
1062
- Special thanks to the Rust and Node.js communities!
1063
-
1064
- ## 🔗 Related Projects
1065
-
1066
- - **[ruvector-core](../ruvector-core)** - Core Rust implementation
1067
- - **[ruvector-wasm](../ruvector-wasm)** - WebAssembly bindings for browsers
1068
- - **[ruvector-cli](../ruvector-cli)** - Command-line interface
1069
- - **[ruvector-bench](../ruvector-bench)** - Benchmarking suite
1070
-
1071
- ---
1072
-
1073
- <div align="center">
1074
-
1075
- **Built by [rUv](https://ruv.io) • Part of the [Ruvector](https://github.com/ruvnet/ruvector) ecosystem**
1076
-
1077
- [![Star on GitHub](https://img.shields.io/github/stars/ruvnet/ruvector?style=social)](https://github.com/ruvnet/ruvector)
1078
- [![npm downloads](https://img.shields.io/npm/dm/ruvector.svg)](https://www.npmjs.com/package/ruvector)
1079
- [![Discord](https://img.shields.io/badge/Discord-Join%20Chat-7289da.svg)](https://discord.gg/ruvnet)
1080
-
1081
- **Status**: Production Ready | **Version**: 0.1.0 | **Performance**: <0.5ms latency
1082
-
1083
- [Get Started](../../docs/guide/GETTING_STARTED.md) • [Documentation](../../docs/README.md) • [API Reference](../../docs/api/NODEJS_API.md) • [Examples](./examples)
1084
-
1085
- </div>