native-vector-store 0.3.9 → 0.4.0

package/README.md

@@ -20,8 +20,9 @@ This design eliminates complex state management, ensures consistent performance,
 - **🚀 High Performance**: C++ implementation with OpenMP SIMD optimization
 - **📦 Arena Allocation**: Memory-efficient storage with 64MB chunks
 - **⚡ Fast Search**: Sub-10ms similarity search for large document collections
+- **🔍 Hybrid Search**: Combines vector similarity (semantic) with BM25 text search (lexical)
 - **🔧 MCP Integration**: Built for Model Context Protocol servers
-- **🌐 Cross-Platform**: Works on Linux, macOS, and Windows
+- **🌐 Cross-Platform**: Works on Linux and macOS (Windows users: use WSL)
 - **📊 TypeScript Support**: Full type definitions included
 - **🔄 Producer-Consumer Loading**: Parallel document loading at 178k+ docs/sec
 
@@ -48,12 +49,11 @@ npm install native-vector-store
   - **Linux**: `sudo apt-get install libgomp1` (Ubuntu/Debian) or `dnf install libgomp` (Fedora)
   - **Alpine**: `apk add libgomp`
   - **macOS**: `brew install libomp`
-  - **Windows**: Included with Visual C++ runtime
+  - **Windows**: Use WSL (Windows Subsystem for Linux)
 
 Prebuilt binaries are included for:
 - Linux (x64, arm64, musl/Alpine) - x64 builds are AWS Lambda compatible (no AVX-512)
 - macOS (x64, arm64/Apple Silicon)
-- Windows (x64)
 
 If building from source, you'll need:
 - Node.js ≥14.0.0
@@ -86,8 +86,16 @@ store.finalize(); // Must call before searching!
 
 // Search for similar documents
 const queryEmbedding = new Float32Array(1536);
+
+// Option 1: Vector-only search (traditional)
 const results = store.search(queryEmbedding, 5); // Top 5 results
 
+// Option 2: Hybrid search (NEW - combines vector + BM25 text search)
+const hybridResults = store.search(queryEmbedding, 5, "your search query text");
+
+// Option 3: BM25 text-only search
+const textResults = store.searchBM25("your search query", 5);
+
 // Results format - array of SearchResult objects, sorted by score (highest first):
 console.log(results);
 // [
@@ -278,6 +286,50 @@ if (process.env.NODE_ENV === 'development') {
 }
 ```
 
+## Hybrid Search
+
+The vector store now supports hybrid search, combining semantic similarity (vector search) with lexical matching (BM25 text search) for improved retrieval accuracy:
+
+```javascript
+const { VectorStore } = require('native-vector-store');
+
+const store = new VectorStore(1536);
+store.loadDir('./documents');
+
+// Hybrid search automatically combines vector and text search
+const queryEmbedding = new Float32Array(1536);
+const results = store.search(
+  queryEmbedding, 
+  10,                               // Top 10 results
+  "machine learning algorithms"    // Query text for BM25
+);
+
+// You can also use individual search methods
+const vectorResults = store.searchVector(queryEmbedding, 10);
+const textResults = store.searchBM25("machine learning", 10);
+
+// Or explicitly control the hybrid weights
+const customResults = store.searchHybrid(
+  queryEmbedding,
+  "machine learning",
+  10,
+  0.3,  // Vector weight (30%)
+  0.7   // BM25 weight (70%)
+);
+
+// Tune BM25 parameters for your corpus
+store.setBM25Parameters(
+  1.2,  // k1: Term frequency saturation (default: 1.2)
+  0.75, // b: Document length normalization (default: 0.75)
+  1.0   // delta: Smoothing parameter (default: 1.0)
+);
+```
+
+Hybrid search is particularly effective for:
+- **Question answering**: BM25 finds documents with exact terms while vectors capture semantic meaning
+- **Knowledge retrieval**: Combines conceptual similarity with keyword matching
+- **Multi-lingual search**: Vectors handle cross-language similarity while BM25 matches exact terms
+
 ## MCP Server Integration
 
 Perfect for building local RAG capabilities in MCP servers:

package/binding.gyp

@@ -2,12 +2,13 @@
   "targets": [
     {
       "target_name": "vector_store",
-      "sources": ["src/binding.cc", "src/vector_store.cpp", "src/vector_store_loader.cpp", "src/vector_store_loader_mmap.cpp", "src/vector_store_loader_adaptive.cpp", "deps/simdjson/simdjson.cpp"],
+      "sources": ["src/binding.cc", "src/vector_store.cpp", "src/simple_tokenizer.cpp", "src/vector_store_loader.cpp", "src/vector_store_loader_mmap.cpp", "src/vector_store_loader_adaptive.cpp", "deps/simdjson/simdjson.cpp"],
       "include_dirs": [
         "<!@(node -p \"require('node-addon-api').include\")",
         "src",
         "deps/simdjson",
-        "deps/atomic_queue"
+        "deps/atomic_queue",
+        "deps"
       ],
       "dependencies": ["<!(node -p \"require('node-addon-api').gyp\")"],
       "cflags_cc": [