native-vector-store 0.3.2 → 0.3.4

package/README.md

@@ -192,16 +192,29 @@ Each JSON file contains self-contained documents with embeddings:
 
 ```json
 {
-  "id": "unique-id",
-  "text": "Document content...",
-  "metadata": {
-    "embedding": [0.1, 0.2, ...],
-    "category": "product",
-    "lastUpdated": "2024-01-01"
+  "id": "unique-id",              // Required: unique document identifier
+  "text": "Document content...",   // Required: searchable text content (or use "content" for Spring AI)
+  "metadata": {                    // Required: metadata object
+    "embedding": [0.1, 0.2, ...],  // Required: array of numbers matching vector dimensions
+    "category": "product",         // Optional: additional metadata
+    "lastUpdated": "2024-01-01"    // Optional: additional metadata
   }
 }
 ```
 
+**Spring AI Compatibility**: You can use `"content"` instead of `"text"` for the document field. The library auto-detects which field name you're using from the first document and optimizes subsequent lookups.
+
+**Common Mistakes:**
+- ❌ Putting `embedding` at the root level instead of inside `metadata`
+- ❌ Using string format for embeddings instead of number array
+- ❌ Missing required fields (`id`, `text`, or `metadata`)
+- ❌ Wrong embedding dimensions (must match VectorStore constructor)
+
+**Validate your JSON format:**
+```bash
+node node_modules/native-vector-store/examples/validate-format.js your-file.json
+```
+
 ### Deployment Strategies
 
 #### Blue-Green Deployment

package/docs/VectorStoreWrapper.html

@@ -815,7 +815,7 @@ Supports both single documents and arrays of documents per file.</p>
 store.loadDir('./knowledge-base');
 // Store is automatically finalized and ready for searches</code></pre>
 
-    <pre class="prettyprint"><code>// Document format in JSON files
+    <pre class="prettyprint"><code>// Standard format with 'text' field
 {
   "id": "doc-123",
   "text": "Document content...",
@@ -825,6 +825,16 @@ store.loadDir('./knowledge-base');
   }
 }</code></pre>
 
+    <pre class="prettyprint"><code>// Spring AI format with 'content' field
+{
+  "id": "doc-456",
+  "content": "Document content...",  // 'content' instead of 'text'
+  "metadata": {
+    "embedding": [0.1, 0.2, ...],
+    "category": "spring-ai"
+  }
+}</code></pre>
+
 
 
         

package/docs/global.html

@@ -137,6 +137,8 @@
         <th>Type</th>
 
         
+        <th>Attributes</th>
+        
 
         
 
@@ -162,6 +164,12 @@
             </td>
 
             
+                <td class="attributes">
+                
+
+                
+                </td>
+            
 
             
 
@@ -185,10 +193,49 @@
             </td>
 
             
+                <td class="attributes">
+                
+                    &lt;optional><br>
+                
+
+                
+                </td>
+            
 
             
 
-            <td class="description last"><p>The text content of the document</p></td>
+            <td class="description last"><p>The text content of the document (mutually exclusive with content)</p></td>
+        </tr>
+
+    
+
+        <tr>
+            
+                <td class="name"><code>content</code></td>
+            
+
+            <td class="type">
+            
+                
+<span class="param-type">string</span>
+
+
+            
+            </td>
+
+            
+                <td class="attributes">
+                
+                    &lt;optional><br>
+                
+
+                
+                </td>
+            
+
+            
+
+            <td class="description last"><p>Alternative to text for Spring AI compatibility (mutually exclusive with text)</p></td>
         </tr>
 
     
@@ -208,6 +255,12 @@
             </td>
 
             
+                <td class="attributes">
+                
+
+                
+                </td>
+            
 
             
 

package/docs/index.html

@@ -212,15 +212,26 @@ const supportResults = supportStore.search(queryEmbedding, 5);
 </code></pre>
 <p>Each JSON file contains self-contained documents with embeddings:</p>
 <pre class="prettyprint source lang-json"><code>{
-  &quot;id&quot;: &quot;unique-id&quot;,
-  &quot;text&quot;: &quot;Document content...&quot;,
-  &quot;metadata&quot;: {
-    &quot;embedding&quot;: [0.1, 0.2, ...],
-    &quot;category&quot;: &quot;product&quot;,
-    &quot;lastUpdated&quot;: &quot;2024-01-01&quot;
+  &quot;id&quot;: &quot;unique-id&quot;,              // Required: unique document identifier
+  &quot;text&quot;: &quot;Document content...&quot;,   // Required: searchable text content (or use &quot;content&quot; for Spring AI)
+  &quot;metadata&quot;: {                    // Required: metadata object
+    &quot;embedding&quot;: [0.1, 0.2, ...],  // Required: array of numbers matching vector dimensions
+    &quot;category&quot;: &quot;product&quot;,         // Optional: additional metadata
+    &quot;lastUpdated&quot;: &quot;2024-01-01&quot;    // Optional: additional metadata
   }
 }
 </code></pre>
+<p><strong>Spring AI Compatibility</strong>: You can use <code>&quot;content&quot;</code> instead of <code>&quot;text&quot;</code> for the document field. The library auto-detects which field name you're using from the first document and optimizes subsequent lookups.</p>
+<p><strong>Common Mistakes:</strong></p>
+<ul>
+<li>❌ Putting <code>embedding</code> at the root level instead of inside <code>metadata</code></li>
+<li>❌ Using string format for embeddings instead of number array</li>
+<li>❌ Missing required fields (<code>id</code>, <code>text</code>, or <code>metadata</code>)</li>
+<li>❌ Wrong embedding dimensions (must match VectorStore constructor)</li>
+</ul>
+<p><strong>Validate your JSON format:</strong></p>
+<pre class="prettyprint source lang-bash"><code>node node_modules/native-vector-store/examples/validate-format.js your-file.json
+</code></pre>
 <h3 id="deployment-strategies">Deployment Strategies</h3>
 <h4 id="blue-green-deployment">Blue-Green Deployment</h4>
 <pre class="prettyprint source lang-javascript"><code>// Load new version without downtime

package/index.js

@@ -3,7 +3,8 @@ const { VectorStore } = require('node-gyp-build')(__dirname);
 /**
  * @typedef {Object} Document
  * @property {string} id - Unique identifier for the document
- * @property {string} text - The text content of the document
+ * @property {string} [text] - The text content of the document (mutually exclusive with content)
+ * @property {string} [content] - Alternative to text for Spring AI compatibility (mutually exclusive with text)
  * @property {Object} metadata - Document metadata
  * @property {number[]} metadata.embedding - The embedding vector for the document
  * @property {*} [metadata.*] - Additional metadata properties
@@ -60,7 +61,7 @@ class VectorStoreWrapper {
    * // Store is automatically finalized and ready for searches
    * 
    * @example
-   * // Document format in JSON files
+   * // Standard format with 'text' field
    * {
    *   "id": "doc-123",
    *   "text": "Document content...",
@@ -69,6 +70,17 @@ class VectorStoreWrapper {
    *     "category": "product"           // Optional: additional metadata
    *   }
    * }
+   * 
+   * @example
+   * // Spring AI format with 'content' field
+   * {
+   *   "id": "doc-456",
+   *   "content": "Document content...",  // 'content' instead of 'text'
+   *   "metadata": {
+   *     "embedding": [0.1, 0.2, ...],
+   *     "category": "spring-ai"
+   *   }
+   * }
    */
   loadDir(path) {}
 

package/lib/index.d.ts

@@ -1,6 +1,7 @@
 export interface Document {
   id: string;
-  text: string;
+  text?: string;      // Either text or content is required
+  content?: string;   // Spring AI compatibility
   metadata: {
     embedding?: number[];
     [key: string]: any;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "native-vector-store",
-  "version": "0.3.2",
+  "version": "0.3.4",
   "description": "High-performance local vector store with SIMD optimization for MCP servers",
   "main": "index.js",
   "types": "lib/index.d.ts",
@@ -13,7 +13,7 @@
   "bugs": {
     "url": "https://github.com/mboros1/native-vector-store/issues"
   },
-  "homepage": "https://github.com/mboros1/native-vector-store#readme",
+  "homepage": "https://mboros1.github.io/native-vector-store/",
   "scripts": {
     "build": "node-gyp rebuild",
     "clean": "node-gyp clean",

package/src/vector_store.cpp

@@ -130,10 +130,58 @@ simdjson::error_code VectorStore::add_document(simdjson::ondemand::object& json_
     // Parse with error handling
     std::string_view id, text;
     auto error = json_doc["id"].get_string().get(id);
-    if (error) return error;
+    if (error) {
+        if (error == simdjson::NO_SUCH_FIELD) {
+            fprintf(stderr, "Missing required field 'id'\n");
+        }
+        return error;
+    }
     
-    error = json_doc["text"].get_string().get(text);
-    if (error) return error;
+    // Auto-detect text field type on first document, then use that for all subsequent documents
+    TextFieldType field_type = text_field_type_.load(std::memory_order_acquire);
+    
+    if (field_type == TextFieldType::UNKNOWN) {
+        // First document - detect field type
+        error = json_doc["text"].get_string().get(text);
+        if (!error) {
+            // Found 'text' field - use it for all documents
+            TextFieldType expected = TextFieldType::UNKNOWN;
+            text_field_type_.compare_exchange_strong(expected, TextFieldType::TEXT, std::memory_order_release);
+        } else if (error == simdjson::NO_SUCH_FIELD) {
+            // Try 'content' field
+            error = json_doc["content"].get_string().get(text);
+            if (!error) {
+                // Found 'content' field - use it for all documents
+                TextFieldType expected = TextFieldType::UNKNOWN;
+                text_field_type_.compare_exchange_strong(expected, TextFieldType::CONTENT, std::memory_order_release);
+            } else {
+                if (error == simdjson::NO_SUCH_FIELD) {
+                    fprintf(stderr, "Missing required field 'text' or 'content'\n");
+                }
+                return error;
+            }
+        } else {
+            return error;
+        }
+    } else if (field_type == TextFieldType::TEXT) {
+        // Use 'text' field directly
+        error = json_doc["text"].get_string().get(text);
+        if (error) {
+            if (error == simdjson::NO_SUCH_FIELD) {
+                fprintf(stderr, "Missing required field 'text' (detected from first document)\n");
+            }
+            return error;
+        }
+    } else { // TextFieldType::CONTENT
+        // Use 'content' field directly
+        error = json_doc["content"].get_string().get(text);
+        if (error) {
+            if (error == simdjson::NO_SUCH_FIELD) {
+                fprintf(stderr, "Missing required field 'content' (detected from first document)\n");
+            }
+            return error;
+        }
+    }
     
     // Calculate sizes
     size_t emb_size = dim_ * sizeof(float);
@@ -147,11 +195,21 @@ simdjson::error_code VectorStore::add_document(simdjson::ondemand::object& json_
     // Process metadata and embedding first
     simdjson::ondemand::object metadata;
     error = json_doc["metadata"].get_object().get(metadata);
-    if (error) return error;
+    if (error) {
+        if (error == simdjson::NO_SUCH_FIELD) {
+            fprintf(stderr, "Missing required field 'metadata'\n");
+        }
+        return error;
+    }
     
     simdjson::ondemand::array emb_array;
     error = metadata["embedding"].get_array().get(emb_array);
-    if (error) return error;
+    if (error) {
+        if (error == simdjson::NO_SUCH_FIELD) {
+            fprintf(stderr, "Missing required field 'embedding' inside 'metadata'\n");
+        }
+        return error;
+    }
     
     // Consume the array before touching anything else  
     size_t i = 0;

package/src/vector_store.h

@@ -74,6 +74,10 @@ private:
     std::atomic<bool> is_finalized_{false};  // Simple flag: false = loading, true = serving
     mutable std::shared_mutex search_mutex_;  // Protects against overlapping OpenMP teams
     
+    // Auto-detect text field name for Spring AI compatibility
+    enum class TextFieldType { UNKNOWN, TEXT, CONTENT };
+    std::atomic<TextFieldType> text_field_type_{TextFieldType::UNKNOWN};
+    
 public:
     explicit VectorStore(size_t dim);
     

package/src/vector_store_loader.cpp

@@ -135,6 +135,14 @@ void VectorStoreLoader::loadDirectory(VectorStore* store, const std::string& pat
                                 if (add_error) {
                                     fprintf(stderr, "Error adding document from %s: %s\n", 
                                            data->filename.c_str(), simdjson::error_message(add_error));
+                                    if (add_error == simdjson::NO_SUCH_FIELD) {
+                                        fprintf(stderr, "  Expected JSON format: {\"id\": string, \"text\": string, \"metadata\": {\"embedding\": [numbers...]}}\n");
+                                        fprintf(stderr, "  Required fields: id, text (or content), metadata.embedding\n");
+                                        fprintf(stderr, "  Note: 'embedding' must be inside 'metadata' object\n");
+                                        fprintf(stderr, "  Note: 'text' and 'content' are interchangeable (Spring AI compatibility)\n");
+                                    } else if (add_error == simdjson::INCORRECT_TYPE) {
+                                        fprintf(stderr, "  Possible causes: wrong embedding dimensions or store already finalized\n");
+                                    }
                                 }
                             }
                         }
@@ -147,6 +155,14 @@ void VectorStoreLoader::loadDirectory(VectorStore* store, const std::string& pat
                             if (add_error) {
                                 fprintf(stderr, "Error adding document from %s: %s\n", 
                                        data->filename.c_str(), simdjson::error_message(add_error));
+                                if (add_error == simdjson::NO_SUCH_FIELD) {
+                                    fprintf(stderr, "  Expected JSON format: {\"id\": string, \"text\": string, \"metadata\": {\"embedding\": [numbers...]}}\n");
+                                    fprintf(stderr, "  Required fields: id, text (or content), metadata.embedding\n");
+                                    fprintf(stderr, "  Note: 'embedding' must be inside 'metadata' object\n");
+                                    fprintf(stderr, "  Note: 'text' and 'content' are interchangeable (Spring AI compatibility)\n");
+                                } else if (add_error == simdjson::INCORRECT_TYPE) {
+                                    fprintf(stderr, "  Possible causes: wrong embedding dimensions or store already finalized\n");
+                                }
                             }
                         }
                     }