@ouim/vectoriadb-server 0.1.1 → 0.1.3

package/README.md

@@ -0,0 +1,200 @@
+# VectoriaDB SDK
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Version](https://img.shields.io/badge/version-1.0.0-blue.svg)]()
+
+A lightweight, JavaScript-only client/server SDK for [VectoriaDB](https://github.com/agentfront/vectoriadb). Forward vector database operations from client applications to a centralized server instance with ease.
+
+> ⚠️ **Not for large production deployments.** This SDK is primarily intended for small applications, prototypes, demos, and hobby projects where quick setup and simplicity are priorities. For mission‑critical or large-scale production use, consider a hardened, production-grade solution.
+
+---
+
+## Features
+
+- **Mirror API**: Client SDK replicates the full VectoriaDB API surface.
+- **Remote Execution**: Perform heavy vector operations on the server; keep your client light.
+- **Smart Filtering**: Send JavaScript functions as filters; they are automatically serialized and executed on the server.
+- **Streaming Support**: Smoothly handle large result sets with built-in chunking.
+- **Robust Connection**: Automatic reconnection, request queueing, and configurable timeouts.
+- **Collection Helpers**: High-level abstractions for simplified document management.
+
+## Project Structure
+
+```text
+vectoriadb-sdk/
+├── server/         # Server implementation (hosts VectoriaDB instance)
+└── client/         # Client implementation (forwards requests via Socket.io)
+```
+
+## Quick Start
+
+### 1. Start the Server
+
+The server requires `vectoriadb` to be installed.
+
+```bash
+cd server
+npm install
+npm start
+```
+
+**Basic Server Configuration (`server/index.js`):**
+
+```javascript
+import VectoriaDBServer from '@ouim/vectoriadb-server'
+
+const server = new VectoriaDBServer({
+  port: 3001,
+  cors: ['http://localhost:3000'],
+  // apiKey: 'your-secure-key' // Optional authentication
+})
+
+await server.listen()
+```
+
+### 2. Connect the Client
+
+```bash
+npm install @ouim/vectoriadb-client
+```
+
+**Basic Usage:**
+
+```javascript
+import VectoriaDB from '@ouim/vectoriadb-client'
+
+const db = new VectoriaDB({
+  serverUrl: 'http://localhost:3001',
+  // apiKey: 'your-secure-key'
+})
+
+await db.initialize()
+
+// Add documents
+await db.add('doc1', 'Vector databases are awesome!', { category: 'tech' })
+
+// Search with semantic similarity
+const results = await db.search('vector database', { topK: 5 })
+console.log(results)
+```
+
+---
+
+## Advanced Search & Filtering
+
+### Remote Function Filtering
+
+One of the most powerful features is the ability to filter results server-side using client-defined functions:
+
+```javascript
+const results = await db.search('cloud computing', {
+  filter: metadata => metadata.category === 'tech' && metadata.priority > 5,
+  threshold: 0.7, // Strict similarity matching
+})
+```
+
+### Collection-Style API
+
+Use `insert` and `query` for a more traditional database feel:
+
+```javascript
+await db.insert('my-collection', [
+  { text: 'Hello World', metadata: { author: 'Alice' } },
+  { text: 'Goodbye World', metadata: { author: 'Bob' } },
+])
+
+const bobDocs = await db.query('my-collection', 'farewell', {
+  filter: m => m.author === 'Bob',
+})
+```
+
+---
+
+## ⚙️ Configuration
+
+### Server Options
+
+| Option            | Description                            | Default    |
+| :---------------- | :------------------------------------- | :--------- |
+| `port`            | Server listening port                  | `3001`     |
+| `host`            | Server host address                    | `0.0.0.0`  |
+| `apiKey`          | Optional key for client authentication | `null`     |
+| `cors`            | Allowed origins (array)                | `[]` (All) |
+| `streamChunkSize` | Max results per chunk for streaming    | `500`      |
+
+### Client Options
+
+| Option           | Description                  | Default      |
+| :--------------- | :--------------------------- | :----------- |
+| `serverUrl`      | URL of the VectoriaDB server | **Required** |
+| `apiKey`         | Authentication key           | `null`       |
+| `requestTimeout` | API request timeout in ms    | `30000`      |
+
+---
+
+## Large Datasets & Streaming
+
+The SDK automatically handles large result sets by chunking data on the server and assembling it on the client. This prevents memory issues and payload limits when retrieving thousands of vectors.
+
+- **Automatic assembly**: You don't need to worry about chunks; the `Promise` resolves only when all data has arrived.
+- **Configurable limits**: Set `streamChunkSize` on the server to tune the chunking behavior.
+
+## Connection Resilience
+
+Built for real-world networks, the client includes:
+
+- **Offline Queueing**: Requests made while the connection is down are queued and sent automatically upon reconnection.
+- **Heartbeat & Health Checks**: Monitors connection status to ensure reliable delivery.
+- **Configurable Timeouts**: Each request can have its own timeout, or use a global default.
+
+---
+
+## Security Note
+
+> [!WARNING] Filter functions passed from the client are serialized and evaluated on the server using `new Function()`. **Never** expose the server to untrusted clients without strict network controls or additional sandboxing.
+
+---
+
+## Docker Integration
+
+When deploying with Docker, ensure you persist the database state:
+
+```yaml
+services:
+  vectoriadb-server:
+    image: node:18
+    # ... other config
+    volumes:
+      - vectoria-data:/app/server/.cache/vectoriadb
+
+volumes:
+  vectoria-data:
+```
+
+## Best Practices
+
+- **Intended use**: Not recommended for large production systems — best suited for prototypes, demos, and hobby projects where quick setup matters.
+- **Batching**: Use `addMany` instead of repeated `add` calls for efficiency.
+- **Timeouts**: Adjust `requestTimeout` for large-scale operations.
+- **Shutdown**: Always call `db.close()` on the client and `server.close()` on the server for graceful termination.
+
+---
+
+## FAQ
+
+**Q: Do I need `vectoriadb` on the client?**  
+A: No! The client only needs `socket.io-client`. It's designed to be used in browsers or lightweight Node environments.
+
+**Q: How does the server-side filtering work?**  
+A: The client SDK stringifies your filter function. The server then reconstructs it using `new Function()`. This is why the filter must be self-contained and not rely on variables outside its scope.
+
+---
+
+## Credits
+
+This SDK builds on and integrates with [VectoriaDB](https://github.com/agentfront/vectoriadb). See the Vectoria product website at [https://agentfront.dev/vectoria](https://agentfront.dev/vectoria) for more information and additional resources.
+
+---
+
+## License
+
+MIT © 2026 VectoriaDB SDK Authors

package/index.js

@@ -1,23 +1,23 @@
 import VectoriaDBServer from './vectoriadb-server.js'
 import { FileStorageAdapter } from 'vectoriadb'
 
-// If run directly, start a demo server
-if (process.argv[1] && process.argv[1].endsWith('index.js')) {
-  const server = new VectoriaDBServer({
-    port: process.env.PORT ? Number(process.env.PORT) : 3001,
-    host: '0.0.0.0',
-    vectoriadbConfig: {
-      // default storage / model - user should override for production
-      storageAdapter: new FileStorageAdapter({ cacheDir: '/data/vectoriadb', namespace: 'default' }),
-    },
-    cors: ['http://localhost:3000'],
-  })
+// // If run directly, start a demo server
+// if (process.argv[1] && process.argv[1].endsWith('index.js')) {
+//   const server = new VectoriaDBServer({
+//     port: process.env.PORT ? Number(process.env.PORT) : 3001,
+//     host: '0.0.0.0',
+//     vectoriadbConfig: {
+//       // default storage / model - user should override for production
+//       storageAdapter: new FileStorageAdapter({ cacheDir: '/data/vectoriadb', namespace: 'default' }),
+//     },
+//     cors: ['http://localhost:3000'],
+//   })
 
-  server.listen().catch(err => {
-    console.error(err)
-    process.exit(1)
-  })
-}
+//   server.listen().catch(err => {
+//     console.error(err)
+//     process.exit(1)
+//   })
+// }
 
-export { VectoriaDBServer }
+export { VectoriaDBServer, FileStorageAdapter }
 export default VectoriaDBServer

package/listen-race-test.js

@@ -0,0 +1,28 @@
+import VectoriaDBServer from './vectoriadb-server.js'
+import { FileStorageAdapter } from 'vectoriadb'
+
+async function run() {
+  const server = new VectoriaDBServer({
+    port: 3002,
+    vectoriadbConfig: {
+      storageAdapter: new FileStorageAdapter({ cacheDir: './.cache/test-vectoriadb', namespace: 'test' }),
+    },
+  })
+
+  try {
+    // call listen() twice concurrently to reproduce the race if present
+    const results = await Promise.all([server.listen(), server.listen()])
+    console.log('listen() calls resolved, same instance:', results[0] === results[1])
+    console.log('server._started =', server._started)
+  } catch (err) {
+    console.error('concurrent listen failed:', err)
+    process.exitCode = 1
+  } finally {
+    await server.close()
+  }
+}
+
+run().catch(err => {
+  console.error(err)
+  process.exit(1)
+})

package/package.json

@@ -1,12 +1,11 @@
 {
   "name": "@ouim/vectoriadb-server",
-  "version": "0.1.1",
+  "version": "0.1.3",
   "type": "module",
   "main": "index.js",
   "scripts": {
     "start": "node index.js",
-    "demo": "node demo.js",
-    "publish": "npm publish --access public"
+    "demo": "node demo.js"
   },
   "dependencies": {
     "@huggingface/transformers": "^3.8.1",

package/vectoriadb-server.js

@@ -19,63 +19,89 @@ export default class VectoriaDBServer {
     this._vectoria = null
     this._sockets = new Set()
     this._started = false
+    this._startPromise = null
   }
 
   async listen() {
-    if (this._started) return
-    // Initialize VectoriaDB (loads models / storage as configured)
-    this._vectoria = new VectoriaDB(this.vectoriadbConfig)
-    if (typeof this._vectoria.initialize === 'function') {
-      await this._vectoria.initialize()
-    }
-
-    this._http = http.createServer((req, res) => {
-      res.writeHead(200, { 'Content-Type': 'text/plain' })
-      res.end('VectoriaDB Server')
-    })
-
-    this._io = new IOServer(this._http, {
-      cors: { origin: this.cors.length ? this.cors : '*', methods: ['GET', 'POST'] },
-      maxHttpBufferSize: 1e7,
-    })
+    // fast-path: already started
+    if (this._started) return this
 
-    const nsp = this._io.of('/vectoriadb')
+    // if a start is already in progress, return the same promise
+    if (this._startPromise) return this._startPromise
 
-    nsp.use((socket, next) => {
-      // simple API key auth if configured
-      if (this.apiKey) {
-        const provided = socket.handshake.auth?.apiKey || socket.handshake.query?.apiKey
-        if (!provided || provided !== this.apiKey) {
-          return next(new Error('Unauthorized'))
+    this._startPromise = (async () => {
+      try {
+        // Initialize VectoriaDB (loads models / storage as configured)
+        this._vectoria = new VectoriaDB(this.vectoriadbConfig)
+        if (typeof this._vectoria.initialize === 'function') {
+          await this._vectoria.initialize()
         }
-      }
-      next()
-    })
-
-    nsp.on('connection', socket => {
-      this._sockets.add(socket)
-      socket.on('disconnect', () => this._sockets.delete(socket))
 
-      socket.on('request', async payload => {
-        // payload: { id, method, params, collection, timestamp }
+        this._http = http.createServer((req, res) => {
+          res.writeHead(200, { 'Content-Type': 'text/plain' })
+          res.end('VectoriaDB Server')
+        })
+
+        this._io = new IOServer(this._http, {
+          cors: { origin: this.cors.length ? this.cors : '*', methods: ['GET', 'POST'] },
+          maxHttpBufferSize: 1e7,
+        })
+
+        const nsp = this._io.of('/vectoriadb')
+
+        nsp.use((socket, next) => {
+          // simple API key auth if configured
+          if (this.apiKey) {
+            const provided = socket.handshake.auth?.apiKey || socket.handshake.query?.apiKey
+            if (!provided || provided !== this.apiKey) {
+              return next(new Error('Unauthorized'))
+            }
+          }
+          next()
+        })
+
+        nsp.on('connection', socket => {
+          this._sockets.add(socket)
+          socket.on('disconnect', () => this._sockets.delete(socket))
+
+          socket.on('request', async payload => {
+            // payload: { id, method, params, collection, timestamp }
+            try {
+              await this._handleRequest(socket, payload)
+            } catch (err) {
+              socket.emit('response', { id: payload?.id ?? null, result: null, error: { message: err.message }, took: 0 })
+            }
+          })
+
+          // allow ping from client
+          socket.on('health', cb => cb && cb({ ok: true, ts: Date.now() }))
+        })
+
+        await new Promise((res, rej) => {
+          this._http.listen(this.port, this.host, err => (err ? rej(err) : res()))
+        })
+
+        this._started = true
+        console.log(`VectoriaDB Server listening on ${this.host}:${this.port}`)
+        return this
+      } catch (err) {
+        // cleanup partial resources on failure
         try {
-          await this._handleRequest(socket, payload)
-        } catch (err) {
-          socket.emit('response', { id: payload?.id ?? null, result: null, error: { message: err.message }, took: 0 })
-        }
-      })
-
-      // allow ping from client
-      socket.on('health', cb => cb && cb({ ok: true, ts: Date.now() }))
-    })
-
-    await new Promise((res, rej) => {
-      this._http.listen(this.port, this.host, err => (err ? rej(err) : res()))
-    })
+          if (this._io) await this._io.close()
+        } catch (e) {}
+        try {
+          if (this._http) this._http.close(() => {})
+        } catch (e) {}
+        throw err
+      }
+    })()
 
-    this._started = true
-    console.log(`VectoriaDB Server listening on ${this.host}:${this.port}`)
-    return this
+    try {
+      return await this._startPromise
+    } finally {
+      // ensure the in-progress marker is cleared after attempt completes
+      this._startPromise = null
+    }
   }
 
   async _handleRequest(socket, payload) {