knowy 1.0.0 → 1.0.3

package/README.md

@@ -1,108 +1,90 @@
-# Knowy
+# 🧠 KnowyEngine
 
-A local-first, zero-config knowledge base engine with semantic vector search. Built on LanceDB and Hugging Face Transformers.
+**KnowyEngine** is a lightweight, high-performance RAG (Retrieval-Augmented Generation) engine built on **LanceDB** and **HuggingFace Transformers**. It simplifies the process of embedding, storing, and retrieving knowledge with support for scoped knowledge bases, custom metadata, and precise text splitting.
 
-## Features
+## 🚀 Features
 
-- 🔍 **Semantic Search** - Find relevant content using natural language queries
-- 💾 **Local-First** - All data stored locally using LanceDB (no cloud required)
-- 🧠 **Built-in Embeddings** - Uses Qwen3-Embedding-0.6B-ONNX for high-quality vectors
-- 📚 **Knowledge Base Organization** - Group data into named knowledge bases
-- ❓ **Q&A Support** - Store question-answer pairs or free text documents
-- 🚀 **Zero Configuration** - Works out of the box with sensible defaults
+* **Hybrid API**: Use it globally for quick actions or scoped for organized KB management.
+* **Smart Splitting**: Recursive text splitter that preserves sentence integrity and prevents \"semantic dilution.\"
+* **Flexible Metadata**: Store any extra data (user IDs, versions, tags) and filter results using SQL-like queries.
+* **Automatic Timestamps**: Every record is stamped with `__sys_created_at_` for easy time-based filtering.
+* **Local-First**: Powered by ONNX-optimized embeddings and LanceDB for lightning-fast local vector search.
 
-## Installation
+---
+
+## 📦 Installation
 
 ```bash
-npm install knowy
+npm install @lancedb/lancedb @huggingface/transformers
 ```
 
-## Quick Start
-
-```javascript
-import { knowy } from 'knowy';
-
-// Initialize the engine
-const kbs = await knowy('./my_database');
-
-// Create or access a knowledge base
-const docs = kbs('documentation');
+## 🛠️ Usage
 
-// Add Q&A pairs
-await docs.addQA('getting-started', 'How do I install?', 'Run npm install knowy');
+### Initialization
 
-// Or add free text
-await docs.addText('api-reference', 'The API supports vector search with cosine similarity...');
+```javascript
+import { knowy } from './knowy.js';
 
-// Search across your knowledge base
-const results = await docs.search('how to install this package?', { limit: 5 });
-console.log(results);
+const kbs = await knowy(\"./my_knowledge_db\");
 ```
 
-## API Reference
-
-### `knowy(dbPath)`
-
-Initialize the Knowy engine.
-
-- `dbPath` (string): Path to the LanceDB database directory (default: `\"./knowy_database\"`)
-
-Returns a `kbs` function with attached methods.
+### Adding Knowledge
 
-### `kbs(kbName)`
+You can use the **Global API** or the **Scoped API**:
 
-Get or create a knowledge base.
-
-Returns a knowledge base object with methods:
-- `addQA(topic, question, answer)` - Add a Q&A pair
-- `addText(topic, text)` - Add free text content
-- `search(query, options)` - Search within this KB
-- `delete()` - Delete this knowledge base
-
-### `kbs.search(query, options)`
+```javascript
+// Global Style
+await kbs.addText(\"hr\", \"benefits\", \"Unlimited vacation policy.\", \"manual.pdf\", { version: 1.2 });
+
+// Scoped Style (Recommended for clean code)
+const legal = kbs(\"legal\");
+await legal.ingest(\"privacy\", longDocumentText, \"privacy_policy.txt\", {
+  chunkSize: 250,
+  overlap: 50,
+  metadata: { classification: \"confidential\" }
+});
+```
 
-Search across all knowledge bases.
+### Advanced Searching & Filtering
 
-- `query` (string): Search query
-- `options.kbs` (string|string[]): Filter by specific knowledge base(s)
-- `options.limit` (number): Max results (default: 10)
+Retrieve the most relevant context while filtering by your custom metadata:
 
-Returns array of results with `kb`, `topic`, `type`, `content`, and `score` (0-1).
 
-### `kbs.list()`
 
-List all knowledge bases.
+```javascript
+// Search a specific KB with a metadata filter
+const results = await kbs(\"legal\").search(\"What is the retention policy?\", {
+  where: \"classification = 'confidential' AND __sys_created_at_ > 1709400000000\",
+  limit: 3
+});
+
+console.log(results[0].content.text);
+console.log(results[0].metadata.classification); // \"confidential\"
+```
 
-## Example: Multi-KB Setup
+### Management
 
 ```javascript
-const kbs = await knowy();
-
-// Separate KBs for different domains
-const codeKB = kbs('code-snippets');
-const docsKB = kbs('documentation');
-const supportKB = kbs('support-tickets');
+// List all Knowledge Bases
+const list = await kbs.list();
 
-// Add content to each...
-await codeKB.addText('auth', 'JWT authentication implementation...');
-await docsKB.addQA('setup', 'Node version?', 'Requires Node 18+');
-await supportKB.addQA('billing', 'How to cancel?', 'Go to settings > billing...');
+// Delete a KB
+await kbs.delete(\"temp_data\");
+```
 
-// Search everything
-const allResults = await kbs.search('authentication', { limit: 10 });
+---
 
-// Search specific KBs only
-const codeOnly = await kbs.search('authentication', { kbs: 'code-snippets' });
-```
+## ⚙️ Configuration
 
-## Search Scoring
+| Option | Default | Description |
+| :--- | :--- | :--- |
+| `chunkSize` | `250` | Maximum characters per chunk. Lower values increase precision. |
+| `overlap` | `80` | Character overlap between chunks to prevent splitting keywords. |
+| `where` | `undefined` | A SQL-string for metadata filtering (e.g., `\"user_id = 5\"`). |
 
-Results include a `score` from 0 to 1, where 1 is a perfect match. The score is calculated using cosine similarity between the query embedding and stored vectors.
+## 🏗️ Architecture
 
-## Requirements
 
-- Node.js >= 18.0.0
 
-## License
+KnowyEngine uses a **Recursive Splitter** to ensure that data is chunked logically at paragraph and sentence boundaries. These chunks are converted into 384-dimensional vectors using the `Qwen3-Embedding-0.6B-ONNX` model, providing a perfect balance between speed and semantic accuracy for local environments.
 
-MIT © littlejustnode

package/knowy.js

@@ -22,69 +22,121 @@ class KnowyEngine {
     return this;
   }
 
+  _recursiveSplit(text, chunkSize, overlap) {
+    const separators = ["\n\n", "\n", ". ", " "];
+    const split = (str, sepIdx) => {
+      if (str.length <= chunkSize || sepIdx >= separators.length) return [str];
+      const sep = separators[sepIdx];
+      const parts = str.split(sep);
+      let chunks = [];
+      let current = "";
+      for (const part of parts) {
+        if ((current + part).length > chunkSize) {
+          if (current) chunks.push(current.trim());
+          current = current.slice(-overlap) + part + sep;
+        } else {
+          current += part + sep;
+        }
+      }
+      if (current) chunks.push(current.trim());
+      return chunks;
+    };
+    return split(text, 0);
+  }
+
   async _getEmbedding(text) {
-    const output = await this.extractor(text, { pooling: 'last_token', normalize: true });
+    const cleaned = text.replace(/\s+/g, ' ').trim();
+    const output = await this.extractor(cleaned, { pooling: 'last_token', normalize: true });
     return Array.from(output.data);
   }
 
-  async _upsert(record) {
+  async _upsert(records) {
     if (!this.table) {
-      this.table = await this.db.createTable(this.tableName, [record]);
+      this.table = await this.db.createTable(this.tableName, records);
     } else {
-      await this.table.add([record]);
+      await this.table.add(records);
     }
   }
 
   getKB(kbName) {
     return {
       name: kbName,
-      addQA: (topic, q, a) => this.addQA(kbName, topic, q, a),
-      addText: (topic, text) => this.addText(kbName, topic, text),
-      search: (query, options = {}) => this.search(query, { ...options, kbs: kbName }),
+      addQA: (topic, q, a, src, meta) => this.addQA(kbName, topic, q, a, src, meta),
+      addText: (topic, txt, src, meta) => this.addText(kbName, topic, txt, src, meta),
+      ingest: (topic, txt, src, opts) => this.ingest(kbName, topic, txt, src, opts),
+      search: (query, opts = {}) => this.search(query, { ...opts, kbs: kbName }),
       delete: () => this.deleteKB(kbName)
     };
   }
 
-  async addQA(kbName, topic, question, answer) {
+  async addQA(kbName, topic, question, answer, source = "manual", metadata = {}) {
     const vector = await this._getEmbedding(question);
-    await this._upsert({
+    await this._upsert([{
       id: crypto.randomUUID(),
       vector, kb_name: kbName, topic, type: 'qa',
-      question, answer, text: "", created_at: new Date().toISOString()
-    });
+      question, answer, text: "", source, 
+      __sys_created_at_: Date.now(),
+      ...metadata 
+    }]);
   }
 
-  async addText(kbName, topic, text) {
+  async addText(kbName, topic, text, source = "note", metadata = {}) {
     const vector = await this._getEmbedding(text);
-    await this._upsert({
+    await this._upsert([{
       id: crypto.randomUUID(),
-      vector, kb_name: kbName, topic, type: 'text',
-      question: "", answer: "", text, created_at: new Date().toISOString()
-    });
+      vector, 
+      kb_name: kbName, 
+      topic, type: 'text',
+      question: "", answer: "", text, source, 
+      __sys_created_at_: Date.now(),
+      ...metadata 
+    }]);
+  }
+
+  async ingest(kbName, topic, text, source = "file", options = {}) {
+    const { chunkSize = 250, overlap = 80, metadata = {} } = options;
+    const chunks = this._recursiveSplit(text, chunkSize, overlap);
+    const records = [];
+    const timestamp = Date.now();
+    for (const chunk of chunks) {
+      const vector = await this._getEmbedding(chunk);
+      records.push({
+        id: crypto.randomUUID(),
+        vector, kb_name: kbName, topic, type: 'text',
+        question: "", answer: "", text: chunk, source, 
+        __sys_created_at_: timestamp,
+        ...metadata 
+      });
+    }
+    await this._upsert(records);
+    return chunks.length;
   }
 
   async search(query, options = {}) {
     if (!this.table) return [];
-    const { kbs, limit = 10 } = options;
+    const { kbs, limit = 5, where } = options;
     const queryVector = await this._getEmbedding(query);
     let request = this.table.vectorSearch(queryVector);
-	request.metricType = "cosine";
-
+    request.metricType = "cosine";
+    let filters = [];
     if (kbs) {
-      const filter = Array.isArray(kbs) 
+      filters.push(Array.isArray(kbs) 
         ? `kb_name IN (${kbs.map(n => `"${n}"`).join(",")})` 
-        : `kb_name = "${kbs}"`;
-      request = request.where(filter);
+        : `kb_name = "${kbs}"`);
     }
-
+    if (where) filters.push(where);
+    if (filters.length > 0) request = request.where(filters.join(" AND "));
     const results = await request.limit(limit).toArray();
-    return results.map(r => ({
-      kb: r.kb_name,
-      topic: r.topic,
-      type: r.type,
-      content: r.type === 'qa' ? { q: r.question, a: r.answer } : { text: r.text },
-      score: 1-(r._distance/2)
-    }));
+    return results.map(r => {
+      const { vector, _distance, kb_name, topic, type, source, question, answer, text, __sys_created_at_, ...extra } = r;
+      return {
+        kb: kb_name, topic, type, source,
+        content: type === 'qa' ? { q: question, a: answer } : { text },
+        score: 1 - (_distance / 2),
+        __sys_created_at_,
+        metadata: extra 
+      };
+    });
   }
 
   async deleteKB(kbName) {
@@ -98,18 +150,18 @@ class KnowyEngine {
   }
 }
 
-/**
- * Main export: initializes engine and returns the kbs function.
- */
 export const knowy = async (dbPath = "./knowy_database") => {
   const engine = new KnowyEngine(dbPath);
   await engine.init();
 
-  // The kbs function
   const kbs = (kbName) => engine.getKB(kbName);
 
-  // Attached methods
+  // Global methods added back
+  kbs.addQA = (kb, topic, q, a, src, meta) => engine.addQA(kb, topic, q, a, src, meta);
+  kbs.addText = (kb, topic, txt, src, meta) => engine.addText(kb, topic, txt, src, meta);
+  kbs.ingest = (kb, topic, txt, src, opts) => engine.ingest(kb, topic, txt, src, opts);
   kbs.search = (query, options) => engine.search(query, options);
+  kbs.delete = (kb) => engine.deleteKB(kb);
   kbs.list = () => engine.listKBs();
 
   return kbs;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "knowy",
-  "version": "1.0.0",
+  "version": "1.0.3",
   "description": "A local-first knowledge base engine with vector search using LanceDB and Hugging Face embeddings",
   "main": "knowy.js",
   "type": "module",
@@ -16,7 +16,7 @@
     "semantic-search",
     "qa",
     "ai",
-	"knowy",
+    "knowy",
     "local-first"
   ],
   "author": "littlejustnode",
@@ -36,4 +36,4 @@
   "engines": {
     "node": ">=18.0.0"
   }
-}
+}