@kedem/okdb 1.0.0

package/LICENSE

@@ -0,0 +1,57 @@
+OKDB LICENSE AGREEMENT
+Version 1.0
+
+Copyright (c) 2025 Kedem Diamant. All rights reserved.
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+1. PERMITTED USE
+
+   You may use, copy, and integrate this software into your own
+   applications free of charge, provided your use is Non-Commercial
+   as defined in Section 2.
+
+2. NON-COMMERCIAL USE
+
+   "Non-Commercial" means use that is not primarily intended for or
+   directed toward commercial advantage or monetary compensation.
+   This includes personal projects, academic research, internal
+   evaluation, and open-source software. It excludes any use in or
+   as part of a product or service that is sold, licensed, or
+   otherwise monetized.
+
+3. COMMERCIAL USE
+
+   Any Commercial use of this software requires a valid commercial
+   license obtained from the copyright holder. Commercial licenses
+   govern the number of deployments, environments, and features
+   available, as enforced by the license activation system embedded
+   in the software.
+
+   To obtain a commercial license, contact: kedem@storita.ai
+
+4. RESTRICTIONS
+
+   Regardless of the nature of your use, you may not:
+
+   (a) remove, disable, alter, or circumvent any license validation,
+       activation, or enforcement mechanism embedded in the software;
+
+   (b) redistribute this software as a standalone database product,
+       managed service, or competitive offering.
+
+5. DISCLAIMER OF WARRANTIES
+
+   THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF
+   MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
+   NON-INFRINGEMENT.
+
+6. LIMITATION OF LIABILITY
+
+   IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM,
+   DAMAGES, OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT, OR
+   OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE
+   SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

package/README.md

@@ -0,0 +1,344 @@
+<img src="banner.jpg" alt="OKDB — Embedded-First · Sync-Native · AI-Ready" width="100%" />
+
+<br/>
+
+<p align="center">
+  <a href="docs/getting-started.md"><strong>Getting Started</strong></a> ·
+  <a href="docs/querying.md">Querying</a> ·
+  <a href="docs/indexes.md">Indexes</a> ·
+  <a href="docs/fts.md">Full-Text Search</a> ·
+  <a href="docs/embeddings.md">Embeddings</a> ·
+  <a href="docs/sync.md">Sync</a> ·
+  <a href="docs/functions.md">Functions</a> ·
+  <a href="docs/pipelines.md">Pipelines</a> ·
+  <a href="docs/http-api.md">HTTP API</a>
+</p>
+
+---
+
+**OKDB** is an embedded, document-centric database for Node.js. No separate server. No network hop. Open a path, write data.
+
+It wraps [LMDB](https://www.symas.com/lmdb) — one of the fastest memory-mapped key-value stores ever built — and layers on top: typed collections, composite secondary indexes, full-text search, vector embeddings, multi-node sync, a durable job queue, file storage, point-in-time time-machine queries, reactive materialized views, and a built-in HTTP API. All in one process. All ACID.
+
+```js
+const OKDB = require('src/okdb');
+
+const db = new OKDB('./mydb');
+await db.open();
+
+await db.ensureType('users', {
+    indexes: [['role'], ['email']],
+});
+
+await db.put('users', 'alice', { name: 'Alice', role: 'admin', email: 'alice@example.com' });
+
+const alice = db.get('users', 'alice'); // synchronous — instant
+const admins = db.query('users', { role: 'admin' }); // MongoDB-style filter
+```
+
+---
+
+## Why OKDB?
+
+Most Node.js apps reach for a separate database server — Postgres, MongoDB, Redis — even when the workload fits comfortably in a single process. OKDB eliminates that infrastructure layer without sacrificing capability.
+
+|                          | OKDB | SQLite | MongoDB | Redis |
+| ------------------------ | :--: | :----: | :-----: | :---: |
+| Embedded (no server)     |  ✅  |   ✅   |   ❌    |  ❌   |
+| Document model           |  ✅  |   ❌   |   ✅    |  ❌   |
+| Composite indexes        |  ✅  |   ✅   |   ✅    |  ❌   |
+| Full-text search         |  ✅  |   ⚠️   |   ✅    |  ❌   |
+| Vector / semantic search |  ✅  |   ⚠️   |   ✅    |  ❌   |
+| Multi-node sync          |  ✅  |   ❌   |   ✅    |  ✅   |
+| Durable job queue        |  ✅  |   ❌   |   ❌    |  ✅   |
+| Synchronous reads        |  ✅  |   ❌   |   ❌    |  ❌   |
+| ACID transactions        |  ✅  |   ✅   |   ✅    |  ❌   |
+| Point-in-time queries    |  ✅  |   ❌   |   ⚠️    |  ❌   |
+| Materialized views       |  ✅  |   ⚠️   |   ✅    |  ❌   |
+
+---
+
+## Features
+
+### ⚡ Synchronous reads
+
+LMDB maps the database file directly into process memory. Reads are pointer dereferences — no `await`, no round-trip, no copy.
+
+### 🔒 ACID transactions
+
+Inherited from LMDB's copy-on-write B+ trees with MVCC. Partial writes are impossible. Crash-safe by default.
+
+### 🗂️ Typed collections + composite indexes
+
+Organise records into named types (like tables or collections). Declare composite secondary indexes and OKDB maintains them automatically inside every write transaction.
+
+```js
+await db.ensureType('orders', {
+    indexes: [
+        ['status'],
+        ['customerId'],
+        ['customerId', 'createdAt'], // composite — fast filtered range scans
+    ],
+});
+
+// Range scan by composite index
+for (const { key, value } of db.byIndex('orders', ['customerId', 'createdAt'], {
+    start: ['cust-1', 0],
+    end: ['cust-1', Date.now()],
+})) {
+    console.log(value);
+}
+```
+
+### 🔍 Full-text search
+
+Built-in inverted-index FTS backed by a dedicated LMDB environment. No Elasticsearch. No external process.
+
+```js
+await db.fts.register('articles', 'content', { fields: ['title', 'body'] });
+await db.fts.ready('articles', 'content');
+
+const results = await db.fts.search('articles', 'content', 'hello world');
+```
+
+### 🤖 Vector embeddings + semantic search
+
+First-class embedding pipeline. Point it at a type and a field, connect an embedder (Ollama, OpenAI, or custom), and OKDB handles indexing automatically. Nearest-neighbour search uses an in-process HNSW graph.
+
+```js
+await db.embeddings.createPipeline('articles', {
+    source_type: 'articles',
+    field: 'body',
+    dims: 1024,
+    embedder: { type: 'ollama', model: 'mxbai-embed-large', url: 'http://localhost:11434' },
+});
+
+const similar = await db.embeddings.search('articles', queryVector, { limit: 10 });
+```
+
+### 🌍 Geo queries
+
+Built-in geospatial index for radius and bounding-box queries. See [docs/indexes.md](docs/indexes.md).
+
+### 🔄 Multi-node sync
+
+Peer-to-peer last-write-wins replication over HTTP. No coordinator. No Raft. Nodes exchange deltas; HLC timestamps resolve conflicts.
+
+```js
+const db = new OKDB('./node1', {
+    sync: { address: 'http://192.168.1.10:8080' },
+});
+await db.open();
+db.http.listen(8080);
+
+// Create an admin token first, then join a peer with it
+await db.sync.join('http://192.168.1.11:8080', { token: 'cluster-bearer-token' });
+```
+
+### 📬 Durable job queue
+
+Crash-safe job queue stored in LMDB. Workers pull jobs, process them, and acknowledge — all without Redis.
+
+### 📁 File / blob storage
+
+SHA-256 content-addressable binary blob storage, integrated with the sync system. See [docs/files.md](docs/files.md).
+
+### 🧩 Sandboxed custom functions
+
+Store JavaScript functions in the database, validate them, and execute them in pooled sandbox runners.
+
+- all functions execute with env-local facades: `ctx.env`, `ctx.env.queue`, `ctx.env.files`
+- mark a function `unsafe: true` to additionally receive `ctx.okdb` for cross-env access
+- scripts use the contract `[async] (ctx) => {}`
+- `require`, `import`, and `console` are rejected
+- logging flows through `ctx.log(...)`
+
+```js
+await db.env('analytics').functions.create({
+    name: 'countEvents',
+    source: '(ctx) => ({ count: ctx.env.getCount("events") })',
+});
+
+const run = await db.env('analytics').functions.run('countEvents');
+console.log(run.result.value.count);
+```
+
+### 🔌 Built-in HTTP API + Admin UI
+
+Zero-config REST API and a browser-based admin UI for data exploration, index management, and real-time monitoring.
+
+```js
+db.http.listen(8080);
+// REST API:  http://localhost:8080/api/type/users
+// Admin UI:  http://localhost:8080/admin/
+```
+
+### 🧠 MCP endpoint (Inspector-ready example)
+
+OKDB also exposes an MCP server over SSE + POST:
+
+- `GET /mcp/sse`
+- `POST /mcp/messages`
+
+For a local working example, run:
+
+```bash
+node examples/mcp-inspector.js
+```
+
+Then in MCP Inspector choose the SSE transport and connect to:
+
+- `http://localhost:8787/mcp/sse`
+
+The server responds with browser CORS headers for common Inspector origins, accepts preflight `OPTIONS` requests, emits the POST endpoint URL over the SSE `endpoint` event, and exposes the MCP session id via the `mcp-session-id` response header. After connecting, try:
+
+- `tools/list`
+- `tools/call` with generated tools such as:
+    - `system_info`
+    - `system_runtime`
+    - `queue_list_jobs`
+    - `processor_list`
+
+OKDB now derives MCP tools primarily from route operation metadata instead of keeping a separate hand-maintained MCP tool list. Routes marked with canonical metadata and `mcp.include !== false` can participate in MCP generation, while streaming/SSE/raw transport routes stay excluded by default.
+
+If you run Inspector from a different origin, allow that origin in `src/features/mcp/okdb-mcp.js`.
+
+### ⚙️ Reactive processors
+
+Composable change handlers — like database triggers, written in JavaScript. React to writes on any type, in the same transaction or asynchronously.
+
+### ⏱️ Time-machine
+
+Full edit history for every record. Ask what any document looked like at any point in the past — OKDB stores diffs automatically and reconstructs the state on demand.
+
+```js
+// Enable once; tracking starts from current clock
+await db.timeMachine.enable();
+
+// Point-in-time read — synchronous, no await
+const snapshot = db.timeMachine.getStateAt('orders', 'ord-42', historicClock);
+
+// Full history for a single record
+const { diffs, head } = db.timeMachine.getHistory('orders', 'ord-42');
+
+// All changes to a type within a clock range — useful for auditing
+for (const change of db.timeMachine.getChanges('orders', fromClock, toClock)) {
+    console.log(change.key, change.diff);
+}
+```
+
+### 📊 Materialized views
+
+Reactive aggregations that update automatically on every write. Declare a view with built-in reducers (`$count`, `$sum`, `$avg`, `$min`, `$max`, `$countBy`) or supply your own, and OKDB keeps the result in sync.
+
+```js
+await db.views.create('orderStats', {
+    type: 'orders',
+    filter: { status: 'completed' },
+    reduce: {
+        total: { $sum: 'amount' },
+        count: { $count: true },
+        avgValue: { $avg: 'amount' },
+        byRegion: { $countBy: 'region' },
+    },
+});
+
+// Read the live result — synchronous
+const stats = db.views.get('orderStats');
+// → { total: 142500, count: 312, avgValue: 456.7, byRegion: { EU: 140, US: 172 } }
+```
+
+### 🔑 Authentication & authorization
+
+Multi-mode security layer built into the HTTP API. No separate auth service needed.
+
+- **Token / session** — bearer tokens with configurable TTL
+- **OIDC / OAuth2** — discovery-based, JWKS validation, email/domain allowlists
+- **IP allowlists** — restrict access by network range
+
+---
+
+## Installation
+
+```bash
+npm install okdb
+```
+
+> **Runtime:** Node.js only. Requires a native LMDB build (handled automatically by the `lmdb` package via node-gyp).
+
+---
+
+## Quick start
+
+```js
+'use strict';
+const OKDB = require('src/okdb');
+
+async function main() {
+    const db = new OKDB('./demo');
+    await db.open();
+
+    await db.ensureType('products', {
+        indexes: [['category'], ['price'], ['category', 'price']],
+    });
+
+    await db.put('products', 'p1', { name: 'Widget', category: 'tools', price: 9.99 });
+    await db.put('products', 'p2', { name: 'Gadget', category: 'tools', price: 19.99 });
+    await db.put('products', 'p3', { name: 'Doohickey', category: 'misc', price: 4.99 });
+
+    // MongoDB-style filter
+    for (const { key, value } of db.query('products', { category: 'tools' })) {
+        console.log(`${key}: ${value.name} — $${value.price}`);
+    }
+
+    // Index range scan
+    for (const { key, value } of db.byIndex('products', ['price'], { start: [0], end: [10] })) {
+        console.log(`${value.name} costs $${value.price}`);
+    }
+
+    await db.close();
+}
+
+main().catch(console.error);
+```
+
+---
+
+## Documentation
+
+| Topic                                      |                                                                     |
+| ------------------------------------------ | ------------------------------------------------------------------- |
+| [Getting Started](docs/getting-started.md) | Installation, opening a database, first type, writes & reads        |
+| [Querying](docs/querying.md)               | Key reads, range scans, index scans, MongoDB filters                |
+| [Indexes](docs/indexes.md)                 | Composite indexes, unique indexes, geo indexes                      |
+| [Transactions](docs/transactions.md)       | Multi-write atomic transactions                                     |
+| [Full-Text Search](docs/fts.md)            | Registering FTS indexes, tokenizer config, search                   |
+| [Embeddings](docs/embeddings.md)           | Vector pipeline, embedders, HNSW search                             |
+| [Sync](docs/sync.md)                       | Cluster setup, joining peers, conflict resolution                   |
+| [Queue](docs/queue.md)                     | Durable job queue, workers, retry                                   |
+| [Files](docs/files.md)                     | Blob storage, upload, download, sync                                |
+| [Custom Functions](docs/functions.md)      | Stored sandboxed JS functions, env/global execution, HTTP routes    |
+| [Pipelines](docs/pipelines.md)             | Composable data-flow orchestration with queue and processor drivers |
+| [HTTP API](docs/http-api.md)               | REST endpoints, auth, admin UI                                      |
+| [Auth](docs/auth-permissions.md)           | Token, OIDC, IP allowlist authentication                            |
+| [Plugins](docs/plugins.md)                 | Extending OKDB with custom features                                 |
+| [Migration](docs/migration.md)             | Schema evolution, data migration                                    |
+| [Architecture](OKDB-ARCHITECTURE.md)       | Deep-dive technical reference                                       |
+
+---
+
+## Use cases
+
+- **Local-first and offline-capable apps** — the database travels with the user
+- **Desktop / Electron apps** — no bundled server process
+- **Edge and IoT services** — minimal footprint, full capability
+- **Data pipelines** — fast ingest, durable queue, automatic vector indexing
+- **AI-powered apps** — embedding storage and semantic search are first-class
+- **Audit trails and compliance** — time-machine records every change; reconstruct any past state on demand
+- **Analytics dashboards** — materialized views keep aggregates live without separate query jobs
+
+---
+
+## License
+
+See [LICENSE](LICENSE) for details.