@kedem/okdb 1.0.0

package/docs/index.md

@@ -0,0 +1,115 @@
+# OKDB
+
+**An embedded, document-centric database for Node.js.** No separate server. No network hop. Just open a path and write data.
+
+OKDB wraps [LMDB](https://www.symas.com/lmdb) — one of the fastest key-value stores ever built — and adds typed collections, composite secondary indexes, full-text search, vector embeddings, multi-node sync, a durable job queue, and binary blob storage. All in one process, all ACID.
+
+---
+
+## What it is
+
+OKDB is a **structured layer on top of LMDB**. The design decision is simple: LMDB gives you memory-mapped, copy-on-write B+ trees with MVCC and crash-safe writes. OKDB adds everything a real application needs on top of that without changing the embedding model.
+
+Think of it as a document store where reads are synchronous (just a pointer dereference into mmap'd memory), writes are atomic, and all the derived structures — indexes, FTS posting lists, vector graphs, changelogs — are maintained automatically inside the same transaction.
+
+---
+
+## Strengths
+
+- :zap: **Reads are synchronous** — no async, no round trips, no copies
+- :lock: **ACID transactions** — inherited from LMDB; no partial state ever observable
+- :package: **Self-contained** — everything in one process, one set of directories
+- :refresh-cw: **Multi-node sync** — LWW replication over HTTP, no coordinator
+- :search: **Vector + FTS + Geo** — search primitives built in, not bolted on
+- :mail: **Durable queue** — jobs survive crashes, stored in the same database
+- :folder: **File storage** — SHA-256 content-addressable blobs, integrated with sync
+
+---
+
+## What it is not
+
+- Not a client/server database (no PostgreSQL-style network protocol)
+- Not designed for browser or Bun (Node.js only)
+- Not an audit-log — the change-log is LWW-deduplicated per key, not an immutable history
+- Not horizontally scalable in the traditional sense — LMDB is one writer at a time per process
+
+---
+
+## Core concepts at a glance
+
+| Concept         | What it is                                                     |
+| --------------- | -------------------------------------------------------------- |
+| **Type**        | A named collection of records (like a table/collection)        |
+| **Key**         | String primary key for each record                             |
+| **Value**       | Any JSON-serialisable object                                   |
+| **Environment** | An isolated LMDB database directory                            |
+| **Index**       | A composite secondary index on one or more fields              |
+| **Clock**       | A monotonic per-environment integer incremented on every write |
+| **Change log**  | An ordered log of all mutations, keyed by clock value          |
+| **Processor**   | A reactive change handler — like triggers, but composable      |
+
+---
+
+## Architecture overview
+
+```
+Your app code
+      │
+      ▼
+   OKDB  (orchestrator)
+      │
+      ├── ~system env     ← node identity, env registry
+      ├── default env     ← your data, indexes, change log
+      ├── queue env       ← job queue
+      ├── embeddings env  ← vector embedding status
+      └── custom envs     ← whatever you create
+```
+
+All environments are LMDB directories on disk. Each has its own clock, changelog, and HLC-based sync cursor.
+
+---
+
+## Quick example
+
+```javascript
+const OKDB = require('src/okdb');
+
+const okdb = new OKDB('./mydb');
+await okdb.open();
+
+await okdb.registerType('notes');
+await okdb.registerIndex('notes', ['author', 'createdAt']);
+
+await okdb.put('notes', 'n1', { author: 'alice', body: 'Hello world', createdAt: Date.now() });
+await okdb.put('notes', 'n2', { author: 'bob', body: 'Hey there', createdAt: Date.now() });
+
+// Synchronous reads
+const note = okdb.get('notes', 'n1');
+
+// Index scan
+for (const { key, value } of okdb.byIndex('notes', ['author'], { prefix: ['alice'] })) {
+    console.log(key, value);
+}
+
+await okdb.close();
+```
+
+---
+
+## Docs map
+
+- [Getting Started](./getting-started.md) — install, open, first writes
+- [Data Model](./data-model.md) — types, keys, environments, storage layout
+- [Querying](./querying.md) — reads, ranges, MongoDB-style filters
+- [Indexes](./indexes.md) — secondary, composite, unique, geo
+- [Transactions](./transactions.md) — batching, versioning, optimistic concurrency
+- [Change Log & Events](./change-log.md) — clock, changelog, processor, events
+- [Sync](./sync.md) — multi-node replication
+- [Full-Text Search](./fts.md) — FTS index, tokenizer, query
+- [Embeddings](./embeddings.md) — vector pipeline, semantic search
+- [Queue](./queue.md) — durable job queue
+- [Files](./files.md) — blob / file storage
+- [Custom Functions](./functions.md) — sandboxed stored JS functions, env/global scope, HTTP execution
+- [HTTP API](./http-api.md) — built-in REST API
+- [Plugins](./plugins.md) — extending OKDB before open
+- [Migration](./migration.md) — export, import, backup, blob integrity

package/docs/indexes.md

@@ -0,0 +1,207 @@
+# Indexes
+
+Secondary indexes let you scan records by field values without touching every record. They are maintained atomically inside every write transaction — there is no window where an index is out of sync with the data.
+
+---
+
+## Registering an index
+
+```javascript
+// Single field
+await okdb.registerIndex('articles', ['author']);
+
+// Composite — multi-field
+await okdb.registerIndex('articles', ['author', 'publishedAt']);
+
+// Unique constraint
+await okdb.registerIndex('users', ['email'], { unique: true });
+
+// Geospatial
+await okdb.registerIndex('places', ['location'], { type: 'geo', precision: 7 });
+```
+
+### MCP note
+
+Index management is exposed through the grouped `okdb_index` MCP tool.
+Use `action: "create"` and send either the canonical `index` string or the convenience `fields` array.
+
+```json
+{
+    "name": "okdb_index",
+    "arguments": {
+        "action": "create",
+        "env": "default",
+        "type": "places",
+        "fields": ["location"],
+        "indexType": "geo",
+        "precision": 7
+    }
+}
+```
+
+For geo indexes, the indexed field should hold a single object shaped like `{ lat, lon }`.
+That means the correct field path is usually `location`, not `location.lat` + `location.lon` as separate fields.
+
+Indexes build **asynchronously** from existing data after registration. Writes during the build are tracked and the index converges to READY. To wait for completion:
+
+```javascript
+await okdb.indexReady('articles', ['author', 'publishedAt']);
+```
+
+Check status without waiting:
+
+```javascript
+okdb.getIndexStatus('articles', ['author']);
+// → 'creating' | 'resetting' | 'ready'
+```
+
+---
+
+## Dropping an index
+
+```javascript
+await okdb.dropIndex('articles', ['author', 'publishedAt']);
+```
+
+The index sub-database is deleted. Existing query code that references it will throw `OKDBIndexNotRegisteredError`.
+
+---
+
+## Index names
+
+Internally, an index is identified by its fields joined with `~`:
+
+```
+['author']               → "author"
+['author', 'publishedAt'] → "author~publishedAt"
+['status', 'priority', 'when'] → "status~priority~when"
+```
+
+You rarely need to deal with index names directly — all API methods accept `string[]` and normalise for you.
+
+---
+
+## Composite indexes
+
+A composite index orders records by the first field, then the second within ties, and so on. This mirrors SQL's multi-column index semantics.
+
+```javascript
+await okdb.registerIndex('orders', ['customerId', 'createdAt']);
+
+// All orders by customer 'c42', newest first
+for (const { key, value } of okdb.byIndex('orders', ['customerId', 'createdAt'], {
+    prefix:  ['c42'],
+    reverse: true,
+    limit:   50,
+})) { ... }
+```
+
+The trailing `null` sentinel in the LMDB key format ensures that a prefix scan on `['c42']` correctly returns all entries for that customer regardless of their `createdAt` value.
+
+---
+
+## Dot-notation for nested fields
+
+Field paths can use dots to reach nested properties:
+
+```javascript
+await okdb.registerIndex('users', ['address.city']);
+
+// Records where address.city === 'Paris'
+for (const { key, value } of okdb.byIndex('users', ['address.city'], { prefix: ['Paris'] })) { ... }
+```
+
+---
+
+## Unique indexes
+
+A unique index prevents two records from having the same value for the indexed fields.
+
+```javascript
+await okdb.registerIndex('users', ['email'], { unique: true });
+
+// This will throw OKDBUniqueConstraintError:
+await okdb.put('users', 'user-2', { email: 'alice@example.com' }); // if 'alice@example.com' already exists
+```
+
+### Sync conflicts
+
+When a duplicate arrives via sync (from another node), OKDB doesn't reject it — sync data is never lost. Instead, a **violation** is recorded, and a **winner** is deterministically chosen:
+
+> Highest HLC timestamp wins. Ties broken by lexicographically largest `origin` node ID.
+
+Violations are stored locally (not synced) and resolve automatically when the losing record is removed or updated.
+
+```javascript
+// List all violations for a type
+const violations = okdb.getUniqueViolations('users');
+
+// Check if any violations exist for a specific index
+const hasProb = okdb.hasUniqueViolations('users', ['email']);
+
+// Get a specific violation
+const v = okdb.getUniqueViolation('users', ['email'], 'alice@example.com');
+// v = { winner, entries, count, detectedAt, ... }
+```
+
+Events emitted:
+
+```javascript
+okdb.events.on('unique:violation', ({ type, index, indexKey, winner }) => { ... });
+okdb.events.on('unique:violation_resolved', ({ type, index, indexKey }) => { ... });
+```
+
+:::note
+Geo indexes cannot be unique. Geohash collisions make uniqueness semantics undefined — the registration will throw.
+:::
+
+---
+
+## Geo indexes
+
+Geospatial indexes use geohash encoding to map `{ lat, lon }` coordinates into a lexicographically ordered string space.
+
+```javascript
+await okdb.registerIndex('restaurants', ['coords'], { type: 'geo', precision: 7 });
+
+await okdb.put('restaurants', 'r1', { name: 'Le Bistro', coords: { lat: 48.87, lon: 2.31 } });
+
+// Radius search
+const results = okdb.geoQuery('restaurants', 'coords', {
+    lat:    48.87,
+    lon:    2.31,
+    radius: 1000,   // metres
+});
+for (const { key, value, distance } of results) { ... }
+```
+
+**Precision guide:**
+
+| Precision | Cell size | Good for      |
+| --------- | --------- | ------------- |
+| 5         | ~4.9km    | Country-level |
+| 6         | ~1.2km    | City-level    |
+| 7         | ~153m     | Neighbourhood |
+| 8         | ~38m      | Street        |
+| 9         | ~5m       | Building      |
+
+Higher precision = more storage, slower builds, higher accuracy.
+
+---
+
+## `ensureType` shorthand
+
+The most concise way to set up a type with indexes in one call:
+
+```javascript
+await okdb.ensureType('products', {
+    indexes: [
+        ['category'],
+        ['category', 'price'],
+        { fields: ['location'], type: 'geo', precision: 6 },
+        { fields: ['sku'], unique: true },
+    ],
+});
+```
+
+This registers the type if missing, registers each index if missing, and is safe to call on startup every time.

package/docs/logging.md

@@ -0,0 +1,116 @@
+# Logging
+
+OKDB has a built-in structured logger available as `okdb.log`. Logs are emitted as plain objects and routed to any number of attached sinks. A console sink is attached by default.
+
+---
+
+## Entry Shape
+
+Every log call produces an entry with this shape:
+
+```js
+{
+  level:   'info' | 'warn' | 'error' | 'debug',
+  msg:     string,           // human-readable message
+  meta:    object,           // system identity set by child loggers
+  context: any,              // user-provided call-site data (second arg)
+  ts:      number            // Date.now()
+}
+```
+
+`meta` contains machine-readable identity fields (feature, env name, function name, etc). `context` contains whatever you pass as the second argument to a log call — it is for human display only and not filtered on.
+
+### Known `meta` Keys
+
+| Key        | Who sets it                | Example values                                                 |
+| ---------- | -------------------------- | -------------------------------------------------------------- |
+| `feature`  | each feature child logger  | `'queue'`, `'auth'`, `'fts'`, `'sync'`, `'functions'`, `'env'` |
+| `env`      | env logger, functions pool | `'demo-places'`                                                |
+| `fn`       | functions pool (per run)   | `'myFunction'`                                                 |
+| `runId`    | functions pool (per run)   | `'abc123'`                                                     |
+| `runnerId` | functions pool (per run)   | `'fn-1'`                                                       |
+
+---
+
+## `okdb.log` — Basic Usage
+
+```js
+okdb.log.info('server started', { port: 3000 });
+okdb.log.warn('slow query', { ms: 420, type: 'events' });
+okdb.log.error('unexpected failure', { err: err.message });
+okdb.log.debug('cache miss', { key });
+```
+
+---
+
+## `okdb.log.child(meta)` — Scoped Loggers
+
+`child(extraMeta)` returns a new logger instance that shares the same sinks but stamps every entry with additional `meta` fields. Use it to create feature or run-scoped loggers.
+
+```js
+// Feature logger — created once, reused for all calls
+const authLog = okdb.log.child({ feature: 'auth' });
+authLog.warn('invalid token');
+// → { level: 'warn', msg: 'invalid token', meta: { feature: 'auth' }, context: undefined, ts }
+
+// Per-environment logger
+const envLog = okdb.log.child({ feature: 'env', env: 'demo-places' });
+envLog.info('env opened');
+// → { ..., meta: { feature: 'env', env: 'demo-places' }, ... }
+
+// Run-scoped child — created per function invocation
+const runLog = okdb.log.child({ feature: 'functions', env, fn, runId, runnerId });
+runLog.info('function invoked');
+runLog.info('result ready', { value: 42 });
+// → { ..., meta: { feature: 'functions', env, fn, runId, runnerId }, context: { value: 42 }, ts }
+```
+
+`child()` does a shallow merge of meta fields. Child meta overrides parent meta for the same key. The returned logger shares the same sink set — attaching or detaching sinks on the root logger affects all children.
+
+---
+
+## Consuming Logs
+
+### `okdb.log.attach(fn)` — Custom Sink
+
+Attach a function to receive every log entry:
+
+```js
+okdb.log.attach((entry) => {
+    // entry: { level, msg, meta, context, ts }
+    myExternalLogger.log(entry);
+});
+```
+
+To detach, call the returned function:
+
+```js
+const detach = okdb.log.attach(fn);
+detach();
+```
+
+### `okdb.events.on('log', fn)` — Event Bus
+
+Entries are also emitted on the OKDB event bus:
+
+```js
+okdb.events.on('log', (entry) => {
+    console.log(entry.meta?.feature, entry.msg);
+});
+```
+
+### `okdb.logs` — Ring Buffer
+
+A circular buffer of recent log entries is available for inspection:
+
+```js
+okdb.logs; // array of recent entries, newest last
+```
+
+---
+
+## `attachConsole()` — Console Output
+
+`okdb.log.attachConsole()` adds a human-readable console sink. It is called automatically when OKDB starts in development mode.
+
+Function-scoped logs (`meta.feature === 'functions'`) are suppressed from the console because they are stored in function run records and displayed in the Admin UI instead. All other features are printed.

package/docs/manifest.json

@@ -0,0 +1,38 @@
+[
+    { "file": "index.md", "label": "Overview" },
+    { "file": "getting-started.md", "label": "Getting Started" },
+    {
+        "label": "Core",
+        "children": [
+            { "file": "data-model.md", "label": "Data Model" },
+            { "file": "querying.md", "label": "Querying" },
+            { "file": "indexes.md", "label": "Indexes" },
+            { "file": "transactions.md", "label": "Transactions" },
+            { "file": "change-log.md", "label": "Change Log & Events" }
+        ]
+    },
+    {
+        "label": "Features",
+        "children": [
+            { "file": "sync.md", "label": "Sync" },
+            { "file": "fts.md", "label": "Full-Text Search" },
+            { "file": "embeddings.md", "label": "Embeddings" },
+            { "file": "pipelines.md", "label": "Pipelines" },
+            { "file": "queue.md", "label": "Queue" },
+            { "file": "files.md", "label": "Files" },
+            { "file": "functions.md", "label": "Custom Functions" },
+            { "file": "ttl.md", "label": "TTL (Time-to-Live)" }
+        ]
+    },
+    {
+        "label": "Reference",
+        "children": [
+            { "file": "auth-and-sync.md", "label": "Auth & Sync" },
+            { "file": "auth-permissions.md", "label": "Auth Permissions" },
+            { "file": "http-api.md", "label": "HTTP API" },
+            { "file": "plugins.md", "label": "Plugins" },
+            { "file": "logging.md", "label": "Logging" },
+            { "file": "migration.md", "label": "Migration" }
+        ]
+    }
+]

package/docs/migration.md

@@ -0,0 +1,154 @@
+# Migration & Export
+
+OKDB's migration feature handles export/import of all types and records as JSON, plus blob export/import for file storage, and blob integrity tools.
+
+---
+
+## Export
+
+```javascript
+const data = await okdb.migrate.export();
+```
+
+Returns an object:
+
+```javascript
+{
+    types: {
+        users: {
+            entries: [
+                { key: 'alice', value: { name: 'Alice', role: 'admin' }, version: 3 },
+                { key: 'bob',   value: { name: 'Bob',   role: 'member' }, version: 1 },
+            ],
+            indexes:    [...],    // index definitions
+            ftsIndexes: {...},    // FTS index definitions
+            schema:     {...},    // JSON Schema { definition, enforce } if set
+        },
+        orders: { entries: [...] },
+    },
+    meta:    {},     // empty unless includeMeta: true
+    changes: [],     // empty unless includeChangeLog: true
+}
+```
+
+**Indexes** are exported and restored on import. **JSON Schemas** (including foreign key `ref` annotations and `enforce` mode) are preserved across export/import. On import, schemas are applied after all data is loaded so `enforce: true` validation succeeds.
+
+### Options
+
+```javascript
+const data = await okdb.migrate.export({
+    includeMeta: false, // include field schema metadata
+    includeChangeLog: false, // include the clock/change log entries
+});
+```
+
+### Export to JSON string
+
+```javascript
+const json = await okdb.migrate.exportJSON({ pretty: true });
+fs.writeFileSync('./backup.json', json);
+```
+
+---
+
+## Import
+
+```javascript
+const data = JSON.parse(fs.readFileSync('./backup.json', 'utf8'));
+await okdb.migrate.import(data);
+```
+
+:::warning
+Import **drops all existing types** before recreating them. This is a destructive operation. Back up your data first.
+:::
+
+Import recreates all types, populates records with their exact versions (preserving optimistic-concurrency state), and optionally restores the change log.
+
+### Import from JSON string
+
+```javascript
+const json = fs.readFileSync('./backup.json', 'utf8');
+await okdb.migrate.importJSON(json);
+```
+
+---
+
+## Blob export / import
+
+File blobs (from the [Files](./files.md) feature) are stored separately from the OKDB types. Exporting and importing blobs is a separate step:
+
+```javascript
+// Export blobs to a target directory
+const { count, bytes } = await okdb.migrate.exportBlobs('./backup-blobs');
+console.log(`Exported ${count} blobs, ${bytes} bytes`);
+
+// Import blobs from a source directory (skips blobs already present)
+const { imported, skipped } = await okdb.migrate.importBlobs('./backup-blobs');
+console.log(`Imported ${imported}, skipped ${skipped} already present`);
+```
+
+The blob directory layout is preserved: `<dir>/<first2>/<rest>` (matching the SHA-256 hex filename structure).
+
+---
+
+## Blob integrity
+
+Validate that blobs on disk match the `~files` metadata records:
+
+```javascript
+const issues = await okdb.migrate.validateBlobIntegrity();
+
+if (issues.length === 0) {
+    console.log('All blobs intact');
+} else {
+    for (const issue of issues) {
+        console.warn(issue.type, issue.message);
+    }
+}
+```
+
+Issue types returned:
+
+| Type           | Description                                                         |
+| -------------- | ------------------------------------------------------------------- |
+| `missing_blob` | A `~files` record references a hash that has no blob on disk        |
+| `zero_refs`    | A blob's ref-count is zero but a `~files` record still points to it |
+| `orphan_blob`  | A blob file on disk has no `~files` record in any environment       |
+
+### Repair ref-counts
+
+If ref-counts get out of sync (e.g. after a crash or manual blob manipulation), rebuild them from scratch:
+
+```javascript
+const { rebuilt } = await okdb.migrate.repairBlobRefs();
+console.log(`Rebuilt ref-counts for ${rebuilt} blobs`);
+```
+
+---
+
+## Full backup workflow
+
+```javascript
+// 1. Export type data
+const json = await okdb.migrate.exportJSON({ pretty: false });
+fs.writeFileSync('./backup/data.json', json);
+
+// 2. Export blobs
+await okdb.migrate.exportBlobs('./backup/blobs');
+
+console.log('Backup complete');
+```
+
+Restore:
+
+```javascript
+// 1. Import type data
+await okdb.migrate.importJSON(fs.readFileSync('./backup/data.json', 'utf8'));
+
+// 2. Import blobs
+await okdb.migrate.importBlobs('./backup/blobs');
+
+// 3. Optional: verify integrity
+const issues = await okdb.migrate.validateBlobIntegrity();
+if (issues.length > 0) console.warn(issues);
+```