@kedem/okdb 1.0.0

package/docs/change-log.md

@@ -0,0 +1,169 @@
+# Change Log & Events
+
+Every write in OKDB — put, remove, registerType, registerIndex — increments a monotonic integer **clock** and appends an entry to the change log. This is the backbone of sync, the processor system, and reactive change handling.
+
+---
+
+## The clock
+
+A per-environment monotonic integer. Starts at 0, increments by 1 on every write transaction.
+
+```javascript
+// Current clock value
+const clock = okdb.getClock();
+
+// Per-type clock (highest clock seen for that type)
+const usersClock = okdb.getClock('users');
+```
+
+---
+
+## Reading the change log
+
+```javascript
+// Get changes between clock 100 and 200 (both inclusive)
+const changes = okdb.getChanges(null, 100, 200);
+for (const change of changes) {
+    console.log(change);
+    // {
+    //   clock:     142,
+    //   id:        'item:users@alice',
+    //   type:      'users',
+    //   key:       'alice',
+    //   action:    'put',           // 'put' | 'remove' | 'type_register' | 'type_drop' | 'index_register' | 'index_drop'
+    //   timestamp: 1709900000000,  // HLC value (wall-clock ms + logical counter)
+    //   origin:    'node-uuid',
+    //   txnId:     'txn-uuid',
+    // }
+}
+
+// Changes for a specific type only
+const userChanges = okdb.getChanges('users', 0, okdb.getClock());
+```
+
+:::note Deduplication
+The change log keeps only the **most recent** entry per primary key. If you write to `users/alice` three times, only the last write appears in the log. This is intentional — `getChanges` returns "what is the current state delta", not a full history.
+:::
+
+---
+
+## Events
+
+OKDB emits events on `okdb.events` (a standard Node.js `EventEmitter`) after each transaction commits.
+
+### Data events
+
+```javascript
+okdb.events.on('item:create', ({ type, key, value, clock, origin }) => {
+    console.log(`Created ${type}/${key}`);
+});
+
+okdb.events.on('item:update', ({ type, key, value, oldValue, clock }) => {
+    console.log(`Updated ${type}/${key}`);
+});
+
+okdb.events.on('item:remove', ({ type, key, oldValue, clock }) => {
+    console.log(`Removed ${type}/${key}`);
+});
+```
+
+### System events
+
+```javascript
+okdb.events.on('system:clock_change', ({ clock, type }) => { ... });
+okdb.events.on('system:clock_change@users', ({ clock }) => { ... }); // type-specific
+okdb.events.on('system:ready', (okdb) => { ... });
+okdb.events.on('system:stopped', () => { ... });
+okdb.events.on('env:opened', ({ name }) => { ... });
+```
+
+Events fire **after** the LMDB transaction is durably committed. Listeners never see speculative data.
+
+:::warning Backpressure
+Events are emitted synchronously via `EventEmitter`. A slow listener blocks subsequent event delivery. For heavy processing, use the **Processor** system instead.
+:::
+
+---
+
+## Processor
+
+The Processor is OKDB's reactive change handler — a composable, configurable, inline-or-queue mechanism for reacting to type mutations.
+
+Think of it as event listeners with superpowers: they can filter by origin, run inline or async, and are managed as named, stoppable handlers.
+
+### Registering a handler
+
+```javascript
+const stop = okdb.env('default').processor.register('orders', {
+    name: 'orders:fulfillment', // human-readable name
+    mode: 'inline', // 'inline' = synchronous, post-commit
+    originMode: 'all', // 'all' | 'local' | 'remote'
+
+    handler(changes) {
+        for (const ch of changes) {
+            if (ch.action !== 'put') continue;
+            if (ch.newValue.status === 'paid') {
+                scheduleShipment(ch.key, ch.newValue);
+            }
+        }
+    },
+});
+
+// Stop the handler
+stop();
+```
+
+### Change object shape
+
+```javascript
+{
+    type:     'orders',
+    key:      'o42',
+    action:   'put',          // 'put' | 'remove'
+    oldValue: { ... },        // previous value (null if new)
+    newValue: { ... },        // new value (null if removed)
+    clock:    182,
+    origin:   'node-uuid',
+    txnId:    'txn-uuid',
+}
+```
+
+### originMode
+
+| Value      | Fires when                           |
+| ---------- | ------------------------------------ |
+| `'all'`    | Any write, local or synced           |
+| `'local'`  | Only writes originating on this node |
+| `'remote'` | Only writes arriving via sync        |
+
+This is useful for reacting differently to local vs. replicated changes — e.g., sending a notification only when your own node creates a record, not when it receives one from a peer.
+
+### Processor on a custom env
+
+Each environment has its own processor:
+
+```javascript
+const env = okdb.env('analytics');
+const stop = env.processor.register('events', {
+    mode: 'inline',
+    originMode: 'all',
+    name: 'analytics:aggregate',
+    handler(changes) {
+        /* ... */
+    },
+});
+```
+
+---
+
+## HLC timestamps
+
+The `timestamp` field on change objects is a **Hybrid Logical Clock** value: a 64-bit integer encoding both wall-clock milliseconds and a logical counter. HLC values are monotonically increasing even when system clocks skew backwards, which makes them safe for LWW conflict resolution in sync.
+
+```javascript
+// Decode an HLC value
+const { OKDBHlc } = require('okdb/okdb-hlc');
+const { wallMs, logical } = OKDBHlc.decode(change.timestamp);
+```
+
+You don't need to manage HLC values manually — OKDB handles them transparently.

package/docs/data-model.md

@@ -0,0 +1,174 @@
+# Data Model
+
+## Types
+
+A **type** is a named collection of records — similar to a table in SQL or a collection in MongoDB.
+
+```javascript
+await okdb.registerType('articles');
+```
+
+Each type gets its own set of LMDB sub-databases: one for data, one for each secondary index, and one for FTS index metadata. All sub-databases live inside the same LMDB environment (directory) and participate in the same transactions.
+
+### Type operations
+
+```javascript
+// Register (throws if already exists)
+await okdb.registerType('articles');
+
+// Idempotent register + index setup
+await okdb.ensureType('articles', {
+    indexes: [['author'], ['author', 'publishedAt'], { fields: ['location'], type: 'geo' }],
+});
+
+// Check existence
+okdb.hasType('articles'); // → boolean
+
+// Drop (deletes all records, indexes, and sub-databases)
+await okdb.dropType('articles');
+```
+
+---
+
+## Records
+
+A **record** is a key-value pair inside a type.
+
+- **Key**: a non-empty string (your choice — UUID, slug, user ID, etc.)
+- **Value**: any JSON-serialisable value
+
+```javascript
+await okdb.put('articles', 'how-lmdb-works', {
+    title: 'How LMDB works',
+    author: 'alice',
+    body: '...',
+    publishedAt: Date.now(),
+    tags: ['database', 'internals'],
+});
+```
+
+Values can also be direct JSON scalars or arrays (`"hello"`, `42`, `true`, `null`, `[]`). Those values are stored and read back normally. However, object-oriented features such as `patch()` and observed field schema only apply to plain object records.
+
+Values are stored with LZ4 compression by default. The raw bytes are LMDB's [ordered-binary](https://github.com/DoctorEvidence/ordered-binary) codec, which preserves type ordering for secondary indexes.
+
+### Versions
+
+Every record implicitly has a **version** — an integer that monotonically increases with each write. OKDB uses LMDB's `useVersions` feature for optimistic concurrency.
+
+```javascript
+const entry = okdb.getEntry('articles', 'how-lmdb-works');
+// entry = { value: {...}, version: 3 }
+```
+
+---
+
+## Environments
+
+An **environment** is an isolated LMDB directory. OKDB opens several by default and you can create more.
+
+### Built-in environments
+
+| Name                | Purpose                     | Synced |
+| ------------------- | --------------------------- | ------ |
+| `~system`           | Node identity, env registry | yes    |
+| `default`           | Your types and data         | yes    |
+| `~<env>:emb:<type>` | Per-type embedding vectors  | yes    |
+
+:::note
+`queue` and `files` live inside each environment — each env has its own queue and file store.
+:::
+
+### Creating custom environments
+
+```javascript
+// Create a new named environment
+const analyticsEnv = await okdb.createEnvironment('analytics', {
+    sync: true, // include in multi-node sync (default: true)
+});
+
+// It has the same API as okdb itself
+await analyticsEnv.registerType('events');
+await analyticsEnv.put('events', 'e1', { action: 'click', ts: Date.now() });
+const evt = analyticsEnv.get('events', 'e1');
+```
+
+Accessing an environment:
+
+```javascript
+okdb.env('default'); // the default env
+okdb.env('analytics'); // custom env by name
+okdb.default; // shorthand for okdb.env('default')
+```
+
+### When to use multiple environments
+
+- Separate high-churn data from user data (e.g. analytics events vs. user records)
+- Independent encryption keys per dataset
+- Selective sync (set `sync: false` for data that should stay local)
+- Physical isolation for compaction or backup
+
+---
+
+## Storage layout
+
+OKDB creates this structure on disk:
+
+```
+mydb/
+  ~system/          ← node identity + env registry
+  default/          ← your types and records
+  ~<env>:emb:<type>/← per-type vector store
+  ~fts/             ← FTS indexes (one sub-dir per index)
+  blobs/            ← file attachment blobs (SHA-256 content-addressable)
+```
+
+Inside each LMDB environment, sub-databases are named by convention:
+
+```
+"clock"               → global monotonic integer
+"types"               → type metadata (field schema, FTS config)
+"changes"             → changeId → latest clock
+"clocks"              → clock → change object (the event log)
+
+per type T:
+  "type:T"            → lifecycle records
+  "type:T:data"       → key → value (useVersions=true)
+  "type:T:index"      → index metadata
+  "type:T:index:I"    → dupSort ordered-binary index entries
+  "type:T:clockToChange" → clock → changeId
+  "type:T:keyToChange"   → key → changeId
+```
+
+---
+
+## Field schema
+
+OKDB observes field types as you write data and maintains a coarse schema (`string | number | boolean | null | array | object`) per field. You can inspect it:
+
+```javascript
+const desc = await okdb.meta.describeType('articles');
+// {
+//   count: 42,
+//   fields: { title: 'string', publishedAt: 'number', tags: 'array', ... },
+//   indexes: ['author', 'author~publishedAt'],
+//   ftsIndexes: []
+// }
+```
+
+Schema is **observed-only** — no validation on write. If you need strict validation, do it at the application layer before calling `put`.
+
+---
+
+## Durability modes
+
+Set at construction time via the `durability` option:
+
+| Mode       | Behaviour                                                | Use when                       |
+| ---------- | -------------------------------------------------------- | ------------------------------ |
+| `strict`   | Full fsync on every write                                | Mission-critical, money        |
+| `balanced` | `overlappingSync` — background sync, still safe on crash | Default; most use cases        |
+| `fast`     | `noSync` — OS controls flush timing                      | Dev, caches, re-generable data |
+
+```javascript
+const okdb = new OKDB('./db', { durability: 'fast' });
+```

package/docs/embeddings.md

@@ -0,0 +1,338 @@
+# Embeddings
+
+OKDB has a built-in vector embedding pipeline. Point it at a type, tell it which field to embed and which ML model to use, and it handles the rest: watching the change log, calling the embedder, storing vectors, and serving nearest-neighbour queries.
+
+---
+
+## Concepts
+
+| Component         | Role                                                  |
+| ----------------- | ----------------------------------------------------- |
+| **Embedder**      | Connection to an ML model (Ollama, OpenAI, or custom) |
+| **Indexer**       | Watches a type's change log, embeds changed docs      |
+| **Vector store**  | Persists `Float32Array` vectors in LMDB               |
+| **Search engine** | In-memory HNSW graph for nearest-neighbour queries    |
+
+These are wired together as **named engines** — persistent service instances whose config lives in OKDB and survive restarts.
+
+Embeddings is now also a **built-in pipeline family** on top of the generic `~pipelines` feature:
+
+- `okdb.embeddings.createPipeline(...)` still provides the specialized one-call setup
+- the created topology is also persisted as an env-local generic pipeline record
+- the generic record stores explicit ordered member roles (`embedder`, `indexer`, optional `worker`, `search`)
+- embeddings HTTP/admin views read those generic records first and only fall back to legacy engine grouping for older pipelines
+- starter templates now exist too: `GET /api/pipelines/templates` exposes recommended embeddings pipeline blueprints such as `embeddings-inline-fake`, `embeddings-inline-ollama`, `embeddings-queue-fake`, and `embeddings-queue-ollama`
+- MCP clients can discover and expand those blueprints through `okdb_pipeline` actions `template_list`, `template_get`, `template_preview`, and `template_create`
+
+---
+
+## Quick start with `createPipeline`
+
+The one-call setup that provisions all four components:
+
+```javascript
+await okdb.embeddings.createPipeline('articles', {
+    source_type: 'articles', // the OKDB type to watch
+    field: 'body', // which field to embed (null = stringify whole record)
+    dims: 1024, // embedding dimensions (must match model)
+    embedder: {
+        type: 'ollama',
+        model: 'mxbai-embed-large',
+        url: 'http://localhost:11434',
+    },
+});
+```
+
+After this call, any `put` to `articles` will automatically trigger embedding. Vectors are stored in the `~default:emb:articles` environment.
+
+The specialized embeddings facade now also writes a generic `~pipelines` record in the source environment. By default:
+
+- the generic pipeline record name is the pipeline `storage_key`
+- `meta.label` preserves the user-facing embeddings pipeline label
+- `meta.family === 'embeddings'` marks the record as embeddings-backed
+
+So a call like:
+
+```javascript
+await okdb.embeddings.createPipeline('articles-body-pipeline', {
+    source_env: 'default',
+    source_type: 'articles',
+    storage_key: 'articles-body',
+    field: 'body',
+    dims: 768,
+    embedder: { type: 'ollama', model: 'mxbai-embed-large' },
+});
+```
+
+creates engines plus a generic pipeline record named `articles-body` in `default/~pipelines`.
+
+This `storage_key` behavior matters when you use the new template registry too:
+
+- the starter input usually asks for a human-facing `name`
+- embeddings template plans default `storage_key` to that same `name`
+- if you override `storage_key`, the persisted generic pipeline record is stored under that value instead
+- `meta.label` still keeps the original display label so admin/MCP clients can show a friendly name
+
+---
+
+## Templates and discovery
+
+Embeddings templates are presets for the same underlying `okdb.embeddings.createPipeline(...)` flow.
+
+You can discover them over HTTP:
+
+```text
+GET  /api/pipelines/templates
+GET  /api/pipelines/templates/:template
+POST /api/pipelines/templates/:template/preview
+POST /api/env/:env/pipelines/templates/:template/create
+```
+
+Built-in recommended embeddings pipeline starters currently include:
+
+- `embeddings-inline-fake`
+- `embeddings-inline-ollama`
+- `embeddings-queue-fake`
+- `embeddings-queue-ollama`
+
+The preview route expands one of those starters into the exact `embeddings-create` plan it would execute, including the derived `storage_key`, `mode`, search `algorithm`, and provider config.
+
+`POST /api/env/:env/pipelines/templates/:template/create` then executes that plan by calling the specialized embeddings facade and returning the resulting generic pipeline record.
+
+If you want to build the pieces separately, the shared engine template registry also exposes direct embeddings engine starters through the engine API:
+
+```text
+GET  /api/engines/types
+GET  /api/engines/templates
+GET  /api/engines/templates/:template
+POST /api/engines/templates/:template/preview
+POST /api/env/:env/engines/templates/:template/create
+```
+
+That shared registry includes starter definitions such as:
+
+- `embedder-fake`, `embedder-ollama`, `embedder-openai`
+- `indexer-inline`, `indexer-queue`
+- `vector-search-flat`, `vector-search-hnsw`
+- `embed-worker-basic`
+
+So the usual choice is:
+
+- use pipeline templates when you want the full recommended embeddings topology in one step
+- use engine templates when you want to compose or manage the embeddings engines individually
+
+### MCP discovery and creation
+
+Embeddings-related MCP behavior is now split across three grouped tools:
+
+- `okdb_pipeline` — use `template_list`, `template_get`, `template_preview`, and `template_create` for full embeddings pipeline starters
+- `okdb_engine` — use `types`, `templates`, `template_get`, `template_preview`, and `template_create` for direct engine discovery and creation
+- `okdb_embeddings` — use `providers`, `models`, `models_by_provider`, `algorithms`, `probe_ollama_models`, and `embedder_models` to inspect provider schemas and model catalogs before choosing a template or creating engines manually
+
+This makes it possible for MCP clients to both discover recommendations and actually execute them, rather than relying on out-of-band docs.
+
+---
+
+## Built-in embedder types
+
+### Ollama (local)
+
+```javascript
+const embedder = {
+    type: 'ollama',
+    model: 'mxbai-embed-large', // or 'nomic-embed-text', etc.
+    url: 'http://localhost:11434',
+};
+```
+
+Requires [Ollama](https://ollama.ai) running locally:
+
+```bash
+ollama pull mxbai-embed-large
+```
+
+### OpenAI (and compatible APIs)
+
+```javascript
+const embedder = {
+    type: 'openai',
+    model: 'text-embedding-3-small',
+    api_key: process.env.OPENAI_API_KEY,
+    // base_url: 'https://api.openai.com/v1'  (default)
+};
+```
+
+### Custom / fake
+
+For development and testing without a real ML service:
+
+```javascript
+okdb.embeddings.registerEmbedderFactory('my-embedder', (config) => ({
+    async embed(input) {
+        // Return a Float32Array of `config.dims` dimensions
+        return new Float32Array(config.dims).fill(0.5);
+    },
+    async health() {
+        return { ok: true };
+    },
+}));
+
+// Then use it in a pipeline:
+await okdb.embeddings.createPipeline('articles', {
+    source_type: 'articles',
+    dims: 128,
+    embedder: { type: 'my-embedder' },
+});
+```
+
+---
+
+## Semantic search
+
+`createPipeline` returns a pipeline object. Use `pipeline.api.search()` to run nearest-neighbour queries:
+
+```javascript
+const pipeline = await okdb.embeddings.createPipeline('articles', {
+    source_type: 'articles',
+    field: 'body',
+    dims: 1024,
+    embedder: { type: 'ollama', model: 'mxbai-embed-large' },
+});
+
+// Text query — the pipeline embeds it then finds nearest neighbours
+const results = await pipeline.api.search('database performance tuning', { limit: 5 });
+for (const { key, score } of results) {
+    console.log(key, score.toFixed(4));
+}
+
+// Float32Array query — skip the embed step
+const vec = new Float32Array(1024).fill(0.1);
+const results2 = await pipeline.api.search(vec, { limit: 10, threshold: 0.7 });
+```
+
+### Search options
+
+| Option      | Default | Description                    |
+| ----------- | ------- | ------------------------------ |
+| `limit`     | 10      | Maximum results                |
+| `threshold` | 0       | Minimum similarity score (0–1) |
+
+---
+
+## Pipeline modes
+
+### Inline (default)
+
+The indexer calls the embedder synchronously in the change-log processor. Simple, no extra infrastructure:
+
+```javascript
+await okdb.embeddings.createPipeline('articles', {
+    source_type: 'articles',
+    field: 'body',
+    dims: 1024,
+    mode: 'inline', // default
+    embedder: { type: 'ollama', model: 'mxbai-embed-large' },
+});
+```
+
+### Queue mode
+
+For high-throughput or external embedder services, the indexer enqueues jobs and a worker pool consumes them concurrently:
+
+```javascript
+await okdb.embeddings.createPipeline('articles', {
+    source_type: 'articles',
+    field: 'body',
+    dims: 1024,
+    mode: 'queue',
+    embedder: { type: 'openai', model: 'text-embedding-3-small', api_key: '...' },
+    worker: {
+        concurrency: 4,
+        pollInterval: 500,
+        ttl: 30_000,
+    },
+});
+```
+
+Queue-mode embeddings records keep the same explicit membership order in the generic pipeline registry, with the worker inserted between the indexer and search members.
+
+---
+
+## Waiting for indexing to complete
+
+```javascript
+// For a pipeline named 'articles' in the default env, the scoped key is 'default:articles'
+const indexer = okdb.embeddings.indexer('default:articles');
+const stats = await indexer.stats();
+// { total, pending, embedded, failed }
+
+// Pipeline api.stats() wraps all four components
+const { api } = await okdb.embeddings.createPipeline('articles', { ... });
+const allStats = await api.stats();
+// { embedder: {...}, indexer: {...}, worker: {...}, search: {...} }
+
+// Wait for the indexer to finish processing all documents
+function waitForIdle(pipelineKey, timeoutMs = 120_000) {
+    return new Promise((resolve, reject) => {
+        const timer = setTimeout(() => reject(new Error('timeout')), timeoutMs);
+        const check = async () => {
+            const api = okdb.embeddings.indexer(pipelineKey);
+            if (!api) return;
+            const { pending } = await api.stats().catch(() => ({ pending: 1 }));
+            if (pending === 0) { clearTimeout(timer); resolve(); }
+        };
+        okdb.events.on(`embeddings:indexer_flushed@indexer@${pipelineKey}`, check);
+        check();
+    });
+}
+
+await waitForIdle('default:articles');
+```
+
+---
+
+## Engines
+
+Pipelines are built on top of the **engines** feature — a generic lifecycle manager for named, persistent services. Each component (embedder, indexer, search, worker) is an engine with a type, name, config, and status stored in `~engines`.
+
+They survive restarts: on `okdb.open()`, all previously-created engines are restored and their `start` methods called.
+
+```javascript
+// List all running engines
+const engines = okdb.engines.list();
+// [{ name, type, status, config, meta }, ...]
+
+// Get a specific engine
+const embedder = okdb.engines.getEngine('embedder', 'articles');
+```
+
+### Shared embedder lifecycle behavior
+
+Embeddings pipelines usually own the env-scoped indexer/search/worker engines, but the embedder is often shared across pipelines and environments.
+
+To preserve that behavior when delegating lifecycle operations through generic pipelines, embeddings-backed pipeline records mark the embedder role as lifecycle-skipped.
+
+That means specialized or generic pipeline `start` / `stop` operations:
+
+- manage env-owned `indexer`, `search`, and optional `worker` members
+- do **not** stop or restart the shared embedder just because one pipeline is paused
+- still preserve the shared embedder on pipeline delete unless the embedder is explicitly managed elsewhere
+
+---
+
+## Storage layout
+
+```
+~default:emb:articles/
+  vec:<storage_key>           ← Float32Array vectors in LMDB
+  doc_status:<storage_key>    ← per-document embedding status
+
+~system/
+  ~emb:models                 ← model catalog (shared)
+```
+
+The vector store persists raw `Float32Array` bytes (not JSON), preserving full floating-point precision. The HNSW graph is rebuilt from these stored vectors on each restart.
+
+:::note
+Vector data (the raw Float32Array values stored in `vec:` types) **is replicated** via the normal change log — embedding is expensive, so every node receives the computed vectors automatically. The in-memory HNSW search graph is rebuilt locally from these synced vectors on each restart.
+:::