@kedem/okdb 1.0.0

package/docs/sync.md

@@ -0,0 +1,156 @@
+# Sync
+
+OKDB includes a peer-to-peer, last-write-wins (LWW) replication system. Nodes sync over HTTP — no central coordinator, no Raft, no consensus protocol. Just pull deltas from peers, compare timestamps, keep the newer one.
+
+---
+
+## How it works
+
+1. Every write increments the local clock and logs a change entry
+2. When the clock advances, a UDP multicast poke (`bus:poke`) is sent on the LAN
+3. Peers that receive the poke (or poll on a timer) pull a **delta** from the changed node
+4. Each incoming change is compared to the local version using HLC timestamp
+5. The newer timestamp wins; ties are broken by origin node ID (lexicographic)
+6. Applied changes are written through the normal write path (with `origin` set to prevent echo-loops)
+
+This is eventual consistency. It's designed for loosely-coupled multi-node topologies: desktop clusters, distributed edge devices, multi-instance app servers.
+
+---
+
+## Setting up a cluster
+
+### Node configuration
+
+```javascript
+const okdb = new OKDB('./node1', {
+    auth: {
+        tokens: { 'cluster-secret': ['admin'] },
+    },
+    sync: {
+        token: 'cluster-secret',
+        address: 'http://192.168.1.10:8080', // this node's public address
+    },
+});
+await okdb.open();
+okdb.http.listen(8080);
+```
+
+Every node needs:
+
+- `auth.tokens` — includes the sync token so peers can call the delta endpoint
+- `sync.token` — shared secret used to authenticate outbound sync HTTP requests
+- `sync.address` — the URL other nodes will use to reach this one
+
+### Joining a cluster
+
+```javascript
+// Node 2 joins node 1
+await okdb.sync.join('http://192.168.1.10:8080');
+```
+
+`join` does three things:
+
+1. Registers node 1 as a known peer in `~sync_nodes` type (which itself syncs)
+2. Pulls an initial delta from node 1
+3. From that point, auto-reconcile keeps them in sync on clock change
+
+### Multi-node example
+
+```javascript
+const nodes = [8080, 8081, 8082].map((port) => {
+    const okdb = new OKDB(`./node-${port}`, {
+        auth: { tokens: { secret: ['admin'] } },
+        sync: { token: 'secret', address: `http://localhost:${port}` },
+    });
+    okdb._port = port;
+    return okdb;
+});
+
+await Promise.all(nodes.map((k) => k.open()));
+for (const node of nodes) node.http.listen(node._port);
+
+// Join node 1 and 2 to node 0
+for (const node of nodes.slice(1)) {
+    await node.sync.join('http://localhost:8080');
+}
+```
+
+---
+
+## Sync internals
+
+### Peer registry
+
+Peers are stored in the `~sync_nodes` OKDB type — which is itself synced. So adding a peer on one node propagates to all other nodes automatically, forming a mesh.
+
+Local peer progress (which clock was last seen from each peer) is stored outside of sync in `~system` — it's local state that doesn't replicate.
+
+### Delta endpoint
+
+```
+GET /api/sync/delta?from_clock=<N>
+Authorization: Bearer <token>
+```
+
+Returns up to `delta_limit` (default 500) change records from clock `N` onwards across all sync-enabled environments. Each change includes `_env` so the receiver routes it to the right environment.
+
+### Multi-environment sync
+
+The delta spans **all `sync:true` environments** simultaneously. The response carries a `clocks` map (`{ default: N, queue: N, ... }`), and each environment's cursor advances independently. Custom environments created via `createEnvironment` are included automatically if `sync: true`.
+
+---
+
+## LWW conflict model
+
+When two nodes write the same key concurrently:
+
+- The **higher HLC timestamp** wins
+- If timestamps are equal, the **lexicographically larger origin node ID** wins
+- This is fully deterministic — every node arrives at the same winner independently
+
+### Anti-echo
+
+Changes that originated on the local node (`origin === okdb.id`) are never re-applied when they arrive back from a peer. This prevents infinite re-broadcast loops.
+
+---
+
+## Sync info and status
+
+```javascript
+const info = okdb.sync.info();
+// {
+//   node_id: '...',
+//   clock: 142,
+//   auto_reconcile: true,
+//   peers: 2,
+//   reconciling: [],
+// }
+
+// List known peers
+const self = okdb.sync.getSelfNode();
+// { id, address, meta, updated }
+```
+
+---
+
+## Caveats
+
+:::warning Wall-clock drift
+LWW depends on timestamps. If two nodes have significantly different system clocks, causally-later writes can lose. NTP keeps this manageable in practice, but be aware of it. HLC mitigates this for rapid local writes, but not cross-node drift.
+:::
+
+:::note Non-syncable data
+Environments with `sync: false` (FTS indexes, vector stores) are excluded from sync. They are rebuilt locally from synced data. This is intentional — posting lists and HNSW graphs are large, computable, and node-local.
+:::
+
+---
+
+## UDP discovery bus
+
+`okdb-bus` sends a UDP multicast datagram on `239.1.2.3:30303` whenever the clock changes. This is how peers on the same LAN discover that new data is available without polling. Loss is acceptable — sync reconciles correctly even if some pokes are dropped.
+
+To disable:
+
+```javascript
+const okdb = new OKDB('./db', { bus: false });
+```

package/docs/transactions.md

@@ -0,0 +1,140 @@
+# Transactions
+
+OKDB supports two write styles: single-operation writes (the default) and explicit batch transactions. Both are fully atomic.
+
+---
+
+## Single-operation writes
+
+Each of these creates and commits its own LMDB transaction internally:
+
+```javascript
+await okdb.put('orders', 'o1', { item: 'widget', qty: 3, status: 'pending' });
+await okdb.create('orders', 'o2', { item: 'gadget', qty: 1, status: 'pending' });
+await okdb.update('orders', 'o1', { item: 'widget', qty: 5, status: 'pending' });
+await okdb.remove('orders', 'o1');
+```
+
+| Method   | Behaviour                                     |
+| -------- | --------------------------------------------- |
+| `put`    | Upsert — create or overwrite                  |
+| `create` | Throws `OKDBAlreadyExistsError` if key exists |
+| `update` | Throws `OKDBNotFoundError` if key absent      |
+| `remove` | Throws `OKDBNotFoundError` if key absent      |
+
+---
+
+## Batch transactions
+
+When you need multiple operations to succeed or fail together, use `okdb.transaction()`:
+
+```javascript
+const txn = okdb.transaction();
+
+txn.put('accounts', 'alice', { balance: 900 });
+txn.put('accounts', 'bob', { balance: 1100 });
+txn.put('transfers', 'tx1', { from: 'alice', to: 'bob', amount: 100, ts: Date.now() });
+
+await txn.commit(); // all three land in one LMDB write transaction
+```
+
+Rollback (discard without writing):
+
+```javascript
+txn.rollback();
+```
+
+### How it works
+
+`OKDBTransaction` accumulates operations in memory. On `commit()`, it replays them all inside a single `okdb.db.transaction()` call. Every secondary index update and change-log entry for all operations is written atomically in that one transaction — there is no partial state.
+
+---
+
+## Optimistic concurrency
+
+Guard writes with a version check using `ifVersion`:
+
+```javascript
+// Read the current version
+const entry = okdb.getEntry('products', 'p42');
+// entry.version = 7
+
+// Only write if the record hasn't changed since we read it
+await okdb.put(
+    'products',
+    'p42',
+    { ...entry.value, stock: entry.value.stock - 1 },
+    {
+        ifVersion: entry.version, // throws OKDBVersionMismatchError if version changed
+    },
+);
+```
+
+This is a classic compare-and-swap. If another writer modified the record between your read and your write, the write throws and you can retry.
+
+In a batch transaction:
+
+```javascript
+const txn = okdb.transaction();
+txn.put('products', 'p42', newValue, { ifVersion: 7 });
+txn.put('products', 'p43', newValue2);
+await txn.commit(); // both succeed or both fail
+```
+
+---
+
+## Error types
+
+| Error                         | Code                   | Thrown by                  |
+| ----------------------------- | ---------------------- | -------------------------- |
+| `OKDBAlreadyExistsError`      | `ALREADY_EXISTS`       | `create()`                 |
+| `OKDBNotFoundError`           | `NOT_FOUND`            | `update()`, `remove()`     |
+| `OKDBVersionMismatchError`    | `VERSION_MISMATCH`     | any write with `ifVersion` |
+| `OKDBUniqueConstraintError`   | `UNIQUE_CONSTRAINT`    | unique index violation     |
+| `OKDBTypeNotRegisteredError`  | `TYPE_NOT_REGISTERED`  | write to unknown type      |
+| `OKDBIndexNotRegisteredError` | `INDEX_NOT_REGISTERED` | query on unknown index     |
+
+All errors extend `OKDBError` and carry a `code` property.
+
+```javascript
+const { OKDBError, OKDBVersionMismatchError } = require('okdb/okdb-error');
+
+try {
+    await okdb.put('products', 'p42', newValue, { ifVersion: oldVersion });
+} catch (err) {
+    if (err instanceof OKDBVersionMismatchError) {
+        // retry logic
+    } else {
+        throw err;
+    }
+}
+```
+
+---
+
+## Timestamp and origin
+
+Every write accepts a `timestamp` and `origin` option used by the sync system:
+
+```javascript
+await okdb.put('items', 'i1', value, {
+    timestamp: Date.now(), // used for LWW conflict resolution
+    origin: 'node-uuid-...', // prevents echo-loops in sync
+});
+```
+
+You rarely need to set these manually — they are used internally by the sync feature.
+
+---
+
+## What's atomic
+
+Everything that happens inside a single `db.transaction()` call is atomic:
+
+1. Primary data written to `type:T:data`
+2. All secondary index entries updated (`type:T:index:I` sub-dbs)
+3. Clock incremented and change log entries written (`clocks`, `changes`, `clockToChange`, `keyToChange`)
+
+No partial state is ever visible. If the process crashes mid-write, LMDB rolls back the transaction on the next open.
+
+Post-commit events (`item:create`, `system:clock_change`, etc.) fire **after** the LMDB transaction has been durably committed — never speculatively.

package/docs/ttl.md

@@ -0,0 +1,375 @@
+# TTL (Time-to-Live)
+
+OKDB supports per-record TTL — set an expiration on any item and it will be automatically removed after the specified duration.
+
+---
+
+## Quick Start
+
+```javascript
+const OKDB = require('src/okdb');
+const db = new OKDB('./my-data');
+await db.open();
+
+await db.registerType('sessions');
+await db.put('sessions', 's1', { user: 'alice', token: 'abc123' });
+
+// Expire this session in 30 minutes
+await db.setTTL('sessions', 's1', 30 * 60 * 1000);
+
+// Check remaining time
+const ttl = db.getTTL('sessions', 's1');
+// { expiresAt: 1712345678000, remainingMs: 1799432 }
+
+// Make the item permanent again
+await db.clearTTL('sessions', 's1');
+```
+
+---
+
+## API Reference
+
+### `setTTL(type, key, ttlMs)`
+
+Sets a TTL on an existing item. The item will be eligible for removal after `ttlMs` milliseconds from now.
+
+| Parameter | Type     | Description                                     |
+| --------- | -------- | ----------------------------------------------- |
+| `type`    | `string` | The type (collection) name                      |
+| `key`     | `string` | The item's primary key                          |
+| `ttlMs`   | `number` | Time-to-live in milliseconds (must be positive) |
+
+**Throws:**
+
+- `OKDBNotFoundError` if the item doesn't exist
+- `OKDBInvalidValueError` if `ttlMs` is not a positive number
+
+```javascript
+await db.setTTL('cache', 'page:/home', 60_000); // expire in 1 minute
+```
+
+Calling `setTTL` on an item that already has a TTL replaces the previous expiration.
+
+### `getTTL(type, key)`
+
+Returns the TTL info for an item, or `null` if no TTL is set.
+
+| Parameter | Type     | Description            |
+| --------- | -------- | ---------------------- |
+| `type`    | `string` | The type name          |
+| `key`     | `string` | The item's primary key |
+
+**Returns:** `{ expiresAt: number, remainingMs: number } | null`
+
+- `expiresAt` — absolute Unix timestamp (ms) when the item expires
+- `remainingMs` — milliseconds until expiration (0 if already expired but not yet swept)
+
+```javascript
+const ttl = db.getTTL('cache', 'page:/home');
+if (ttl && ttl.remainingMs < 5000) {
+    console.log('About to expire!');
+}
+```
+
+### `clearTTL(type, key)`
+
+Removes the TTL from an item, making it permanent. The item itself is not deleted.
+
+```javascript
+await db.clearTTL('sessions', 's1');
+```
+
+### `sweepExpiredTTL(batchSize?)`
+
+Manually triggers a sweep of all expired items across all types.
+
+| Parameter   | Type     | Default | Description                            |
+| ----------- | -------- | ------- | -------------------------------------- |
+| `batchSize` | `number` | `1000`  | Max items to remove per type per sweep |
+
+**Returns:** `{ removed: number, types: { [typeName]: number } }`
+
+```javascript
+const result = await db.sweepExpiredTTL();
+console.log(`Removed ${result.removed} expired items`);
+// result.types = { sessions: 3, cache: 12 }
+```
+
+### `ttlStats(type?)`
+
+Returns aggregate TTL statistics for the default environment, optionally filtered to a single type.
+
+| Parameter | Type     | Description                                |
+| --------- | -------- | ------------------------------------------ |
+| `type`    | `string` | _(optional)_ Filter stats to a single type |
+
+**Returns:** `{ enabled, totalEntries, expiredEntries, nextExpiry, byType }`
+
+| Field            | Type           | Description                                         |
+| ---------------- | -------------- | --------------------------------------------------- |
+| `enabled`        | `boolean`      | Whether any TTL sub-databases exist                 |
+| `totalEntries`   | `number`       | Total number of TTL entries                         |
+| `expiredEntries` | `number`       | Entries that have already expired but not yet swept |
+| `nextExpiry`     | `number\|null` | Timestamp (ms) of the soonest expiry, or `null`     |
+| `byType`         | `object`       | `{ [typeName]: count }` breakdown                   |
+
+```javascript
+const stats = db.ttlStats();
+// { enabled: true, totalEntries: 15, expiredEntries: 2,
+//   nextExpiry: 1712345678000, byType: { sessions: 10, cache: 5 } }
+
+const sessionStats = db.ttlStats('sessions');
+// filtered to sessions only
+```
+
+### `listTTL(opts?)`
+
+Lists all TTL entries sorted by expiry time (soonest first).
+
+| Parameter     | Type     | Description                                   |
+| ------------- | -------- | --------------------------------------------- |
+| `opts.type`   | `string` | _(optional)_ Filter to a single type          |
+| `opts.limit`  | `number` | _(optional, default 100)_ Max items to return |
+| `opts.offset` | `number` | _(optional, default 0)_ Skip this many items  |
+
+**Returns:** `{ total, items }`
+
+Each item: `{ type, key, expiresAt, remainingMs }`
+
+```javascript
+const result = db.listTTL({ type: 'sessions', limit: 10 });
+// { total: 42, items: [ { type: 'sessions', key: 's1', expiresAt: ..., remainingMs: ... }, ... ] }
+```
+
+---
+
+## Background Sweep
+
+Expired items are not deleted the instant they expire. Instead, they are removed by a background sweep timer or by calling `sweepExpiredTTL()` manually.
+
+### Start / Stop
+
+```javascript
+const env = db.env('default');
+
+// Sweep every 60 seconds (default)
+env.startTTLSweep();
+
+// Sweep every 10 seconds
+env.startTTLSweep(10_000);
+
+// Stop the background sweep
+env.stopTTLSweep();
+```
+
+The sweep timer is automatically stopped when the database is closed. The timer is created with `unref()` so it won't keep your process alive.
+
+### Sweep Behaviour
+
+- Iterates all types, scanning the TTL index for entries where `expiresAt <= Date.now()`
+- Removes each expired item using the standard `_remove()` path (indexes, foreign keys, change log all updated)
+- Emits a `ttl:expired` event for each removed item
+- Respects `batchSize` to avoid long-running transactions
+- Safe to call concurrently — duplicate removal attempts are handled gracefully
+
+---
+
+## Transactions
+
+TTL operations can be included in batch transactions for atomic create-and-expire patterns:
+
+```javascript
+const txn = db.transaction();
+
+txn.put('sessions', 's1', { user: 'alice', token: 'abc123' });
+txn.setTTL('sessions', 's1', 30 * 60 * 1000); // 30 min
+
+txn.put('sessions', 's2', { user: 'bob', token: 'xyz789' });
+txn.setTTL('sessions', 's2', 60 * 60 * 1000); // 1 hour
+
+await txn.commit(); // both items and their TTLs set atomically
+```
+
+Also works with the `txn()` helper:
+
+```javascript
+await db.txn(async (t) => {
+    t.put('cache', 'k1', { data: 'hello' });
+    t.setTTL('cache', 'k1', 30_000);
+});
+```
+
+Clear TTL within a transaction:
+
+```javascript
+const txn = db.transaction();
+txn.clearTTL('sessions', 's1');
+await txn.commit();
+```
+
+---
+
+## Events
+
+TTL operations emit events on `db.events`:
+
+| Event         | Payload                         | When                            |
+| ------------- | ------------------------------- | ------------------------------- |
+| `ttl:set`     | `{ type, key, expiresAt, env }` | A TTL is set or updated         |
+| `ttl:clear`   | `{ type, key, env }`            | A TTL is cleared                |
+| `ttl:expired` | `{ type, key, env }`            | An item is removed by the sweep |
+
+```javascript
+const { EVENTS } = require('okdb/okdb-enums');
+
+db.events.on(EVENTS.TTL_EXPIRED, ({ type, key }) => {
+    console.log(`Expired: ${type}/${key}`);
+});
+```
+
+---
+
+## HTTP API
+
+### Get item TTL
+
+```
+GET /api/:env/type/:type/item/:key/ttl
+```
+
+Returns `{ result: { expiresAt, remainingMs } }` or `{ result: null }`.
+
+### Set item TTL
+
+```
+PUT /api/:env/type/:type/item/:key/ttl
+Content-Type: application/json
+
+{ "ttl": 60000 }
+```
+
+### Clear item TTL
+
+```
+DELETE /api/:env/type/:type/item/:key/ttl
+```
+
+### Put item with TTL
+
+The standard PUT item route accepts an optional `ttl` field:
+
+```
+PUT /api/:env/type/:type/item/:key
+Content-Type: application/json
+
+{ "value": { "user": "alice" }, "ttl": 60000 }
+```
+
+### Sweep expired items
+
+```
+POST /api/:env/ttl/sweep
+Content-Type: application/json
+
+{ "batchSize": 500 }
+```
+
+Returns `{ result: { removed: 5, types: { sessions: 3, cache: 2 } } }`.
+
+### TTL stats
+
+```
+GET /api/:env/ttl/stats
+GET /api/:env/ttl/stats?type=sessions
+```
+
+Returns `{ result: { enabled, totalEntries, expiredEntries, nextExpiry, byType } }`.
+
+### List TTL entries
+
+```
+GET /api/:env/ttl/list
+GET /api/:env/ttl/list?type=sessions&limit=20&offset=0
+```
+
+Returns `{ result: { total, items: [{ type, key, expiresAt, remainingMs }] } }`.
+
+### GET item includes TTL
+
+When a TTL is set on an item, the `GET /api/:env/type/:type/item/:key` response includes it:
+
+```json
+{
+    "result": {
+        "key": "s1",
+        "value": { "user": "alice" },
+        "ttl": { "expiresAt": 1712345678000, "remainingMs": 1799432 }
+    }
+}
+```
+
+---
+
+## Environments
+
+TTL works independently per environment:
+
+```javascript
+await db.createEnvironment('analytics');
+const analytics = db.env('analytics');
+
+await analytics.registerType('events');
+await analytics.put('events', 'e1', { name: 'pageview' });
+await analytics.setTTL('events', 'e1', 86_400_000); // 24h
+
+analytics.startTTLSweep(60_000); // independent sweep timer
+```
+
+---
+
+## Stored Functions
+
+TTL methods are available in the stored function facade:
+
+```javascript
+// Inside a stored function
+async function handler({ db }) {
+    await db.setTTL('sessions', key, 30 * 60 * 1000);
+    const ttl = db.getTTL('sessions', key);
+    await db.clearTTL('sessions', key);
+    await db.sweepExpiredTTL();
+}
+```
+
+---
+
+## How It Works
+
+Each registered type gets two additional LMDB sub-databases:
+
+| Sub-database      | Key                          | Value                         | Purpose                        |
+| ----------------- | ---------------------------- | ----------------------------- | ------------------------------ |
+| `type:T:ttl`      | `expiresAt` (ordered-binary) | `primaryKey` (ordered-binary) | Range scan for sweep (dupSort) |
+| `type:T:ttlByKey` | `primaryKey`                 | `expiresAt`                   | O(1) lookup for get/clear      |
+
+- **Setting a TTL** writes to both sub-databases. If a previous TTL exists, the old entry in `ttl` is removed first.
+- **Clearing a TTL** removes from both sub-databases.
+- **Removing an item** (`db.remove()`) automatically clears its TTL entries.
+- **Updating an item** (`db.put()`, `db.update()`) does **not** change its TTL. The TTL must be explicitly updated with `setTTL()`.
+- **Sweeping** does a range scan on `type:T:ttl` for all entries where `expiresAt <= now`, then removes the corresponding items.
+
+---
+
+## Persistence
+
+TTL metadata is stored in LMDB alongside the data. It survives restarts — items that expired while the database was closed will be swept on the next `sweepExpiredTTL()` call (or by the background timer).
+
+---
+
+## Use Cases
+
+- **Session management** — expire user sessions after inactivity
+- **Cache layer** — use OKDB as a persistent cache with automatic eviction
+- **Rate limiting** — store rate-limit counters with short TTLs
+- **Temporary data** — upload staging, preview tokens, OTP codes
+- **CDN-like caching** — serve cached responses with configurable freshness