@kedem/okdb 1.9.1 → 2.0.0

package/docs/deployment.md

@@ -0,0 +1,155 @@
+# Roles & Deployment
+
+okdb 2.0 has one runtime model: **the process is the control unit**. Every process that opens
+a data path declares — at construction — which background work it runs, and okdb never forks
+or supervises processes for you. Scale-out is always the same move: **run more okdb processes
+on the same path**; the per-processor 1-of-N lease distributes the work automatically, with no
+coordinator.
+
+If you only ever run one process, you can stop reading: `new OKDB(path)` is a full
+do-everything node and the defaults are correct.
+
+---
+
+## The role flags
+
+```js
+new OKDB(path, {
+    processors: true, // claim processor leases + drain derived work (default true)
+    engines: true, // run embeddings / vector-search engines (default true)
+    compaction: true, // eligible to claim the per-env compaction lease (default true)
+});
+```
+
+| Flag         | `true` (default)                                                                                                          | `false` (passive)                                                                            |
+| ------------ | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
+| `processors` | Claim every unclaimed `single` processor lease; drain FTS / views / time-machine / materializer / embeddings on this loop | Claim nothing. Reads, writes, and **inline** processors (indexes, schema, FK, TTL) still run |
+| `engines`    | Run queue worker engines, pipeline workers, embeddings engines                                                            | None of those start here                                                                     |
+| `compaction` | Eligible to claim the per-env compaction lease                                                                            | Never auto-compacts (`env.compact()` still works when called explicitly)                     |
+
+Three things to internalize:
+
+- **Roles are policy, not placement.** A passive node is a full read/write citizen — its writes
+  land in the changelog and are indexed by whichever participating node holds the lease. Inline
+  processors are mandatory on every writer regardless of role.
+- **The lease is the load balancer.** With N participating nodes, each `single` processor is
+  claimed by exactly one; a dead node's leases lapse and survivors re-claim. Under sustained
+  backlog holders rotate work in cooperative hold windows. See [Processors](./processors.md).
+- **Ephemeral processes should be passive.** A CLI one-shot or short-lived script that grabs a
+  1-of-N lease and exits mid-quantum just thrashes it — pass `processors: false`.
+
+---
+
+## Topologies
+
+### 1. One embedded process (the default)
+
+```js
+const db = new OKDB('./data');
+await db.open();
+```
+
+Serves your app's reads/writes and does all derived work on its own loop, in bounded quanta.
+This is crash-safe and correct on its own; you add processes when you decide you need them.
+
+### 2. Embedded + HTTP
+
+```js
+db.http.listen(8080); // returns a plain http.Server
+```
+
+Same single process, plus the REST API and admin UI. The embedded server is always
+single-process — `http.listen(port, { workers })` throws `HTTP_CLUSTER_REMOVED`.
+
+### 3. N capable nodes (the CLI's shape)
+
+Run N identical full-role processes on the same path — either via `bin/okdb` (shared listen
+socket, thin passive supervisor; see [HTTP Clustering](./http-cluster.md)) or as N independent
+`new OKDB(path)` + `http.listen()` processes behind your own load balancer. Each node serves
+**and** processes; the leases spread the processors across them and fail over when a node dies.
+
+### 4. Dedicated workers + a passive serving node
+
+When you want the serving loop insulated from heavy backfill/indexing work:
+
+```js
+// serving process — loop only serves; never claims, never compacts
+const front = new OKDB(path, { processors: false, engines: false, compaction: false });
+front.http.listen(8080);
+
+// N worker processes — headless, do all the processing
+const worker = new OKDB(path, { processors: true }); // engines/compaction default true
+```
+
+Workers need no ports: coordination is the shared LMDB + the UDP bus. Placement and respawn
+are your launcher's job (systemd, pm2, docker, k8s, a PowerShell script — anything that can
+start N processes).
+
+**Boot order tip:** start processes one at a time — wait until a node has fully opened (e.g.
+it bound its port, or touched a ready-file of your choosing) before starting the next, so
+concurrent first-opens of the same envs never race.
+
+---
+
+## Dynamic participation: `processors.start()` / `processors.stop()`
+
+The `processors` constructor option is only the **initial** state — participation is
+controlled at runtime:
+
+```js
+db.processors.start(); // begin participating: un-gate claiming, run the deferred view boot
+db.processors.stop(); //  cease: finish in-flight quanta, release leases (peers fail over)
+```
+
+Both are idempotent and resolve `true` when the state actually changed. `stop()` never
+touches reads, writes, or inline processors — the node stays a full read/write citizen.
+
+**Fast startup** falls out of this: open without participating, start once your service is up.
+
+```js
+const db = new OKDB(path, { processors: false });
+await db.open(); // fast — no view boot, no FTS/TM drains competing with your startup
+
+app.listen(PORT, async () => {
+    await db.processors.start(); // now begin claiming + draining
+});
+```
+
+**Maintenance windows** too: `stop()` during a heavy backfill to keep this node's loop free,
+`start()` after — the leases redistribute to peers and back via normal failover.
+
+---
+
+## Controlling processors at runtime
+
+Static role flags decide _participation_; the running system is controlled per processor
+through durable desired-state (applies across every process, no IPC):
+
+- **Node-level participation** — `db.processors.start()` / `stop()` (see above); this-process,
+  in-memory.
+- **Pause / resume / retry / cursor reset** — per processor, from the admin UI's Processors
+  view or `stop.pause()` / `stop.resume()` on the registration handle. A durable pause
+  survives restarts and re-claims (cross-process, unlike node-level start/stop).
+- **Mode switching** — `setMode(logicalKey, mode)` flips a processor between sync/async at a
+  clock boundary. See [Processors](./processors.md).
+- **Observability** — `GET /api/processors/status` (per-processor durable lag — the honest
+  backlog signal from any node), `GET /api/processes/tree` (the census of OS processes on the
+  root), and the per-env Write queue (`depth`, `oldestPendingMs` — the stall tell). These are
+  the signals an external autoscaler would watch; the scaling _policy_ lives with your
+  launcher, not inside okdb.
+- **`db.pressure()`** — the composite of those signals for THIS node (writer stall/depth, max
+  durable lag, queue backlog, loop lag, and a normalized `score`), cached 250 ms. Feed it to a
+  queue consumer's `admission` option (`() => db.pressure().score < 1`) for load-aware
+  consumption, or poll it from your orchestrator. See docs/queue.md → "Load-aware consumers".
+
+---
+
+## Liveness guarantees (what you can rely on)
+
+- Cross-process wakes ride the UDP bus; because UDP is lossy, every participating node also
+  runs an idle **catch-up tick** (`OKDB_PROCESSOR_CATCHUP_MS`, default 15 s) that drains any
+  backlog a lost poke left behind — a write from a short-lived process is picked up within
+  one tick, not "whenever the next write happens".
+- The **changelog is never pruned past the slowest processor cursor**, so a lagging drain
+  finds its entries intact instead of silently skipping writes. A durably paused processor
+  therefore pins changelog growth — visible and recoverable, by design.

package/docs/diagnostics.md

@@ -0,0 +1,115 @@
+# Diagnostics — finding crashes, hangs, and stalls
+
+okdb's hardest failures are timing races on LMDB's native memory map: a write or read that
+touches an env after it was closed/unmapped for a compaction swap or `removeEnvironment`. They
+surface as:
+
+- **`The environment is already closed`** — an uncaught JS error thrown from inside lmdb's own
+  deferred callback (no okdb stack at throw time). The "crash".
+- **`0xC0000005` / access violation** (Windows) / **SIGSEGV** (POSIX) — a native read/write
+  after unmap. The "hard crash".
+- **A hang / "halt"** — a compaction swap that never completes because something pins the env
+  across the swap (a busy slot, a wedged commit).
+
+These are rare and timing-dependent, so the fix is **resident, opt-in instrumentation** you can
+turn on in the failing deployment and leave on until it fires. Three independent tools, each a
+single env var. **None has any cost when off.**
+
+> **Before anything else: are you running current code?** These races have been fixed
+> incrementally. okdb ships as a **build** (`dist/` / `release/` / npm package), not `src/`. If
+> your deployment's build predates the fix, you are chasing a ghost. Rebuild
+> (`npm run build` / `npm run build:release`) and redeploy, then reproduce. Check the build date
+> against the relevant commits before spending time on instrumentation.
+
+---
+
+## 1. `OKDB_DIAG` — the write-orphan ring + drain-stall watchdog
+
+The primary tool for the "already closed" orphan and the compaction "halt". JS-level, works on
+the main process **and** slot/worker isolates.
+
+| Env var                | Default     | Effect                                                                                                                                                                |
+| ---------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `OKDB_DIAG=1`          | off         | Enable the write-origin ring, the orphan crash-dump handler, and the drain-stall watchdog.                                                                            |
+| `OKDB_DIAG_STACK=1`    | off         | Capture a JS stack **at each write**, so a dump names the exact call site (doc commit / cursor save / view-fts-tm replay). A few µs/write — fine for a debugging run. |
+| `OKDB_DIAG_DIR=<path>` | stderr only | Also append every dump to `<path>/okdb-diag.<pid>.log` (survives the crash).                                                                                          |
+| `OKDB_DIAG_RING=N`     | 256         | Size of the recent-writes ring.                                                                                                                                       |
+| `OKDB_DIAG_STALL_MS=N` | 8000        | A **swap** drain held longer than this is dumped as a stall (re-dumped every `N` ms while it persists).                                                               |
+| `OKDB_DIAG_WATCH_MS=N` | 2000        | Stall-watchdog poll interval.                                                                                                                                         |
+| `OKDB_DIAG_SURVIVE=1`  | off         | After dumping an orphan, **do not exit** — ride through it (the orphaned write is lost, but the process keeps running). Use to gather more than one dump per run.     |
+
+**On an orphan** (`already closed` / invalid-read-txn signature) you get the recent write
+origins, newest first — the orphaning call site is named (with `OKDB_DIAG_STACK=1`):
+
+```
+=== OKDB_DIAG ORPHAN (env-closed write) @ ... pid=19932 ===
+error: Error: The environment is already closed
+recent writes (newest first):
+  #4821 -3ms env=default raw-put
+      at OKDBProcessor._saveCursor (.../okdb-processor.js:...)
+      ...
+  #4820 -4ms env=default commit
+      ...
+drain state (envs with a non-zero drain):
+  env=default phase=SWAP drainAge=120ms activeWriters=1 writer{depth=1 oldestPendingMs=118 ...}
+```
+
+**On a stall** the watchdog dumps _why_ the swap can't finish — which writer is in flight and
+which slots are pinning the env:
+
+```
+=== OKDB_DIAG STALL: swap drain held 8400ms on env=default @ ... ===
+  env=default phase=SWAP drainAge=8400ms activeWriters=0 writer{depth=0 oldestPendingMs=0 parkedNow=0}
+    slots PINNING this env: #3/fts/fts:default:Order
+```
+
+**Live, no crash needed:** `db.diag()` / `env.diag()` return the same snapshot
+(`{recentWrites, drain, ...}`) for an ad-hoc tap or an admin endpoint.
+
+---
+
+## 2. `OKDB_FATAL_REPORT` — native fault report
+
+For the **native** crash (`0xC0000005` / SIGSEGV) that `OKDB_DIAG` can't catch (it's not a JS
+exception). Makes Node write a diagnostic report naming the **faulting thread's** JS + native
+stack, env, and thread id.
+
+| Env var                   | Effect                                                                   |
+| ------------------------- | ------------------------------------------------------------------------ |
+| `OKDB_FATAL_REPORT=1`     | Write a Node report on a fatal native error.                             |
+| `OKDB_REPORT_DIR=<path>`  | Where to write reports (created if missing).                             |
+| `OKDB_FATAL_REPORT_ENV=1` | Include env vars in the report (off by default — they may hold secrets). |
+
+---
+
+## 3. `OKDB_NATIVE_OP_LOG` — the unmap timeline
+
+For an **open-vs-close race** on lmdb-js's shared (process-global, refcounted) native env: a
+synchronous breadcrumb right before each risky native op that unmaps/remaps memory
+(`db.close`, env swap/rename, `removeEnvironment`, residency evict, reopen, slot stop) **and**
+the matching open. Pairs with `OKDB_FATAL_REPORT`: the report gives the faulting stack, this
+gives the cross-thread open/close timeline.
+
+| Env var                     | Effect                                                            |
+| --------------------------- | ----------------------------------------------------------------- |
+| `OKDB_NATIVE_OP_LOG=<path>` | Append a one-line breadcrumb per risky native op.                 |
+| `OKDB_NATIVE_OP_STACK=1`    | Also append the JS stack of each op (names the exact close path). |
+
+---
+
+## Triage workflow
+
+1. **Confirm the build is current** (see the note above). Most "still crashing after a fix"
+   reports are a stale build.
+2. Reproduce with **all three** on:
+   `OKDB_DIAG=1 OKDB_DIAG_STACK=1 OKDB_FATAL_REPORT=1 OKDB_NATIVE_OP_LOG=ops.log OKDB_NATIVE_OP_STACK=1 OKDB_DIAG_DIR=diag/ OKDB_REPORT_DIR=reports/`
+3. On a crash:
+    - `already closed` → `OKDB_DIAG` dump names the write site. The fix is to route that write
+      through `OKDBWriter` with a **synchronous** `putSync`/`removeSync` inside the txn (an async
+      `db.put` inside a writer `childTransaction` escapes into lmdb's deferred event-turn-batch and
+      orphans on close).
+    - `0xC0000005` → cross-reference the `OKDB_FATAL_REPORT` faulting stack with the
+      `OKDB_NATIVE_OP_LOG` timeline to find the close that raced the read/write.
+4. On a hang: the `OKDB_DIAG` **stall** dump names the writer/slot pinning the swap.
+5. For a deterministic local repro, `OKDB_SLOTS_INLINE=1` runs slot quanta inline on the owner
+   loop — it makes several of these races reproduce 100% instead of intermittently.

package/docs/embeddings.md

@@ -237,7 +237,7 @@ await okdb.embeddings.createPipeline('articles', {
 
 ### Queue mode
 
-For high-throughput or external embedder services, the indexer enqueues jobs and the worker population claims and processes them concurrently:
+For high-throughput or external embedder services, the indexer enqueues jobs and queue consumers (the `embed-worker`, a [queue](queue.md) consumer) claim and process them concurrently across whatever nodes run the embeddings engine:
 
 ```javascript
 await okdb.embeddings.createPipeline('articles', {

package/docs/functions.md

@@ -9,7 +9,7 @@ They are designed for:
 - stable HTTP entry points
 - MCP-accessible orchestration routines
 
-Custom functions are **stored**, **versioned**, **validated**, **audited**, and executed in **pooled sandbox runners**.
+Custom functions are **stored**, **versioned**, **validated**, **audited**, and executed in a per-process **sandbox thread**.
 
 ---
 
@@ -110,32 +110,35 @@ Use `ctx.log(...)` instead of `console`.
 
 ## Runtime model
 
-A function invocation is a **durable work unit**, never a closure: `functions.run()` writes a
-`~fn:requests` row and the response comes back as a `~fn:responses` row (the UDP bus carries a
-wake hint so dispatch is ~ms). Because the stored function's script lives in the database, the
-work is fully portable — whoever claims the row can run it. Execution **never blocks the
-caller's main loop**; untrusted user code always runs off-loop.
-
-Where it runs depends on whether a [worker population](worker-fleet.md) exists:
-
-- **With workers** — a worker claims the request row and executes the user code in its
-  **sandbox thread**: a dedicated `worker_thread` with its own okdb instance, heap-capped from
-  `runtime.memoryMb` via `resourceLimits`, and `terminate()` as the timeout/wedge watchdog.
-  One sandbox per claimer; invocations run concurrently in it, capped. Containment is
-  thread-level — a wedged function is `terminate()`d without taking the worker's other claims
-  down.
-- **No workers (embedded / scale 0)** — the instance claims its own requests and runs them in a
-  **fork pool** of child-process runners (warm between calls, idle-reaped, one run at a time per
-  runner). This is the zero-config default and the **memory fence**: a thread heap cap is
-  process-fatal on OOM, so only a separate process can contain a memory-hog function
-  (`FUNCTION_OOM`) without crashing the host. Embedders who prefer thread-cheapness can opt into
-  the sandbox locally; the fork pool can be forced with `OKDB_FN_LEGACY_POOL=1`.
-
-The **transport is identical** either way (durable `~fn` rows + bus fast path) — only the
-execution site differs (sandbox thread vs. runner process). Both: scripts are validated and
-compile-checked before storage; timeouts kill/replace the executor; memory is capped; every run
-is written to a run ledger. This makes functions suitable for operational entry points without
-paying spawn cost on every call.
+A function runs on **the node it was asked on**, in that node's per-process **sandbox thread**:
+a dedicated `worker_thread` with its own okdb instance, heap-capped from `runtime.memoryMb` via
+`resourceLimits`, with `terminate()` as the timeout/wedge watchdog. One sandbox per node,
+lazy-spawned and idle-reaped; invocations run concurrently in it, capped. Execution **never
+blocks the caller's main loop** — untrusted user code always runs off-loop — and containment is
+thread-level: a wedged function is `terminate()`d without taking the node's other work down.
+This is what makes "run the function locally" safe, so it no longer needs a separate worker
+process.
+
+Sandbox concurrency is capped **per process** (`OKDB_FN_SANDBOX_CONCURRENCY`, default 4):
+invocations beyond the cap queue at the claim layer. Host-level CPU budgeting is the operator's
+job — size the number of okdb processes (and their sandbox caps) to the machine.
+
+Two invocation modes:
+
+- **Synchronous** (`functions.run()` / HTTP execution) — routed straight to the local sandbox,
+  no `~fn:requests` round-trip, lowest latency.
+- **Durable / run-eventually** (survive the caller, store the result) — a [queue](queue.md) job
+  on a designated node. The `~fn:requests` / `~fn:responses` substrate is retained as the
+  durable invocation mode (the UDP bus carries a wake hint so dispatch is ~ms).
+
+Either way: scripts are validated and compile-checked before storage; timeouts terminate and
+replace the executor; memory is capped; every run is written to a run ledger. This makes
+functions suitable for operational entry points without paying spawn cost on every call.
+
+> **Removed in 2.0:** the anonymous worker-population dispatch and the child-process **fork
+> pool** (`OKDB_FN_LEGACY_POOL`). Functions execute in the asking node's local sandbox. See
+> [Upgrading to 2.0](upgrade-2.0.md). If you don't want every HTTP node carrying a sandbox,
+> designate which nodes serve functions — placement is yours.
 
 ---
 

package/docs/getting-started.md

@@ -17,7 +17,7 @@ Dependencies that OKDB needs internally (`lmdb`, `sift`, `hnswlib-node`, etc.) a
 ## Opening a database
 
 ```javascript
-const OKDB = require('src/okdb');
+const OKDB = require('@kedem/okdb');
 
 const okdb = new OKDB('./mydb', options);
 await okdb.open();
@@ -63,9 +63,20 @@ const okdb = new OKDB('./mydb', {
         token: 'cluster-secret',
         address: 'http://localhost:8080', // this node's public address
     },
+
+    // Role flags (okdb 2.0) — what background work THIS process runs.
+    // All default to true; a plain `new OKDB(path)` is a full do-everything node.
+    // `processors` is the INITIAL participation only — okdb.processors.start()/stop()
+    // flip it at runtime (open with false + start() later = fast startup).
+    processors: true, // claim processor leases and drain derived work (FTS, views, …)
+    engines: true, // run embeddings / vector-search engines
+    compaction: true, // eligible to claim the per-env compaction lease
 });
 ```
 
+See [Roles & Deployment](./deployment.md) for what the role flags mean and how to run
+multi-process topologies (N capable nodes, dedicated workers + passive serving nodes).
+
 ---
 
 ## Your first type
@@ -76,7 +87,7 @@ A **type** is OKDB's equivalent of a collection or table. Records in a type shar
 await okdb.registerType('users');
 ```
 
-Types are idempotent to register — calling `registerType` on an already-registered type throws. Use `ensureType` for idempotent setup:
+`registerType` throws if the type already exists. Use `ensureType` for idempotent setup:
 
 ```javascript
 await okdb.ensureType('users', {
@@ -197,7 +208,7 @@ process.on('SIGTERM', () => okdb.close().then(() => process.exit(0)));
 
 ```javascript
 'use strict';
-const OKDB = require('src/okdb');
+const OKDB = require('@kedem/okdb');
 
 async function main() {
     const okdb = new OKDB('./demo');

package/docs/http-api.md

@@ -600,7 +600,7 @@ OKDB's HTTP layer is **framework-agnostic**. `okdb.http.listen()` is entirely op
 
 ```javascript
 const express = require('express');
-const OKDB = require('src/okdb');
+const OKDB = require('@kedem/okdb');
 
 const app = express();
 const okdb = new OKDB('./mydb');
@@ -643,7 +643,7 @@ app.listen(3000);
 
 ```javascript
 const fastify = require('fastify')({ logger: false });
-const OKDB = require('src/okdb');
+const OKDB = require('@kedem/okdb');
 
 const okdb = new OKDB('./mydb');
 await okdb.open();
@@ -684,7 +684,7 @@ await fastify.listen({ port: 3000 });
 
 ```javascript
 const http = require('http');
-const OKDB = require('src/okdb');
+const OKDB = require('@kedem/okdb');
 
 const okdb = new OKDB('./mydb');
 await okdb.open();

package/docs/http-cluster.md

@@ -1,12 +1,21 @@
 # HTTP Clustering
 
-By default, the `okdb` CLI runs HTTP serving across **multiple worker processes** so that
-request/response latency is never starved by background work (indexing, view rebuilds,
-compaction, async catch-up). The processing tier runs in a single **primary** process; HTTP +
-SSE are served by **N passive worker processes** that share the listen socket.
-
-This is a **CLI-only default**. The embedded library (`new OKDB(...).http.listen(port)`) stays
-single-process unless you opt in — see [Embedded clustering](#embedded-clustering).
+By default, the `okdb` CLI runs across **N identical capable worker processes** that share
+the listen socket: each one serves HTTP + SSE **and** participates in processing. There is **no
+privileged processor** — the 1-of-N `OKDBLock` lease distributes claims across the workers
+(cooperative hold-window rotation load-shares under sustained backlog), and each worker drains
+its claims on its own loop in bounded quanta, so a heavy drain on one worker never blocks the
+others. A dead worker's claims fail over to the survivors. The Node-cluster primary is a thin
+**passive supervisor** (it forks/respawns the workers and seeds the shared token secret; it
+does not serve or process).
+
+This is a **CLI-only feature** (`okdb` / `bin/okdb`). The embedded library
+(`new OKDB(...).http.listen(port)`) is always single-process — `http.listen({workers})` was
+removed in 2.0. For multi-process embedding, run **N independent full nodes** (each its own
+`new OKDB(path).http.listen(port)`) on the same data path behind your own load balancer: they
+already coordinate through the shared LMDB + bus + the 1-of-N lease exactly like the CLI's
+workers (this is the same "N capable nodes" shape, just supervised by your balancer instead of
+`bin/okdb`). See [Embedded clustering](#embedded-clustering).
 
 ---
 
@@ -14,36 +23,40 @@ single-process unless you opt in — see [Embedded clustering](#embedded-cluster
 
 ```
                 ┌──────────────────────────────────────────────┐
-                │ primary  (full role — engines, async          │
-                │          processors, compaction:'active', bus)│
-                │  • runs ALL processing / indexing / compaction │
-                │  • HTTP-SILENT (never serves requests itself)  │
+                │ supervisor  (PASSIVE — processors:false,       │
+                │             engines:false, compaction:passive) │
+                │  • forks/respawns/decommissions the workers    │
+                │  • owns the ~processes registry (publishes a   │
+                │    row per worker; workers are registry-passive)│
                 │  • opens its OKDB BEFORE forking (seeds the     │
                 │    shared __tokenSecret so workers agree)       │
+                │  • HTTP-SILENT; does NOT process                │
                 └───────────────┬──────────────────────────────┘
                                 │ forks N
         ┌───────────────┬───────┴───────┬───────────────┐
         ▼               ▼               ▼               ▼
    ┌─────────┐    ┌─────────┐     ┌─────────┐     ┌─────────┐
-   │ worker  │    │ worker  │ ... │ worker  │     │ worker  │   passive role:
-   │ HTTP+SSE│    │ HTTP+SSE│     │ HTTP+SSE│     │ HTTP+SSE│   engines:false
-   └─────────┘    └─────────┘     └─────────┘     └─────────┘   asyncProcessors:false
-        └───────────────┴── shared listen socket ──┴───────────┘ compaction:'passive'
-                                                                  bus:true
+   │ worker  │    │ worker  │ ... │ worker  │     │ worker  │   CAPABLE role:
+   │HTTP+proc│    │HTTP+proc│     │HTTP+proc│     │HTTP+proc│   processors:true
+   └─────────┘    └─────────┘     └─────────┘     └─────────┘   (serve + process +
+        └───────────────┴── shared listen socket ──┴───────────┘  compact)
 ```
 
-- **Primary = processor.** Exactly one process runs the processing pool, async processors, and
-  compaction. It is HTTP-silent; it never serves requests. (The role flag is the belt; the
-  `OKDBLock` proc-lease is the backstop — no worker ever processes.)
-- **Workers = passive HTTP servers.** Each forked worker opens a **passive** OKDB
-  (`engines:false`, `asyncProcessors:false`, `compaction:'passive'`, `bus:true`) and serves
-  HTTP + SSE on the shared socket. Workers do real reads/writes — inline index/schema/FK/TTL
-  updates run synchronously inside the writer regardless of role — but never run background
-  processing. The primary's async processors pick up the changelog via the UDP bus poke.
-- **`workers:1`** (or `--no-cluster`) is byte-identical to the pre-cluster single-process path.
-
-Worker writes update indexes correctly; the primary's processors then index FTS / refresh
-views / run the materializer / etc. from the shared changelog.
+- **Workers = N identical capable nodes.** Each forked worker opens a **full-role** OKDB
+  (`processors:true`) and both serves HTTP + SSE on the shared socket **and** processes — it
+  claims the 1-of-N `OKDBLock` lease, builds/indexes FTS + views, and drains its claims on its
+  own loop in bounded quanta. Claims **distribute** across the workers (no
+  privileged processor); a dead worker's leases lapse and the survivors re-claim. Workers are
+  **registry-passive** (the supervisor publishes their `~processes` rows; a capable worker
+  reports its processing facet to the supervisor over IPC so cluster processing stays visible).
+- **Supervisor = passive.** It opens a passive OKDB (`processors:false`, `engines:false`,
+  `compaction:false`) only to seed the shared `__tokenSecret` before forking and to own the
+  registry. It is HTTP-silent and never processes.
+- **`workers:1`** (or `--no-cluster`) is byte-identical to the single-process path: one full
+  capable process serves + processes.
+
+A write on any worker is indexed by whichever worker holds the lease (cross-process POKE + the
+shared changelog); bounded quanta keep that indexing from monopolizing the holder's loop.
 
 ---
 
@@ -59,9 +72,9 @@ The CLI resolves the worker count from the first source present, highest priorit
 | 4        | `http.workers`      | `.kdbconfig` config file.                      |
 | 5        | _default_           | `max(1, cores − 1)` where `cores = os.cpus()`. |
 
-A non-integer or `< 1` explicit value clamps to `1` with a warning. The default reserves one
-core for the processing primary and serves HTTP from the rest; on a single-core box it collapses
-to one process.
+A non-integer or `< 1` explicit value clamps to `1` with a warning. The default leaves one core
+for the OS / the thin passive supervisor and runs N capable workers on the rest (each serves HTTP
+_and_ processes); on a single-core box it collapses to one process.
 
 ```bash
 okdb                       # default: max(1, cores − 1) HTTP workers
@@ -107,16 +120,21 @@ client on a given worker will and won't see:
   reconstruct data-change events (item put/remove, index, view, type). A write committed on
   worker A surfaces in worker B's SSE within the poke window. The bus carries the **signal**;
   the **data** is read from shared LMDB — the changelog is the single source of truth.
-- **Processing progress is primary-owned, forwarded over IPC.** Ephemeral processing events
-  (view-rebuild progress, FTS/index lifecycle, processor lifecycle) originate only in the
-  primary (the sole processor). The primary forwards a curated set to all workers over
-  **cluster IPC**, plus a workers / processing status snapshot. Workers serve
-  `/api/processing/status` and `/api/cluster/status` from that snapshot.
-- **Worker-local logs stay local.** A log line emitted inside worker A is **not** replicated to
-  worker B's SSE. Logs are per-process; only data changes (via the changelog) and primary
-  progress (via IPC) are coherent across the cluster.
-
-`/api/cluster/status` returns the cluster shape (primary pid, worker pids, uptime).
+- **Processing progress is per-worker; the cluster-wide view is the durable registry.** There is
+  no privileged processor — each capable worker processes its own claimed leases and emits its own
+  ephemeral processing events (view-rebuild progress, FTS/index lifecycle, processor lifecycle) to
+  **its own** SSE. The cluster-wide picture is **read** from the durable `~processes` registry:
+  each worker reports its processing facet to the supervisor over **cluster IPC**, the supervisor
+  stamps it onto the worker's registry row, and any node serves `/api/processing/status`,
+  `/api/processors/status`, and `/api/cluster/status` from that shared registry. So a processing
+  event on worker A is not mirrored into worker B's live SSE, but A's processor **state** is
+  visible cluster-wide via the registry-sourced status routes.
+- **Worker logs are forwarded to the supervisor's log pipeline** over cluster IPC (tagged per
+  worker) so the operator log is unified; they are **not** mirrored into other workers' SSE
+  streams (SSE log frames are per-process). Only data changes (via the changelog) are coherent
+  across every worker's SSE.
+
+`/api/cluster/status` returns the cluster shape (supervisor pid, worker pids, uptime).
 
 ---
 
@@ -132,26 +150,41 @@ client on a given worker will and won't see:
 
 ## Embedded clustering
 
-The **library default is single-process** and unchanged. `new OKDB(...).http.listen(port)` with
-no `workers` option returns a plain single-process `http.Server`. To opt an embedded app into
-clustering, pass `workers` (and a passive-worker OKDB factory) to `http.listen`:
+The **library default is single-process** and unchanged. `new OKDB(...).http.listen(port)`
+returns a plain single-process `http.Server`.
+
+> **Removed in 2.0:** the embedded `http.listen(port, { workers, makeOkdb, primaryOkdb })` API
+> (it **throws `HTTP_CLUSTER_REMOVED`**). A library node never forks — HTTP clustering is a
+> **launcher** concern. See [Upgrading to 2.0](upgrade-2.0.md).
+
+Two ways to cluster an embedded app — both are the same **"N identical capable nodes"** shape
+(every node serves _and_ processes; the 1-of-N lease distributes the work; no privileged processor):
+
+1. **N independent nodes behind your balancer** (simplest). Run your entry as N separate processes,
+   each a full `new OKDB(path).http.listen(port_i)`, and put a load balancer in front. They
+   coordinate through the shared LMDB + bus + the lease — no Node `cluster`, no supervisor needed.
+2. **Plain Node `cluster`** (one listen port, shared socket). Fork your entry; each worker opens a
+   **capable** OKDB and calls `http.listen(port)` — a cluster worker's `listen` shares the socket
+   natively, with no okdb involvement.
 
 ```js
-okdb.http.listen(port, {
-    workers: 4,
-    primaryOkdb: okdb, // full-role processor, opened before listen() (open-before-fork)
-    makeOkdb: async () => {
-        const w = new OKDB(dbPath, {
-            engines: false,
-            asyncProcessors: false,
-            compaction: 'passive',
-            bus: true,
-        });
-        await w.open();
-        return w;
-    },
-});
+const cluster = require('node:cluster');
+
+if (cluster.isPrimary) {
+    // Thin supervisor: open a passive node BEFORE forking to seed the shared __tokenSecret,
+    // then fork N capable workers. It does not serve or process.
+    const supervisor = new OKDB(dbPath, { processors: false, compaction: false, http: false });
+    await supervisor.open();
+    for (let i = 0; i < 4; i++) cluster.fork();
+} else {
+    // Each worker = an identical CAPABLE node: serves HTTP AND processes (claims leases,
+    // pool-drains heavy work off its loop). The lease distributes processing across them.
+    const w = new OKDB(dbPath, { processors: true });
+    await w.open();
+    w.http.listen(port); // shared socket via Node cluster; bus auto-derives from the shared path
+}
 ```
 
-The same shmbuf requirement applies — only cluster an embedded app on a host with the native
-`shmbuf` binding.
+The same shmbuf requirement applies — only cluster on a host with the native `shmbuf` binding.
+For a turnkey cluster (supervisor, respawn, registry rows, log forwarding), `bin/okdb` does all of
+this for you.