@kedem/okdb 1.8.17 → 1.9.0

package/docs/change-log.md

@@ -115,22 +115,22 @@ stop();
 
 ### Register options reference
 
-| Option               | Type       | Default   | Description                                                                                                                                                                                                                                   |
-| -------------------- | ---------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `handler`            | `function` | required  | Async callback receiving `(changes[], info)`                                                                                                                                                                                                  |
-| `mode`               | `string`   | `'async'` | `'async'` \| `'inline'` \| `'worker'`                                                                                                                                                                                                         |
-| `bootstrap`          | `string`   | `'log'`   | `'snapshot'` \| `'log'` — how the initial backfill runs                                                                                                                                                                                       |
-| `originMode`         | `string`   | `'self'`  | `'all'` \| `'self'` \| `'remote'`                                                                                                                                                                                                             |
-| `batchSize`          | `number`   | `256`     | Max changes delivered per handler call                                                                                                                                                                                                        |
-| `hydrateValues`      | `boolean`  | `true`    | Attach current document value to each change                                                                                                                                                                                                  |
-| `cursorKey`          | `string`   | `null`    | Persist the cursor under this key so progress survives restarts                                                                                                                                                                               |
-| `lockMode`           | `string`   | `null`    | `'exclusive'` — serialise concurrent flushes for this processor                                                                                                                                                                               |
-| `failOnHandlerError` | `boolean`  | `false`   | Put the processor into error state on handler exception                                                                                                                                                                                       |
-| `flushDebounce`      | `number`   | `0`       | Trailing-edge debounce in ms. Rapid write bursts coalesce into a single flush after `flushDebounce` ms of quiet. `0` = flush immediately (default).                                                                                           |
-| `flushInterval`      | `number`   | `null`    | Start a `setInterval` that ticks `_scheduleFlush` every `flushInterval` ms after bootstrap. Useful for polling-style processors or `originMode: 'remote'` subscriptions where local writes don't trigger `_onAfterCommit`. `null` = disabled. |
-| `name`               | `string`   | `null`    | Human-readable label shown in admin UI and status                                                                                                                                                                                             |
-| `meta`               | `object`   | `{}`      | Arbitrary metadata attached to status objects                                                                                                                                                                                                 |
-| `leaseTtlMs`         | `number`   | `30000`   | Lease TTL for cross-process exclusive execution                                                                                                                                                                                               |
+| Option               | Type       | Default    | Description                                                                                                                                                                                                                                   |
+| -------------------- | ---------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `handler`            | `function` | required   | Async callback receiving `(changes[], info)`                                                                                                                                                                                                  |
+| `mode`               | `string`   | `'fanout'` | `'fanout'` (N-of-N) \| `'single'` (1-of-N) \| `'inline'` (every writer). Old names `'async'`→`'fanout'` and `'worker'`→`'single'` are accepted as deprecated aliases.                                                                         |
+| `bootstrap`          | `string`   | `'log'`    | `'snapshot'` \| `'log'` — how the initial backfill runs                                                                                                                                                                                       |
+| `originMode`         | `string`   | `'self'`   | `'all'` \| `'self'` \| `'remote'`                                                                                                                                                                                                             |
+| `batchSize`          | `number`   | `256`      | Max changes delivered per handler call                                                                                                                                                                                                        |
+| `hydrateValues`      | `boolean`  | `true`     | Attach current document value to each change                                                                                                                                                                                                  |
+| `cursorKey`          | `string`   | `null`     | Persist the cursor under this key so progress survives restarts                                                                                                                                                                               |
+| `lockMode`           | `string`   | `null`     | `'exclusive'` — serialise concurrent flushes for this processor                                                                                                                                                                               |
+| `failOnHandlerError` | `boolean`  | `false`    | Put the processor into error state on handler exception                                                                                                                                                                                       |
+| `flushDebounce`      | `number`   | `0`        | Trailing-edge debounce in ms. Rapid write bursts coalesce into a single flush after `flushDebounce` ms of quiet. `0` = flush immediately (default).                                                                                           |
+| `flushInterval`      | `number`   | `null`     | Start a `setInterval` that ticks `_scheduleFlush` every `flushInterval` ms after bootstrap. Useful for polling-style processors or `originMode: 'remote'` subscriptions where local writes don't trigger `_onAfterCommit`. `null` = disabled. |
+| `name`               | `string`   | `null`     | Human-readable label shown in admin UI and status                                                                                                                                                                                             |
+| `meta`               | `object`   | `{}`       | Arbitrary metadata attached to status objects                                                                                                                                                                                                 |
+| `leaseTtlMs`         | `number`   | `30000`    | Lease TTL for cross-process exclusive execution                                                                                                                                                                                               |
 
 `stop.status()` (alias `stop.getStatus()`) returns the live processor state including `flushDebounce` and `flushInterval`. `stop.pause()` clears both timers; `stop.resume()` restarts the interval if configured.
 

package/docs/embeddings.md

@@ -237,7 +237,7 @@ await okdb.embeddings.createPipeline('articles', {
 
 ### Queue mode
 
-For high-throughput or external embedder services, the indexer enqueues jobs and a worker pool consumes them concurrently:
+For high-throughput or external embedder services, the indexer enqueues jobs and the worker population claims and processes them concurrently:
 
 ```javascript
 await okdb.embeddings.createPipeline('articles', {

package/docs/functions.md

@@ -110,31 +110,32 @@ Use `ctx.log(...)` instead of `console`.
 
 ## Runtime model
 
-Functions execute in **isolated child processes** (runners), not in the main OKDB process.
-
-The pool is **dynamic**:
-
-- runners spawn on demand when `functions.run()` is called
-- runners stay warm between invocations — no spawn overhead on repeated calls
-- idle runners are reaped automatically — no wasted resources when quiet
-- one function runs at a time per runner
-
-**Per-env isolation**: each environment has its own pool with its own runner processes. Functions in different envs never share a runner.
-
-**Pool option**:
-
-- `maxRunners` — maximum number of concurrent runners (optional, defaults to unlimited)
-
-When using queue-worker engines with `concurrency: N`, the pool grows automatically to match — no manual sizing needed.
-
-Other behavior:
-
-- scripts are validated and compile-checked before storage
-- timeouts kill and replace the runner
-- memory is capped per runner process
-- every run is written to a run ledger
-
-This makes the feature suitable for operational entry points without paying process spawn cost on every call.
+A function invocation is a **durable work unit**, never a closure: `functions.run()` writes a
+`~fn:requests` row and the response comes back as a `~fn:responses` row (the UDP bus carries a
+wake hint so dispatch is ~ms). Because the stored function's script lives in the database, the
+work is fully portable — whoever claims the row can run it. Execution **never blocks the
+caller's main loop**; untrusted user code always runs off-loop.
+
+Where it runs depends on whether a [worker population](worker-fleet.md) exists:
+
+- **With workers** — a worker claims the request row and executes the user code in its
+  **sandbox thread**: a dedicated `worker_thread` with its own okdb instance, heap-capped from
+  `runtime.memoryMb` via `resourceLimits`, and `terminate()` as the timeout/wedge watchdog.
+  One sandbox per claimer; invocations run concurrently in it, capped. Containment is
+  thread-level — a wedged function is `terminate()`d without taking the worker's other claims
+  down.
+- **No workers (embedded / scale 0)** — the instance claims its own requests and runs them in a
+  **fork pool** of child-process runners (warm between calls, idle-reaped, one run at a time per
+  runner). This is the zero-config default and the **memory fence**: a thread heap cap is
+  process-fatal on OOM, so only a separate process can contain a memory-hog function
+  (`FUNCTION_OOM`) without crashing the host. Embedders who prefer thread-cheapness can opt into
+  the sandbox locally; the fork pool can be forced with `OKDB_FN_LEGACY_POOL=1`.
+
+The **transport is identical** either way (durable `~fn` rows + bus fast path) — only the
+execution site differs (sandbox thread vs. runner process). Both: scripts are validated and
+compile-checked before storage; timeouts kill/replace the executor; memory is capped; every run
+is written to a run ledger. This makes functions suitable for operational entry points without
+paying spawn cost on every call.
 
 ---
 

package/docs/http-cluster.md

@@ -0,0 +1,157 @@
+# HTTP Clustering
+
+By default, the `okdb` CLI runs HTTP serving across **multiple worker processes** so that
+request/response latency is never starved by background work (indexing, view rebuilds,
+compaction, async catch-up). The processing tier runs in a single **primary** process; HTTP +
+SSE are served by **N passive worker processes** that share the listen socket.
+
+This is a **CLI-only default**. The embedded library (`new OKDB(...).http.listen(port)`) stays
+single-process unless you opt in — see [Embedded clustering](#embedded-clustering).
+
+---
+
+## Process model
+
+```
+                ┌──────────────────────────────────────────────┐
+                │ primary  (full role — engines, async          │
+                │          processors, compaction:'active', bus)│
+                │  • runs ALL processing / indexing / compaction │
+                │  • HTTP-SILENT (never serves requests itself)  │
+                │  • opens its OKDB BEFORE forking (seeds the     │
+                │    shared __tokenSecret so workers agree)       │
+                └───────────────┬──────────────────────────────┘
+                                │ forks N
+        ┌───────────────┬───────┴───────┬───────────────┐
+        ▼               ▼               ▼               ▼
+   ┌─────────┐    ┌─────────┐     ┌─────────┐     ┌─────────┐
+   │ worker  │    │ worker  │ ... │ worker  │     │ worker  │   passive role:
+   │ HTTP+SSE│    │ HTTP+SSE│     │ HTTP+SSE│     │ HTTP+SSE│   engines:false
+   └─────────┘    └─────────┘     └─────────┘     └─────────┘   asyncProcessors:false
+        └───────────────┴── shared listen socket ──┴───────────┘ compaction:'passive'
+                                                                  bus:true
+```
+
+- **Primary = processor.** Exactly one process runs the processing pool, async processors, and
+  compaction. It is HTTP-silent; it never serves requests. (The role flag is the belt; the
+  `OKDBLock` proc-lease is the backstop — no worker ever processes.)
+- **Workers = passive HTTP servers.** Each forked worker opens a **passive** OKDB
+  (`engines:false`, `asyncProcessors:false`, `compaction:'passive'`, `bus:true`) and serves
+  HTTP + SSE on the shared socket. Workers do real reads/writes — inline index/schema/FK/TTL
+  updates run synchronously inside the writer regardless of role — but never run background
+  processing. The primary's async processors pick up the changelog via the UDP bus poke.
+- **`workers:1`** (or `--no-cluster`) is byte-identical to the pre-cluster single-process path.
+
+Worker writes update indexes correctly; the primary's processors then index FTS / refresh
+views / run the materializer / etc. from the shared changelog.
+
+---
+
+## The `workers` knob
+
+The CLI resolves the worker count from the first source present, highest priority first:
+
+| Priority | Source              | Notes                                          |
+| -------- | ------------------- | ---------------------------------------------- |
+| 1        | `--no-cluster`      | Forces `workers:1` (single-process escape).    |
+| 2        | `--workers N`       | Explicit flag.                                 |
+| 3        | `OKDB_HTTP_WORKERS` | Environment variable.                          |
+| 4        | `http.workers`      | `.kdbconfig` config file.                      |
+| 5        | _default_           | `max(1, cores − 1)` where `cores = os.cpus()`. |
+
+A non-integer or `< 1` explicit value clamps to `1` with a warning. The default reserves one
+core for the processing primary and serves HTTP from the rest; on a single-core box it collapses
+to one process.
+
+```bash
+okdb                       # default: max(1, cores − 1) HTTP workers
+okdb --workers 4           # 4 workers
+okdb --no-cluster          # single process (back-compat)
+OKDB_HTTP_WORKERS=3 okdb   # 3 workers
+```
+
+```json
+// .kdbconfig
+{ "http": { "workers": 4 } }
+```
+
+---
+
+## shmbuf requirement (hard gate)
+
+Multi-process clustering is **only safe with the native `shmbuf` binding**. The per-env Hybrid
+Logical Clock and changelog clock counter live in a **cross-process shared-memory segment**
+(`shmbuf`). Without the native addon, `shmbuf` falls back to an in-process `SharedArrayBuffer`
+that cannot coordinate clocks across forked workers — two processes could mint the same HLC /
+clock value and corrupt causal ordering.
+
+Therefore, **if `shmbuf` is in fallback mode at startup, the CLI forces `workers:1`** regardless
+of any flag, env var, or config, and logs a warning. This gate is non-negotiable.
+
+Check readiness with `okdb doctor`:
+
+```
+✓ cluster: ready (7 workers)                                         # native present
+✓ cluster: unavailable (shmbuf native missing) — running single-process   # fallback
+```
+
+---
+
+## SSE / log coherence boundary
+
+The cluster exposes two distinct event domains. Knowing which is which explains what an SSE
+client on a given worker will and won't see:
+
+- **Data changes are cluster-wide.** Each worker runs a **change-feed**: it subscribes to the
+  UDP bus `SYSTEM_POKE` signal and, per poke, reads the shared changelog (`getChanges`) to
+  reconstruct data-change events (item put/remove, index, view, type). A write committed on
+  worker A surfaces in worker B's SSE within the poke window. The bus carries the **signal**;
+  the **data** is read from shared LMDB — the changelog is the single source of truth.
+- **Processing progress is primary-owned, forwarded over IPC.** Ephemeral processing events
+  (view-rebuild progress, FTS/index lifecycle, processor lifecycle) originate only in the
+  primary (the sole processor). The primary forwards a curated set to all workers over
+  **cluster IPC**, plus a workers / processing status snapshot. Workers serve
+  `/api/processing/status` and `/api/cluster/status` from that snapshot.
+- **Worker-local logs stay local.** A log line emitted inside worker A is **not** replicated to
+  worker B's SSE. Logs are per-process; only data changes (via the changelog) and primary
+  progress (via IPC) are coherent across the cluster.
+
+`/api/cluster/status` returns the cluster shape (primary pid, worker pids, uptime).
+
+---
+
+## Lifecycle
+
+- **Worker death → primary respawns** it with bounded backoff (`min(4000, 250·2^failures)` ms)
+  and a crash-loop guard (≥ 5 deaths / 10 s per slot stops respawning that slot).
+- **Primary death → workers exit** (no orphan listeners or locks), via `cluster` disconnect plus
+  a POSIX `ppid===1` backstop.
+- **Shutdown** drains in-flight requests, then closes envs, with a 3000 ms hard-exit backstop.
+
+---
+
+## Embedded clustering
+
+The **library default is single-process** and unchanged. `new OKDB(...).http.listen(port)` with
+no `workers` option returns a plain single-process `http.Server`. To opt an embedded app into
+clustering, pass `workers` (and a passive-worker OKDB factory) to `http.listen`:
+
+```js
+okdb.http.listen(port, {
+    workers: 4,
+    primaryOkdb: okdb, // full-role processor, opened before listen() (open-before-fork)
+    makeOkdb: async () => {
+        const w = new OKDB(dbPath, {
+            engines: false,
+            asyncProcessors: false,
+            compaction: 'passive',
+            bus: true,
+        });
+        await w.open();
+        return w;
+    },
+});
+```
+
+The same shmbuf requirement applies — only cluster an embedded app on a host with the native
+`shmbuf` binding.

package/docs/index.md

@@ -108,6 +108,7 @@ await okdb.close();
 - [Full-Text Search](./fts.md) — FTS index, tokenizer, query
 - [Embeddings](./embeddings.md) — vector pipeline, semantic search
 - [Queue](./queue.md) — durable job queue
+- [Live Subscriptions](./subscriptions.md) — signal-SSE change streams with durable sessions
 - [Files](./files.md) — blob / file storage
 - [Custom Functions](./functions.md) — sandboxed stored JS functions, env/global scope, HTTP execution
 - [HTTP API](./http-api.md) — built-in REST API

package/docs/manifest.json

@@ -18,6 +18,7 @@
             { "file": "fts.md", "label": "Full-Text Search" },
             { "file": "embeddings.md", "label": "Embeddings" },
             { "file": "views.md", "label": "Materialized Views" },
+            { "file": "subscriptions.md", "label": "Live Subscriptions" },
             { "file": "pipelines.md", "label": "Pipelines" },
             { "file": "queue.md", "label": "Queue" },
             { "file": "files.md", "label": "Files" },
@@ -31,8 +32,11 @@
             { "file": "auth-and-sync.md", "label": "Auth & Sync" },
             { "file": "auth-permissions.md", "label": "Auth Permissions" },
             { "file": "http-api.md", "label": "HTTP API" },
+            { "file": "http-cluster.md", "label": "HTTP Clustering" },
+            { "file": "process-registry.md", "label": "Process Registry" },
             { "file": "plugins.md", "label": "Plugins" },
-            { "file": "processors.md", "label": "Processors (custom workers)" },
+            { "file": "processors.md", "label": "Processors (modes & registration)" },
+            { "file": "worker-fleet.md", "label": "Workers (population & placement)" },
             { "file": "logging.md", "label": "Logging" },
             { "file": "migration.md", "label": "Migration" }
         ]

package/docs/process-registry.md

@@ -0,0 +1,185 @@
+# Process Registry
+
+The process registry is a root-wide **task manager** for every okdb process sharing one data
+directory: the HTTP-cluster primary and its workers, queue/function child processes, a standalone
+CLI server, and opted-in embedded nodes. It gives the admin UI (System → **Processes**) and the HTTP API
+a single, live view of the whole process tree — and a durable, safe way to control it (pause/resume,
+decommission, retry/reset, hard-kill).
+
+It generalizes the same **durable-state-as-coordination** model used by live subscriptions: the
+authority for every action is a durable record; the UDP bus only carries a lossy "go re-read it now"
+hint. A forged or dropped bus packet can therefore neither trigger nor lose a control action.
+
+> **Opt-in.** The registry is off by default so embedders and tests are unaffected. `bin/okdb` and the
+> HTTP cluster turn it on automatically; a library embedder opts in with `new OKDB(path, { processes: … })`.
+
+---
+
+## The model
+
+### A process is a `node`; processors are a facet
+
+The primary entity is the **process**, not the processor. Each long-lived process publishes one
+`node` row:
+
+```js
+{
+  nodeKey,            // identity: `${pid}:${startedAt}` (wall-clock startedAt — same-host, see Liveness)
+  kind,               // primary | http-server | queue-worker | fn-runner | cli | standalone | embedded
+  pid, ppid, host,
+  supervisorKey,      // nodeKey of the supervising process (null = a root) → builds the tree
+  slot, slotKey,      // supervised position; slotKey is the STABLE durable control key (see below)
+  role,               // { engines, asyncProcessors, compaction, http }
+  listenAddr,
+  caps,               // { controllable, decommissionable, killable, killConfirm? } — see Capabilities
+  lifecycle,          // { state: 'running' | 'decommissioning', desiredVersionApplied }
+  processing,         // FACET (owners only): { pools[], processors[], cap, policy }
+  heartbeatAt, expiresAt,
+}
+```
+
+A processor (FTS, views, materializer, time-machine, …) is **not** a separate entity — it is summarized
+on the owner's `processing` facet. The same is true of process-backend pool workers: they are real OS
+processes but are summarized on `processing.pools[]`, not given their own rows.
+
+### Two stores, split by lifetime
+
+| Store                     | Where                                               | Lifetime / durability                                   | Holds                                               |
+| ------------------------- | --------------------------------------------------- | ------------------------------------------------------- | --------------------------------------------------- |
+| **Status** (`~processes`) | a dedicated env (`sync:false`, `durability:'fast'`) | **disposable** — crash-loss _is_ the liveness mechanism | one self-refreshed `node` row per live process      |
+| **Desired** (`~system`)   | `~processes:desired/*`, `~processes:cmd/*` sub-DBs  | **durable** — the authority for every control action    | pause/decommission/kill intent + retry/reset epochs |
+
+`~processes` is off the change-feed (the `~` env name) and is read by key/range, never tailed. Status
+writes emit **no** signal — the panel polls. Only desired-state/command writes emit a `PROC` hint.
+
+### Supervisor-published registration
+
+Children (http-servers, queue/fn runners) lack a stable cross-restart identity and some run with the
+bus off, so they do **not** self-register. Their **supervisor** — which already tracks each child's
+pid/slot and heartbeats it — writes the child's `node` row (`publishChildRow`) and removes it on exit.
+Roots (primary/cli/standalone) and opted-in embedded nodes self-publish via their heartbeat.
+
+### Liveness — wall-clock, read-time staleness
+
+Each writer refreshes its rows every `OKDB_WORKERS_HEARTBEAT_MS` (default 10000); a row's
+`expiresAt = now + 3×heartbeat`. Readers drop any row whose `expiresAt` is in the past
+(`listNodes` filters at read time) — so a crashed publisher simply ages out; nothing has to notice the
+death. Liveness deliberately uses **wall-clock `Date.now()`**, not the HLC: it is a same-host TTL
+deadline, not a causal stamp, and there is no clock skew between processes on one host.
+
+### `PROC` — a reconcile hint, never a command
+
+A control write emits `BUS_EVENTS.PROC` = `{ kind: 'reconcile', scope, key }` on the UDP bus. The bus
+framing is a shared-secret prefix, **not** an authenticated MAC, so the payload carries no command and
+no authority — only "re-read the durable `~system` record for `scope`/`key` now." The owner applies the
+durable record on the hint (fast path) **and** on its heartbeat (backstop), so a dropped hint
+self-heals within one beat.
+
+---
+
+## HTTP API
+
+All routes require `system:read` (views) or `system:write` (control).
+
+> **Naming.** The canonical surface is **`processes`** — `okdb.processes` and
+> `GET /api/processes[/tree]`. `/api/processes*` is the route surface (the former `okdb.fleet` accessor alias was removed). The tables
+> below still use `/api/processes*`; `/api/processes*` is equivalent for the read views.
+
+| Method & path                                        | Purpose                                                        |
+| ---------------------------------------------------- | -------------------------------------------------------------- |
+| `GET /api/processes` (or `/api/processes`)           | flat list of live processes (grouped, each with `instances[]`) |
+| `GET /api/processes/tree` (or `/api/processes/tree`) | nested process → instance → work tree (`supervisorKey`)        |
+| `POST /api/processors/:id/pause`/`resume`            | durable pause/resume of a processor (desired-state)            |
+| `POST /api/processors/:id/retry`                     | retry an errored processor (durable command epoch)             |
+| `POST /api/processors/cursor/reset`                  | reset a processor cursor to 0 (durable + command epoch)        |
+| `POST /api/processes/:nodeKey/decommission`          | gracefully stop a supervised process and keep it down          |
+| `POST /api/processes/:nodeKey/recommission`          | reverse a decommission/kill — the supervisor re-forks the slot |
+| `POST /api/processes/:nodeKey/kill?force=1`          | hard-kill (guarded; see Control tiers)                         |
+
+When the registry is off (single-process / embedders), `/api/processes[/tree]` returns a degenerate
+single-root node so the API still answers.
+
+---
+
+## Control tiers
+
+All three tiers write a durable record behind HTTP auth, then emit a `PROC` hint. The owner (or the
+target's supervisor) reconciles it.
+
+1. **Soft — pause/resume** (`controllable` kinds). Durable `~processes:desired/{proc,pool}/<key>` with a
+   monotonic `version`; the owner applies it when `version > applied`. Rapid toggles converge to the
+   last version (no flip-flop); concurrent writes from two nodes are last-writer-wins by `version`.
+2. **Imperative — retry/reset** (`~processes:cmd/<logicalKey>`, monotonic `epoch`). The owner re-bootstraps
+   the processor via `processor.restart`; `reset` also zeroes the persisted `~proc:state` cursor (so it
+   takes effect on the owner's next start even while it is offline). `epoch > applied` guards against
+   replay.
+3. **Lifecycle — decommission / hard-kill** (routed through the **supervisor**, not by fighting the
+   respawn loop):
+    - **Decommission** (`decommissionable` kinds): the supervisor marks the slot dead (no respawn) and
+      cleanly `disconnect()`s the child to drain in-flight work. Keyed by the **stable** `slotKey`
+      (`cluster:<port>/<slot>`), so it **survives a supervisor restart** — a restarted primary reloads
+      the dead slots before its fork loop and does not revive them. **Recommission** clears the record
+      and re-forks.
+    - **Hard-kill** (`POST …/kill`): the last resort, guarded. Requires explicit `?force=1`. Rejected
+      for non-killable kinds. Killing a **primary** additionally requires `?confirm=1` because it tears
+      down the whole cluster (its workers self-exit on the primary's death). The supervisor `SIGKILL`s
+      its own child; a root reads its own durable kill record and self-terminates.
+
+Every decommission / recommission / kill writes a structured **audit** entry to `~log` (feature
+`workers`: actor, target `nodeKey`+`kind`, action, result), surfaced in System → Logs.
+
+### Capabilities
+
+`caps` is published per kind and **enforced by the routes**:
+
+| kind                                         | controllable | decommissionable | killable | kill confirm |
+| -------------------------------------------- | ------------ | ---------------- | -------- | ------------ |
+| `primary`                                    | if owner     | no (root)        | yes      | cluster-wide |
+| `cli` / `standalone`                         | if owner     | no (root)        | yes      | confirm      |
+| `http-server` / `queue-worker` / `fn-runner` | no           | yes              | yes      | force        |
+| `embedded`                                   | no           | no               | **no**   | —            |
+
+Roots have no supervisor to route a decommission through — stop a root via its own shutdown or a
+confirmed kill. `embedded` is **observe-only**: okdb must never `process.exit` a host application, so an
+embedded node is never killable and never self-terminates.
+
+---
+
+## Scope: same-host processes vs sync peers
+
+The registry covers **same-host, same-root** processes — they share the LMDB files, the UDP bus, and
+the `EnvSharedState` shmbuf. **Cross-machine sync-replication peers are a separate axis** (no shared
+LMDB; "kill" there would be a sync-protocol RPC) and are surfaced by the Sync view, not the process registry.
+
+---
+
+## Embedding
+
+```js
+// Observe-only: appear in the registry, expose nothing controllable.
+const db = new OKDB('./data', { processes: { register: true, kind: 'embedded' } });
+
+// A library server that wants the full registry (and is the processor owner):
+const db = new OKDB('./data', { processes: { kind: 'standalone', listenAddr: ':8484' } });
+```
+
+`register: true` makes a process publish its own row even when it is not an `asyncProcessors` owner.
+`bin/okdb` sets `kind` automatically (`primary` / `http-server`); embedders pass it explicitly.
+
+---
+
+## Operational notes
+
+- **Bus secret & kill safety.** The `PROC` hint is only a nudge, so a weak/default bus secret cannot
+  forge a control action — the durable `~system` record (written behind HTTP auth) is the sole
+  authority. Still, set `OKDB_BUS_SECRET` in production so unrelated processes don't cross-poke.
+- **Hard-kill kill-switch.** The irreversible hard-kill tier is enabled by default but can be locked
+  down — `OKDB_PROCESS_KILL=off` or `new OKDB(path, { processes: { kill: false } })`. **Recommended in
+  production.** When off, every node's `caps.killable` is `false`, so the UI hides the Kill button and
+  the route rejects it (409). The graceful, reversible decommission/recommission tier is unaffected.
+- **`okdb doctor`** reports a `processes: N (M supervised)` line when a server is already running on the
+  data directory (read as a passive reader; skipped otherwise).
+- **Heartbeat tuning.** `OKDB_WORKERS_HEARTBEAT_MS` (default 10000) sets the publish cadence and, ×3, the
+  liveness TTL. Lower it for snappier death detection at the cost of more status writes.
+
+See also: [HTTP Clustering](http-cluster.md), [Processors](processors.md).