@dikolab/kbdb 0.1.6 → 0.2.1

package/docs/goals/worker-daemon.md

@@ -0,0 +1,1447 @@
+# Worker Daemon
+
+*A detached TypeScript daemon that loads the WASM
+module, caches indexes and content objects in memory,
+and serves database operations over local IPC.*
+
+## Goal
+
+The worker daemon (`src/worker.ts`) is a detached
+background process that provides knowledge base content
+retrieval and caching management. It is accessible
+**only** by worker client connections over local IPC
+(see [Worker Client](worker-client.md)).
+
+The daemon:
+
+1. Loads and compiles the `kb-worker` WASM module once
+   at startup.
+2. Holds indexes and content objects in an in-memory
+   cache managed by the WASM module.
+3. Listens for IPC requests from worker client
+   connections.
+4. Processes all database operations through the WASM
+   module.
+5. Returns results over IPC.
+
+The daemon is spawned on demand by the worker client
+module and shuts itself down after a configurable idle
+timeout. It is not a persistent server -- it is a
+short-lived background process that exists only to
+amortize WASM compilation cost and share cached data
+across worker client connections.
+
+The daemon has no knowledge of its callers. It does not
+know whether a request originates from a CLI invocation,
+an MCP server, or any other consumer. It speaks only
+the JSON-RPC 2.0 protocol defined in this document.
+
+### Why a Separate Process
+
+The daemon runs as a separate OS process rather than an
+in-process worker thread to provide:
+
+- **Process isolation** -- The daemon survives the exit
+  of the process that spawned it. It stays alive as
+  long as the idle timeout allows, serving any worker
+  client that connects.
+- **Cross-connection sharing** -- Multiple concurrent
+  worker client connections share the same daemon and
+  its warm cache for the same database directory.
+- **Independent lifecycle** -- The daemon starts and
+  stops independently of any single consumer.
+- **Single WASM compilation** -- The daemon compiles
+  the WASM module once at startup. All connected worker
+  clients reuse the compiled module, avoiding repeated
+  compilation overhead.
+
+## Context
+
+Every worker daemon operates within a **context** --
+the absolute path to the knowledge base database
+directory (e.g. `${target_dir}/.kbdb`). The context is
+the fundamental scope for all daemon operations. It
+determines:
+
+- **Which database is served** -- The daemon loads the
+  WASM module and operates exclusively on the database
+  at its context path. It never accesses other
+  databases.
+- **Which daemon to connect to** -- Each context gets
+  its own independent daemon process. Worker client
+  connections targeting the same database share one
+  daemon; different databases run separate daemons.
+- **Which configuration applies** -- Daemon
+  configuration (cache limits, timeouts) is read from
+  `worker.toml` inside the context directory.
+
+### Context ID
+
+Each context has a **context ID** (`ctx`) -- a
+**SHA-256 hash of the database directory's absolute
+path**, truncated to 16 lowercase hexadecimal
+characters. The truncated hash provides 64 bits of
+entropy -- sufficient to disambiguate all database
+directories on a single machine.
+
+Example: `/home/user/project/.kbdb` hashes to
+`ctx = "a1b2c3d4e5f67890"`.
+
+The context ID serves as:
+
+- **PID file name** -- `kbdb-{ctx}.pid` in the OS
+  temporary directory.
+- **IPC socket/pipe name** --
+  `kbdb-{ctx}.sock` (Unix) or
+  `\\.\pipe\kbdb-{ctx}` (Windows).
+- **Crash log file name** -- `kbdb-{ctx}.crash` in
+  the OS temporary directory, written on startup
+  failure.
+- **Request validation** -- Every IPC request includes
+  the `ctx` param. The daemon verifies it matches its
+  own context and rejects misrouted requests with error
+  code `-32001` (context mismatch).
+
+A single machine may host multiple knowledge bases.
+Each database directory is an independent context with
+its own daemon, its own cache, and its own WASM module
+instance. There is no shared state across contexts.
+
+## Daemon Startup
+
+When the worker client spawns a new daemon (see
+[Worker Client -- Spawning](worker-client.md)), the
+daemon process performs the following startup sequence:
+
+1. Reads `worker.toml` from the context directory for
+   configuration (falls back to defaults if absent).
+2. Computes the SHA-256 of the context path to derive
+   the context ID.
+3. Validates startup preconditions (see Startup
+   Preconditions below). If any check fails, the daemon
+   prints an error to stderr, writes a crash log to
+   `{tmpdir}/kbdb-{ctx}.crash`, and exits with code 1.
+   It does **not** write a PID file or create an IPC
+   socket.
+4. Compiles and instantiates the `kb-worker` WASM
+   module (see WASM Module below).
+5. Calls `worker_init` on the WASM module to open the
+   database context.
+6. Writes its PID to `kbdb-{ctx}.pid` in the OS
+   temporary directory.
+7. Creates the IPC socket (`kbdb-{ctx}.sock` on Unix,
+   named pipe on Windows).
+8. Begins listening for connections.
+
+The daemon only writes its PID file and creates the
+IPC socket after all preconditions pass and the WASM
+module is loaded. If the daemon exits before step 6,
+the worker client's PID-file poll times out, and the
+worker client reads the crash log (or the daemon's
+stderr) to report the failure reason.
+
+### Startup Preconditions
+
+The daemon validates the following conditions before
+proceeding with WASM module loading and IPC setup. If
+any condition fails, the daemon prints a structured
+error message to stderr, writes a crash log, and exits
+with exit code `1`.
+
+**1. Database directory exists**
+
+The context path must be an existing directory. If the
+directory does not exist, the daemon prints to stderr:
+
+```
+kbdb-worker: database directory not found: /path/to/.kbdb
+```
+
+The daemon does not create the directory. Database
+initialisation is the responsibility of the CLI
+(`kbdb --init`) or the `initDirectory` IPC method.
+
+**2. Database directory is readable and writable**
+
+The daemon must have both read and write access to the
+context directory. The daemon checks both R_OK and
+W_OK access flags.
+
+If the directory is not accessible:
+
+```
+kbdb-worker: cannot access database directory: /path/to/.kbdb
+  -- permission denied
+```
+
+The daemon requires write access because database
+operations (add, update, remove sections) modify files
+in the context directory. A read-only daemon mode is
+not supported.
+
+**3. Database version is compatible**
+
+After confirming the directory exists and is
+accessible, the daemon reads `catalog.toml` and checks
+the `version` field against `DB_FORMAT_VERSION` (fixed
+at `1` in `daemon-defaults.constant.ts`).
+
+- **version == DB_FORMAT_VERSION** -- compatible,
+  proceed with startup.
+- **version > DB_FORMAT_VERSION** -- the database was
+  created or migrated by a newer version of kbdb that
+  this daemon cannot handle. The daemon prints to
+  stderr:
+
+  ```
+  kbdb-worker: database version 3 is newer than kbdb
+    version 1 -- upgrade kbdb to open this database
+  ```
+
+- **version < DB_FORMAT_VERSION** (or absent) -- the
+  database needs migration. The daemon prints to
+  stderr:
+
+  ```
+  kbdb-worker: database version 0 requires migration
+    to version 1 -- run `kbdb --migrate` before
+    starting the daemon
+  ```
+
+  The daemon does not auto-migrate. Migration is an
+  explicit user action.
+
+### Startup Failure Behaviour
+
+When a precondition fails (or WASM loading fails):
+
+1. The daemon prints the error message to **stderr**.
+2. The daemon writes a crash log to
+   `{tmpdir}/kbdb-{ctx}.crash` (JSON with `error`
+   and `timestamp` fields).
+3. The daemon exits with exit code **1**.
+4. No PID file is written.
+5. No IPC socket is created.
+
+The crash log and stderr output are the mechanisms for
+communicating startup failures. The crash log persists
+after the daemon exits so the worker client can read it
+even after the subprocess has gone.
+
+## Connection Tracking
+
+The daemon tracks active client connections via an
+internal counter. The idle timer is only active when
+the connection count is zero. The timer resets whenever
+a client connects, disconnects, or sends a request.
+The daemon stays alive as long as at least one client
+is connected, regardless of the idle timer.
+
+## Graceful Shutdown
+
+On shutdown (idle timeout or OS signal SIGTERM/SIGINT),
+the daemon:
+
+1. Clears the idle timer.
+2. Closes the IPC server, stopping new connections.
+3. Clears the WASM module references.
+4. Removes its PID file (`kbdb-{ctx}.pid`).
+5. Removes the IPC socket file (`kbdb-{ctx}.sock` on
+   Unix).
+6. Removes the crash log (`kbdb-{ctx}.crash`) if it
+   exists.
+7. Calls `process.exit(0)`.
+
+## WASM Module
+
+The daemon loads a single WASM module: `kb-worker.wasm`
+(file: `kbdb_worker_bg.wasm`). This module statically
+links the `fs-database` and `query-parser` crates, so
+only one module needs to be compiled and instantiated.
+
+```
+kbdb_worker_bg.wasm
+  (statically links fs-database + query-parser)
+       ↕
+  host I/O imports (TypeScript)
+```
+
+### WASM Module: kb-worker.wasm
+
+`kb-worker.wasm` is the single module loaded by the
+daemon. Its responsibilities:
+
+- **Query orchestration** -- Parses queries, performs
+  index lookups, scores and ranks results.
+- **Write orchestration** -- Tokenizes content per
+  field and stores pre-computed term frequencies,
+  positions, and token counts in the database.
+- **Cache management** -- Manages the LRU cache of
+  sections, indexes, and manifests internally.
+- **Result tokens** -- Manages async query tokens for
+  IPC callers.
+
+### Module Loading
+
+When the daemon starts (after all startup preconditions
+pass), it:
+
+1. Creates a deferred `MemoryRef` (initially `null`)
+   and `AllocatorRef` (initially `null`).
+2. Builds the host I/O import record using
+   `createHostImports`, passing the context path and
+   both deferred refs.
+3. Reads `kbdb_worker_bg.wasm` from the WASM build
+   output directory (`kb-worker/kbdb_worker_bg.wasm`).
+4. Compiles the WASM bytes with `WebAssembly.compile`.
+5. Instantiates the module with `WebAssembly.instantiate`,
+   providing `{ env: { ...hostImports } }`.
+6. Wires the `MemoryRef` to the module's exported
+   `memory` and the `AllocatorRef` to the module's
+   exported `kbdb_alloc`.
+7. Calls `worker_init(pathBytes)` to open the database
+   context (passes `.` as the path, resolved relative
+   to the context path).
+
+The `WasmModules` record tracks only two fields:
+
+```typescript
+interface WasmModules {
+   readonly memory: WebAssembly.Memory;
+   readonly kbWorker: WebAssembly.Instance;
+}
+```
+
+The module exports its own linear memory (not imported
+externally). Host I/O imports read and write it through
+the deferred `MemoryRef`.
+
+### Host I/O Imports
+
+The `kb-worker.wasm` module calls back into TypeScript
+for all filesystem operations. The host import record
+(`createHostImports`) provides:
+
+| Import name       | Description                                |
+|-------------------|--------------------------------------------|
+| `host_read_file`  | Read file bytes; out-pointer protocol      |
+| `host_write_file` | Write bytes to file                        |
+| `host_rename`     | Rename (atomic swap) a file                |
+| `host_remove_file`| Delete a file                              |
+| `host_list_dir`   | List directory entries; out-pointer protocol |
+| `host_mkdir`      | Create directory recursively               |
+| `host_file_exists`| Check if a file exists (returns 1 or 0)    |
+
+All paths passed from WASM are relative; the host
+resolves them against the `contextPath`. Out-pointer
+functions (`host_read_file`, `host_list_dir`) allocate
+a buffer in WASM memory via `allocator.alloc` and write
+the pointer and length back to WASM-provided addresses.
+
+### WASM Memory Management
+
+The TypeScript layer uses four WASM exports to manage
+shared memory:
+
+| Export            | Signature                        | Purpose               |
+|-------------------|----------------------------------|-----------------------|
+| `kbdb_alloc`      | `(size: i32) -> i32`             | Allocate WASM memory  |
+| `kbdb_free`       | `(ptr: i32, size: i32) -> void`  | Free WASM memory      |
+| `get_return_ptr`  | `() -> i32`                      | Read return slot ptr  |
+| `get_return_len`  | `() -> i32`                      | Read return slot len  |
+
+For call conventions that pass input data:
+
+1. Encode the input as bytes.
+2. Allocate a buffer with `kbdb_alloc(len)`.
+3. Write the bytes into WASM memory at the returned
+   pointer.
+4. Call the WASM export with `(ptr, len)`.
+5. Free the input buffer with `kbdb_free(ptr, len)`.
+6. Read the return slot with `get_return_ptr()` and
+   `get_return_len()`, copy the bytes, then free
+   the return buffer.
+
+Zero-argument WASM exports (no input data) are called
+directly; only the return slot is read afterward.
+
+Return code `-1` (as a signed i32) signals an error;
+the return slot contains a UTF-8 error message.
+
+## Configuration
+
+Daemon configuration is stored in
+`${context_path}/worker.toml`. The daemon reads it at
+startup via `readConfig`. If the file is absent or
+unreadable, all defaults apply. The `DaemonConfig`
+interface uses camelCase properties:
+
+```typescript
+interface DaemonConfig {
+   readonly contextPath: string;
+   readonly itemLimit: number;
+   readonly indexLimit: number;
+   readonly timeoutMs: number;
+   readonly itemTtlMs: number;
+}
+```
+
+Settings (TOML keys map to camelCase properties):
+
+| TOML key       | Property      | Description                          | Default        |
+|----------------|---------------|--------------------------------------|----------------|
+| `item_limit`   | `itemLimit`   | Max content objects held in cache    | 1000           |
+| `index_limit`  | `indexLimit`  | Max index segments held in cache     | 100            |
+| `timeout_ms`   | `timeoutMs`   | Daemon idle timeout in milliseconds  | 300000 (5 min) |
+| `item_ttl_ms`  | `itemTtlMs`   | Per-item cache TTL in milliseconds   | 60000 (1 min)  |
+
+The `contextPath` field is always set from the runtime
+argument; it is not read from `worker.toml`.
+
+Example `worker.toml`:
+
+```toml
+item_limit = 2000
+index_limit = 200
+timeout_ms = 600000
+item_ttl_ms = 120000
+```
+
+Configuration is set once when the daemon spawns and
+does not change until the daemon shuts down and a new
+one is spawned.
+
+## IPC Transport
+
+The daemon listens on a local IPC channel:
+
+- **Linux / macOS** -- Unix domain socket at
+  `{tmpdir}/kbdb-{ctx}.sock`
+  (e.g., `/tmp/kbdb-a1b2c3d4e5f67890.sock`).
+- **Windows** -- Named pipe at
+  `\\.\pipe\kbdb-{ctx}`.
+
+The IPC path is computed by the platform-specific
+`getIpcPath(ctx)` function.
+
+Both Deno and Node.js support Unix domain sockets and
+Windows named pipes natively:
+
+- **Node.js** -- `net.createServer()` with
+  `server.listen(path)` and
+  `net.createConnection(path)`.
+
+No third-party IPC library is required. The daemon
+creates the IPC socket at startup and removes it on
+shutdown.
+
+## IPC Protocol
+
+Communication uses **JSON-RPC 2.0** over the IPC
+channel. Each request is a JSON-RPC call with a method
+name and params; each response is a JSON-RPC result or
+error.
+
+Messages are delimited by newlines. Each JSON-RPC
+message is a single line of JSON followed by `\n`. This
+provides simple framing over the stream-oriented IPC
+transport.
+
+Every request includes a `ctx` param -- the context ID.
+The daemon verifies the `ctx` matches its own context
+and returns a JSON-RPC error (`-32001`, "context
+mismatch") if it does not.
+
+## Method Routing
+
+The daemon dispatches each IPC method to one of three
+categories:
+
+1. **Daemon-state methods** -- handled directly in
+   TypeScript without touching WASM.
+2. **Config methods** (`config_{key}`) -- read from
+   `DaemonConfig` in TypeScript.
+3. **WASM methods** -- routed through `WASM_METHOD_MAP`
+   to a named `kb-worker.wasm` export.
+
+### WASM Method Map
+
+The daemon accepts both camelCase and snake_case aliases
+for each WASM operation:
+
+| IPC method(s)                           | WASM export              |
+|-----------------------------------------|--------------------------|
+| `search`, `query`                       | `worker_search`          |
+| `addSection`, `add_section`             | `worker_add_section`     |
+| `updateSection`, `update_section`       | `worker_update_section`  |
+| `removeSection`, `remove_section`       | `worker_remove_section`  |
+| `readSections`, `read_sections`         | `worker_read_sections`   |
+| `groupSections`, `group_sections`       | `worker_group_sections`  |
+| `removeGrouping`, `remove_grouping`     | `worker_remove_grouping` |
+| `retrieveDocument`, `retrieve_doc`      | `worker_retrieve_doc`    |
+| `content`, `compose`                    | `worker_compose`         |
+| `integrityCheck`, `integrity_check`     | `worker_check_integrity` |
+| `gc`                                    | `worker_garbage_collect` |
+| `rebuildIndexes`, `rebuild_indexes`     | `worker_rebuild_indexes` |
+| `initDirectory`, `init_directory`       | `worker_init_directory`  |
+| `migrateDatabase`                       | (not yet supported)      |
+| `query`                                 | `worker_query`           |
+| `get_result_status`                     | `worker_result_status`   |
+| `get_result`                            | `worker_get_result`      |
+| `invalidate`                            | `worker_invalidate`      |
+| `invalidateAll`, `invalidate_all`       | `worker_invalidate_all`  |
+| `dbStatus`, `db_status`                 | `worker_status`          |
+| `listByType`, `list_by_type`            | `worker_list_by_type`    |
+| `listDocumentsByType`                   | `worker_list_docs_by_type` |
+| `recall`                                | `worker_recall`          |
+
+`migrateDatabase` is listed in the map but throws
+`IpcError(-32601, 'migrateDatabase is not yet
+supported')` when called.
+
+## IPC API -- Daemon-State Methods
+
+These methods are handled in TypeScript without WASM
+calls.
+
+### `status`
+
+Returns the daemon's current state.
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "status",
+   "params": { "ctx": "<context-id>" },
+   "id": 1
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 1,
+   "result": "busy"
+}
+```
+
+Result: `"busy"` if the connection count is greater
+than zero, `"idle"` otherwise.
+
+### `idle_at`
+
+Returns the Unix timestamp (in seconds) when the last
+transaction completed.
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "idle_at",
+   "params": { "ctx": "<context-id>" },
+   "id": 2
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 2,
+   "result": 1750012800
+}
+```
+
+Result: integer (Unix epoch seconds). `0` if no
+transaction has completed since startup.
+
+### `config_{key}`
+
+Returns a configuration value. Method name is `config_`
+followed by the TOML key name.
+
+| Method               | DaemonConfig property | Returns                      |
+|----------------------|-----------------------|------------------------------|
+| `config_timeout_ms`  | `timeoutMs`           | Daemon idle timeout in ms    |
+| `config_item_ttl_ms` | `itemTtlMs`           | Per-item cache TTL in ms     |
+| `config_item_limit`  | `itemLimit`           | Max cached content objects   |
+| `config_index_limit` | `indexLimit`          | Max cached index segments    |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "config_timeout_ms",
+   "params": { "ctx": "<context-id>" },
+   "id": 3
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 3,
+   "result": 300000
+}
+```
+
+### `checkVersion`
+
+Reads `catalog.toml` from the given path (or the
+daemon's own context path) and returns a version status
+object. Does not require the daemon to be fully started.
+
+**Request params:**
+
+| Param        | Type   | Required | Description                       |
+|--------------|--------|----------|-----------------------------------|
+| `ctx`        | string | yes      | Context ID                        |
+| `targetPath` | string | no       | Path to read catalog from; defaults to `contextPath` |
+
+**Response (`VersionStatus`):**
+
+| Field             | Type   | Values                                       |
+|-------------------|--------|----------------------------------------------|
+| `status`          | string | `"compatible"` \| `"needs_migration"` \| `"too_new"` |
+| `currentVersion`  | number | Version found in catalog (0 if absent)       |
+| `requiredVersion` | number | `DB_FORMAT_VERSION` (currently `1`)          |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "checkVersion",
+   "params": { "ctx": "<context-id>" },
+   "id": 4
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 4,
+   "result": {
+      "status": "compatible",
+      "currentVersion": 1,
+      "requiredVersion": 1
+   }
+}
+```
+
+## IPC API -- WASM Read Methods
+
+### `search` / `query`
+
+Submits a full-text search query. The daemon encodes
+the params, calls `worker_search`, and decodes the
+result synchronously. Returns a `ScoredResult[]` array.
+
+**Request params:**
+
+| Param    | Type   | Required | Description                   |
+|----------|--------|----------|-------------------------------|
+| `ctx`    | string | yes      | Context ID                    |
+| `query`  | string | yes      | Query text                    |
+| `mode`   | string | no       | `"and"` (default) or `"or"`  |
+| `limit`  | number | no       | Max results (default 20)      |
+| `offset` | number | no       | Zero-based result offset (default 0) |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "search",
+   "params": {
+      "ctx": "<context-id>",
+      "query": "TypeScript architecture",
+      "mode": "and",
+      "limit": 10
+   },
+   "id": 5
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 5,
+   "result": {
+      "items": [
+         {
+            "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+            "heading": "Architecture Overview",
+            "type": "text/markdown",
+            "docids": ["doc-5678"],
+            "snippet": "The system is organized...",
+            "matched_terms": ["architecture", "system"],
+            "score": 8.7432
+         }
+      ],
+      "total": 42,
+      "offset": 0,
+      "limit": 20,
+      "has_more": true
+   }
+}
+```
+
+Note: the IPC layer keeps `score` (CLI uses it). The
+MCP layer strips it.
+
+Result: `PagedSearchResult` decoded from the WASM
+binary return slot.
+
+### `get_result_status`
+
+Checks the status of a previously submitted async
+operation by its numeric token.
+
+**Request params:**
+
+| Param   | Type   | Description            |
+|---------|--------|------------------------|
+| `ctx`   | string | Context ID             |
+| `token` | number | Token from prior call  |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "get_result_status",
+   "params": { "ctx": "<context-id>", "token": 42 },
+   "id": 6
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 6,
+   "result": { "status": 2 }
+}
+```
+
+Result: `{ status: number }` -- the raw status code
+from `worker_result_status`.
+
+### `get_result`
+
+Retrieves the result for a previously submitted async
+operation by its numeric token. Decodes the return slot
+as `ScoredResult[]`.
+
+**Request params:**
+
+| Param   | Type   | Description            |
+|---------|--------|------------------------|
+| `ctx`   | string | Context ID             |
+| `token` | number | Token from prior call  |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "get_result",
+   "params": { "ctx": "<context-id>", "token": 42 },
+   "id": 7
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 7,
+   "result": [
+      { "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3", "score": 0.95 }
+   ]
+}
+```
+
+### `readSections` / `read_sections`
+
+Reads multiple sections by kb-id. Returns an array of
+`SectionRecord` objects decoded from the WASM binary.
+
+**Request params:**
+
+| Param    | Type     | Description                    |
+|----------|----------|--------------------------------|
+| `ctx`    | string   | Context ID                     |
+| `kbids`  | string[] | List of section IDs to read    |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "readSections",
+   "params": {
+      "ctx": "<context-id>",
+      "kbids": [
+         "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+         "p4q5r6s7t8u9v0w1x2y3z4a5b6"
+      ]
+   },
+   "id": 8
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 8,
+   "result": [
+      {
+         "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+         "docids": ["doc-5678"],
+         "type": "text/markdown",
+         "title": "Architecture Overview",
+         "description": "High-level overview",
+         "content": "Section body text..."
+      }
+   ]
+}
+```
+
+### `recall`
+
+Progressive context expansion for one or more
+sections. Returns section content plus, depending
+on the `depth` parameter, parent documents,
+sibling sections, forward references, and
+back-references from the reference graph index.
+
+**Request params:**
+
+| Param   | Type     | Required | Description              |
+|---------|----------|----------|--------------------------|
+| `ctx`   | string   | yes      | Context ID               |
+| `kbid`  | string   | no*      | Single section ID        |
+| `kbids` | string[] | no*      | Batch section IDs        |
+| `depth` | number   | no       | Expansion depth (0--3, default 0) |
+
+*At least one of `kbid` or `kbids` required.
+
+The daemon calls `worker_recall` in the WASM
+engine, which reads sections, optionally looks up
+parent documents, sibling sections, forward
+references, and back-references from
+`references.idx` depending on the requested depth.
+
+Result: `RecallResult` (single) or
+`RecallResult[]` (batch) decoded from the WASM
+binary return slot.
+
+### `retrieveDocument` / `retrieve_doc`
+
+Retrieves a document manifest by document ID. Returns
+a decoded `DocumentManifest`.
+
+**Request params:**
+
+| Param   | Type   | Description        |
+|---------|--------|--------------------|
+| `ctx`   | string | Context ID         |
+| `docid` | string | Document ID        |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "retrieveDocument",
+   "params": {
+      "ctx": "<context-id>",
+      "docid": "doc-5678"
+   },
+   "id": 9
+}
+
+// Response (decoded DocumentManifest)
+{
+   "jsonrpc": "2.0", "id": 9,
+   "result": { ... }
+}
+```
+
+### `dbStatus` / `db_status`
+
+Returns database status information. The WASM module
+(`worker_status`) writes a key-value text block to the
+return slot; the daemon decodes it and returns it as an
+object.
+
+**Request params:** `ctx` only.
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "dbStatus",
+   "params": { "ctx": "<context-id>" },
+   "id": 10
+}
+
+// Response (decoded key-value text)
+{
+   "jsonrpc": "2.0", "id": 10,
+   "result": { "sections": "42", "documents": "5" }
+}
+```
+
+### `listByType` / `list_by_type`
+
+Returns all sections whose `sectionType` field
+matches the given type string. The daemon encodes the
+params, calls `worker_list_by_type`, and decodes the
+result as an array of section records.
+
+**Request params:**
+
+| Param         | Type   | Required | Description                       |
+|---------------|--------|----------|-----------------------------------|
+| `ctx`         | string | yes      | Context ID                        |
+| `sectionType` | string | yes      | Section type to filter by         |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "listByType",
+   "params": {
+      "ctx": "<context-id>",
+      "sectionType": "skill"
+   },
+   "id": 23
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 23,
+   "result": [
+      {
+         "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+         "docids": [],
+         "type": "skill",
+         "title": "summarize-document",
+         "description": "Summarize a document into
+            key points",
+         "content": "{\"name\":\"summarize-document\",...}"
+      }
+   ]
+}
+```
+
+Result: array of `SectionRecord` objects filtered by
+the requested `sectionType`. Returns an empty array
+when no sections match.
+
+### `listDocumentsByType`
+
+Returns all document manifests whose `type` field
+matches the given document type string. The daemon
+encodes the params, calls `worker_list_docs_by_type`,
+and decodes the result as an array of document
+manifests.
+
+**Request params:**
+
+| Param     | Type   | Required | Description                        |
+|-----------|--------|----------|------------------------------------|
+| `ctx`     | string | yes      | Context ID                         |
+| `docType` | string | yes      | Document type to filter by         |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "listDocumentsByType",
+   "params": {
+      "ctx": "<context-id>",
+      "docType": "agent"
+   },
+   "id": 24
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 24,
+   "result": [
+      {
+         "docid": "agent-code-reviewer",
+         "title": "Code Reviewer Agent",
+         "type": "agent",
+         "sections": [
+            { "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3", "position": 0 },
+            { "kbid": "p4q5r6s7t8u9v0w1x2y3z4a5b6", "position": 1 }
+         ]
+      }
+   ]
+}
+```
+
+Result: array of document manifest objects filtered
+by the requested `docType`. Returns an empty array
+when no documents match. Documents without a `type`
+field are regular documents and are not returned by
+this method.
+
+## IPC API -- WASM Write Methods
+
+### `addSection` / `add_section`
+
+Adds a new section to the database. The params are
+binary-encoded and passed to `worker_add_section`. The
+return slot contains the new section's kb-id as UTF-8.
+
+**Request params:**
+
+| Param         | Type   | Required | Description                                   |
+|---------------|--------|----------|-----------------------------------------------|
+| `ctx`         | string | yes      | Context ID                                    |
+| `sectionType` | string | yes      | Section type (`"text"`, `"image"`, `"code"`, etc.) |
+| `mimeType`    | string | no       | MIME type                                     |
+| `content`     | string | yes      | Section content                               |
+| `title`       | string | no       | Section title                                 |
+| `description` | string | no       | Section description                           |
+| `docid`       | string | no       | Document ID to attach this section to         |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "addSection",
+   "params": {
+      "ctx": "<context-id>",
+      "sectionType": "text",
+      "mimeType": "text/markdown",
+      "content": "# Architecture\n\nOverview...",
+      "title": "Architecture Overview",
+      "docid": "doc-5678"
+   },
+   "id": 11
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 11,
+   "result": { "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3" }
+}
+```
+
+### `updateSection` / `update_section`
+
+Updates an existing section. The params are
+binary-encoded and passed to `worker_update_section`.
+Returns the new kb-id (content hash may change).
+
+**Request params:**
+
+| Param         | Type   | Required | Description                            |
+|---------------|--------|----------|----------------------------------------|
+| `ctx`         | string | yes      | Context ID                             |
+| `kbid`        | string | yes      | Existing section ID to replace         |
+| `sectionType` | string | yes      | New section type                       |
+| `mimeType`    | string | no       | New MIME type                          |
+| `content`     | string | yes      | New content                            |
+| `title`       | string | no       | New title                              |
+| `description` | string | no       | New description                        |
+| `docid`       | string | no       | Document ID                            |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "updateSection",
+   "params": {
+      "ctx": "<context-id>",
+      "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+      "sectionType": "text",
+      "content": "Updated content...",
+      "title": "Updated Title"
+   },
+   "id": 12
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 12,
+   "result": { "kbid": "x9y8z7w6v5u4t3s2r1q0p9o8n7" }
+}
+```
+
+### `removeSection` / `remove_section`
+
+Removes a section by kb-id. The kb-id is UTF-8 encoded
+and passed to `worker_remove_section`. The return flag
+is decoded by `decodeRemoveFlag`.
+
+**Request params:**
+
+| Param   | Type   | Description             |
+|---------|--------|-------------------------|
+| `ctx`   | string | Context ID              |
+| `kbid`  | string | Section ID to remove    |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "removeSection",
+   "params": {
+      "ctx": "<context-id>",
+      "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3"
+   },
+   "id": 13
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 13,
+   "result": { "removed": true }
+}
+```
+
+### `groupSections` / `group_sections`
+
+Groups a set of sections into a named document. Params
+are binary-encoded and passed to `worker_group_sections`.
+Returns the new document ID.
+
+**Request params:**
+
+| Param   | Type     | Description                       |
+|---------|----------|-----------------------------------|
+| `ctx`   | string   | Context ID                        |
+| `title` | string   | Document title                    |
+| `kbids` | string[] | Section IDs to group              |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "groupSections",
+   "params": {
+      "ctx": "<context-id>",
+      "title": "Architecture Guide",
+      "kbids": [
+         "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+         "p4q5r6s7t8u9v0w1x2y3z4a5b6"
+      ]
+   },
+   "id": 14
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 14,
+   "result": { "docid": "doc-5678" }
+}
+```
+
+### `removeGrouping` / `remove_grouping`
+
+Removes a document grouping (the manifest), leaving
+the underlying sections intact. The document ID is
+UTF-8 encoded and passed to `worker_remove_grouping`.
+
+**Request params:**
+
+| Param   | Type   | Description          |
+|---------|--------|----------------------|
+| `ctx`   | string | Context ID           |
+| `docid` | string | Document ID to ungroup |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "removeGrouping",
+   "params": {
+      "ctx": "<context-id>",
+      "docid": "doc-5678"
+   },
+   "id": 15
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 15,
+   "result": { "ok": true }
+}
+```
+
+### `content` / `compose`
+
+Composes a rendered Markdown document from one or more
+kbid or docid identifiers. The identifiers are
+binary-encoded and passed to `worker_compose`. The
+return slot is decoded by `decodeComposeOutput`.
+
+**Request params:**
+
+| Param   | Type     | Description                            |
+|---------|----------|----------------------------------------|
+| `ctx`   | string   | Context ID                             |
+| `ids`   | string[] | kbids or docids to compose             |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "compose",
+   "params": {
+      "ctx": "<context-id>",
+      "ids": [
+         "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+         "doc-5678"
+      ]
+   },
+   "id": 16
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 16,
+   "result": {
+      "data": "# Architecture Guide\n\n...",
+      "total": 4096
+   }
+}
+```
+
+Result fields:
+
+| Field   | Type   | Description                                   |
+|---------|--------|-----------------------------------------------|
+| `data`  | string | Rendered Markdown string (`output.markdown`)  |
+| `total` | number | Total character count (`output.totalChars`)   |
+
+### `invalidate`
+
+Evicts a single cached entry by kb-id. The kb-id is
+UTF-8 encoded and passed to `worker_invalidate`.
+
+**Request params:**
+
+| Param   | Type   | Description            |
+|---------|--------|------------------------|
+| `ctx`   | string | Context ID             |
+| `kbid`  | string | Section ID to evict    |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "invalidate",
+   "params": {
+      "ctx": "<context-id>",
+      "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3"
+   },
+   "id": 17
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 17,
+   "result": { "ok": true }
+}
+```
+
+### `invalidateAll` / `invalidate_all`
+
+Evicts the entire WASM cache by calling
+`worker_invalidate_all` with no arguments.
+
+**Request params:** `ctx` only.
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "invalidateAll",
+   "params": { "ctx": "<context-id>" },
+   "id": 18
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 18,
+   "result": { "ok": true }
+}
+```
+
+### `initDirectory` / `init_directory`
+
+Initialises a new database directory. The path is
+UTF-8 encoded and passed to `worker_init_directory`.
+When `path` is omitted, the daemon uses its own
+`targetDir`.
+
+**Request params:**
+
+| Param   | Type   | Required | Description                                  |
+|---------|--------|----------|----------------------------------------------|
+| `ctx`   | string | yes      | Context ID                                   |
+| `path`  | string | no       | Path to initialise; defaults to `targetDir`  |
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "initDirectory",
+   "params": { "ctx": "<context-id>" },
+   "id": 19
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 19,
+   "result": { "ok": true }
+}
+```
+
+## IPC API -- Maintenance Methods
+
+Maintenance methods perform database-wide operations.
+They call WASM and return results directly.
+
+### `integrityCheck` / `integrity_check`
+
+Verifies the integrity of the database. Calls
+`worker_check_integrity` with no arguments. The return
+slot contains a key-value text block decoded by
+`decodeKeyValueText`.
+
+**Request params:** `ctx` only.
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "integrityCheck",
+   "params": { "ctx": "<context-id>" },
+   "id": 20
+}
+
+// Response (decoded key-value text as object)
+{
+   "jsonrpc": "2.0", "id": 20,
+   "result": { "ok": "true", "sections_checked": "42" }
+}
+```
+
+### `gc`
+
+Removes section files not referenced by any document
+manifest. Calls `worker_garbage_collect` with no
+arguments. The return slot contains a key-value text
+block decoded by `decodeKeyValueText`.
+
+**Request params:** `ctx` only.
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "gc",
+   "params": { "ctx": "<context-id>" },
+   "id": 21
+}
+
+// Response (decoded key-value text as object)
+{
+   "jsonrpc": "2.0", "id": 21,
+   "result": { "removed_sections": "3", "removed_bytes": "15360" }
+}
+```
+
+### `rebuildIndexes` / `rebuild_indexes`
+
+Reconstructs all index files from section files and
+document manifests on disk. Calls
+`worker_rebuild_indexes` with no arguments. Returns
+`{ ok: true }` on success.
+
+**Request params:** `ctx` only.
+
+```json
+// Request
+{
+   "jsonrpc": "2.0",
+   "method": "rebuildIndexes",
+   "params": { "ctx": "<context-id>" },
+   "id": 22
+}
+
+// Response
+{
+   "jsonrpc": "2.0", "id": 22,
+   "result": { "ok": true }
+}
+```
+
+## Error Responses
+
+JSON-RPC 2.0 error format for invalid requests:
+
+```json
+{
+   "jsonrpc": "2.0", "id": 11,
+   "error": {
+      "code": -32601,
+      "message": "Method not found: unknownMethod"
+   }
+}
+```
+
+Standard JSON-RPC 2.0 error codes apply. Application
+errors use codes in the `-32000` to `-32099` range:
+
+| Code     | Meaning                              |
+|----------|--------------------------------------|
+| `-32001` | Context mismatch                     |
+| `-32700` | Parse error (malformed JSON line)    |
+| `-32601` | Method not found                     |
+| `-32000` | WASM call error (message from WASM)  |
+
+Startup errors (`-32004` through `-32007` in previous
+versions of this document) are not returned over IPC
+because the daemon exits before creating the IPC
+socket. They are communicated via stderr and the crash
+log.
+
+## Source Files
+
+The worker daemon module lives in
+`src/shared/worker-daemon/` with the following
+structure:
+
+```
+src/shared/worker-daemon/
+   classes/
+      worker-daemon.class.ts       # WorkerDaemon class
+   functions/
+      create-host-imports.function.ts  # WASM host I/O imports
+      create-ipc-server.function.ts    # IPC socket server
+      load-wasm-modules.function.ts    # WASM compilation/instantiation
+      read-config.function.ts          # worker.toml parser
+      run-worker.function.ts           # entry-point logic
+      start-daemon.function.ts         # startup orchestration
+   constants/
+      daemon-defaults.constant.ts      # defaults + DB_FORMAT_VERSION
+   typings/
+      daemon-config.interface.ts       # DaemonConfig
+      ipc-handler.interface.ts         # IpcRequest, IpcResponse, IpcHandler
+      wasm-modules.interface.ts        # WasmModules
+   wasm/
+      kb-worker/
+         kbdb_worker_bg.wasm           # compiled WASM (gitignored)
+```
+
+The `run-worker.function.ts` implements the entry-point
+logic (argument validation and calling `startDaemon`).
+The `start-daemon.function.ts` orchestrates the full
+startup sequence: reads config, computes the context ID,
+constructs `WorkerDaemon`, checks preconditions, starts
+the daemon, and installs signal handlers.
+
+## Distribution
+
+The worker daemon (`src/worker.ts` and its compiled
+`kbdb-worker.js`) ships as part of the npm/JSR package.
+No per-platform native binaries are required.
+
+The package includes one WASM module:
+
+- `kb-worker/kbdb_worker_bg.wasm` -- single module
+  that statically links the query-parser and
+  fs-database crates
+
+It is compiled from Rust via `wasm-pack` and works on
+all platforms. The daemon loads and compiles it at
+startup using the runtime's `WebAssembly` API. The
+daemon runs on the same runtime (Node.js or Deno) as
+the worker client that spawns it.