@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/goals/worker-daemon.md

@@ -1,1547 +0,0 @@
-# Worker Daemon
-
-*A detached TypeScript daemon that loads the WASM
-module, caches indexes and content objects in memory,
-and serves database operations over local IPC.*
-
-## Goal
-
-The worker daemon (`src/worker.ts`) is a detached
-background process that provides knowledge base content
-retrieval and caching management. It is accessible
-**only** by worker client connections over local IPC
-(see [Worker Client](worker-client.md)).
-
-The daemon:
-
-1. Loads and compiles the WASM modules once at startup
-   (`kb-worker` for database operations, `embedding`
-   for default TF-IDF vector computation).
-2. Holds indexes and content objects in an in-memory
-   cache managed by the WASM module.
-3. Listens for IPC requests from worker client
-   connections.
-4. Processes all database operations through the WASM
-   module.
-5. Returns results over IPC.
-
-The daemon is spawned on demand by the worker client
-module and shuts itself down after a configurable idle
-timeout. It is not a persistent server -- it is a
-short-lived background process that exists only to
-amortize WASM compilation cost and share cached data
-across worker client connections.
-
-The daemon has no knowledge of its callers. It does not
-know whether a request originates from a CLI invocation,
-an MCP server, or any other consumer. It speaks only
-the JSON-RPC 2.0 protocol defined in this document.
-
-### Why a Separate Process
-
-The daemon runs as a separate OS process rather than an
-in-process worker thread to provide:
-
-- **Process isolation** -- The daemon survives the exit
-  of the process that spawned it. It stays alive as
-  long as the idle timeout allows, serving any worker
-  client that connects.
-- **Cross-connection sharing** -- Multiple concurrent
-  worker client connections share the same daemon and
-  its warm cache for the same database directory.
-- **Independent lifecycle** -- The daemon starts and
-  stops independently of any single consumer.
-- **Single WASM compilation** -- The daemon compiles
-  the WASM module once at startup. All connected worker
-  clients reuse the compiled module, avoiding repeated
-  compilation overhead.
-
-## Context
-
-Every worker daemon operates within a **context** --
-the absolute path to the knowledge base database
-directory (e.g. `${target_dir}/.kbdb`). The context is
-the fundamental scope for all daemon operations. It
-determines:
-
-- **Which database is served** -- The daemon loads the
-  WASM module and operates exclusively on the database
-  at its context path. It never accesses other
-  databases.
-- **Which daemon to connect to** -- Each context gets
-  its own independent daemon process. Worker client
-  connections targeting the same database share one
-  daemon; different databases run separate daemons.
-- **Which configuration applies** -- Daemon
-  configuration (cache limits, timeouts) is read from
-  `worker.toml` inside the context directory.
-
-### Context ID
-
-Each context has a **context ID** (`ctx`) -- a
-**SHA-256 hash of the database directory's absolute
-path**, truncated to 16 lowercase hexadecimal
-characters. The truncated hash provides 64 bits of
-entropy -- sufficient to disambiguate all database
-directories on a single machine.
-
-Example: `/home/user/project/.kbdb` hashes to
-`ctx = "a1b2c3d4e5f67890"`.
-
-The context ID serves as:
-
-- **PID file name** -- `kbdb-{ctx}.pid` in the OS
-  temporary directory.
-- **IPC socket/pipe name** --
-  `kbdb-{ctx}.sock` (Unix) or
-  `\\.\pipe\kbdb-{ctx}` (Windows).
-- **Crash log file name** -- `kbdb-{ctx}.crash` in
-  the OS temporary directory, written on startup
-  failure.
-- **Request validation** -- Every IPC request includes
-  the `ctx` param. The daemon verifies it matches its
-  own context and rejects misrouted requests with error
-  code `-32001` (context mismatch).
-
-A single machine may host multiple knowledge bases.
-Each database directory is an independent context with
-its own daemon, its own cache, and its own WASM module
-instance. There is no shared state across contexts.
-
-## Write Serialization
-
-The daemon serializes write operations using an
-exclusive file lock on `kbdb-{ctx}.lock` in the OS
-temporary directory. The lock file uses the same
-context ID as the PID file and IPC socket.
-
-- **Read operations** proceed without acquiring the
-  lock. Concurrent reads are fully parallel.
-- **Write operations** acquire the exclusive lock
-  before calling the WASM write export. If another
-  write is already in progress, the new write queues
-  behind the lock and waits.
-- **Lock duration** -- the lock is held only for the
-  duration of the single write operation and released
-  immediately after it completes. It is not held
-  across multiple operations or across client
-  connections.
-
-This serialization model is equivalent to SQLite's
-default locking: readers never block, and writers
-take turns one at a time.
-
-Write methods subject to this lock are:
-`addSection`, `updateSection`, `removeSection`,
-`groupSections`, `removeGrouping`, `initDirectory`,
-`gc`, and `rebuildIndexes`.
-
-## Daemon Startup
-
-When the worker client spawns a new daemon (see
-[Worker Client -- Spawning](worker-client.md)), the
-daemon process performs the following startup sequence:
-
-1. Reads `worker.toml` from the context directory for
-   configuration (falls back to defaults if absent).
-2. Computes the SHA-256 of the context path to derive
-   the context ID.
-3. Validates startup preconditions (see Startup
-   Preconditions below). If any check fails, the daemon
-   prints an error to stderr, writes a crash log to
-   `{tmpdir}/kbdb-{ctx}.crash`, and exits with code 1.
-   It does **not** write a PID file or create an IPC
-   socket.
-4. Compiles and instantiates the `kb-worker` WASM
-   module (see WASM Module below).
-5. Calls `worker_init` on the WASM module to open the
-   database context.
-6. Calls `db_check_integrity` on the WASM module to
-   run a startup health check. If `ok` is false, the
-   daemon logs a structured warning to stderr listing
-   the error count and error types. The daemon
-   continues operating -- health issues are not fatal
-   to startup. The health status (`integrity_ok`,
-   `integrity_errors`) is included in subsequent
-   `status` responses.
-7. Removes the stale IPC socket if present.
-8. Creates the IPC socket (`kbdb-{ctx}.sock` on Unix,
-   named pipe on Windows) and begins listening
-   (async, with backlog 256).
-9. Writes its PID to `kbdb-{ctx}.pid` in the OS
-   temporary directory.
-
-The daemon writes its PID file only after the IPC
-socket is bound and listening. This guarantees the
-worker client's PID-file poll succeeds only when
-the socket is ready to accept connections. If the
-daemon exits before step 9, the worker client's
-PID-file poll times out, and the worker client
-reads the crash log (or the daemon's stderr) to
-report the failure reason.
-
-### Startup Preconditions
-
-The daemon validates the following conditions before
-proceeding with WASM module loading and IPC setup. If
-any condition fails, the daemon prints a structured
-error message to stderr, writes a crash log, and exits
-with exit code `1`.
-
-**1. Database directory exists**
-
-The context path must be an existing directory. If the
-directory does not exist, the daemon prints to stderr:
-
-```
-kbdb-worker: database directory not found: /path/to/.kbdb
-```
-
-The daemon does not create the directory. Database
-initialisation is the responsibility of the CLI
-(`kbdb --init`) or the `initDirectory` IPC method.
-
-**2. Database directory is readable and writable**
-
-The daemon must have both read and write access to the
-context directory. The daemon checks both R_OK and
-W_OK access flags.
-
-If the directory is not accessible:
-
-```
-kbdb-worker: cannot access database directory: /path/to/.kbdb
-  -- permission denied
-```
-
-The daemon requires write access because database
-operations (add, update, remove sections) modify files
-in the context directory. A read-only daemon mode is
-not supported.
-
-**3. Database version is compatible**
-
-After confirming the directory exists and is
-accessible, the daemon reads `catalog.toml` and checks
-the `version` field against `DB_FORMAT_VERSION` (fixed
-at `1` in `daemon-defaults.constant.ts`).
-
-- **version == DB_FORMAT_VERSION** -- compatible,
-  proceed with startup.
-- **version > DB_FORMAT_VERSION** -- the database was
-  created or migrated by a newer version of kbdb that
-  this daemon cannot handle. The daemon prints to
-  stderr:
-
-  ```
-  kbdb-worker: database version 3 is newer than kbdb
-    version 1 -- upgrade kbdb to open this database
-  ```
-
-- **version < DB_FORMAT_VERSION** (or absent) -- the
-  database needs migration. The daemon prints to
-  stderr:
-
-  ```
-  kbdb-worker: database version 0 requires migration
-    to version 1 -- run `kbdb --migrate` before
-    starting the daemon
-  ```
-
-  The daemon does not auto-migrate. Migration is an
-  explicit user action.
-
-### Startup Failure Behaviour
-
-When a precondition fails (or WASM loading fails):
-
-1. The daemon prints the error message to **stderr**.
-2. The daemon writes a crash log to
-   `{tmpdir}/kbdb-{ctx}.crash` (JSON with `error`
-   and `timestamp` fields).
-3. The daemon exits with exit code **1**.
-4. No PID file is written.
-5. No IPC socket is created.
-
-The crash log and stderr output are the mechanisms for
-communicating startup failures. The crash log persists
-after the daemon exits so the worker client can read it
-even after the subprocess has gone.
-
-## Connection Tracking
-
-The daemon tracks active client connections via an
-internal counter. The idle timer is only active when
-the connection count is zero. The timer resets whenever
-a client connects, disconnects, or sends a request.
-The daemon stays alive as long as at least one client
-is connected, regardless of the idle timer.
-
-`SO_KEEPALIVE` (30-second interval) provides a
-secondary mechanism for detecting dead connections
-beyond the counter-based idle timer. If a client
-process crashes without closing the socket, the OS
-will detect the dead peer within ~30 seconds and
-close the connection, allowing the daemon to
-decrement its counter and eventually idle-shutdown.
-
-## Graceful Shutdown
-
-On shutdown (idle timeout or OS signal SIGTERM/SIGINT),
-the daemon:
-
-1. Clears the idle timer.
-2. Closes the IPC server, stopping new connections.
-3. Clears the WASM module references.
-4. Removes its PID file (`kbdb-{ctx}.pid`).
-5. Removes the IPC socket file (`kbdb-{ctx}.sock` on
-   Unix).
-6. Removes the crash log (`kbdb-{ctx}.crash`) if it
-   exists.
-7. Calls `process.exit(0)`.
-
-## WASM Modules
-
-The daemon loads four WASM modules at startup. Three
-share a single `WebAssembly.Memory` and are wired
-together via imports; the fourth is a standalone
-pure-computation module.
-
-```
-query-parser.wasm  (standalone)
-fs-database.wasm   (standalone)
-default-embedding.wasm (standalone, no host imports)
-       ↓
-kb-worker.wasm
-  (imports query-parser + fs-database)
-       ↕
-  host I/O imports (TypeScript)
-```
-
-### WASM Module: kb-worker.wasm
-
-`kb-worker.wasm` is the orchestration module. Its
-responsibilities:
-
-- **Query orchestration** -- Parses queries, performs
-  index lookups, scores and ranks results.
-- **Write orchestration** -- Tokenizes content per
-  field and stores pre-computed term frequencies,
-  positions, and token counts in the database.
-- **Cache management** -- Manages the LRU cache of
-  sections, indexes, and manifests internally.
-- **Result tokens** -- Manages async query tokens for
-  IPC callers.
-
-### Module Loading
-
-When the daemon starts (after all startup preconditions
-pass), it:
-
-1. Creates a deferred `MemoryRef` (initially `null`)
-   and `AllocatorRef` (initially `null`).
-2. Builds the host I/O import record using
-   `createHostImports`, passing the context path and
-   both deferred refs.
-3. Reads `kbdb_worker_bg.wasm` from the WASM build
-   output directory (`kb-worker/kbdb_worker_bg.wasm`).
-4. Compiles the WASM bytes with `WebAssembly.compile`.
-5. Instantiates the module with `WebAssembly.instantiate`,
-   providing `{ env: { ...hostImports } }`.
-6. Wires the `MemoryRef` to the module's exported
-   `memory` and the `AllocatorRef` to the module's
-   exported `kbdb_alloc`.
-7. Calls `worker_init(pathBytes)` to open the database
-   context (passes `.` as the path, resolved relative
-   to the context path).
-
-The `WasmModules` record tracks only two fields:
-
-```typescript
-interface WasmModules {
-   readonly memory: WebAssembly.Memory;
-   readonly kbWorker: WebAssembly.Instance;
-}
-```
-
-The module exports its own linear memory (not imported
-externally). Host I/O imports read and write it through
-the deferred `MemoryRef`.
-
-### Host I/O Imports
-
-The `kb-worker.wasm` module calls back into TypeScript
-for all filesystem operations. The host import record
-(`createHostImports`) provides:
-
-| Import name       | Description                                |
-|-------------------|--------------------------------------------|
-| `host_read_file`  | Read file bytes; out-pointer protocol      |
-| `host_write_file` | Write bytes to file                        |
-| `host_rename`     | Rename (atomic swap) a file                |
-| `host_remove_file`| Delete a file                              |
-| `host_list_dir`   | List directory entries; out-pointer protocol |
-| `host_mkdir`      | Create directory recursively               |
-| `host_file_exists`| Check if a file exists (returns 1 or 0)    |
-
-All paths passed from WASM are relative; the host
-resolves them against the `contextPath`. Out-pointer
-functions (`host_read_file`, `host_list_dir`) allocate
-a buffer in WASM memory via `allocator.alloc` and write
-the pointer and length back to WASM-provided addresses.
-
-### WASM Memory Management
-
-The TypeScript layer uses four WASM exports to manage
-shared memory:
-
-| Export            | Signature                        | Purpose               |
-|-------------------|----------------------------------|-----------------------|
-| `kbdb_alloc`      | `(size: i32) -> i32`             | Allocate WASM memory  |
-| `kbdb_free`       | `(ptr: i32, size: i32) -> void`  | Free WASM memory      |
-| `get_return_ptr`  | `() -> i32`                      | Read return slot ptr  |
-| `get_return_len`  | `() -> i32`                      | Read return slot len  |
-
-For call conventions that pass input data:
-
-1. Encode the input as bytes.
-2. Allocate a buffer with `kbdb_alloc(len)`.
-3. Write the bytes into WASM memory at the returned
-   pointer.
-4. Call the WASM export with `(ptr, len)`.
-5. Free the input buffer with `kbdb_free(ptr, len)`.
-6. Read the return slot with `get_return_ptr()` and
-   `get_return_len()`, copy the bytes, then free
-   the return buffer.
-
-Zero-argument WASM exports (no input data) are called
-directly; only the return slot is read afterward.
-
-Return code `-1` (as a signed i32) signals an error;
-the return slot contains a UTF-8 error message.
-
-## Configuration
-
-Daemon configuration is stored in
-`${context_path}/worker.toml`. The daemon reads it at
-startup via `readConfig`. If the file is absent or
-unreadable, all defaults apply. The `DaemonConfig`
-interface uses camelCase properties:
-
-```typescript
-interface DaemonConfig {
-   readonly contextPath: string;
-   readonly itemLimit: number;
-   readonly indexLimit: number;
-   readonly timeoutMs: number;
-   readonly itemTtlMs: number;
-}
-```
-
-Settings (TOML keys map to camelCase properties):
-
-| TOML key       | Property      | Description                          | Default        |
-|----------------|---------------|--------------------------------------|----------------|
-| `item_limit`   | `itemLimit`   | Max content objects held in cache    | 1000           |
-| `index_limit`  | `indexLimit`  | Max index segments held in cache     | 100            |
-| `timeout_ms`   | `timeoutMs`   | Daemon idle timeout in milliseconds  | 300000 (5 min) |
-| `item_ttl_ms`  | `itemTtlMs`   | Per-item cache TTL in milliseconds   | 60000 (1 min)  |
-
-The `contextPath` field is always set from the runtime
-argument; it is not read from `worker.toml`.
-
-Example `worker.toml`:
-
-```toml
-item_limit = 2000
-index_limit = 200
-timeout_ms = 600000
-item_ttl_ms = 120000
-```
-
-Configuration is set once when the daemon spawns and
-does not change until the daemon shuts down and a new
-one is spawned.
-
-### Client vs Daemon Timeouts
-
-The `KBDB_DAEMON_TIMEOUT` environment variable (or
-`--timeout-ms` CLI flag) controls the **client-side
-request timeout** -- how long a CLI or MCP client
-waits for a response from the daemon before giving
-up and optionally retrying.
-
-The `timeout_ms` setting in `worker.toml` controls
-the **daemon-side idle timeout** -- how long the
-daemon stays alive after all client connections have
-closed.
-
-These are independent. A long-running query may
-exceed the client request timeout while the daemon
-continues processing. Conversely, a short idle
-timeout shuts the daemon down quickly between
-invocations but has no effect on in-flight
-requests.
-
-## IPC Transport
-
-The daemon listens on a local IPC channel:
-
-- **Linux / macOS** -- Unix domain socket at
-  `{tmpdir}/kbdb-{ctx}.sock`
-  (e.g., `/tmp/kbdb-a1b2c3d4e5f67890.sock`).
-- **Windows** -- Named pipe at
-  `\\.\pipe\kbdb-{ctx}`.
-
-The IPC path is computed by the platform-specific
-`getIpcPath(ctx)` function.
-
-Both Deno and Node.js support Unix domain sockets and
-Windows named pipes natively:
-
-- **Node.js** -- `net.createServer()` with
-  `server.listen(path)` and
-  `net.createConnection(path)`.
-
-No third-party IPC library is required. The daemon
-creates the IPC socket at startup and removes it on
-shutdown.
-
-### Socket Options
-
-- **Backlog**: `256`. Handles connection bursts from
-  concurrent CLI invocations and benchmarks without
-  dropping connections.
-- **SO_KEEPALIVE**: Enabled on every accepted
-  connection with a 30-second interval. Detects dead
-  clients without relying solely on the idle timer.
-  Both the server and client enable keepalive (see
-  [Worker Client](worker-client.md)).
-
-## IPC Protocol
-
-Communication uses **JSON-RPC 2.0** over the IPC
-channel. Each request is a JSON-RPC call with a method
-name and params; each response is a JSON-RPC result or
-error.
-
-Messages are delimited by newlines. Each JSON-RPC
-message is a single line of JSON followed by `\n`. This
-provides simple framing over the stream-oriented IPC
-transport.
-
-Every request includes a `ctx` param -- the context ID.
-The daemon verifies the `ctx` matches its own context
-and returns a JSON-RPC error (`-32001`, "context
-mismatch") if it does not.
-
-## Method Routing
-
-The daemon dispatches each IPC method to one of three
-categories:
-
-1. **Daemon-state methods** -- handled directly in
-   TypeScript without touching WASM.
-2. **Config methods** (`config_{key}`) -- read from
-   `DaemonConfig` in TypeScript.
-3. **WASM methods** -- routed through `WASM_METHOD_MAP`
-   to a named `kb-worker.wasm` export.
-
-### WASM Method Map
-
-The daemon accepts both camelCase and snake_case aliases
-for each WASM operation:
-
-| IPC method(s)                           | WASM export              |
-|-----------------------------------------|--------------------------|
-| `search`, `query`                       | `worker_search`          |
-| `addSection`, `add_section`             | `worker_add_section`     |
-| `updateSection`, `update_section`       | `worker_update_section`  |
-| `removeSection`, `remove_section`       | `worker_remove_section`  |
-| `readSections`, `read_sections`         | `worker_read_sections`   |
-| `groupSections`, `group_sections`       | `worker_group_sections`  |
-| `removeGrouping`, `remove_grouping`     | `worker_remove_grouping` |
-| `retrieveDocument`, `retrieve_doc`      | `worker_retrieve_doc`    |
-| `content`, `compose`                    | `worker_compose`         |
-| `integrityCheck`, `integrity_check`     | `worker_check_integrity` |
-| `gc`                                    | `worker_garbage_collect` |
-| `rebuildIndexes`, `rebuild_indexes`     | `worker_rebuild_indexes` |
-| `initDirectory`, `init_directory`       | `worker_init_directory`  |
-| `migrateDatabase`                       | (not yet supported)      |
-| `query`                                 | `worker_query`           |
-| `get_result_status`                     | `worker_result_status`   |
-| `get_result`                            | `worker_get_result`      |
-| `invalidate`                            | `worker_invalidate`      |
-| `invalidateAll`, `invalidate_all`       | `worker_invalidate_all`  |
-| `dbStatus`, `db_status`                 | `worker_status`          |
-| `listByType`, `list_by_type`            | `worker_list_by_type`    |
-| `listDocumentsByType`                   | `worker_list_docs_by_type` |
-| `recall`                                | `worker_recall`          |
-| `export`                                | `db_export`              |
-
-`migrateDatabase` is listed in the map but throws
-`IpcError(-32601, 'migrateDatabase is not yet
-supported')` when called.
-
-## IPC API -- Daemon-State Methods
-
-These methods are handled in TypeScript without WASM
-calls.
-
-### `status`
-
-Returns the daemon's current state.
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "status",
-   "params": { "ctx": "<context-id>" },
-   "id": 1
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 1,
-   "result": {
-      "state": "busy",
-      "integrity_ok": true,
-      "integrity_errors": []
-   }
-}
-```
-
-Result fields:
-
-| Field              | Type     | Description                               |
-|--------------------|----------|-------------------------------------------|
-| `state`            | string   | `"busy"` if connections > 0, else `"idle"` |
-| `integrity_ok`     | boolean  | `true` if startup health check passed     |
-| `integrity_errors` | string[] | Error types from `db_check_integrity`;    |
-|                    |          | empty when `integrity_ok` is `true`       |
-
-### `idle_at`
-
-Returns the Unix timestamp (in seconds) when the last
-transaction completed.
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "idle_at",
-   "params": { "ctx": "<context-id>" },
-   "id": 2
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 2,
-   "result": 1750012800
-}
-```
-
-Result: integer (Unix epoch seconds). `0` if no
-transaction has completed since startup.
-
-### `config_{key}`
-
-Returns a configuration value. Method name is `config_`
-followed by the TOML key name.
-
-| Method               | DaemonConfig property | Returns                      |
-|----------------------|-----------------------|------------------------------|
-| `config_timeout_ms`  | `timeoutMs`           | Daemon idle timeout in ms    |
-| `config_item_ttl_ms` | `itemTtlMs`           | Per-item cache TTL in ms     |
-| `config_item_limit`  | `itemLimit`           | Max cached content objects   |
-| `config_index_limit` | `indexLimit`          | Max cached index segments    |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "config_timeout_ms",
-   "params": { "ctx": "<context-id>" },
-   "id": 3
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 3,
-   "result": 300000
-}
-```
-
-### `checkVersion`
-
-Reads `catalog.toml` from the given path (or the
-daemon's own context path) and returns a version status
-object. Does not require the daemon to be fully started.
-
-**Request params:**
-
-| Param        | Type   | Required | Description                       |
-|--------------|--------|----------|-----------------------------------|
-| `ctx`        | string | yes      | Context ID                        |
-| `targetPath` | string | no       | Path to read catalog from; defaults to `contextPath` |
-
-**Response (`VersionStatus`):**
-
-| Field             | Type   | Values                                       |
-|-------------------|--------|----------------------------------------------|
-| `status`          | string | `"compatible"` \| `"needs_migration"` \| `"too_new"` |
-| `currentVersion`  | number | Version found in catalog (0 if absent)       |
-| `requiredVersion` | number | `DB_FORMAT_VERSION` (currently `1`)          |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "checkVersion",
-   "params": { "ctx": "<context-id>" },
-   "id": 4
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 4,
-   "result": {
-      "status": "compatible",
-      "currentVersion": 1,
-      "requiredVersion": 1
-   }
-}
-```
-
-## IPC API -- WASM Read Methods
-
-### `search` / `query`
-
-Submits a full-text search query. The daemon encodes
-the params, calls `worker_search`, and decodes the
-result synchronously. Returns a `ScoredResult[]` array.
-
-**Request params:**
-
-| Param    | Type   | Required | Description                   |
-|----------|--------|----------|-------------------------------|
-| `ctx`    | string | yes      | Context ID                    |
-| `query`  | string | yes      | Query text                    |
-| `mode`   | string | no       | `"and"` (default) or `"or"`  |
-| `limit`  | number | no       | Max results (default 20)      |
-| `offset` | number | no       | Zero-based result offset (default 0) |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "search",
-   "params": {
-      "ctx": "<context-id>",
-      "query": "TypeScript architecture",
-      "mode": "and",
-      "limit": 10
-   },
-   "id": 5
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 5,
-   "result": {
-      "items": [
-         {
-            "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-            "heading": "Architecture Overview",
-            "type": "text/markdown",
-            "docids": ["doc-5678"],
-            "snippet": "The system is organized...",
-            "matched_terms": ["architecture", "system"],
-            "score": 8.7432
-         }
-      ],
-      "total": 42,
-      "offset": 0,
-      "limit": 20,
-      "has_more": true
-   }
-}
-```
-
-Note: the IPC layer keeps `score` (CLI uses it). The
-MCP layer strips it.
-
-Result: `PagedSearchResult` decoded from the WASM
-binary return slot.
-
-### `get_result_status`
-
-Checks the status of a previously submitted async
-operation by its numeric token.
-
-**Request params:**
-
-| Param   | Type   | Description            |
-|---------|--------|------------------------|
-| `ctx`   | string | Context ID             |
-| `token` | number | Token from prior call  |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "get_result_status",
-   "params": { "ctx": "<context-id>", "token": 42 },
-   "id": 6
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 6,
-   "result": { "status": 2 }
-}
-```
-
-Result: `{ status: number }` -- the raw status code
-from `worker_result_status`.
-
-### `get_result`
-
-Retrieves the result for a previously submitted async
-operation by its numeric token. Decodes the return slot
-as `ScoredResult[]`.
-
-**Request params:**
-
-| Param   | Type   | Description            |
-|---------|--------|------------------------|
-| `ctx`   | string | Context ID             |
-| `token` | number | Token from prior call  |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "get_result",
-   "params": { "ctx": "<context-id>", "token": 42 },
-   "id": 7
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 7,
-   "result": [
-      { "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3", "score": 0.95 }
-   ]
-}
-```
-
-### `readSections` / `read_sections`
-
-Reads multiple sections by kb-id. Returns an array of
-`SectionRecord` objects decoded from the WASM binary.
-
-**Request params:**
-
-| Param    | Type     | Description                    |
-|----------|----------|--------------------------------|
-| `ctx`    | string   | Context ID                     |
-| `kbids`  | string[] | List of section IDs to read    |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "readSections",
-   "params": {
-      "ctx": "<context-id>",
-      "kbids": [
-         "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-         "p4q5r6s7t8u9v0w1x2y3z4a5b6"
-      ]
-   },
-   "id": 8
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 8,
-   "result": [
-      {
-         "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-         "docids": ["doc-5678"],
-         "type": "text/markdown",
-         "title": "Architecture Overview",
-         "description": "High-level overview",
-         "content": "Section body text..."
-      }
-   ]
-}
-```
-
-### `recall`
-
-Progressive context expansion for one or more
-sections. Returns section content plus, depending
-on the `depth` parameter, parent documents,
-sibling sections, forward references, and
-back-references from the reference graph index.
-
-**Request params:**
-
-| Param   | Type     | Required | Description              |
-|---------|----------|----------|--------------------------|
-| `ctx`   | string   | yes      | Context ID               |
-| `kbid`  | string   | no*      | Single section ID        |
-| `kbids` | string[] | no*      | Batch section IDs        |
-| `depth` | number   | no       | Expansion depth (0--3, default 0) |
-
-*At least one of `kbid` or `kbids` required.
-
-The daemon calls `worker_recall` in the WASM
-engine, which reads sections, optionally looks up
-parent documents, sibling sections, forward
-references, and back-references from
-`references.idx` depending on the requested depth.
-
-Result: `RecallResult` (single) or
-`RecallResult[]` (batch) decoded from the WASM
-binary return slot.
-
-### `retrieveDocument` / `retrieve_doc`
-
-Retrieves a document manifest by document ID. Returns
-a decoded `DocumentManifest`.
-
-**Request params:**
-
-| Param   | Type   | Description        |
-|---------|--------|--------------------|
-| `ctx`   | string | Context ID         |
-| `docid` | string | Document ID        |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "retrieveDocument",
-   "params": {
-      "ctx": "<context-id>",
-      "docid": "doc-5678"
-   },
-   "id": 9
-}
-
-// Response (decoded DocumentManifest)
-{
-   "jsonrpc": "2.0", "id": 9,
-   "result": { ... }
-}
-```
-
-### `dbStatus` / `db_status`
-
-Returns database status information. The WASM module
-(`worker_status`) writes a key-value text block to the
-return slot; the daemon decodes it and returns it as an
-object.
-
-**Request params:** `ctx` only.
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "dbStatus",
-   "params": { "ctx": "<context-id>" },
-   "id": 10
-}
-
-// Response (decoded key-value text)
-{
-   "jsonrpc": "2.0", "id": 10,
-   "result": { "sections": "42", "documents": "5" }
-}
-```
-
-### `listByType` / `list_by_type`
-
-Returns all sections whose `sectionType` field
-matches the given type string. The daemon encodes the
-params, calls `worker_list_by_type`, and decodes the
-result as an array of section records.
-
-**Request params:**
-
-| Param         | Type   | Required | Description                       |
-|---------------|--------|----------|-----------------------------------|
-| `ctx`         | string | yes      | Context ID                        |
-| `sectionType` | string | yes      | Section type to filter by         |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "listByType",
-   "params": {
-      "ctx": "<context-id>",
-      "sectionType": "skill"
-   },
-   "id": 23
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 23,
-   "result": [
-      {
-         "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-         "docids": [],
-         "type": "skill",
-         "title": "summarize-document",
-         "description": "Summarize a document into
-            key points",
-         "content": "{\"name\":\"summarize-document\",...}"
-      }
-   ]
-}
-```
-
-Result: array of `SectionRecord` objects filtered by
-the requested `sectionType`. Returns an empty array
-when no sections match.
-
-### `listDocumentsByType`
-
-Returns all document manifests whose `type` field
-matches the given document type string. The daemon
-encodes the params, calls `worker_list_docs_by_type`,
-and decodes the result as an array of document
-manifests.
-
-**Request params:**
-
-| Param     | Type   | Required | Description                        |
-|-----------|--------|----------|------------------------------------|
-| `ctx`     | string | yes      | Context ID                         |
-| `docType` | string | yes      | Document type to filter by         |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "listDocumentsByType",
-   "params": {
-      "ctx": "<context-id>",
-      "docType": "agent"
-   },
-   "id": 24
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 24,
-   "result": [
-      {
-         "docid": "agent-code-reviewer",
-         "title": "Code Reviewer Agent",
-         "type": "agent",
-         "sections": [
-            { "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3", "position": 0 },
-            { "kbid": "p4q5r6s7t8u9v0w1x2y3z4a5b6", "position": 1 }
-         ]
-      }
-   ]
-}
-```
-
-Result: array of document manifest objects filtered
-by the requested `docType`. Returns an empty array
-when no documents match. Documents without a `type`
-field are regular documents and are not returned by
-this method.
-
-## IPC API -- WASM Write Methods
-
-### `addSection` / `add_section`
-
-Adds a new section to the database. The params are
-binary-encoded and passed to `worker_add_section`. The
-return slot contains the new section's kb-id as UTF-8.
-
-**Request params:**
-
-| Param         | Type   | Required | Description                                   |
-|---------------|--------|----------|-----------------------------------------------|
-| `ctx`         | string | yes      | Context ID                                    |
-| `sectionType` | string | yes      | Section type (`"text"`, `"image"`, `"code"`, etc.) |
-| `mimeType`    | string | no       | MIME type                                     |
-| `content`     | string | yes      | Section content                               |
-| `title`       | string | no       | Section title                                 |
-| `description` | string | no       | Section description                           |
-| `docid`       | string | no       | Document ID to attach this section to         |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "addSection",
-   "params": {
-      "ctx": "<context-id>",
-      "sectionType": "text",
-      "mimeType": "text/markdown",
-      "content": "# Architecture\n\nOverview...",
-      "title": "Architecture Overview",
-      "docid": "doc-5678"
-   },
-   "id": 11
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 11,
-   "result": { "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3" }
-}
-```
-
-### `updateSection` / `update_section`
-
-Updates an existing section. The params are
-binary-encoded and passed to `worker_update_section`.
-Returns the new kb-id (content hash may change).
-
-**Request params:**
-
-| Param         | Type   | Required | Description                            |
-|---------------|--------|----------|----------------------------------------|
-| `ctx`         | string | yes      | Context ID                             |
-| `kbid`        | string | yes      | Existing section ID to replace         |
-| `sectionType` | string | yes      | New section type                       |
-| `mimeType`    | string | no       | New MIME type                          |
-| `content`     | string | yes      | New content                            |
-| `title`       | string | no       | New title                              |
-| `description` | string | no       | New description                        |
-| `docid`       | string | no       | Document ID                            |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "updateSection",
-   "params": {
-      "ctx": "<context-id>",
-      "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-      "sectionType": "text",
-      "content": "Updated content...",
-      "title": "Updated Title"
-   },
-   "id": 12
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 12,
-   "result": { "kbid": "x9y8z7w6v5u4t3s2r1q0p9o8n7" }
-}
-```
-
-### `removeSection` / `remove_section`
-
-Removes a section by kb-id. The kb-id is UTF-8 encoded
-and passed to `worker_remove_section`. The return flag
-is decoded by `decodeRemoveFlag`.
-
-**Request params:**
-
-| Param   | Type   | Description             |
-|---------|--------|-------------------------|
-| `ctx`   | string | Context ID              |
-| `kbid`  | string | Section ID to remove    |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "removeSection",
-   "params": {
-      "ctx": "<context-id>",
-      "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3"
-   },
-   "id": 13
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 13,
-   "result": { "removed": true }
-}
-```
-
-### `groupSections` / `group_sections`
-
-Groups a set of sections into a named document. Params
-are binary-encoded and passed to `worker_group_sections`.
-Returns the new document ID.
-
-**Request params:**
-
-| Param   | Type     | Description                       |
-|---------|----------|-----------------------------------|
-| `ctx`   | string   | Context ID                        |
-| `title` | string   | Document title                    |
-| `kbids` | string[] | Section IDs to group              |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "groupSections",
-   "params": {
-      "ctx": "<context-id>",
-      "title": "Architecture Guide",
-      "kbids": [
-         "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-         "p4q5r6s7t8u9v0w1x2y3z4a5b6"
-      ]
-   },
-   "id": 14
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 14,
-   "result": { "docid": "doc-5678" }
-}
-```
-
-### `removeGrouping` / `remove_grouping`
-
-Removes a document grouping (the manifest), leaving
-the underlying sections intact. The document ID is
-UTF-8 encoded and passed to `worker_remove_grouping`.
-
-**Request params:**
-
-| Param   | Type   | Description          |
-|---------|--------|----------------------|
-| `ctx`   | string | Context ID           |
-| `docid` | string | Document ID to ungroup |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "removeGrouping",
-   "params": {
-      "ctx": "<context-id>",
-      "docid": "doc-5678"
-   },
-   "id": 15
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 15,
-   "result": { "ok": true }
-}
-```
-
-### `content` / `compose`
-
-Composes a rendered Markdown document from one or more
-kbid or docid identifiers. The identifiers are
-binary-encoded and passed to `worker_compose`. The
-return slot is decoded by `decodeComposeOutput`.
-
-**Request params:**
-
-| Param   | Type     | Description                            |
-|---------|----------|----------------------------------------|
-| `ctx`   | string   | Context ID                             |
-| `ids`   | string[] | kbids or docids to compose             |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "compose",
-   "params": {
-      "ctx": "<context-id>",
-      "ids": [
-         "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-         "doc-5678"
-      ]
-   },
-   "id": 16
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 16,
-   "result": {
-      "data": "# Architecture Guide\n\n...",
-      "total": 4096
-   }
-}
-```
-
-Result fields:
-
-| Field   | Type   | Description                                   |
-|---------|--------|-----------------------------------------------|
-| `data`  | string | Rendered Markdown string (`output.markdown`)  |
-| `total` | number | Total character count (`output.totalChars`)   |
-
-### `invalidate`
-
-Evicts a single cached entry by kb-id. The kb-id is
-UTF-8 encoded and passed to `worker_invalidate`.
-
-**Request params:**
-
-| Param   | Type   | Description            |
-|---------|--------|------------------------|
-| `ctx`   | string | Context ID             |
-| `kbid`  | string | Section ID to evict    |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "invalidate",
-   "params": {
-      "ctx": "<context-id>",
-      "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3"
-   },
-   "id": 17
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 17,
-   "result": { "ok": true }
-}
-```
-
-### `invalidateAll` / `invalidate_all`
-
-Evicts the entire WASM cache by calling
-`worker_invalidate_all` with no arguments.
-
-**Request params:** `ctx` only.
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "invalidateAll",
-   "params": { "ctx": "<context-id>" },
-   "id": 18
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 18,
-   "result": { "ok": true }
-}
-```
-
-### `initDirectory` / `init_directory`
-
-Initialises a new database directory. The path is
-UTF-8 encoded and passed to `worker_init_directory`.
-When `path` is omitted, the daemon uses its own
-`targetDir`.
-
-**Request params:**
-
-| Param   | Type   | Required | Description                                  |
-|---------|--------|----------|----------------------------------------------|
-| `ctx`   | string | yes      | Context ID                                   |
-| `path`  | string | no       | Path to initialise; defaults to `targetDir`  |
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "initDirectory",
-   "params": { "ctx": "<context-id>" },
-   "id": 19
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 19,
-   "result": { "ok": true }
-}
-```
-
-## IPC API -- Maintenance Methods
-
-Maintenance methods perform database-wide operations.
-They call WASM and return results directly.
-
-### `integrityCheck` / `integrity_check`
-
-Verifies the integrity of the database. Calls
-`worker_check_integrity` with no arguments. The return
-slot contains a key-value text block decoded by
-`decodeKeyValueText`.
-
-**Request params:** `ctx` only.
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "integrityCheck",
-   "params": { "ctx": "<context-id>" },
-   "id": 20
-}
-
-// Response (decoded key-value text as object)
-{
-   "jsonrpc": "2.0", "id": 20,
-   "result": { "ok": "true", "sections_checked": "42" }
-}
-```
-
-### `gc`
-
-Removes section files not referenced by any document
-manifest. Calls `worker_garbage_collect` with no
-arguments. The return slot contains a key-value text
-block decoded by `decodeKeyValueText`.
-
-**Request params:** `ctx` only.
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "gc",
-   "params": { "ctx": "<context-id>" },
-   "id": 21
-}
-
-// Response (decoded key-value text as object)
-{
-   "jsonrpc": "2.0", "id": 21,
-   "result": { "removed_sections": "3", "removed_bytes": "15360" }
-}
-```
-
-### `rebuildIndexes` / `rebuild_indexes`
-
-Reconstructs all index files from section files and
-document manifests on disk. Calls
-`worker_rebuild_indexes` with no arguments. Returns
-`{ ok: true }` on success.
-
-**Request params:** `ctx` only.
-
-```json
-// Request
-{
-   "jsonrpc": "2.0",
-   "method": "rebuildIndexes",
-   "params": { "ctx": "<context-id>" },
-   "id": 22
-}
-
-// Response
-{
-   "jsonrpc": "2.0", "id": 22,
-   "result": { "ok": true }
-}
-```
-
-## Error Responses
-
-JSON-RPC 2.0 error format for invalid requests:
-
-```json
-{
-   "jsonrpc": "2.0", "id": 11,
-   "error": {
-      "code": -32601,
-      "message": "Method not found: unknownMethod"
-   }
-}
-```
-
-Standard JSON-RPC 2.0 error codes apply. Application
-errors use codes in the `-32000` to `-32099` range:
-
-| Code     | Meaning                              |
-|----------|--------------------------------------|
-| `-32001` | Context mismatch                     |
-| `-32700` | Parse error (malformed JSON line)    |
-| `-32601` | Method not found                     |
-| `-32000` | WASM call error (message from WASM)  |
-
-Startup errors (`-32004` through `-32007` in previous
-versions of this document) are not returned over IPC
-because the daemon exits before creating the IPC
-socket. They are communicated via stderr and the crash
-log.
-
-## Source Files
-
-The worker daemon module lives in
-`src/shared/worker-daemon/` with the following
-structure:
-
-```
-src/shared/worker-daemon/
-   classes/
-      worker-daemon.class.ts       # WorkerDaemon class
-   functions/
-      create-host-imports.function.ts  # WASM host I/O imports
-      create-ipc-server.function.ts    # IPC socket server
-      load-wasm-modules.function.ts    # WASM compilation/instantiation
-      read-config.function.ts          # worker.toml parser
-      run-worker.function.ts           # entry-point logic
-      start-daemon.function.ts         # startup orchestration
-   constants/
-      daemon-defaults.constant.ts      # defaults + DB_FORMAT_VERSION
-   typings/
-      daemon-config.interface.ts       # DaemonConfig
-      ipc-handler.interface.ts         # IpcRequest, IpcResponse, IpcHandler
-      wasm-modules.interface.ts        # WasmModules
-   wasm/
-      kb-worker/
-         kbdb_worker_bg.wasm           # compiled WASM (gitignored)
-```
-
-The `run-worker.function.ts` implements the entry-point
-logic (argument validation and calling `startDaemon`).
-The `start-daemon.function.ts` orchestrates the full
-startup sequence: reads config, computes the context ID,
-constructs `WorkerDaemon`, checks preconditions, starts
-the daemon, and installs signal handlers.
-
-## Distribution
-
-The worker daemon (`src/worker.ts` and its compiled
-`kbdb-worker.js`) ships as part of the npm/JSR package.
-No per-platform native binaries are required.
-
-The package includes four WASM modules:
-
-- `kb-worker/kbdb_worker_bg.wasm` -- orchestration
-  module that imports query-parser and fs-database
-- `query-parser/query_parser_bg.wasm` -- tokenization
-  and query parsing
-- `fs-database/fs_database_bg.wasm` -- filesystem
-  storage engine
-- `default-embedding/kbdb_default_embedding_bg.wasm`
-  -- built-in TF-IDF embedding (standalone, no host
-  imports)
-
-All are compiled from Rust via `wasm-pack` and work
-on all platforms. The daemon loads and compiles them
-at startup using the runtime's `WebAssembly` API. The
-daemon runs on the same runtime (Node.js or Deno) as
-the worker client that spawns it.