@dikolab/kbdb 0.1.6 → 0.2.0

package/docs/modules/typescript/shared/worker-daemon/classes.md

@@ -0,0 +1,239 @@
+# worker-daemon/classes -- WorkerDaemon
+
+*Core daemon class that loads WASM modules, manages
+the IPC server, routes requests, and handles lifecycle
+(startup, idle tracking, graceful shutdown).*
+
+## Source
+
+`src/shared/worker-daemon/classes/`
+
+## Classes
+
+### `WorkerDaemon`
+
+**File:** `worker-daemon.class.ts`
+
+The main daemon class. Orchestrates WASM module
+loading, IPC server creation, request routing, idle
+timeout tracking, and graceful shutdown.
+
+#### Constructor
+
+```ts
+class WorkerDaemon {
+   constructor(
+      ctx: string,
+      targetDir: string,
+      config: DaemonConfig,
+      wasmDir?: string,
+   );
+}
+```
+
+| Parameter   | Type           | Description             |
+|-------------|----------------|-------------------------|
+| `ctx`       | `string`       | Context ID (16-hex)     |
+| `targetDir` | `string`       | Absolute path to the    |
+|             |                | `.kbdb` directory       |
+| `config`    | `DaemonConfig` | Parsed config from      |
+|             |                | `worker.toml`           |
+| `wasmDir`   | `string?`      | Path to WASM build      |
+|             |                | output directory        |
+
+#### Properties
+
+| Property          | Type           | Description            |
+|-------------------|----------------|------------------------|
+| `ctx`             | `string`       | Context ID             |
+| `targetDir`       | `string`       | Database directory     |
+| `config`          | `DaemonConfig` | Daemon config          |
+| `connectionCount` | `number`       | Active IPC connections |
+
+#### Public Methods
+
+##### `checkPreconditions`
+
+```ts
+checkPreconditions(): void;
+```
+
+Validates startup preconditions before WASM module
+loading. Checks that:
+
+1. Database directory exists.
+2. Directory is readable and writable.
+3. `catalog.toml` reports a compatible version.
+
+Throws on any failure. `startDaemon` catches the
+error, prints to stderr, and exits with code 1.
+
+##### `start`
+
+```ts
+async start(): Promise<void>;
+```
+
+Startup sequence:
+
+1. Calls `loadWasmModules(config, wasmDir)` to
+   compile and instantiate the WASM modules.
+2. Initializes the worker context.
+3. Writes the PID file (`kbdb-{ctx}.pid`) to the
+   OS temporary directory.
+4. Calls `createIpcServer(ipcPath, handler)` to
+   bind the IPC socket.
+5. Begins listening for connections and resets the
+   idle timer.
+
+##### `stop`
+
+```ts
+stop(): void;
+```
+
+Graceful shutdown sequence:
+
+1. Clears the idle timer.
+2. Stops accepting new connections.
+3. Purges WASM module references.
+4. Removes the PID file.
+5. Removes the IPC socket file.
+6. Calls `process.exit(0)`.
+
+##### `handleRequest`
+
+```ts
+async handleRequest(
+   request: IpcRequest,
+): Promise<IpcResponse>;
+```
+
+Handles a single JSON-RPC 2.0 request. Validates
+context, resets the idle timer, routes to the
+appropriate handler, and returns a response.
+Requests whose `ctx` param does not match the
+daemon's context ID are rejected with error
+code `-32001`.
+
+##### `incrementConnections` / `decrementConnections`
+
+```ts
+incrementConnections(): void;
+decrementConnections(): void;
+```
+
+Adjust the active connection count. Called by the
+IPC server on connect/disconnect events.
+
+##### `injectModulesForTesting`
+
+```ts
+injectModulesForTesting(modules: WasmModules): void;
+```
+
+Injects pre-built WASM modules for unit testing.
+Must not be called in production code.
+
+#### Private Behavior
+
+##### Idle Timeout Tracking
+
+The daemon maintains an internal idle timer. The
+timer resets on every:
+
+- Client connection.
+- Client disconnection.
+- IPC request received.
+
+When the timer reaches `config.timeoutMs` with no
+active connections, the daemon calls `stop()`.
+
+##### Connection Counting
+
+Tracks the number of active client connections. The
+daemon stays alive regardless of the idle timer as
+long as at least one connection is open.
+
+##### Startup Preconditions
+
+Before WASM modules are loaded, the daemon checks
+three preconditions. Failure causes `startDaemon` to
+print the error to stderr and exit with code 1:
+
+1. Database directory exists.
+2. Directory is readable and writable.
+3. `catalog.toml` version is compatible.
+
+##### Request Routing
+
+Routes incoming JSON-RPC 2.0 requests. Daemon-state
+methods are handled inline; all other methods are
+looked up in `WASM_METHOD_MAP`:
+
+| IPC Method           | Handler / WASM Export          |
+|----------------------|--------------------------------|
+| `status`             | daemon state (busy/idle)       |
+| `idle_at`            | last idle Unix timestamp       |
+| `config_{key}`       | `DaemonConfig` property lookup |
+| `checkVersion`       | catalog version check          |
+| `search`             | `worker_search`                |
+| `recall`             | `worker_recall`                |
+| `addSection`         | `worker_add_section`           |
+| `updateSection`      | `worker_update_section`        |
+| `removeSection`      | `worker_remove_section`        |
+| `readSections`       | `worker_read_sections`         |
+| `groupSections`      | `worker_group_sections`        |
+| `removeGrouping`     | `worker_remove_grouping`       |
+| `retrieveDocument`   | `worker_retrieve_doc`          |
+| `content`            | `worker_compose`               |
+| `integrityCheck`     | `worker_check_integrity`       |
+| `gc`                 | `worker_garbage_collect`       |
+| `rebuildIndexes`     | `worker_rebuild_indexes`       |
+| `initDirectory`      | `worker_init_directory`        |
+| `dbStatus`           | `worker_status`                |
+| `invalidate`         | `worker_invalidate`            |
+| `invalidateAll`      | `worker_invalidate_all`        |
+| `get_result_status`  | `worker_result_status`         |
+| `get_result`         | `worker_get_result`            |
+| `query`              | `worker_query`                 |
+| `listByType` / `list_by_type` | `worker_list_by_type` |
+| `listDocumentsByType` | `worker_list_docs_by_type`   |
+
+Snake_case aliases (`add_section`, `remove_section`,
+etc.) are also accepted and map to the same exports.
+
+##### Context Validation
+
+Every IPC request includes a `ctx` param. The daemon
+validates it against its own context ID before
+processing. Mismatches are rejected with JSON-RPC
+error code `-32001` (context mismatch).
+
+##### `handleListByType(params)`
+
+Handles `listByType` / `list_by_type` IPC requests.
+Receives `{ ctx, sectionType }` and delegates to
+`worker_list_by_type` in the WASM engine, which
+scans the sections directory and returns all records
+whose `type` field matches `sectionType`.
+
+Returns: array of section records filtered by type.
+
+##### `handleListDocumentsByType(params)`
+
+Handles `listDocumentsByType` IPC requests.
+Receives `{ ctx, docType }` and delegates to
+`worker_list_docs_by_type` in the WASM engine,
+which scans the documents directory and returns all
+manifests whose `type` field matches `docType`.
+
+Returns: array of document manifests filtered by
+type.
+
+## Testing
+
+```
+src/shared/worker-daemon/
+└── worker-daemon.spec.ts
+```

package/docs/modules/typescript/shared/worker-daemon/constants.md

@@ -0,0 +1,37 @@
+# worker-daemon/constants -- Daemon Defaults
+
+*Default values for daemon configuration when
+`worker.toml` is missing or incomplete.*
+
+## Source
+
+`src/shared/worker-daemon/constants/`
+
+### `daemon-defaults`
+
+**File:** `daemon-defaults.constant.ts`
+
+```ts
+const DEFAULT_ITEM_LIMIT = 1000;
+const DEFAULT_INDEX_LIMIT = 100;
+const DEFAULT_TIMEOUT_MS = 300_000;
+const DEFAULT_ITEM_TTL_MS = 60_000;
+```
+
+| Constant              | Value     | Description    |
+|-----------------------|-----------|----------------|
+| `DEFAULT_ITEM_LIMIT`  | `1000`    | Max content    |
+|                       |           | objects held   |
+|                       |           | in cache       |
+| `DEFAULT_INDEX_LIMIT` | `100`     | Max index      |
+|                       |           | segments held  |
+|                       |           | in cache       |
+| `DEFAULT_TIMEOUT_MS`  | `300000`  | Daemon idle    |
+|                       |           | timeout in ms  |
+|                       |           | (5 minutes)    |
+| `DEFAULT_ITEM_TTL_MS` | `60000`   | Per-item cache |
+|                       |           | TTL in ms      |
+|                       |           | (1 minute)     |
+
+These constants are used by `readConfig()` to fill
+missing keys when parsing `worker.toml`.

package/docs/modules/typescript/shared/worker-daemon/functions.md

@@ -0,0 +1,221 @@
+# worker-daemon/functions -- Daemon Functions
+
+*Worker entry point, daemon entry point, host I/O
+imports, WASM module loading, IPC server creation,
+and configuration reading.*
+
+## Source
+
+`src/shared/worker-daemon/functions/`
+
+## Functions
+
+### `runWorker`
+
+**File:** `run-worker.function.ts`
+
+Worker daemon entry point extracted from
+`src/worker.ts`. Validates the context path argument,
+resolves it to an absolute path, checks existence, and
+delegates to `startDaemon`.
+
+```ts
+async function runWorker(
+   argv: string[],
+   deps: WorkerDeps,
+): Promise<WorkerResult>;
+```
+
+```ts
+interface WorkerDeps {
+   existsSync: (path: string) => boolean;
+   startDaemon: (
+      contextPath: string,
+      wasmDir?: string,
+   ) => Promise<void>;
+   wasmDir?: string;
+}
+
+interface WorkerResult {
+   error?: string;
+   exitCode: number;
+}
+```
+
+Returns a `WorkerResult` without touching process
+globals, making it fully testable.
+
+### `startDaemon`
+
+**File:** `start-daemon.function.ts`
+
+Entry point for the daemon process. Called from
+`runWorker` after path validation.
+
+```ts
+async function startDaemon(
+   contextPath: string,
+   wasmDir?: string,
+): Promise<void>;
+```
+
+1. Calls `readConfig(contextPath)` to load
+   `worker.toml`.
+2. Computes the context ID from `contextPath`
+   (SHA-256 prefix, 16 hex characters).
+3. Creates a new `WorkerDaemon(ctx, contextPath,
+   config, wasmDir)`.
+4. Calls `daemon.checkPreconditions()`. Failures
+   are written to stderr and a crash log, then
+   `process.exit(1)` is called.
+5. Calls `daemon.start()`.
+6. Registers OS signal handlers (`SIGTERM`,
+   `SIGINT`) that call `daemon.stop()`.
+
+### `createHostImports`
+
+**File:** `create-host-imports.function.ts`
+
+Creates the host I/O import functions required by
+`kb-worker.wasm`. All functions access WASM linear
+memory through a deferred `MemoryRef` that is wired
+up after module instantiation.
+
+```ts
+function createHostImports(
+   contextPath: string,
+   memoryRef?: MemoryRef,
+   fs?: FsAdapter,
+   allocator?: AllocatorRef,
+): Record<string, WebAssembly.ImportValue>;
+```
+
+| Parameter     | Description                        |
+|---------------|------------------------------------|
+| `contextPath` | Absolute path to `.kbdb` directory |
+| `memoryRef`   | Deferred WASM memory reference     |
+| `fs`          | Filesystem adapter (default: real  |
+|               | `node:fs` operations)              |
+| `allocator`   | Deferred WASM allocator reference  |
+
+**Host imports provided:**
+
+| Import name       | Description                     |
+|-------------------|---------------------------------|
+| `host_read_file`  | Read a file into WASM memory    |
+| `host_write_file` | Write WASM memory to a file     |
+| `host_rename`     | Rename a file                   |
+| `host_remove_file`| Delete a file                   |
+| `host_list_dir`   | List directory entries          |
+| `host_mkdir`      | Create a directory (recursive)  |
+| `host_file_exists`| Check if a file exists          |
+
+All paths received from WASM are resolved relative
+to `contextPath`.
+
+**Supporting interfaces:**
+
+```ts
+interface FsAdapter {
+   readFileSync(path: string): Buffer;
+   writeFileSync(path: string, data: Uint8Array): void;
+   renameSync(oldPath: string, newPath: string): void;
+   unlinkSync(path: string): void;
+   readdirSync(path: string): string[];
+   mkdirSync(path: string): void;
+   existsSync(path: string): boolean;
+}
+
+interface MemoryRef {
+   memory: WebAssembly.Memory | null;
+}
+
+interface AllocatorRef {
+   alloc: ((size: number) => number) | null;
+}
+```
+
+### `loadWasmModules`
+
+**File:** `load-wasm-modules.function.ts`
+
+Compiles and instantiates the `kb-worker.wasm` module
+along with its host imports.
+
+```ts
+async function loadWasmModules(
+   config: DaemonConfig,
+   wasmDir?: string,
+): Promise<WasmModules>;
+```
+
+Loading sequence:
+
+1. Builds `createHostImports` with a deferred
+   `MemoryRef` and `AllocatorRef`.
+2. Compiles and instantiates `kb-worker.wasm`,
+   passing host imports.
+3. Wires the exported `memory` and `kbdb_alloc`
+   function back into the deferred refs.
+
+Returns a `WasmModules` object containing the
+instantiated module and its exported memory.
+
+### `createIpcServer`
+
+**File:** `create-ipc-server.function.ts`
+
+Creates a local IPC server bound to the given socket
+path and returns the `net.Server` instance.
+
+```ts
+function createIpcServer(
+   ipcPath: string,
+   handler: IpcHandler,
+): net.Server;
+```
+
+Platform behavior:
+
+- **Linux / macOS** -- Unix domain socket.
+- **Windows** -- Named pipe.
+
+Messages are newline-delimited JSON-RPC 2.0. Each
+message is a single line of JSON followed by `\n`.
+
+Incoming requests are parsed and forwarded to
+`handler`. Responses are serialized and written back
+to the connection.
+
+### `readConfig`
+
+**File:** `read-config.function.ts`
+
+Reads daemon configuration from `worker.toml` in
+the `.kbdb` directory.
+
+```ts
+function readConfig(
+   contextPath: string,
+): DaemonConfig;
+```
+
+Reads `${contextPath}/worker.toml`. Missing keys
+are filled with defaults from
+[daemon-defaults.constant.ts](constants.md). If the
+file does not exist, returns a `DaemonConfig` with
+all defaults.
+
+## Testing
+
+Each function has a `.spec.ts` at the module root:
+
+```
+src/shared/worker-daemon/
+├── run-worker.spec.ts
+├── start-daemon.spec.ts
+├── create-host-imports.spec.ts
+├── load-wasm-modules.spec.ts
+├── create-ipc-server.spec.ts
+└── read-config.spec.ts
+```

package/docs/modules/typescript/shared/worker-daemon/module.md

@@ -0,0 +1,112 @@
+# worker-daemon -- Daemon Process Module
+
+*Background process that loads WASM modules once,
+listens on IPC, serves JSON-RPC 2.0 requests, manages
+cache + context + idle timeout. Spawned on demand by
+worker-client, shuts down on idle timeout or OS
+signal.*
+
+## Source
+
+`src/shared/worker-daemon/`
+
+## Responsibility
+
+The worker-daemon module implements a detached
+background process that:
+
+- Loads and compiles the three WASM modules once at
+  startup (`query-parser.wasm`, `fs-database.wasm`,
+  `kb-worker.wasm`).
+- Reads configuration from `worker.toml` in the
+  context directory.
+- Listens on a local IPC socket (Unix domain socket
+  or Windows named pipe) for JSON-RPC 2.0 requests.
+- Maintains an in-memory LRU cache with TTL expiry
+  for indexes, content objects, and manifests.
+- Tracks active connections and resets the idle timer
+  on every connect, disconnect, or request.
+- Shuts down gracefully on idle timeout or OS signal.
+
+The daemon is **not** a persistent server. It is a
+short-lived process spawned by
+[worker-client](../worker-client/module.md) that
+exists only to amortize WASM compilation cost and
+share cached data across connections.
+
+See [Worker Daemon](../../../../goals/worker-daemon.md)
+for the full behavioral specification.
+
+## Directory Layout
+
+```
+src/shared/worker-daemon/
+├── index.ts
+├── classes/
+│   └── worker-daemon.class.ts
+├── functions/
+│   ├── create-host-imports.function.ts
+│   ├── create-ipc-server.function.ts
+│   ├── load-wasm-modules.function.ts
+│   ├── read-config.function.ts
+│   ├── run-worker.function.ts
+│   └── start-daemon.function.ts
+├── typings/
+│   ├── daemon-config.interface.ts
+│   ├── ipc-handler.interface.ts
+│   └── wasm-modules.interface.ts
+├── constants/
+│   └── daemon-defaults.constant.ts
+└── *.spec.ts
+```
+
+## Exports
+
+- **[classes.md](classes.md)** -- `WorkerDaemon`.
+- **[functions.md](functions.md)** -- `startDaemon`,
+  `runWorker`, `createHostImports`, `loadWasmModules`,
+  `createIpcServer`, `readConfig`.
+- **[typings.md](typings.md)** -- `DaemonConfig`,
+  `IpcHandler`, `IpcRequest`, `IpcResponse`,
+  `IpcErrorDetail`, `WasmModules`.
+- **[constants.md](constants.md)** --
+  `DEFAULT_ITEM_LIMIT`, `DEFAULT_INDEX_LIMIT`,
+  `DEFAULT_TIMEOUT_MS`, `DEFAULT_ITEM_TTL_MS`.
+
+## Startup Sequence
+
+```
+src/worker.ts (entry point)
+  │
+  ├─ startDaemon(contextPath)
+  │    ├─ readConfig(contextPath)
+  │    │    └─ reads worker.toml from .kbdb dir
+  │    ├─ new WorkerDaemon(ctx, config)
+  │    └─ daemon.start()
+  │         ├─ loadWasmModules(config)
+  │         │    ├─ allocate shared WebAssembly.Memory
+  │         │    ├─ compile query-parser.wasm ─┐
+  │         │    ├─ compile fs-database.wasm ──┤
+  │         │    │  (parallel, standalone,     │
+  │         │    │   shared memory)            │
+  │         │    └─ compile kb-worker.wasm ────┘
+  │         │       (imports both modules)
+  │         ├─ init cache
+  │         ├─ write PID file
+  │         │    └─ kb-worker-{ctx}.pid
+  │         ├─ createIpcServer(ctx, handler)
+  │         │    └─ kb-worker-{ctx}.sock
+  │         └─ listen for connections
+  └─ daemon running
+```
+
+## Shutdown Sequence
+
+On idle timeout or OS signal (`SIGTERM`, `SIGINT`):
+
+1. Stop accepting new connections.
+2. Wait for in-flight requests to complete.
+3. Purge all cached data.
+4. Remove PID file from OS temporary directory.
+5. Remove IPC socket file (Unix only).
+6. Exit process.