@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/modules/rust/shared/error/types.md

@@ -0,0 +1,302 @@
+# types::error -- Error Types
+
+*Hierarchical error enums shared across all three
+WASM modules, mapping to JSON-RPC 2.0 error codes at
+the IPC boundary.*
+
+## File
+
+`engine/crates/shared/src/error/error.rs`
+
+## Overview
+
+Errors originate in the standalone modules
+(`fs-database`, `query-parser`) and propagate upward
+through `kb-worker` to the TypeScript daemon, which
+translates them into JSON-RPC 2.0 error responses.
+Defining all error types in `kbdb-shared` ensures a
+single hierarchy with consistent `From`
+implementations across crate boundaries.
+
+See [Worker Daemon -- Error Responses](
+../../../../../goals/worker-daemon.md) for the JSON-RPC
+2.0 error codes at the IPC boundary.
+
+## Error Hierarchy
+
+```
+WorkerError           (kb-worker.wasm -> TypeScript)
+├── Fs(FsError)       (from fs-database.wasm)
+├── Index(IndexError) (from fs-database.wasm)
+├── Parse(ParseError) (from query-parser.wasm)
+├── CacheFull
+├── InvalidToken(String)
+└── ContextMismatch { expected, actual }
+```
+
+`WorkerError` is the top-level error returned by
+`kb-worker.wasm` exports. It wraps lower-level
+errors from the standalone modules.
+
+## Type Definitions
+
+### `FsError`
+
+```rust
+#[derive(Debug)]
+pub enum FsError {
+   NotFound(String),
+   PermissionDenied(String),
+   IoError(String),
+   ParseError(String),
+   ChecksumMismatch {
+      expected: String,
+      actual: String,
+   },
+}
+```
+
+Errors from `fs-database.wasm` file operations.
+
+| Variant            | Cause                          |
+|--------------------|--------------------------------|
+| `NotFound`         | Section, document, or file path does not exist |
+| `PermissionDenied` | Host I/O import returned permission error |
+| `IoError`          | Host I/O import returned generic error |
+| `ParseError`       | TOML header or catalog parsing failed |
+| `ChecksumMismatch` | Section file content does not match its stored checksum |
+
+`NotFound` carries the path or kb-id that was
+requested. `ChecksumMismatch` carries both the
+expected and actual checksum strings for diagnostics.
+
+### `IndexError`
+
+```rust
+#[derive(Debug)]
+pub enum IndexError {
+   Corrupted(String),
+   KeyNotFound(String),
+}
+```
+
+Errors from `fs-database.wasm` index operations.
+
+| Variant      | Cause                             |
+|--------------|-----------------------------------|
+| `Corrupted`  | B+ tree structure is invalid, page size mismatch, or binary data is unreadable |
+| `KeyNotFound`| Term not found in the inverted index (not necessarily an error -- a query for a term with no matches) |
+
+`Corrupted` indicates the index files need a
+rebuild. The daemon can recover by calling
+`worker_rebuild_indexes`.
+
+### `ParseError`
+
+```rust
+#[derive(Debug)]
+pub enum ParseError {
+   InvalidSyntax(String),
+   EmptyQuery,
+   UnmatchedQuote(usize),
+}
+```
+
+Errors from `query-parser.wasm` query parsing.
+
+| Variant          | Cause                          |
+|------------------|--------------------------------|
+| `InvalidSyntax`  | Unrecognized query operator or malformed expression |
+| `EmptyQuery`     | Query string is empty or contains only stop words |
+| `UnmatchedQuote` | Opening `"` without a closing `"` -- carries the character position |
+
+### `WorkerError`
+
+```rust
+#[derive(Debug)]
+pub enum WorkerError {
+   Fs(FsError),
+   Index(IndexError),
+   Parse(ParseError),
+   CacheFull,
+   InvalidToken(String),
+   ContextMismatch {
+      expected: String,
+      actual: String,
+   },
+}
+```
+
+Top-level error returned by `kb-worker.wasm` exports.
+
+| Variant          | Cause                          |
+|------------------|--------------------------------|
+| `Fs(FsError)`    | Storage operation failed       |
+| `Index(IndexError)` | Index operation failed      |
+| `Parse(ParseError)` | Query parsing failed        |
+| `CacheFull`      | Cache limit reached and eviction failed (should not happen with LRU, but defensive) |
+| `InvalidToken`   | Result token does not exist or has expired |
+| `ContextMismatch`| Request's `ctx` param does not match the daemon's context |
+
+## `From` Implementations
+
+```rust
+impl From<FsError> for WorkerError {
+   fn from(e: FsError) -> Self {
+      WorkerError::Fs(e)
+   }
+}
+
+impl From<IndexError> for WorkerError {
+   fn from(e: IndexError) -> Self {
+      WorkerError::Index(e)
+   }
+}
+
+impl From<ParseError> for WorkerError {
+   fn from(e: ParseError) -> Self {
+      WorkerError::Parse(e)
+   }
+}
+```
+
+These conversions allow the `?` operator to propagate
+errors naturally across module boundaries:
+
+```rust
+fn search(...) -> Result<SearchResult, WorkerError> {
+   let plan = parse_query(query)?;   // ParseError -> WorkerError
+   let raw = execute_plan(&plan)?;   // IndexError -> WorkerError
+   let section = read_section(id)?;  // FsError -> WorkerError
+   // ...
+}
+```
+
+## JSON-RPC 2.0 Mapping
+
+The TypeScript daemon maps `WorkerError` variants to
+JSON-RPC 2.0 error codes:
+
+| WorkerError variant | JSON-RPC code | JSON-RPC message     |
+|---------------------|---------------|----------------------|
+| `Fs(NotFound)`      | `-32000`      | `"Not found"`        |
+| `Fs(PermissionDenied)` | `-32000`   | `"Permission denied"`|
+| `Fs(IoError)`       | `-32000`      | `"I/O error"`        |
+| `Fs(ParseError)`    | `-32000`      | `"Parse error"`      |
+| `Fs(ChecksumMismatch)` | `-32000`  | `"Checksum mismatch"`|
+| `Index(Corrupted)`  | `-32000`      | `"Index corrupted"`  |
+| `Index(KeyNotFound)`| `-32000`      | `"Key not found"`    |
+| `Parse(InvalidSyntax)` | `-32000`  | `"Invalid query syntax"` |
+| `Parse(EmptyQuery)` | `-32000`      | `"Empty query"`      |
+| `Parse(UnmatchedQuote)` | `-32000` | `"Unmatched quote"`  |
+| `CacheFull`         | `-32000`      | `"Cache full"`       |
+| `InvalidToken`      | `-32002`      | `"Invalid token"`    |
+| `ContextMismatch`   | `-32001`      | `"Context mismatch"` |
+
+Codes `-32001` (context mismatch) and `-32002`
+(invalid token) are application-specific codes
+defined in the [Worker Daemon](
+../../../../../goals/worker-daemon.md) specification.
+All other errors use the generic application error
+code `-32000`.
+
+## Error Relay -- WASM to Host
+
+WASM errors must reach the TypeScript process's
+stderr before the WASM instance is cleaned up. The
+relay sequence:
+
+1. A WASM export (`worker_init`, `worker_search`,
+   etc.) encounters an error and returns a
+   `WorkerError` via the shared memory return
+   convention (`set_return(ptr, len)` with the
+   `Encode`-serialized error).
+2. The TypeScript daemon **reads the error from
+   shared memory immediately** -- before any cleanup,
+   cache purge, or WASM teardown.
+3. The daemon deserializes the `WorkerError`, maps it
+   to the appropriate JSON-RPC error code and
+   human-readable message.
+4. **For IPC-connected errors:** The daemon sends the
+   JSON-RPC error response to the client over IPC,
+   then resumes normal operation (no teardown).
+5. **For fatal startup errors** (precondition failures
+   during `worker_init`): The daemon writes the error
+   message to **stderr**, then proceeds with shutdown
+   (purge cache -> remove PID file -> remove socket
+   -> exit). The WASM modules remain instantiated
+   until the error is fully read and relayed.
+
+**Key invariant:** the TypeScript host always reads
+and relays the WASM error before the WASM instance
+is purged. No error is lost to premature cleanup.
+
+## Daemon-Level Error Codes
+
+Beyond the WASM-origin error codes (`-32000` through
+`-32002`), the daemon defines additional error codes
+for startup and client-side failures:
+
+| Code     | Meaning                              | Origin | Relay path |
+|----------|--------------------------------------|--------|------------|
+| `-32001` | Context mismatch                     | WASM (`WorkerError::ContextMismatch`) | IPC response |
+| `-32002` | Invalid token                        | WASM (`WorkerError::InvalidToken`) | IPC response |
+| `-32003` | Daemon unavailable                   | Worker client (spawn failure) | `WorkerClientError` thrown to caller |
+| `-32004` | Database directory not found         | WASM `worker_init` or TS precondition | stderr -> `WorkerClientError.daemonStderr` |
+| `-32005` | Database directory not accessible    | WASM `worker_init` or TS precondition | stderr -> `WorkerClientError.daemonStderr` |
+| `-32006` | Database version too new             | WASM `worker_init` | stderr -> `WorkerClientError.daemonStderr` |
+| `-32007` | Database version too old (needs migration) | WASM `worker_init` | stderr -> `WorkerClientError.daemonStderr` |
+
+Codes `-32004` through `-32007` are startup errors.
+They are never returned over IPC because the daemon
+exits before creating the socket. The daemon writes
+the error to stderr; the worker client captures
+stderr during the spawn window and includes it in
+`WorkerClientError.daemonStderr`. This relay ensures
+the user sees the specific failure reason even though
+the daemon runs as a separate detached process.
+
+## `Display` and `Error` Implementations
+
+All error types implement `std::fmt::Display` and
+`std::error::Error`:
+
+```rust
+impl fmt::Display for FsError {
+   fn fmt(&self, f: &mut fmt::Formatter<'_>)
+      -> fmt::Result
+   {
+      match self {
+         Self::NotFound(p) =>
+            write!(f, "not found: {p}"),
+         Self::PermissionDenied(p) =>
+            write!(f, "permission denied: {p}"),
+         Self::IoError(msg) =>
+            write!(f, "I/O error: {msg}"),
+         Self::ParseError(msg) =>
+            write!(f, "parse error: {msg}"),
+         Self::ChecksumMismatch { expected, actual } =>
+            write!(
+               f,
+               "checksum mismatch: expected {expected}, got {actual}",
+            ),
+      }
+   }
+}
+```
+
+`WorkerError::source()` returns the wrapped inner
+error for `Fs`, `Index`, and `Parse` variants,
+enabling error chain inspection.
+
+## Testing
+
+Unit tests:
+
+- `From` conversions: each error type converts to
+  the correct `WorkerError` variant.
+- `Display` formatting: each variant produces a
+  human-readable message.
+- `source()` chain: `WorkerError::Fs(e).source()`
+  returns `Some(&e)`.
+- All variants are `Debug`-printable without panic.

package/docs/modules/rust/shared/kbid/module.md

@@ -0,0 +1,24 @@
+# kbid -- Content-Addressed Identifier
+
+*Content-addressed 26-character section identifier
+derived from SHA-256 hashing.*
+
+## Source
+
+`engine/crates/shared/src/kbid/`
+
+## Responsibility
+
+Defines the `KbId` newtype and its construction
+methods. Every section in the knowledge base is
+identified by a deterministic, collision-resistant,
+URL-safe string derived from the section's content.
+
+See [Database -- Sections](../../../../goals/database.md)
+for the behavioral specification of content-
+addressable storage.
+
+## Exports
+
+- **[types.md](types.md)** -- `KbId` newtype,
+  `from_content`, `new`, `shard_prefix`, `KbIdError`.

package/docs/modules/rust/shared/kbid/types.md

@@ -0,0 +1,147 @@
+# types::kbid -- Knowledge Base Section Identifier
+
+*Newtype wrapper around the 26-character,
+content-addressed section identifier used throughout
+the system.*
+
+## File
+
+`engine/crates/shared/src/kbid/kbid.rs`
+
+## Overview
+
+Every section in the knowledge base is identified by
+a **kb-id** -- a deterministic, collision-resistant,
+URL-safe string derived from the SHA-256 hash of the
+section's content. The `KbId` type encodes these
+invariants at the type level so that raw strings
+cannot be passed where a validated identifier is
+expected.
+
+See [Database -- Sections](../../../../../goals/database.md)
+for the behavioral specification of content-
+addressable storage.
+
+## Type Definition
+
+```rust
+#[derive(Clone, Debug, PartialEq, Eq, Hash)]
+pub struct KbId(String);
+```
+
+The inner `String` is private. Construction goes
+through `KbId::new` or `KbId::from_content`, both of
+which validate or derive the identifier. This
+prevents accidental construction from arbitrary
+strings.
+
+## Construction
+
+### `KbId::from_content`
+
+```rust
+impl KbId {
+   pub fn from_content(content: &[u8]) -> Self;
+}
+```
+
+Derives the kb-id from raw content bytes:
+
+1. Compute SHA-256 of `content`.
+2. Truncate the digest to 16 bytes (128 bits).
+3. Encode with base32 (Crockford variant, lowercase).
+4. Result: 26 alphanumeric characters.
+
+For text and code sections, `content` is the raw
+UTF-8 bytes. For image sections, `content` is the
+base64-encoded string bytes (not raw binary), so
+identical base64 input always produces the same
+kb-id.
+
+This function is infallible -- any byte slice
+produces a valid kb-id.
+
+### `KbId::new`
+
+```rust
+impl KbId {
+   pub fn new(s: &str) -> Result<Self, KbIdError>;
+}
+```
+
+Validates an existing identifier string:
+
+- Must be exactly 26 characters.
+- Must contain only lowercase alphanumeric characters
+  (`[a-z0-9]`).
+
+Returns `KbIdError::InvalidLength` or
+`KbIdError::InvalidCharacter` on failure.
+
+## Methods
+
+```rust
+impl KbId {
+   pub fn as_str(&self) -> &str;
+
+   pub fn shard_prefix(&self) -> &str;
+}
+```
+
+### `as_str`
+
+Returns the inner string slice. Used when passing
+the kb-id to I/O functions, serializers, or display.
+
+### `shard_prefix`
+
+Returns the first two characters of the kb-id. Used
+by `fs-database` to compute the hash-sharded
+directory path: `sections/{prefix}/{kbid}.sec`.
+
+```rust
+let id = KbId::from_content(b"hello");
+assert_eq!(id.shard_prefix().len(), 2);
+```
+
+## Trait Implementations
+
+| Trait        | Purpose                          |
+|--------------|----------------------------------|
+| `Clone`      | Cache entries need owned copies  |
+| `Debug`      | Diagnostic output                |
+| `PartialEq` / `Eq` | Identity comparison       |
+| `Hash`       | HashMap/HashSet keys (cache, index lookups) |
+| `Display`    | Formats as the raw 26-char string |
+| `Encode`     | Binary serialization for shared memory (see [encode](../encode/traits.md)) |
+
+`Ord` / `PartialOrd` are **not** derived. Kb-ids
+have no meaningful ordering -- they are content
+hashes. If sorting is needed (e.g. deterministic
+output), use `as_str()` for lexicographic comparison
+explicitly.
+
+## Error Type
+
+```rust
+#[derive(Debug)]
+pub enum KbIdError {
+   InvalidLength(usize),
+   InvalidCharacter(char, usize),
+}
+```
+
+## Testing
+
+Unit tests:
+
+- `from_content` produces 26-char lowercase
+  alphanumeric output for various inputs.
+- Identical content always produces the same kb-id.
+- Different content produces different kb-ids.
+- `shard_prefix` returns the first two characters.
+- `new` accepts valid 26-char strings.
+- `new` rejects strings that are too short, too long,
+  or contain uppercase/special characters.
+- `Encode` round-trip: encode then decode produces
+  an equal `KbId`.

package/docs/modules/rust/shared/memory/functions.md

@@ -0,0 +1,209 @@
+# functions::alloc -- Allocator and Read/Write Helpers
+
+*Allocator exports and shared-memory read/write
+helpers for passing data between the TypeScript host
+and WASM modules.*
+
+## File
+
+`engine/crates/shared/src/memory/alloc.rs`
+
+## Overview
+
+WASM modules and the TypeScript host share a single
+`WebAssembly.Memory` buffer. All data exchange --
+strings, byte slices, structured records -- passes
+through this buffer using `(pointer, length)` pairs.
+This module provides:
+
+1. **Allocator exports** (`kbdb_alloc`, `kbdb_free`)
+   that the TypeScript host calls to manage buffers.
+2. **Read helpers** that safely reconstruct Rust types
+   from raw memory pointers.
+3. **Write helpers** that serialize Rust types into
+   the shared buffer and return pointer/length pairs.
+
+Every WASM module re-exports `kbdb_alloc` and
+`kbdb_free` from this module so the host can allocate
+memory in any module's address space.
+
+## Allocator Exports
+
+```rust
+use std::alloc::{alloc, dealloc, Layout};
+
+#[no_mangle]
+pub extern "C" fn kbdb_alloc(size: u32) -> *mut u8 {
+   let layout = Layout::from_size_align(
+      size as usize, 1,
+   ).unwrap();
+   unsafe { alloc(layout) }
+}
+
+#[no_mangle]
+pub extern "C" fn kbdb_free(
+   ptr: *mut u8, size: u32,
+) {
+   let layout = Layout::from_size_align(
+      size as usize, 1,
+   ).unwrap();
+   unsafe { dealloc(ptr, layout) }
+}
+```
+
+### Usage from TypeScript
+
+The TypeScript daemon uses this protocol to pass
+data into WASM functions:
+
+```
+1. host calls kbdb_alloc(size) → ptr
+2. host writes bytes into memory at ptr
+3. host calls wasm_function(ptr, size)
+4. wasm reads from ptr
+5. host calls kbdb_free(ptr, size)
+```
+
+For return values:
+
+```
+1. wasm calls kbdb_alloc(size) internally → ptr
+2. wasm writes result into ptr
+3. wasm returns (ptr, size) to host
+4. host reads bytes from memory at ptr
+5. host calls kbdb_free(ptr, size)
+```
+
+### Alignment
+
+Alignment is `1` (byte-aligned). The data exchanged
+is always byte streams (UTF-8 strings, binary
+encoded structs). No multi-byte alignment is needed
+at the allocation level -- the `Encode` trait handles
+field alignment internally.
+
+### Null and Zero
+
+`kbdb_alloc(0)` returns a non-null, dangling pointer
+(per Rust's allocator contract). Callers must not
+dereference zero-length allocations. `kbdb_free` with
+size `0` is a no-op.
+
+## Read Helpers
+
+Safe functions that reconstruct Rust types from raw
+shared-memory pointers.
+
+```rust
+pub unsafe fn read_str<'a>(
+   ptr: *const u8, len: u32,
+) -> &'a str;
+
+pub unsafe fn read_bytes<'a>(
+   ptr: *const u8, len: u32,
+) -> &'a [u8];
+```
+
+### `read_str`
+
+Interprets the byte range `[ptr, ptr + len)` as
+UTF-8. Panics if the bytes are not valid UTF-8.
+
+This is used for string arguments from the host
+(file paths, query strings, MIME types). The host
+always writes valid UTF-8 -- a panic here indicates
+a host bug.
+
+### `read_bytes`
+
+Returns the byte range `[ptr, ptr + len)` as a
+slice. No UTF-8 validation. Used for binary data
+(section content, encoded structs).
+
+## Write Helpers
+
+Functions that serialize Rust types into the shared
+buffer and return `(pointer, length)` pairs for the
+caller to pass back to the host or another module.
+
+```rust
+pub fn write_str(s: &str) -> (*const u8, u32);
+
+pub fn write_bytes(data: &[u8]) -> (*const u8, u32);
+
+pub fn write_encoded<T: Encode>(
+   value: &T,
+) -> (*const u8, u32);
+```
+
+### `write_str`
+
+Allocates `s.len()` bytes via `kbdb_alloc`, copies
+the UTF-8 bytes, and returns the pointer and length.
+The caller (or host) is responsible for freeing the
+allocation.
+
+### `write_bytes`
+
+Same as `write_str` but for arbitrary byte slices.
+
+### `write_encoded`
+
+Encodes `value` using the `Encode` trait (see
+[encode](../encode/traits.md)), allocates a
+buffer for the result, copies the encoded bytes, and
+returns the pointer and length. Convenience wrapper
+for returning structured data from WASM exports.
+
+## Return Value Convention
+
+WASM functions can only return scalar values. For
+functions that return variable-length data, the
+convention is a **two-slot return buffer**:
+
+```rust
+pub fn set_return(ptr: *const u8, len: u32);
+pub fn get_return_ptr() -> *const u8;
+pub fn get_return_len() -> u32;
+```
+
+A pair of module-global slots holds the most recent
+return value. The WASM export writes `(ptr, len)`
+into the slots via `set_return`, returns a status
+code (`0` for success, negative for error), and the
+host reads the actual data via `get_return_ptr` and
+`get_return_len`.
+
+This avoids multi-value returns, which have uneven
+support across WASM runtimes.
+
+## Safety
+
+All read functions are `unsafe` because they
+dereference raw pointers. The safety contract:
+
+- `ptr` must point into the shared
+  `WebAssembly.Memory` buffer.
+- `ptr + len` must not exceed the buffer's current
+  size.
+- The memory region must not be freed or
+  reallocated while the returned reference is alive.
+
+The write functions are safe -- they allocate fresh
+buffers and copy data in.
+
+## Testing
+
+Unit tests (native Rust, not WASM):
+
+- `kbdb_alloc` + `kbdb_free` round-trip without
+  panic for various sizes (1, 256, 65536).
+- `kbdb_alloc(0)` returns non-null.
+- `write_str` + `read_str` round-trip preserves
+  content for ASCII, Unicode, and empty strings.
+- `write_bytes` + `read_bytes` round-trip preserves
+  arbitrary byte sequences.
+- `write_encoded` + `Encode::decode` round-trip for
+  sample types.
+- Return value slots: `set_return` → `get_return_ptr`
+  / `get_return_len` retrieves the same values.

package/docs/modules/rust/shared/memory/module.md

@@ -0,0 +1,24 @@
+# memory -- Memory Management
+
+*Allocator exports and shared-memory read/write
+helpers for passing data between the TypeScript host
+and WASM modules.*
+
+## Source
+
+`engine/crates/shared/src/memory/`
+
+## Responsibility
+
+Provides the shared-memory allocation protocol used
+by all three WASM modules. Every WASM export that
+exchanges variable-length data (strings, byte
+slices, encoded structs) goes through this module's
+allocator and read/write helpers.
+
+## Exports
+
+- **[functions.md](functions.md)** -- `kbdb_alloc`,
+  `kbdb_free`, `read_str`, `read_bytes`, `write_str`,
+  `write_bytes`, `write_encoded`, `set_return`,
+  `get_return_ptr`, `get_return_len`.