@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/modules/rust/shared/encode/traits.md

@@ -1,304 +0,0 @@
-# traits::encode -- Binary Encoding Trait
-
-*Compact binary serialization trait for structured
-data passed through shared `WebAssembly.Memory`
-between WASM modules.*
-
-## File
-
-`engine/crates/shared/src/encode/encode.rs`
-
-## Overview
-
-When `kb-worker.wasm` passes a `QueryPlan` to
-`fs-database.wasm`, or receives a `Vec<RawMatch>`
-back, the data crosses a module boundary through
-shared memory. JSON would work but adds parsing
-overhead on every call. The `Encode` trait provides
-a minimal, zero-copy-friendly binary format that
-both sides compile from the same type definitions --
-no schema negotiation, no version headers, no
-self-description.
-
-This is **not** a general-purpose serializer. It
-handles only the specific data shapes defined in
-`kbdb-shared`'s types. If a type changes, both the
-encoder and decoder are recompiled from the same
-source, so wire compatibility is guaranteed by the
-build system.
-
-## Trait Definition
-
-```rust
-pub trait Encode: Sized {
-   fn encode(&self, buf: &mut Vec<u8>);
-
-   fn decode(buf: &[u8])
-      -> Result<Self, DecodeError>;
-
-   fn encoded_size(&self) -> usize;
-}
-```
-
-### `encode`
-
-Appends the binary representation of `self` to
-`buf`. The caller pre-allocates or grows the buffer.
-Encoding is deterministic -- the same value always
-produces the same bytes.
-
-### `decode`
-
-Reads a value from the beginning of `buf`. Returns
-the decoded value and advances past the consumed
-bytes (via the return value's size). Returns
-`DecodeError` if the buffer is too short or contains
-invalid data.
-
-### `encoded_size`
-
-Returns the number of bytes `encode` will append.
-Used to pre-allocate buffers for write helpers in
-[memory/functions](../memory/functions.md).
-
-## Wire Format
-
-### Scalars
-
-| Rust type | Wire format         | Bytes |
-|-----------|---------------------|-------|
-| `u8`      | Raw byte            | 1     |
-| `u16`     | Little-endian       | 2     |
-| `u32`     | Little-endian       | 4     |
-| `u64`     | Little-endian       | 8     |
-| `i32`     | Little-endian       | 4     |
-| `f32`     | Little-endian IEEE 754 | 4  |
-| `bool`    | `0x00` / `0x01`     | 1     |
-
-### Strings
-
-```
-[len: u32] [utf8_bytes: len]
-```
-
-Length-prefixed. `len` is the byte count (not
-character count). Empty strings encode as
-`[0x00 0x00 0x00 0x00]` (4 bytes, zero length).
-
-### `Vec<T>`
-
-```
-[count: u32] [item_0] [item_1] ... [item_n]
-```
-
-Count-prefixed. Each item is encoded using `T`'s
-`Encode` implementation. Empty vecs encode as
-`[0x00 0x00 0x00 0x00]`.
-
-### `Option<T>`
-
-```
-[tag: u8] [value: T]?
-```
-
-- `0x00` = `None` (no value bytes follow).
-- `0x01` = `Some` (value bytes follow).
-
-### Tuples `(A, B)`
-
-```
-[a: A] [b: B]
-```
-
-Concatenated, no separator. Both sides know the
-types from the schema.
-
-### Enums
-
-```
-[variant: u8] [fields...]
-```
-
-One-byte discriminant followed by the variant's
-fields (if any). Variant numbering follows
-declaration order (`0, 1, 2, ...`).
-
-## Implementations
-
-### `KbId`
-
-```
-[len: u32] [utf8_bytes: 26]
-```
-
-Always 30 bytes (4-byte length prefix + 26-byte
-identifier). The length prefix is always `26` but
-included for consistency with the string format.
-
-### `Term`
-
-```
-[len: u32] [utf8_bytes: len]
-```
-
-Standard string encoding. Variable length.
-
-### `SectionType`
-
-```
-[variant: u8] [mime_str...]?
-```
-
-- `0x00` = `Text` (no additional data).
-- `0x01` = `Code` + MIME string encoding.
-- `0x02` = `Image` + MIME string encoding.
-
-### `QueryPlan`
-
-```
-[terms: Vec<Term>]
-[phrases: Vec<Vec<Term>>]
-[exclusions: Vec<Term>]
-[mode: u8]
-```
-
-`mode`: `0x00` = `And`, `0x01` = `Or`.
-
-### `FieldTf`
-
-```
-[heading: f32]
-[body: f32]
-[code: f32]
-```
-
-Fixed 12 bytes.
-
-### `FieldTokenCounts`
-
-```
-[heading: u32]
-[body: u32]
-[code: u32]
-```
-
-Fixed 12 bytes.
-
-### `RawMatch`
-
-```
-[kbid: KbId]
-[field_tfs: Vec<(Term, FieldTf)>]
-[positions: Vec<(Term, Vec<u32>)>]
-[token_count: u32]
-[field_token_counts: FieldTokenCounts]
-```
-
-### `SectionRecord`
-
-```
-[kbid: KbId]
-[docids: Vec<String>]
-[section_type: SectionType]
-[title: Option<String>]
-[description: Option<String>]
-[content: Vec<u8>]          -- as [len: u32] [bytes]
-[size: u32]
-[created_at: String]
-[checksum: String]
-[token_count: u32]
-[heading_token_count: u32]
-[body_token_count: u32]
-[code_token_count: u32]
-```
-
-### `DocumentManifest`
-
-```
-[docid: String]
-[title: String]
-[created_at: String]
-[sections: Vec<DocumentSection>]
-```
-
-### `DocumentSection`
-
-```
-[kbid: KbId]
-[position: u32]
-```
-
-### `CorpusStats`
-
-```
-[total_sections: u32]
-[total_terms: u32]
-[avg_section_length: f32]
-[avg_heading_length: f32]
-[avg_body_length: f32]
-[avg_code_length: f32]
-```
-
-Fixed 24 bytes.
-
-## Error Type
-
-```rust
-#[derive(Debug)]
-pub enum DecodeError {
-   BufferTooShort {
-      expected: usize,
-      available: usize,
-   },
-   InvalidUtf8(usize),
-   InvalidVariant(u8),
-}
-```
-
-| Variant          | Cause                          |
-|------------------|--------------------------------|
-| `BufferTooShort` | Not enough bytes to read the next field |
-| `InvalidUtf8`    | String bytes are not valid UTF-8 at the given offset |
-| `InvalidVariant` | Enum discriminant is out of range |
-
-## Design Decisions
-
-**Why not serde + bincode?** The `serde` ecosystem
-adds significant binary size to WASM modules.
-`bincode` pulls in `serde` as a dependency. The data
-shapes here are small and known at compile time --
-a hand-rolled trait with ~200 lines of impl is
-lighter than the dependency tree.
-
-**Why not JSON?** JSON parsing on every cross-module
-call adds CPU overhead. The inverted index may
-return thousands of `RawMatch` entries per query.
-Binary encoding avoids string escaping, number
-parsing, and allocation overhead.
-
-**Why length-prefixed?** Simplicity. No delimiters
-to escape, no backtracking. The decoder reads the
-length, allocates, reads the payload, advances. Each
-field is independent.
-
-**Why little-endian?** WASM is little-endian. Native
-byte order avoids byte-swapping.
-
-## Testing
-
-Unit tests:
-
-- Round-trip for every `Encode` implementation:
-  encode then decode produces an equal value.
-- `encoded_size` matches actual `encode` output
-  length for all types.
-- `DecodeError::BufferTooShort` when buffer is
-  truncated mid-field.
-- `DecodeError::InvalidVariant` when enum
-  discriminant is out of range.
-- `DecodeError::InvalidUtf8` when string bytes are
-  not valid UTF-8.
-- Edge cases: empty `Vec`, `None` options, zero-
-  length strings, maximum `u32` values.
-- Large payloads: `Vec<RawMatch>` with 1000 entries
-  encodes and decodes correctly.

package/docs/modules/rust/shared/error/module.md

@@ -1,28 +0,0 @@
-# error -- Error Types
-
-*Hierarchical error enums shared across all three
-WASM modules, mapping to JSON-RPC 2.0 error codes at
-the IPC boundary.*
-
-## Source
-
-`engine/crates/shared/src/error/`
-
-## Responsibility
-
-Defines the unified error hierarchy so that errors
-originating in `fs-database` or `query-parser`
-propagate through `kb-worker` to the TypeScript
-daemon with consistent `From` conversions and
-JSON-RPC 2.0 code mapping.
-
-See [Worker Daemon -- Error Responses](
-../../../../goals/worker-daemon.md) for the JSON-RPC
-2.0 error codes at the IPC boundary.
-
-## Exports
-
-- **[types.md](types.md)** -- `FsError`,
-  `IndexError`, `ParseError`, `WorkerError`. `From`
-  conversions, `Display`/`Error` implementations,
-  JSON-RPC 2.0 code mapping.

package/docs/modules/rust/shared/error/types.md

@@ -1,302 +0,0 @@
-# types::error -- Error Types
-
-*Hierarchical error enums shared across all three
-WASM modules, mapping to JSON-RPC 2.0 error codes at
-the IPC boundary.*
-
-## File
-
-`engine/crates/shared/src/error/error.rs`
-
-## Overview
-
-Errors originate in the standalone modules
-(`fs-database`, `query-parser`) and propagate upward
-through `kb-worker` to the TypeScript daemon, which
-translates them into JSON-RPC 2.0 error responses.
-Defining all error types in `kbdb-shared` ensures a
-single hierarchy with consistent `From`
-implementations across crate boundaries.
-
-See [Worker Daemon -- Error Responses](
-../../../../../goals/worker-daemon.md) for the JSON-RPC
-2.0 error codes at the IPC boundary.
-
-## Error Hierarchy
-
-```
-WorkerError           (kb-worker.wasm -> TypeScript)
-├── Fs(FsError)       (from fs-database.wasm)
-├── Index(IndexError) (from fs-database.wasm)
-├── Parse(ParseError) (from query-parser.wasm)
-├── CacheFull
-├── InvalidToken(String)
-└── ContextMismatch { expected, actual }
-```
-
-`WorkerError` is the top-level error returned by
-`kb-worker.wasm` exports. It wraps lower-level
-errors from the standalone modules.
-
-## Type Definitions
-
-### `FsError`
-
-```rust
-#[derive(Debug)]
-pub enum FsError {
-   NotFound(String),
-   PermissionDenied(String),
-   IoError(String),
-   ParseError(String),
-   ChecksumMismatch {
-      expected: String,
-      actual: String,
-   },
-}
-```
-
-Errors from `fs-database.wasm` file operations.
-
-| Variant            | Cause                          |
-|--------------------|--------------------------------|
-| `NotFound`         | Section, document, or file path does not exist |
-| `PermissionDenied` | Host I/O import returned permission error |
-| `IoError`          | Host I/O import returned generic error |
-| `ParseError`       | TOML header or catalog parsing failed |
-| `ChecksumMismatch` | Section file content does not match its stored checksum |
-
-`NotFound` carries the path or kb-id that was
-requested. `ChecksumMismatch` carries both the
-expected and actual checksum strings for diagnostics.
-
-### `IndexError`
-
-```rust
-#[derive(Debug)]
-pub enum IndexError {
-   Corrupted(String),
-   KeyNotFound(String),
-}
-```
-
-Errors from `fs-database.wasm` index operations.
-
-| Variant      | Cause                             |
-|--------------|-----------------------------------|
-| `Corrupted`  | B+ tree structure is invalid, page size mismatch, or binary data is unreadable |
-| `KeyNotFound`| Term not found in the inverted index (not necessarily an error -- a query for a term with no matches) |
-
-`Corrupted` indicates the index files need a
-rebuild. The daemon can recover by calling
-`worker_rebuild_indexes`.
-
-### `ParseError`
-
-```rust
-#[derive(Debug)]
-pub enum ParseError {
-   InvalidSyntax(String),
-   EmptyQuery,
-   UnmatchedQuote(usize),
-}
-```
-
-Errors from `query-parser.wasm` query parsing.
-
-| Variant          | Cause                          |
-|------------------|--------------------------------|
-| `InvalidSyntax`  | Unrecognized query operator or malformed expression |
-| `EmptyQuery`     | Query string is empty or contains only stop words |
-| `UnmatchedQuote` | Opening `"` without a closing `"` -- carries the character position |
-
-### `WorkerError`
-
-```rust
-#[derive(Debug)]
-pub enum WorkerError {
-   Fs(FsError),
-   Index(IndexError),
-   Parse(ParseError),
-   CacheFull,
-   InvalidToken(String),
-   ContextMismatch {
-      expected: String,
-      actual: String,
-   },
-}
-```
-
-Top-level error returned by `kb-worker.wasm` exports.
-
-| Variant          | Cause                          |
-|------------------|--------------------------------|
-| `Fs(FsError)`    | Storage operation failed       |
-| `Index(IndexError)` | Index operation failed      |
-| `Parse(ParseError)` | Query parsing failed        |
-| `CacheFull`      | Cache limit reached and eviction failed (should not happen with LRU, but defensive) |
-| `InvalidToken`   | Result token does not exist or has expired |
-| `ContextMismatch`| Request's `ctx` param does not match the daemon's context |
-
-## `From` Implementations
-
-```rust
-impl From<FsError> for WorkerError {
-   fn from(e: FsError) -> Self {
-      WorkerError::Fs(e)
-   }
-}
-
-impl From<IndexError> for WorkerError {
-   fn from(e: IndexError) -> Self {
-      WorkerError::Index(e)
-   }
-}
-
-impl From<ParseError> for WorkerError {
-   fn from(e: ParseError) -> Self {
-      WorkerError::Parse(e)
-   }
-}
-```
-
-These conversions allow the `?` operator to propagate
-errors naturally across module boundaries:
-
-```rust
-fn search(...) -> Result<SearchResult, WorkerError> {
-   let plan = parse_query(query)?;   // ParseError -> WorkerError
-   let raw = execute_plan(&plan)?;   // IndexError -> WorkerError
-   let section = read_section(id)?;  // FsError -> WorkerError
-   // ...
-}
-```
-
-## JSON-RPC 2.0 Mapping
-
-The TypeScript daemon maps `WorkerError` variants to
-JSON-RPC 2.0 error codes:
-
-| WorkerError variant | JSON-RPC code | JSON-RPC message     |
-|---------------------|---------------|----------------------|
-| `Fs(NotFound)`      | `-32000`      | `"Not found"`        |
-| `Fs(PermissionDenied)` | `-32000`   | `"Permission denied"`|
-| `Fs(IoError)`       | `-32000`      | `"I/O error"`        |
-| `Fs(ParseError)`    | `-32000`      | `"Parse error"`      |
-| `Fs(ChecksumMismatch)` | `-32000`  | `"Checksum mismatch"`|
-| `Index(Corrupted)`  | `-32000`      | `"Index corrupted"`  |
-| `Index(KeyNotFound)`| `-32000`      | `"Key not found"`    |
-| `Parse(InvalidSyntax)` | `-32000`  | `"Invalid query syntax"` |
-| `Parse(EmptyQuery)` | `-32000`      | `"Empty query"`      |
-| `Parse(UnmatchedQuote)` | `-32000` | `"Unmatched quote"`  |
-| `CacheFull`         | `-32000`      | `"Cache full"`       |
-| `InvalidToken`      | `-32002`      | `"Invalid token"`    |
-| `ContextMismatch`   | `-32001`      | `"Context mismatch"` |
-
-Codes `-32001` (context mismatch) and `-32002`
-(invalid token) are application-specific codes
-defined in the [Worker Daemon](
-../../../../../goals/worker-daemon.md) specification.
-All other errors use the generic application error
-code `-32000`.
-
-## Error Relay -- WASM to Host
-
-WASM errors must reach the TypeScript process's
-stderr before the WASM instance is cleaned up. The
-relay sequence:
-
-1. A WASM export (`worker_init`, `worker_search`,
-   etc.) encounters an error and returns a
-   `WorkerError` via the shared memory return
-   convention (`set_return(ptr, len)` with the
-   `Encode`-serialized error).
-2. The TypeScript daemon **reads the error from
-   shared memory immediately** -- before any cleanup,
-   cache purge, or WASM teardown.
-3. The daemon deserializes the `WorkerError`, maps it
-   to the appropriate JSON-RPC error code and
-   human-readable message.
-4. **For IPC-connected errors:** The daemon sends the
-   JSON-RPC error response to the client over IPC,
-   then resumes normal operation (no teardown).
-5. **For fatal startup errors** (precondition failures
-   during `worker_init`): The daemon writes the error
-   message to **stderr**, then proceeds with shutdown
-   (purge cache -> remove PID file -> remove socket
-   -> exit). The WASM modules remain instantiated
-   until the error is fully read and relayed.
-
-**Key invariant:** the TypeScript host always reads
-and relays the WASM error before the WASM instance
-is purged. No error is lost to premature cleanup.
-
-## Daemon-Level Error Codes
-
-Beyond the WASM-origin error codes (`-32000` through
-`-32002`), the daemon defines additional error codes
-for startup and client-side failures:
-
-| Code     | Meaning                              | Origin | Relay path |
-|----------|--------------------------------------|--------|------------|
-| `-32001` | Context mismatch                     | WASM (`WorkerError::ContextMismatch`) | IPC response |
-| `-32002` | Invalid token                        | WASM (`WorkerError::InvalidToken`) | IPC response |
-| `-32003` | Daemon unavailable                   | Worker client (spawn failure) | `WorkerClientError` thrown to caller |
-| `-32004` | Database directory not found         | WASM `worker_init` or TS precondition | stderr -> `WorkerClientError.daemonStderr` |
-| `-32005` | Database directory not accessible    | WASM `worker_init` or TS precondition | stderr -> `WorkerClientError.daemonStderr` |
-| `-32006` | Database version too new             | WASM `worker_init` | stderr -> `WorkerClientError.daemonStderr` |
-| `-32007` | Database version too old (needs migration) | WASM `worker_init` | stderr -> `WorkerClientError.daemonStderr` |
-
-Codes `-32004` through `-32007` are startup errors.
-They are never returned over IPC because the daemon
-exits before creating the socket. The daemon writes
-the error to stderr; the worker client captures
-stderr during the spawn window and includes it in
-`WorkerClientError.daemonStderr`. This relay ensures
-the user sees the specific failure reason even though
-the daemon runs as a separate detached process.
-
-## `Display` and `Error` Implementations
-
-All error types implement `std::fmt::Display` and
-`std::error::Error`:
-
-```rust
-impl fmt::Display for FsError {
-   fn fmt(&self, f: &mut fmt::Formatter<'_>)
-      -> fmt::Result
-   {
-      match self {
-         Self::NotFound(p) =>
-            write!(f, "not found: {p}"),
-         Self::PermissionDenied(p) =>
-            write!(f, "permission denied: {p}"),
-         Self::IoError(msg) =>
-            write!(f, "I/O error: {msg}"),
-         Self::ParseError(msg) =>
-            write!(f, "parse error: {msg}"),
-         Self::ChecksumMismatch { expected, actual } =>
-            write!(
-               f,
-               "checksum mismatch: expected {expected}, got {actual}",
-            ),
-      }
-   }
-}
-```
-
-`WorkerError::source()` returns the wrapped inner
-error for `Fs`, `Index`, and `Parse` variants,
-enabling error chain inspection.
-
-## Testing
-
-Unit tests:
-
-- `From` conversions: each error type converts to
-  the correct `WorkerError` variant.
-- `Display` formatting: each variant produces a
-  human-readable message.
-- `source()` chain: `WorkerError::Fs(e).source()`
-  returns `Some(&e)`.
-- All variants are `Debug`-printable without panic.

package/docs/modules/rust/shared/kbid/module.md

@@ -1,24 +0,0 @@
-# kbid -- Content-Addressed Identifier
-
-*Content-addressed 26-character section identifier
-derived from SHA-256 hashing.*
-
-## Source
-
-`engine/crates/shared/src/kbid/`
-
-## Responsibility
-
-Defines the `KbId` newtype and its construction
-methods. Every section in the knowledge base is
-identified by a deterministic, collision-resistant,
-URL-safe string derived from the section's content.
-
-See [Database -- Sections](../../../../goals/database.md)
-for the behavioral specification of content-
-addressable storage.
-
-## Exports
-
-- **[types.md](types.md)** -- `KbId` newtype,
-  `from_content`, `new`, `shard_prefix`, `KbIdError`.