@dikolab/kbdb 0.1.6 → 0.2.0

package/docs/modules/rust/shared/corpus/types.md

@@ -0,0 +1,141 @@
+# types::corpus -- Corpus Statistics
+
+*Lightweight corpus statistics struct passed between
+modules for IDF computation and BM25F length
+normalization.*
+
+## File
+
+`engine/crates/shared/src/corpus/corpus.rs`
+
+## Overview
+
+Ranking functions need corpus-wide statistics to
+compute inverse document frequency (IDF) and
+normalize for section length. The full corpus data
+(including the per-term document frequency map) lives
+in `fs-database`'s `CorpusStats` struct in the
+`index::corpus` module. This shared type is a
+**lightweight view** -- just the scalar values needed
+by `query-parser`'s ranking exports.
+
+`kb-worker` reads the full stats from `fs-database`,
+extracts the fields needed for ranking, and passes
+this struct to `query-parser`.
+
+See [Query Parser -- Ranking](
+../../../../../goals/query-parser.md) for how these
+values are used in BM25F scoring.
+
+## Type Definition
+
+```rust
+pub struct CorpusStats {
+   pub total_sections: u32,
+   pub total_terms: u32,
+   pub avg_section_length: f32,
+   pub avg_heading_length: f32,
+   pub avg_body_length: f32,
+   pub avg_code_length: f32,
+}
+```
+
+| Field                | Description                    |
+|----------------------|--------------------------------|
+| `total_sections`     | Number of sections with at least one indexed term |
+| `total_terms`        | Total number of distinct terms in the inverted index |
+| `avg_section_length` | Average total token count per indexed section |
+| `avg_heading_length` | Average heading-field token count per indexed section |
+| `avg_body_length`    | Average body-field token count per indexed section |
+| `avg_code_length`    | Average code-field token count per indexed section |
+
+### `total_sections`
+
+Count of sections that have at least one indexed
+term. Image sections are included when their
+`title` or `description` produces indexed terms.
+Used as the `N` value in IDF:
+`idf(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)`.
+
+### `total_terms`
+
+Number of unique terms across the entire index.
+Informational -- not directly used in ranking, but
+useful for status reporting and diagnostics.
+
+### `avg_section_length`
+
+Average number of tokens per indexed section. Used
+by BM25F for overall length normalization.
+
+Computed as:
+`sum(token_count for each indexed section) / total_sections`
+
+### `avg_heading_length`
+
+Average heading-field token count per indexed
+section. Used by BM25F for per-field length
+normalization of the heading component (title
+content).
+
+Computed as:
+`sum(heading_token_count for each indexed section) / total_sections`
+
+### `avg_body_length`
+
+Average body-field token count per indexed section.
+Used by BM25F for per-field length normalization
+of the body component (body text + description).
+
+Computed as:
+`sum(body_token_count for each indexed section) / total_sections`
+
+### `avg_code_length`
+
+Average code-field token count per indexed section.
+Used by BM25F for per-field length normalization
+of the code component (source code content).
+
+Computed as:
+`sum(code_token_count for each indexed section) / total_sections`
+
+## Why a Separate Shared Type
+
+`fs-database`'s internal `CorpusStats` struct (in
+`index::corpus`) contains additional fields -- the
+full `HashMap<Term, u32>` of per-term document
+frequencies, incremental update methods, and
+serialization logic for the `corpus.dat` binary file.
+That struct is private to `fs-database`.
+
+The shared `CorpusStats` here contains only the
+scalar fields that cross module boundaries. This
+keeps the shared type small and avoids exposing
+`fs-database` internals to `query-parser`.
+
+When `kb-worker` needs per-term IDF values, it calls
+`db_corpus_stats` (which returns this shared struct)
+and separately calls `db_execute_plan` (which returns
+per-term frequencies in `RawMatch`). The full
+document frequency map never leaves `fs-database`.
+
+## Trait Implementations
+
+| Trait    | Purpose                          |
+|----------|----------------------------------|
+| `Clone`  | Cached alongside index segments  |
+| `Debug`  | Diagnostic output                |
+| `Copy`   | Small struct, stack-friendly     |
+| `Encode` | Binary serialization for shared memory (see [encode](../encode/traits.md)) |
+
+## Testing
+
+Unit tests:
+
+- Construction with valid values.
+- `Copy` semantics: assignment creates independent
+  value.
+- `Encode` round-trip preserves all six fields
+  including `f32` precision.
+- Edge cases: `total_sections = 0` (empty database),
+  all average lengths `0.0`.

package/docs/modules/rust/shared/document/module.md

@@ -0,0 +1,25 @@
+# document -- Document Records
+
+*Data structures for document manifests and document-
+level retrieval results.*
+
+## Source
+
+`engine/crates/shared/src/document/`
+
+## Responsibility
+
+Defines the document manifest and record types that
+group sections under a title with a defined order.
+Shared across `fs-database` (manifest storage),
+`kb-worker` (cache entries and retrieval), and the
+IPC boundary.
+
+See [Database -- Documents](
+../../../../goals/database.md) for the behavioral
+specification of document manifests.
+
+## Exports
+
+- **[types.md](types.md)** -- `DocumentManifest`,
+  `DocumentSection`, `DocumentRecord`.

package/docs/modules/rust/shared/document/types.md

@@ -0,0 +1,154 @@
+# types::document -- Document Records
+
+*Data structures for document manifests and document-
+level retrieval results shared across WASM modules.*
+
+## File
+
+`engine/crates/shared/src/document/document.rs`
+
+## Overview
+
+A **document** groups sections under a title with a
+defined order. The document itself contains no content
+-- it references sections by kb-id. These types are
+shared across `fs-database` (manifest storage),
+`kb-worker` (cache entries and retrieval), and
+the IPC boundary (JSON-RPC responses).
+
+See [Database -- Documents](
+../../../../../goals/database.md) for the behavioral
+specification of document manifests.
+
+## Type Definitions
+
+### `DocumentManifest`
+
+```rust
+pub struct DocumentManifest {
+   pub docid: String,
+   pub title: String,
+   pub created_at: String,
+   pub sections: Vec<DocumentSection>,
+}
+```
+
+Represents the TOML manifest file at
+`documents/{docid}.toml`. Contains metadata and an
+ordered list of section references.
+
+`sections` is ordered by `position` -- the manifest
+preserves the original section order within the
+document.
+
+### `DocumentSection`
+
+```rust
+pub struct DocumentSection {
+   pub kbid: KbId,
+   pub position: u32,
+}
+```
+
+A reference from a document to a section. `position`
+is a zero-based ordinal that determines the section's
+display order when the document is retrieved.
+
+### `DocumentRecord`
+
+```rust
+pub struct DocumentRecord {
+   pub docid: String,
+   pub title: String,
+   pub created_at: String,
+   pub sections: Vec<SectionRecord>,
+}
+```
+
+A fully resolved document -- the manifest plus the
+actual content of each referenced section. Produced
+by `kb-worker` when a caller requests a document
+via `worker_retrieve_doc`. The `sections` vector
+contains full `SectionRecord` entries (with content
+bodies) in position order.
+
+This type is used at the IPC boundary for the
+`retrieve document` endpoint and in the `get_result`
+response with `type: "content"`.
+
+## Methods
+
+### `DocumentManifest`
+
+```rust
+impl DocumentManifest {
+   pub fn section_count(&self) -> u32;
+
+   pub fn contains(&self, kbid: &KbId) -> bool;
+
+   pub fn replace_section(
+      &mut self,
+      old_kbid: &KbId,
+      new_kbid: &KbId,
+   ) -> bool;
+
+   pub fn remove_section(
+      &mut self,
+      kbid: &KbId,
+   ) -> bool;
+
+   pub fn reposition(&mut self);
+}
+```
+
+#### `section_count`
+
+Returns the number of section references.
+
+#### `contains`
+
+Checks whether a kb-id is in the section list.
+
+#### `replace_section`
+
+Replaces `old_kbid` with `new_kbid` in the section
+list, preserving position. Returns `true` if the
+replacement was made, `false` if `old_kbid` was not
+found. Used by `fs-database` during section updates
+(content-addressable storage means a content change
+produces a new kb-id).
+
+#### `remove_section`
+
+Removes a section reference by kb-id. Returns `true`
+if removed. Does **not** reposition remaining
+sections -- call `reposition()` afterward if
+contiguous positions are required.
+
+#### `reposition`
+
+Reassigns `position` values to `0, 1, 2, ...` based
+on current vector order. Called after
+`remove_section` to close gaps.
+
+## Trait Implementations
+
+| Trait    | Types                                   |
+|----------|-----------------------------------------|
+| `Clone`  | All three types                         |
+| `Debug`  | All three types                         |
+| `Encode` | All three types (see [encode](../encode/traits.md)) |
+
+## Testing
+
+Unit tests:
+
+- `replace_section` swaps kb-ids correctly and
+  preserves position.
+- `replace_section` returns `false` for missing
+  kb-ids.
+- `remove_section` + `reposition` closes position
+  gaps.
+- `contains` matches on kb-id.
+- `Encode` round-trip for `DocumentManifest` and
+  `DocumentRecord` with multiple sections.

package/docs/modules/rust/shared/encode/module.md

@@ -0,0 +1,22 @@
+# encode -- Binary Encoding
+
+*Compact binary serialization for structured data
+passed through shared `WebAssembly.Memory` between
+WASM modules.*
+
+## Source
+
+`engine/crates/shared/src/encode/`
+
+## Responsibility
+
+Defines the `Encode` trait and its implementations
+for all shared types. Provides the wire format used
+whenever structured data crosses a WASM module
+boundary through shared memory.
+
+## Exports
+
+- **[traits.md](traits.md)** -- `Encode` trait
+  definition, wire format specification, per-type
+  implementations, `DecodeError`.

package/docs/modules/rust/shared/encode/traits.md

@@ -0,0 +1,304 @@
+# traits::encode -- Binary Encoding Trait
+
+*Compact binary serialization trait for structured
+data passed through shared `WebAssembly.Memory`
+between WASM modules.*
+
+## File
+
+`engine/crates/shared/src/encode/encode.rs`
+
+## Overview
+
+When `kb-worker.wasm` passes a `QueryPlan` to
+`fs-database.wasm`, or receives a `Vec<RawMatch>`
+back, the data crosses a module boundary through
+shared memory. JSON would work but adds parsing
+overhead on every call. The `Encode` trait provides
+a minimal, zero-copy-friendly binary format that
+both sides compile from the same type definitions --
+no schema negotiation, no version headers, no
+self-description.
+
+This is **not** a general-purpose serializer. It
+handles only the specific data shapes defined in
+`kbdb-shared`'s types. If a type changes, both the
+encoder and decoder are recompiled from the same
+source, so wire compatibility is guaranteed by the
+build system.
+
+## Trait Definition
+
+```rust
+pub trait Encode: Sized {
+   fn encode(&self, buf: &mut Vec<u8>);
+
+   fn decode(buf: &[u8])
+      -> Result<Self, DecodeError>;
+
+   fn encoded_size(&self) -> usize;
+}
+```
+
+### `encode`
+
+Appends the binary representation of `self` to
+`buf`. The caller pre-allocates or grows the buffer.
+Encoding is deterministic -- the same value always
+produces the same bytes.
+
+### `decode`
+
+Reads a value from the beginning of `buf`. Returns
+the decoded value and advances past the consumed
+bytes (via the return value's size). Returns
+`DecodeError` if the buffer is too short or contains
+invalid data.
+
+### `encoded_size`
+
+Returns the number of bytes `encode` will append.
+Used to pre-allocate buffers for write helpers in
+[memory/functions](../memory/functions.md).
+
+## Wire Format
+
+### Scalars
+
+| Rust type | Wire format         | Bytes |
+|-----------|---------------------|-------|
+| `u8`      | Raw byte            | 1     |
+| `u16`     | Little-endian       | 2     |
+| `u32`     | Little-endian       | 4     |
+| `u64`     | Little-endian       | 8     |
+| `i32`     | Little-endian       | 4     |
+| `f32`     | Little-endian IEEE 754 | 4  |
+| `bool`    | `0x00` / `0x01`     | 1     |
+
+### Strings
+
+```
+[len: u32] [utf8_bytes: len]
+```
+
+Length-prefixed. `len` is the byte count (not
+character count). Empty strings encode as
+`[0x00 0x00 0x00 0x00]` (4 bytes, zero length).
+
+### `Vec<T>`
+
+```
+[count: u32] [item_0] [item_1] ... [item_n]
+```
+
+Count-prefixed. Each item is encoded using `T`'s
+`Encode` implementation. Empty vecs encode as
+`[0x00 0x00 0x00 0x00]`.
+
+### `Option<T>`
+
+```
+[tag: u8] [value: T]?
+```
+
+- `0x00` = `None` (no value bytes follow).
+- `0x01` = `Some` (value bytes follow).
+
+### Tuples `(A, B)`
+
+```
+[a: A] [b: B]
+```
+
+Concatenated, no separator. Both sides know the
+types from the schema.
+
+### Enums
+
+```
+[variant: u8] [fields...]
+```
+
+One-byte discriminant followed by the variant's
+fields (if any). Variant numbering follows
+declaration order (`0, 1, 2, ...`).
+
+## Implementations
+
+### `KbId`
+
+```
+[len: u32] [utf8_bytes: 26]
+```
+
+Always 30 bytes (4-byte length prefix + 26-byte
+identifier). The length prefix is always `26` but
+included for consistency with the string format.
+
+### `Term`
+
+```
+[len: u32] [utf8_bytes: len]
+```
+
+Standard string encoding. Variable length.
+
+### `SectionType`
+
+```
+[variant: u8] [mime_str...]?
+```
+
+- `0x00` = `Text` (no additional data).
+- `0x01` = `Code` + MIME string encoding.
+- `0x02` = `Image` + MIME string encoding.
+
+### `QueryPlan`
+
+```
+[terms: Vec<Term>]
+[phrases: Vec<Vec<Term>>]
+[exclusions: Vec<Term>]
+[mode: u8]
+```
+
+`mode`: `0x00` = `And`, `0x01` = `Or`.
+
+### `FieldTf`
+
+```
+[heading: f32]
+[body: f32]
+[code: f32]
+```
+
+Fixed 12 bytes.
+
+### `FieldTokenCounts`
+
+```
+[heading: u32]
+[body: u32]
+[code: u32]
+```
+
+Fixed 12 bytes.
+
+### `RawMatch`
+
+```
+[kbid: KbId]
+[field_tfs: Vec<(Term, FieldTf)>]
+[positions: Vec<(Term, Vec<u32>)>]
+[token_count: u32]
+[field_token_counts: FieldTokenCounts]
+```
+
+### `SectionRecord`
+
+```
+[kbid: KbId]
+[docids: Vec<String>]
+[section_type: SectionType]
+[title: Option<String>]
+[description: Option<String>]
+[content: Vec<u8>]          -- as [len: u32] [bytes]
+[size: u32]
+[created_at: String]
+[checksum: String]
+[token_count: u32]
+[heading_token_count: u32]
+[body_token_count: u32]
+[code_token_count: u32]
+```
+
+### `DocumentManifest`
+
+```
+[docid: String]
+[title: String]
+[created_at: String]
+[sections: Vec<DocumentSection>]
+```
+
+### `DocumentSection`
+
+```
+[kbid: KbId]
+[position: u32]
+```
+
+### `CorpusStats`
+
+```
+[total_sections: u32]
+[total_terms: u32]
+[avg_section_length: f32]
+[avg_heading_length: f32]
+[avg_body_length: f32]
+[avg_code_length: f32]
+```
+
+Fixed 24 bytes.
+
+## Error Type
+
+```rust
+#[derive(Debug)]
+pub enum DecodeError {
+   BufferTooShort {
+      expected: usize,
+      available: usize,
+   },
+   InvalidUtf8(usize),
+   InvalidVariant(u8),
+}
+```
+
+| Variant          | Cause                          |
+|------------------|--------------------------------|
+| `BufferTooShort` | Not enough bytes to read the next field |
+| `InvalidUtf8`    | String bytes are not valid UTF-8 at the given offset |
+| `InvalidVariant` | Enum discriminant is out of range |
+
+## Design Decisions
+
+**Why not serde + bincode?** The `serde` ecosystem
+adds significant binary size to WASM modules.
+`bincode` pulls in `serde` as a dependency. The data
+shapes here are small and known at compile time --
+a hand-rolled trait with ~200 lines of impl is
+lighter than the dependency tree.
+
+**Why not JSON?** JSON parsing on every cross-module
+call adds CPU overhead. The inverted index may
+return thousands of `RawMatch` entries per query.
+Binary encoding avoids string escaping, number
+parsing, and allocation overhead.
+
+**Why length-prefixed?** Simplicity. No delimiters
+to escape, no backtracking. The decoder reads the
+length, allocates, reads the payload, advances. Each
+field is independent.
+
+**Why little-endian?** WASM is little-endian. Native
+byte order avoids byte-swapping.
+
+## Testing
+
+Unit tests:
+
+- Round-trip for every `Encode` implementation:
+  encode then decode produces an equal value.
+- `encoded_size` matches actual `encode` output
+  length for all types.
+- `DecodeError::BufferTooShort` when buffer is
+  truncated mid-field.
+- `DecodeError::InvalidVariant` when enum
+  discriminant is out of range.
+- `DecodeError::InvalidUtf8` when string bytes are
+  not valid UTF-8.
+- Edge cases: empty `Vec`, `None` options, zero-
+  length strings, maximum `u32` values.
+- Large payloads: `Vec<RawMatch>` with 1000 entries
+  encodes and decodes correctly.

package/docs/modules/rust/shared/error/module.md

@@ -0,0 +1,28 @@
+# error -- Error Types
+
+*Hierarchical error enums shared across all three
+WASM modules, mapping to JSON-RPC 2.0 error codes at
+the IPC boundary.*
+
+## Source
+
+`engine/crates/shared/src/error/`
+
+## Responsibility
+
+Defines the unified error hierarchy so that errors
+originating in `fs-database` or `query-parser`
+propagate through `kb-worker` to the TypeScript
+daemon with consistent `From` conversions and
+JSON-RPC 2.0 code mapping.
+
+See [Worker Daemon -- Error Responses](
+../../../../goals/worker-daemon.md) for the JSON-RPC
+2.0 error codes at the IPC boundary.
+
+## Exports
+
+- **[types.md](types.md)** -- `FsError`,
+  `IndexError`, `ParseError`, `WorkerError`. `From`
+  conversions, `Display`/`Error` implementations,
+  JSON-RPC 2.0 code mapping.