@dikolab/kbdb 0.5.0 → 0.6.0

package/docs/modules/rust/shared/content-composer/types.md

@@ -1,93 +0,0 @@
-# content-composer -- Types
-
-## `ComposeRequest`
-
-Input to the composition engine. Contains all
-pre-fetched data needed to produce the composed
-Markdown -- no I/O occurs during composition.
-
-```rust
-pub struct ComposeRequest {
-   pub main_ids: Vec<ResolvedId>,
-   pub sections: HashMap<String, SectionData>,
-   pub documents: HashMap<String, DocumentData>,
-}
-```
-
-| Field       | Description                              |
-|-------------|------------------------------------------|
-| `main_ids`  | Ordered list of resolved identifiers forming the primary content |
-| `sections`  | All pre-fetched sections, keyed by kbid  |
-| `documents` | All pre-fetched documents, keyed by docid|
-
-## `ResolvedId`
-
-Discriminated identifier from the resolution phase.
-
-```rust
-pub enum ResolvedId {
-   Section(String),
-   Document(String),
-}
-```
-
-## `SectionData`
-
-Pre-fetched section content with extracted content
-markers.
-
-```rust
-pub struct SectionData {
-   pub kbid: String,
-   pub section_type: SectionType,
-   pub content: String,
-   pub markers: Vec<ExtractedContentMarker>,
-}
-```
-
-| Field          | Description                           |
-|----------------|---------------------------------------|
-| `kbid`         | Content-addressed section identifier  |
-| `section_type` | MIME-based type (`SectionType` from `section` module) |
-| `content`      | Raw section body text                 |
-| `markers`      | Pre-extracted content markers from `extract_content_markers()` |
-
-## `DocumentData`
-
-Pre-fetched document manifest.
-
-```rust
-pub struct DocumentData {
-   pub docid: String,
-   pub title: String,
-   pub section_kbids: Vec<String>,
-}
-```
-
-| Field          | Description                           |
-|----------------|---------------------------------------|
-| `docid`        | Document identifier                   |
-| `title`        | Document title for heading rendering  |
-| `section_kbids`| Ordered section kbids in this document|
-
-## `ComposeResult`
-
-Output of the composition engine.
-
-```rust
-pub struct ComposeResult {
-   pub markdown: String,
-   pub total_chars: usize,
-}
-```
-
-| Field        | Description                             |
-|--------------|-----------------------------------------|
-| `markdown`   | Composed Markdown string                |
-| `total_chars` | Character count of `markdown` (not byte count) |
-
-## `RenderedSection`
-
-Internal rendering type used by `render.rs` during
-Phase 2 composition. Tracks per-section rendering
-state.

package/docs/modules/rust/shared/content-parser/functions.md

@@ -1,72 +0,0 @@
-# content-parser -- Functions
-
-## `extract_content_markers`
-
-```rust
-pub fn extract_content_markers(text: &str)
-   -> Vec<ExtractedContentMarker>
-```
-
-Scans the input text and returns all recognized
-content markers in document order (lowest to highest
-byte offset). Does not modify the input. Returns an
-empty `Vec` if no markers are found.
-
-The function performs a single left-to-right scan.
-It is non-recursive -- it does not follow references
-or read other sections.
-
-### Tests
-
-Unit tests in
-`engine/crates/shared/src/content-parser/extract.rs`:
-
-- Empty input returns empty vec.
-- Single `$(image: {kbid})` returns one marker with
-  correct type, value, offset, and length.
-- All six content marker types recognized
-  individually.
-- Multiple markers in sequence, returned in document
-  order.
-- Adjacent markers with no space between them.
-- Unrecognized `$(formula: ...)` produces no marker.
-- Unclosed marker (no `)`) produces no marker.
-- Unicode text surrounding markers -- offsets are
-  byte-accurate.
-- Marker-only input (no surrounding text).
-- Whitespace variations inside markers (e.g. extra
-  spaces after colon).
-
-## `strip_content_markers`
-
-```rust
-pub fn strip_content_markers(text: &str) -> String
-```
-
-Returns a new string with all recognized content
-markers removed. Whitespace left by marker removal
-is preserved; the caller should normalise whitespace
-if needed.
-
-This function replaces the earlier implementation in
-`shared/src/pipeline/markers.rs`, which only
-recognized `$(image:)` and `$(code:)`. The shared
-version recognizes all six content marker types.
-
-Both functions share the same scanning logic
-internally. `strip_content_markers` is equivalent to
-removing every character range identified by
-`extract_content_markers`.
-
-### Tests
-
-Unit tests in
-`engine/crates/shared/src/content-parser/strip.rs`:
-
-- Empty input returns empty string.
-- Plain text with no markers returns unchanged.
-- All six content marker types stripped.
-- Multiple markers stripped in one pass.
-- Unrecognized markers left untouched.
-- Unclosed markers left untouched.
-- Unicode text preserved around stripped markers.

package/docs/modules/rust/shared/content-parser/module.md

@@ -1,99 +0,0 @@
-# content-parser -- Content Marker Extraction
-
-*Format-agnostic parser that extracts `$(type: value)`
-embedding and link markers from section content,
-producing structured reference lists for result set
-resolution.*
-
-## Module
-
-| Field       | Value                                     |
-|-------------|-------------------------------------------|
-| Rust module | `content_parser`                          |
-| Source path | `engine/crates/shared/src/content-parser/`|
-| Crate       | `kbdb-shared` (rlib)                      |
-
-The module is declared in `lib.rs` with a path
-attribute because the source directory uses a
-hyphenated name:
-
-```rust
-#[path = "content-parser/mod.rs"]
-pub mod content_parser;
-```
-
-## Directory Layout
-
-```
-engine/crates/shared/src/content-parser/
-   mod.rs          -- re-exports
-   types.rs        -- ContentMarkerType,
-                      ExtractedContentMarker
-   extract.rs      -- extract_content_markers()
-   strip.rs        -- strip_content_markers()
-```
-
-## Purpose
-
-The content parser scans any UTF-8 text for
-`$(type: value)` markers and returns a structured
-list of extracted references. It is format-agnostic
--- the marker syntax is independent of the
-surrounding text format (markdown, source code,
-plain text).
-
-The parser performs a **single left-to-right scan**
-of the input. It is **non-recursive** -- it does not
-follow references or read other sections. Recursive
-reference resolution is the caller's responsibility
-(typically `kb-worker`).
-
-## Integration Points
-
-### Query Parser
-
-The `query-parser` crate imports
-`strip_content_markers()` from
-`kbdb_shared::content_parser` for tokenization
-Pipeline Stage 1. This replaces the crate's earlier
-local regex that only recognized `$(image:)` and
-`$(code:)`. The shared version recognizes all six
-content marker types.
-
-### Filesystem Database
-
-The `fs-database` crate imports
-`extract_content_markers()` from
-`kbdb_shared::content_parser` for integrity
-checking. The integrity checker extracts all markers
-and verifies that each referenced kb-id or doc-id
-exists and has the expected type.
-
-### Worker Orchestration
-
-The `kb-worker` crate imports
-`extract_content_markers()` from
-`kbdb_shared::content_parser` to build the
-`references` array in result sets. When a search
-or retrieval request returns sections, `kb-worker`
-scans each section's content, follows `$(text:)` and
-`$(document:)` references recursively using a visited
-set, and returns a flat deduplicated `references`
-array alongside the primary result data.
-
-## Sub-documents
-
-- [types.md](types.md) -- `ContentMarkerType`,
-  `ExtractedContentMarker`
-- [functions.md](functions.md) --
-  `extract_content_markers()`,
-  `strip_content_markers()`
-
-## Testing
-
-Tests live inline in
-`engine/crates/shared/src/content-parser/extract.rs`
-and
-`engine/crates/shared/src/content-parser/strip.rs`
-(`#[cfg(test)]` modules). See
-[functions.md](functions.md) for test descriptions.

package/docs/modules/rust/shared/content-parser/types.md

@@ -1,71 +0,0 @@
-# content-parser -- Types
-
-## `ContentMarkerType`
-
-Enum identifying the six recognized content marker
-types.
-
-```rust
-pub enum ContentMarkerType {
-   /// `$(image: {kbid})` -- image section embedding.
-   Image,
-   /// `$(code: {kbid})` -- code section embedding.
-   Code,
-   /// `$(text: {kbid})` -- text section embedding.
-   Text,
-   /// `$(document: {docid})` -- document embedding.
-   Document,
-   /// `$(link: {kbid})` -- section cross-reference.
-   Link,
-   /// `$(doc-link: {docid})` -- document
-   /// cross-reference.
-   DocLink,
-}
-```
-
-Each variant maps to a `$(type:` prefix:
-
-| Variant    | Prefix         | Value type |
-|------------|----------------|------------|
-| `Image`    | `$(image:`     | kbid       |
-| `Code`     | `$(code:`      | kbid       |
-| `Text`     | `$(text:`      | kbid       |
-| `Document` | `$(document:`  | docid      |
-| `Link`     | `$(link:`      | kbid       |
-| `DocLink`  | `$(doc-link:`  | docid      |
-
-## `ExtractedContentMarker`
-
-A single marker extracted from text content.
-
-```rust
-pub struct ExtractedContentMarker {
-   /// The marker category.
-   pub marker_type: ContentMarkerType,
-   /// The value inside the marker (kb-id or doc-id).
-   pub value: String,
-   /// Byte offset of the marker start (`$`) in the
-   /// original text.
-   pub offset: usize,
-   /// Byte length of the entire marker, including
-   /// the `$(` and `)` delimiters.
-   pub length: usize,
-}
-```
-
-## Syntax Rules
-
-Content markers follow these parsing rules:
-
-1. A marker starts with `$(` and ends with the next
-   `)`.
-2. The type identifier is matched against the six
-   recognized prefixes listed above.
-3. Unrecognized prefixes (e.g. `$(formula:`) are
-   **skipped** -- no `ExtractedContentMarker` is
-   produced.
-4. Unclosed markers (no closing `)` before end of
-   input) are **skipped**.
-5. The value is the text between the colon-space and
-   the closing `)`, trimmed of leading and trailing
-   whitespace.

package/docs/modules/rust/shared/corpus/module.md

@@ -1,24 +0,0 @@
-# corpus -- Corpus Statistics
-
-*Lightweight corpus statistics passed between modules
-for IDF computation and BM25F length normalization.*
-
-## Source
-
-`engine/crates/shared/src/corpus/`
-
-## Responsibility
-
-Defines the lightweight `CorpusStats` view passed
-from `fs-database` to `query-parser` via `kb-worker`
-for ranking computations. Contains only the scalar
-values needed for BM25F -- the full document
-frequency map stays private to `fs-database`.
-
-See [Query Parser -- Ranking](
-../../../../goals/query-parser.md) for how these
-values are used in BM25F scoring.
-
-## Exports
-
-- **[types.md](types.md)** -- `CorpusStats`.

package/docs/modules/rust/shared/corpus/types.md

@@ -1,141 +0,0 @@
-# types::corpus -- Corpus Statistics
-
-*Lightweight corpus statistics struct passed between
-modules for IDF computation and BM25F length
-normalization.*
-
-## File
-
-`engine/crates/shared/src/corpus/corpus.rs`
-
-## Overview
-
-Ranking functions need corpus-wide statistics to
-compute inverse document frequency (IDF) and
-normalize for section length. The full corpus data
-(including the per-term document frequency map) lives
-in `fs-database`'s `CorpusStats` struct in the
-`index::corpus` module. This shared type is a
-**lightweight view** -- just the scalar values needed
-by `query-parser`'s ranking exports.
-
-`kb-worker` reads the full stats from `fs-database`,
-extracts the fields needed for ranking, and passes
-this struct to `query-parser`.
-
-See [Query Parser -- Ranking](
-../../../../../goals/query-parser.md) for how these
-values are used in BM25F scoring.
-
-## Type Definition
-
-```rust
-pub struct CorpusStats {
-   pub total_sections: u32,
-   pub total_terms: u32,
-   pub avg_section_length: f32,
-   pub avg_heading_length: f32,
-   pub avg_body_length: f32,
-   pub avg_code_length: f32,
-}
-```
-
-| Field                | Description                    |
-|----------------------|--------------------------------|
-| `total_sections`     | Number of sections with at least one indexed term |
-| `total_terms`        | Total number of distinct terms in the inverted index |
-| `avg_section_length` | Average total token count per indexed section |
-| `avg_heading_length` | Average heading-field token count per indexed section |
-| `avg_body_length`    | Average body-field token count per indexed section |
-| `avg_code_length`    | Average code-field token count per indexed section |
-
-### `total_sections`
-
-Count of sections that have at least one indexed
-term. Image sections are included when their
-`title` or `description` produces indexed terms.
-Used as the `N` value in IDF:
-`idf(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)`.
-
-### `total_terms`
-
-Number of unique terms across the entire index.
-Informational -- not directly used in ranking, but
-useful for status reporting and diagnostics.
-
-### `avg_section_length`
-
-Average number of tokens per indexed section. Used
-by BM25F for overall length normalization.
-
-Computed as:
-`sum(token_count for each indexed section) / total_sections`
-
-### `avg_heading_length`
-
-Average heading-field token count per indexed
-section. Used by BM25F for per-field length
-normalization of the heading component (title
-content).
-
-Computed as:
-`sum(heading_token_count for each indexed section) / total_sections`
-
-### `avg_body_length`
-
-Average body-field token count per indexed section.
-Used by BM25F for per-field length normalization
-of the body component (body text + description).
-
-Computed as:
-`sum(body_token_count for each indexed section) / total_sections`
-
-### `avg_code_length`
-
-Average code-field token count per indexed section.
-Used by BM25F for per-field length normalization
-of the code component (source code content).
-
-Computed as:
-`sum(code_token_count for each indexed section) / total_sections`
-
-## Why a Separate Shared Type
-
-`fs-database`'s internal `CorpusStats` struct (in
-`index::corpus`) contains additional fields -- the
-full `HashMap<Term, u32>` of per-term document
-frequencies, incremental update methods, and
-serialization logic for the `corpus.dat` binary file.
-That struct is private to `fs-database`.
-
-The shared `CorpusStats` here contains only the
-scalar fields that cross module boundaries. This
-keeps the shared type small and avoids exposing
-`fs-database` internals to `query-parser`.
-
-When `kb-worker` needs per-term IDF values, it calls
-`db_corpus_stats` (which returns this shared struct)
-and separately calls `db_execute_plan` (which returns
-per-term frequencies in `RawMatch`). The full
-document frequency map never leaves `fs-database`.
-
-## Trait Implementations
-
-| Trait    | Purpose                          |
-|----------|----------------------------------|
-| `Clone`  | Cached alongside index segments  |
-| `Debug`  | Diagnostic output                |
-| `Copy`   | Small struct, stack-friendly     |
-| `Encode` | Binary serialization for shared memory (see [encode](../encode/traits.md)) |
-
-## Testing
-
-Unit tests:
-
-- Construction with valid values.
-- `Copy` semantics: assignment creates independent
-  value.
-- `Encode` round-trip preserves all six fields
-  including `f32` precision.
-- Edge cases: `total_sections = 0` (empty database),
-  all average lengths `0.0`.

package/docs/modules/rust/shared/document/module.md

@@ -1,25 +0,0 @@
-# document -- Document Records
-
-*Data structures for document manifests and document-
-level retrieval results.*
-
-## Source
-
-`engine/crates/shared/src/document/`
-
-## Responsibility
-
-Defines the document manifest and record types that
-group sections under a title with a defined order.
-Shared across `fs-database` (manifest storage),
-`kb-worker` (cache entries and retrieval), and the
-IPC boundary.
-
-See [Database -- Documents](
-../../../../goals/database.md) for the behavioral
-specification of document manifests.
-
-## Exports
-
-- **[types.md](types.md)** -- `DocumentManifest`,
-  `DocumentSection`, `DocumentRecord`.

package/docs/modules/rust/shared/document/types.md

@@ -1,154 +0,0 @@
-# types::document -- Document Records
-
-*Data structures for document manifests and document-
-level retrieval results shared across WASM modules.*
-
-## File
-
-`engine/crates/shared/src/document/document.rs`
-
-## Overview
-
-A **document** groups sections under a title with a
-defined order. The document itself contains no content
--- it references sections by kb-id. These types are
-shared across `fs-database` (manifest storage),
-`kb-worker` (cache entries and retrieval), and
-the IPC boundary (JSON-RPC responses).
-
-See [Database -- Documents](
-../../../../../goals/database.md) for the behavioral
-specification of document manifests.
-
-## Type Definitions
-
-### `DocumentManifest`
-
-```rust
-pub struct DocumentManifest {
-   pub docid: String,
-   pub title: String,
-   pub created_at: String,
-   pub sections: Vec<DocumentSection>,
-}
-```
-
-Represents the TOML manifest file at
-`documents/{docid}.toml`. Contains metadata and an
-ordered list of section references.
-
-`sections` is ordered by `position` -- the manifest
-preserves the original section order within the
-document.
-
-### `DocumentSection`
-
-```rust
-pub struct DocumentSection {
-   pub kbid: KbId,
-   pub position: u32,
-}
-```
-
-A reference from a document to a section. `position`
-is a zero-based ordinal that determines the section's
-display order when the document is retrieved.
-
-### `DocumentRecord`
-
-```rust
-pub struct DocumentRecord {
-   pub docid: String,
-   pub title: String,
-   pub created_at: String,
-   pub sections: Vec<SectionRecord>,
-}
-```
-
-A fully resolved document -- the manifest plus the
-actual content of each referenced section. Produced
-by `kb-worker` when a caller requests a document
-via `worker_retrieve_doc`. The `sections` vector
-contains full `SectionRecord` entries (with content
-bodies) in position order.
-
-This type is used at the IPC boundary for the
-`retrieve document` endpoint and in the `get_result`
-response with `type: "content"`.
-
-## Methods
-
-### `DocumentManifest`
-
-```rust
-impl DocumentManifest {
-   pub fn section_count(&self) -> u32;
-
-   pub fn contains(&self, kbid: &KbId) -> bool;
-
-   pub fn replace_section(
-      &mut self,
-      old_kbid: &KbId,
-      new_kbid: &KbId,
-   ) -> bool;
-
-   pub fn remove_section(
-      &mut self,
-      kbid: &KbId,
-   ) -> bool;
-
-   pub fn reposition(&mut self);
-}
-```
-
-#### `section_count`
-
-Returns the number of section references.
-
-#### `contains`
-
-Checks whether a kb-id is in the section list.
-
-#### `replace_section`
-
-Replaces `old_kbid` with `new_kbid` in the section
-list, preserving position. Returns `true` if the
-replacement was made, `false` if `old_kbid` was not
-found. Used by `fs-database` during section updates
-(content-addressable storage means a content change
-produces a new kb-id).
-
-#### `remove_section`
-
-Removes a section reference by kb-id. Returns `true`
-if removed. Does **not** reposition remaining
-sections -- call `reposition()` afterward if
-contiguous positions are required.
-
-#### `reposition`
-
-Reassigns `position` values to `0, 1, 2, ...` based
-on current vector order. Called after
-`remove_section` to close gaps.
-
-## Trait Implementations
-
-| Trait    | Types                                   |
-|----------|-----------------------------------------|
-| `Clone`  | All three types                         |
-| `Debug`  | All three types                         |
-| `Encode` | All three types (see [encode](../encode/traits.md)) |
-
-## Testing
-
-Unit tests:
-
-- `replace_section` swaps kb-ids correctly and
-  preserves position.
-- `replace_section` returns `false` for missing
-  kb-ids.
-- `remove_section` + `reposition` closes position
-  gaps.
-- `contains` matches on kb-id.
-- `Encode` round-trip for `DocumentManifest` and
-  `DocumentRecord` with multiple sections.

package/docs/modules/rust/shared/encode/module.md

@@ -1,22 +0,0 @@
-# encode -- Binary Encoding
-
-*Compact binary serialization for structured data
-passed through shared `WebAssembly.Memory` between
-WASM modules.*
-
-## Source
-
-`engine/crates/shared/src/encode/`
-
-## Responsibility
-
-Defines the `Encode` trait and its implementations
-for all shared types. Provides the wire format used
-whenever structured data crosses a WASM module
-boundary through shared memory.
-
-## Exports
-
-- **[traits.md](traits.md)** -- `Encode` trait
-  definition, wire format specification, per-type
-  implementations, `DecodeError`.