@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/modules/rust/fs-database/types/section.md

@@ -1,127 +0,0 @@
-# types::section -- Section Header and Parsing
-
-*The `SectionHeader` struct and `IntegrityReport`
-struct, plus TOML frontmatter parse/serialize and
-hash-sharded path resolution.*
-
-## Source
-
-`engine/crates/fs-database/src/section/`
-
-## Type Definitions
-
-### `SectionHeader`
-
-```rust
-pub struct SectionHeader {
-   pub kbid: KbId,
-   pub docids: Vec<String>,
-   pub section_type: SectionType,
-   pub title: Option<String>,
-   pub description: Option<String>,
-   pub size: u32,
-   pub created_at: String,
-   pub checksum: String,
-   pub token_count: u32,
-   pub heading_token_count: u32,
-   pub body_token_count: u32,
-   pub code_token_count: u32,
-   pub source_key: Option<String>,
-   pub source_path: Option<String>,
-   pub source_mtime: Option<String>,
-   pub updated_at: Option<String>,
-   pub tags: Vec<String>,
-   pub supersedes: Option<KbId>,
-}
-```
-
-TOML frontmatter fields from a section file. Parsed
-by `parse_section_file`, serialized by
-`serialize_section_file`.
-
-`docids` lists the document IDs this section belongs
-to (may be empty, one, or many -- many-to-many
-relationship). `title` is required for image sections
-and optional for text/code sections. `description` is
-always optional. Token counts are computed by
-`query-parser.wasm` during per-field tokenization
-and stored in the header for query-time retrieval.
-See [FieldTokenCounts](../../shared/query/types.md)
-for the shared struct.
-
-`source_key` is a stable logical identity derived
-from `sha256("${source_path}\0${heading}")`,
-truncated to 16 hex chars. Separate from `kbid`
-(content hash) -- identifies *what the section is
-about* rather than *what it currently says*. `null`
-for stdin input. `source_path` is the original file
-path that was learned. `source_mtime` is the ISO 8601
-modified time of the source file at learn time.
-`updated_at` is the ISO 8601 timestamp of the last
-supersession (`null` on first create). `tags` are
-user-supplied strings for scoping and filtering.
-`supersedes` is the kbid of the section this one
-replaced (`null` on first create). See
-[Database -- Logical Identity](../../../goals/database.md).
-
-### `IntegrityReport`
-
-```rust
-pub struct IntegrityReport {
-   pub checksum_failures: Vec<KbId>,
-   pub orphaned_sections: Vec<KbId>,
-   pub orphaned_documents: Vec<String>,
-   pub broken_refs: Vec<(KbId, String, KbId)>,
-   pub type_mismatches: Vec<(KbId, String, KbId)>,
-   pub circular_refs: Vec<(KbId, Vec<KbId>)>,
-   pub divergent_content: Vec<(String, Vec<KbId>)>,
-}
-```
-
-Returned by `check_integrity`. `broken_refs`
-entries are `(source_kbid, marker_type, target_kbid)`
-covering all six marker types (`text`, `code`,
-`image`, `document`, `link`, `doc-link`).
-`type_mismatches` are `(source_kbid, marker_type,
-target_kbid)` where the target section type does
-not match the marker type. `circular_refs` are
-`(start_kbid, cycle_path)` for reference chains
-that form loops. `divergent_content` entries are
-`(source_key, conflicting_kbids)` -- sections
-sharing the same `source_key` that have different
-content, indicating contradictory versions that
-should be resolved via supersession.
-
-## Functions
-
-### Frontmatter Parsing (`section/parse.rs`)
-
-```rust
-pub fn parse_section_file(data: &[u8])
-   -> Result<(SectionHeader, &[u8]), ParseError>;
-
-pub fn serialize_section_file(
-   header: &SectionHeader,
-   body: &[u8],
-) -> Vec<u8>;
-```
-
-Splits on `---\n` separator. Header is TOML; body
-is raw bytes stored verbatim.
-
-### Content Hashing (`section/hash.rs`)
-
-```rust
-pub fn compute_kbid(content: &[u8]) -> KbId;
-```
-
-SHA-256 → truncate to 16 bytes → base32 (Crockford)
-→ 26-character kb-id.
-
-### Path Resolution (`section/shard.rs`)
-
-```rust
-pub fn section_path(kbid: &KbId) -> String;
-```
-
-Returns `sections/{first_two_chars}/{kbid}.sec`.

package/docs/modules/rust/kb-worker/functions/compose.md

@@ -1,57 +0,0 @@
-# functions::compose -- Content Composition Flow
-
-*Orchestrates content composition: resolve identifiers,
-fetch content, compose rendered Markdown.*
-
-## Source
-
-`engine/crates/kb-worker/src/orchestrate/compose.rs`
-
-## Exported WASM Function
-
-### `worker_compose`
-
-```rust
-pub fn worker_compose(
-   ids_ptr: *const u8,
-   ids_len: u32,
-) -> u64;
-```
-
-Composes a rendered Markdown document from one or
-more kbid or docid identifiers. Returns a result
-token (same async pattern as `worker_query`).
-
-## Compose Flow
-
-```
-1. Receive identifiers from IPC `compose` method
-         ↓
-2. Resolve identifiers via fs-database
-   (try kbid first, then docid)
-         ↓
-3. Expand documents to section kbids, deduplicate
-         ↓
-4. Fetch section/document content from fs-database
-         ↓
-5. Extract content markers recursively using
-   extract_content_markers() from content-parser
-         ↓
-6. Build ComposeRequest with all resolved data
-         ↓
-7. Call compose_content() from content-composer
-         ↓
-8. Return ComposeResult to daemon
-```
-
-The caller retrieves the composed document via
-`worker_get_result` with type `"content"`.
-
-Individual sections fetched during composition are
-cached. The composed result itself is NOT cached --
-it is computed fresh on each request because
-composition depends on the full set of input
-identifiers.
-
-See [Content Composer](../../shared/content-composer/module.md)
-for the full composition specification.

package/docs/modules/rust/kb-worker/functions/module.md

@@ -1,20 +0,0 @@
-# kb-worker/functions -- Orchestration Functions
-
-*Functions that coordinate calls between
-query-parser.wasm and fs-database.wasm to implement
-full database operations.*
-
-## Modules
-
-- **[search](search.md)** -- Search orchestration.
-  Parse query → build plan → execute → rank →
-  return results.
-- **[write](write.md)** -- Write orchestration.
-  Add/update/remove text, code, and image sections.
-- **[retrieve](retrieve.md)** -- Read orchestration.
-  Retrieve sections and documents, with cache.
-  Progressive context expansion (`worker_recall`).
-  Status reporting.
-- **[compose](compose.md)** -- Content composition.
-  Resolve identifiers, fetch content, compose
-  rendered Markdown via content-composer.

package/docs/modules/rust/kb-worker/functions/retrieve.md

@@ -1,194 +0,0 @@
-# functions::retrieve -- Read & Status Orchestration
-
-*Read sections and documents from cache or disk.
-Report database status.*
-
-## Source
-
-`engine/crates/kb-worker/src/orchestrate/retrieve.rs`
-
-## Exported WASM Functions
-
-### `worker_read_sections`
-
-```rust
-pub fn worker_read_sections(
-   kbid_ptr: *const u8,
-   kbid_len: u32,
-) -> u64;
-```
-
-Read one or more sections by KbId. Returns Encode'd
-`Vec<SectionRecord>`.
-
-### `worker_retrieve_doc`
-
-```rust
-pub fn worker_retrieve_doc(
-   doc_id_ptr: *const u8,
-   doc_id_len: u32,
-) -> u64;
-```
-
-Read a document manifest and all its sections.
-Returns Encode'd `DocumentRecord`.
-
-### `worker_status`
-
-```rust
-pub fn worker_status() -> u64;
-```
-
-Returns Encode'd `StatusResult` (see
-[types/status](../types/status.md)).
-
-### `worker_result_status`
-
-```rust
-pub fn worker_result_status(
-   token_ptr: *const u8,
-   token_len: u32,
-) -> u32;
-```
-
-Poll an async token. Returns `0` = busy,
-`1` = success, `2` = fail.
-
-### `worker_get_result`
-
-```rust
-pub fn worker_get_result(
-   token_ptr: *const u8,
-   token_len: u32,
-) -> u64;
-```
-
-Fetch the result buffer for a completed token.
-Returns `(ptr, len)`. Consumes the token (one-shot).
-
-### `worker_invalidate`
-
-```rust
-pub fn worker_invalidate(
-   keys_ptr: *const u8,
-   keys_len: u32,
-) -> u32;
-```
-
-Evict specific cache entries. Returns count evicted.
-
-### `worker_invalidate_all`
-
-```rust
-pub fn worker_invalidate_all() -> u32;
-```
-
-Evict entire cache. Returns count evicted.
-
-## `worker_recall` -- Progressive Context Expansion
-
-```rust
-pub fn worker_recall(
-   params_ptr: *const u8,
-   params_len: u32,
-) -> u64;
-```
-
-WASM export for the `recall` IPC method. Accepts
-a serialized `RecallParams` (kbid or kbids array,
-depth 0--3). Returns serialized `RecallResult`
-(single) or `RecallResult[]` (batch) via the shared
-memory return slot.
-
-### Recall Flow
-
-```
-1. Receive kbid(s) + depth
-         |
-2. For each kbid: read section from cache
-   (disk fallback)
-   -> SectionRecord with heading, type, docids,
-      content
-         |
-3. If depth == 0: return section data
-         |
-4. If depth >= 1: for each docid in section's
-   docids, read document manifest -> list of all
-   section kbids with headings and types.
-   Query references.idx for back-references
-   (entries where target_id == kbid and
-   direction == back).
-         |
-5. If depth >= 2: read full content of sibling
-   sections (other sections in the same documents).
-   Query references.idx for forward references
-   (entries where source_kbid == kbid and
-   direction == forward).
-         |
-6. If depth >= 3: for each forward-referenced
-   section, read its full content. Query
-   references.idx for their back-references.
-         |
-   Return RecallResult(s) via shared memory
-```
-
-**Cache interaction:** Recall is read-heavy and
-benefits from the LRU cache. Section reads at
-depth 0 hit the cache populated by recent searches.
-Sibling and reference reads at depth 2--3 may cause
-cache misses on first access.
-
-**Reference graph lookup:** Forward and back
-references are read from `references.idx` via
-`fs-database.wasm` index lookup functions. This is
-O(1) per kbid via the B+ tree index -- no content
-re-parsing needed.
-
----
-
-## Retrieve Flow
-
-```
-1. Receive KbId
-         ↓
-2. Check cache  →  hit?  →  return cached
-         ↓ miss
-3. db_read_section  →  SectionRecord from disk
-         ↓
-4. Put into cache
-         ↓
-   Return SectionRecord
-```
-
-Document retrieval follows the same pattern but calls
-`db_read_document` and caches the manifest separately
-from individual sections.
-
-## Maintenance Exports
-
-### `worker_check_integrity`
-
-```rust
-pub fn worker_check_integrity() -> u64;
-```
-
-Delegates to `db_check_integrity`. Returns Encode'd
-`IntegrityReport`.
-
-### `worker_garbage_collect`
-
-```rust
-pub fn worker_garbage_collect() -> u64;
-```
-
-Delegates to `db_garbage_collect`. Returns count of
-removed orphans.
-
-### `worker_rebuild_indexes`
-
-```rust
-pub fn worker_rebuild_indexes() -> u64;
-```
-
-Delegates to `db_index_rebuild`. Invalidates entire
-cache afterward.

package/docs/modules/rust/kb-worker/functions/search.md

@@ -1,173 +0,0 @@
-# functions::search -- Search Orchestration
-
-*End-to-end search flow: query string in, ranked
-results out.*
-
-## Source
-
-`engine/crates/kb-worker/src/orchestrate/search.rs`
-
-## Exported WASM Functions
-
-### `worker_search`
-
-```rust
-pub fn worker_search(
-   query_ptr: *const u8,
-   query_len: u32,
-   limit: u32,
-) -> u64;
-```
-
-Synchronous search. Returns `(ptr, len)` packed into
-`u64`. The result buffer contains Encode'd ranked
-matches.
-
-### `worker_query`
-
-```rust
-pub fn worker_query(
-   query_ptr: *const u8,
-   query_len: u32,
-   limit: u32,
-) -> u64;
-```
-
-Async search. Returns a `(ptr, len)` token string.
-Poll with `worker_result_status`, fetch with
-`worker_get_result`.
-
-## Search Flow (13 Steps)
-
-```
-1. Receive query string from TypeScript IPC
-         ↓
-2. qp_tokenize  →  token stream
-         ↓
-3. qp_parse_query  →  QueryPlan
-         ↓
-4. db_execute_plan  →  Vec<RawMatch>
-         ↓
-5. db_corpus_stats  →  CorpusStats
-         ↓
-6. qp_score_bm25f  →  base scores
-         ↓
-7. qp_score_proximity  →  proximity boost
-         ↓
-8. qp_score_zone  →  zone/field boost
-         ↓
-9. Sort by combined score descending
-         ↓
-10. Apply offset: skip first `offset` entries
-         ↓
-11. Truncate to `limit`
-         ↓
-12. Shape results: for each scored section, read
-    heading, type, and docids from the section
-    header; generate KWIC snippet (~150 chars
-    centered on the highest-scoring matched term
-    position from the positional index)
-         ↓
-13. Build paged envelope: wrap shaped items with
-    `total` (pre-offset/limit count), `offset`,
-    `limit`, `has_more`
-         ↓
-    Return PagedSearchResult via shared memory
-```
-
-### Step Details
-
-1. **Receive query** -- TypeScript writes the UTF-8
-   query into shared memory and calls
-   `worker_search(ptr, len, limit)`.
-
-2. **Tokenize** -- Calls `qp_tokenize` from
-   query-parser. Produces a `TokenStream` with
-   normalized, stemmed tokens.
-
-3. **Parse** -- Calls `qp_parse_query`. Interprets
-   boolean operators, phrase markers, field
-   qualifiers. Produces a `QueryPlan`.
-
-4. **Execute plan** -- Calls `db_execute_plan` from
-   fs-database. Walks the inverted index and
-   positional index. Returns `Vec<RawMatch>` with
-   per-field TFs (`field_tfs`), positions,
-   `token_count`, and `field_token_counts` per
-   matching section.
-
-5. **Corpus stats** -- Calls `db_corpus_stats`. Gets
-   document count, term frequencies for IDF
-   computation, and per-field average lengths
-   (`avg_heading_length`, `avg_body_length`,
-   `avg_code_length`) for BM25F normalization.
-
-6. **BM25F** -- Calls `qp_score_bm25f`. Computes
-   per-field BM25 using `FieldTf` values from each
-   `RawMatch` and per-field corpus averages. Field
-   weights (heading 2.0x, body 1.0x, code 1.5x) are
-   applied at query time. See
-   [Query Result](../../../../goals/query-result.md)
-   for the full scoring specification.
-
-7. **Proximity** -- Calls `qp_score_proximity`.
-   Boosts documents where query terms appear close
-   together.
-
-8. **Zone scoring** -- Calls `qp_score_zone`. Boosts
-   matches in high-value zones (title, headings,
-   first paragraph).
-
-9. **Sort** -- Combines all scores, sorts descending.
-
-10. **Apply offset** -- Skips the first `offset`
-    entries from the sorted result list. Enables
-    cursor-based pagination without re-executing the
-    query.
-
-11. **Truncate** -- Takes at most `limit` entries
-    from the offset-adjusted list.
-
-12. **Shape results** -- For each scored section,
-    reads heading, type, and docids from the section
-    header. Generates a KWIC snippet (~150 chars
-    centered on the highest-scoring matched term
-    position from the positional index). See
-    [Snippet Generation](#snippet-generation) below.
-
-13. **Build paged envelope** -- Wraps shaped items
-    with `total` (pre-offset/limit count), `offset`,
-    `limit`, and `has_more`. Encodes as
-    `PagedSearchResult` via the `Encode` trait into
-    shared memory.
-
----
-
-### Snippet Generation
-
-Snippets are generated during step 12 (result
-shaping), using data already available from the
-search pipeline:
-
-- The positional index provides the token offset
-  of each matched term in each section.
-- The section content is read from cache (or disk
-  fallback).
-
-**Algorithm:**
-
-1. For each matched section, find the position of
-   the highest-scoring matched term (the term with
-   the highest BM25F contribution).
-2. Convert the token position to a byte offset in
-   the section content.
-3. Extract ~150 characters centered on that byte
-   offset. If the offset is near the start or end,
-   shift the window accordingly.
-4. Trim to word boundaries and add `...` ellipsis
-   at truncated boundaries.
-5. If the highest-scoring match is in the heading
-   (not body), use the first ~150 characters of
-   the body content instead.
-6. For image sections (`image/*`) or sections with
-   no body content, the snippet is empty string.