@dikolab/kbdb 0.1.6 → 0.2.0

package/docs/modules/rust/kb-worker/functions/retrieve.md

@@ -0,0 +1,194 @@
+# functions::retrieve -- Read & Status Orchestration
+
+*Read sections and documents from cache or disk.
+Report database status.*
+
+## Source
+
+`engine/crates/kb-worker/src/orchestrate/retrieve.rs`
+
+## Exported WASM Functions
+
+### `worker_read_sections`
+
+```rust
+pub fn worker_read_sections(
+   kbid_ptr: *const u8,
+   kbid_len: u32,
+) -> u64;
+```
+
+Read one or more sections by KbId. Returns Encode'd
+`Vec<SectionRecord>`.
+
+### `worker_retrieve_doc`
+
+```rust
+pub fn worker_retrieve_doc(
+   doc_id_ptr: *const u8,
+   doc_id_len: u32,
+) -> u64;
+```
+
+Read a document manifest and all its sections.
+Returns Encode'd `DocumentRecord`.
+
+### `worker_status`
+
+```rust
+pub fn worker_status() -> u64;
+```
+
+Returns Encode'd `StatusResult` (see
+[types/status](../types/status.md)).
+
+### `worker_result_status`
+
+```rust
+pub fn worker_result_status(
+   token_ptr: *const u8,
+   token_len: u32,
+) -> u32;
+```
+
+Poll an async token. Returns `0` = busy,
+`1` = success, `2` = fail.
+
+### `worker_get_result`
+
+```rust
+pub fn worker_get_result(
+   token_ptr: *const u8,
+   token_len: u32,
+) -> u64;
+```
+
+Fetch the result buffer for a completed token.
+Returns `(ptr, len)`. Consumes the token (one-shot).
+
+### `worker_invalidate`
+
+```rust
+pub fn worker_invalidate(
+   keys_ptr: *const u8,
+   keys_len: u32,
+) -> u32;
+```
+
+Evict specific cache entries. Returns count evicted.
+
+### `worker_invalidate_all`
+
+```rust
+pub fn worker_invalidate_all() -> u32;
+```
+
+Evict entire cache. Returns count evicted.
+
+## `worker_recall` -- Progressive Context Expansion
+
+```rust
+pub fn worker_recall(
+   params_ptr: *const u8,
+   params_len: u32,
+) -> u64;
+```
+
+WASM export for the `recall` IPC method. Accepts
+a serialized `RecallParams` (kbid or kbids array,
+depth 0--3). Returns serialized `RecallResult`
+(single) or `RecallResult[]` (batch) via the shared
+memory return slot.
+
+### Recall Flow
+
+```
+1. Receive kbid(s) + depth
+         |
+2. For each kbid: read section from cache
+   (disk fallback)
+   -> SectionRecord with heading, type, docids,
+      content
+         |
+3. If depth == 0: return section data
+         |
+4. If depth >= 1: for each docid in section's
+   docids, read document manifest -> list of all
+   section kbids with headings and types.
+   Query references.idx for back-references
+   (entries where target_id == kbid and
+   direction == back).
+         |
+5. If depth >= 2: read full content of sibling
+   sections (other sections in the same documents).
+   Query references.idx for forward references
+   (entries where source_kbid == kbid and
+   direction == forward).
+         |
+6. If depth >= 3: for each forward-referenced
+   section, read its full content. Query
+   references.idx for their back-references.
+         |
+   Return RecallResult(s) via shared memory
+```
+
+**Cache interaction:** Recall is read-heavy and
+benefits from the LRU cache. Section reads at
+depth 0 hit the cache populated by recent searches.
+Sibling and reference reads at depth 2--3 may cause
+cache misses on first access.
+
+**Reference graph lookup:** Forward and back
+references are read from `references.idx` via
+`fs-database.wasm` index lookup functions. This is
+O(1) per kbid via the B+ tree index -- no content
+re-parsing needed.
+
+---
+
+## Retrieve Flow
+
+```
+1. Receive KbId
+         ↓
+2. Check cache  →  hit?  →  return cached
+         ↓ miss
+3. db_read_section  →  SectionRecord from disk
+         ↓
+4. Put into cache
+         ↓
+   Return SectionRecord
+```
+
+Document retrieval follows the same pattern but calls
+`db_read_document` and caches the manifest separately
+from individual sections.
+
+## Maintenance Exports
+
+### `worker_check_integrity`
+
+```rust
+pub fn worker_check_integrity() -> u64;
+```
+
+Delegates to `db_check_integrity`. Returns Encode'd
+`IntegrityReport`.
+
+### `worker_garbage_collect`
+
+```rust
+pub fn worker_garbage_collect() -> u64;
+```
+
+Delegates to `db_garbage_collect`. Returns count of
+removed orphans.
+
+### `worker_rebuild_indexes`
+
+```rust
+pub fn worker_rebuild_indexes() -> u64;
+```
+
+Delegates to `db_index_rebuild`. Invalidates entire
+cache afterward.

package/docs/modules/rust/kb-worker/functions/search.md

@@ -0,0 +1,173 @@
+# functions::search -- Search Orchestration
+
+*End-to-end search flow: query string in, ranked
+results out.*
+
+## Source
+
+`engine/crates/kb-worker/src/orchestrate/search.rs`
+
+## Exported WASM Functions
+
+### `worker_search`
+
+```rust
+pub fn worker_search(
+   query_ptr: *const u8,
+   query_len: u32,
+   limit: u32,
+) -> u64;
+```
+
+Synchronous search. Returns `(ptr, len)` packed into
+`u64`. The result buffer contains Encode'd ranked
+matches.
+
+### `worker_query`
+
+```rust
+pub fn worker_query(
+   query_ptr: *const u8,
+   query_len: u32,
+   limit: u32,
+) -> u64;
+```
+
+Async search. Returns a `(ptr, len)` token string.
+Poll with `worker_result_status`, fetch with
+`worker_get_result`.
+
+## Search Flow (13 Steps)
+
+```
+1. Receive query string from TypeScript IPC
+         ↓
+2. qp_tokenize  →  token stream
+         ↓
+3. qp_parse_query  →  QueryPlan
+         ↓
+4. db_execute_plan  →  Vec<RawMatch>
+         ↓
+5. db_corpus_stats  →  CorpusStats
+         ↓
+6. qp_score_bm25f  →  base scores
+         ↓
+7. qp_score_proximity  →  proximity boost
+         ↓
+8. qp_score_zone  →  zone/field boost
+         ↓
+9. Sort by combined score descending
+         ↓
+10. Apply offset: skip first `offset` entries
+         ↓
+11. Truncate to `limit`
+         ↓
+12. Shape results: for each scored section, read
+    heading, type, and docids from the section
+    header; generate KWIC snippet (~150 chars
+    centered on the highest-scoring matched term
+    position from the positional index)
+         ↓
+13. Build paged envelope: wrap shaped items with
+    `total` (pre-offset/limit count), `offset`,
+    `limit`, `has_more`
+         ↓
+    Return PagedSearchResult via shared memory
+```
+
+### Step Details
+
+1. **Receive query** -- TypeScript writes the UTF-8
+   query into shared memory and calls
+   `worker_search(ptr, len, limit)`.
+
+2. **Tokenize** -- Calls `qp_tokenize` from
+   query-parser. Produces a `TokenStream` with
+   normalized, stemmed tokens.
+
+3. **Parse** -- Calls `qp_parse_query`. Interprets
+   boolean operators, phrase markers, field
+   qualifiers. Produces a `QueryPlan`.
+
+4. **Execute plan** -- Calls `db_execute_plan` from
+   fs-database. Walks the inverted index and
+   positional index. Returns `Vec<RawMatch>` with
+   per-field TFs (`field_tfs`), positions,
+   `token_count`, and `field_token_counts` per
+   matching section.
+
+5. **Corpus stats** -- Calls `db_corpus_stats`. Gets
+   document count, term frequencies for IDF
+   computation, and per-field average lengths
+   (`avg_heading_length`, `avg_body_length`,
+   `avg_code_length`) for BM25F normalization.
+
+6. **BM25F** -- Calls `qp_score_bm25f`. Computes
+   per-field BM25 using `FieldTf` values from each
+   `RawMatch` and per-field corpus averages. Field
+   weights (heading 2.0x, body 1.0x, code 1.5x) are
+   applied at query time. See
+   [Query Result](../../../../goals/query-result.md)
+   for the full scoring specification.
+
+7. **Proximity** -- Calls `qp_score_proximity`.
+   Boosts documents where query terms appear close
+   together.
+
+8. **Zone scoring** -- Calls `qp_score_zone`. Boosts
+   matches in high-value zones (title, headings,
+   first paragraph).
+
+9. **Sort** -- Combines all scores, sorts descending.
+
+10. **Apply offset** -- Skips the first `offset`
+    entries from the sorted result list. Enables
+    cursor-based pagination without re-executing the
+    query.
+
+11. **Truncate** -- Takes at most `limit` entries
+    from the offset-adjusted list.
+
+12. **Shape results** -- For each scored section,
+    reads heading, type, and docids from the section
+    header. Generates a KWIC snippet (~150 chars
+    centered on the highest-scoring matched term
+    position from the positional index). See
+    [Snippet Generation](#snippet-generation) below.
+
+13. **Build paged envelope** -- Wraps shaped items
+    with `total` (pre-offset/limit count), `offset`,
+    `limit`, and `has_more`. Encodes as
+    `PagedSearchResult` via the `Encode` trait into
+    shared memory.
+
+---
+
+### Snippet Generation
+
+Snippets are generated during step 12 (result
+shaping), using data already available from the
+search pipeline:
+
+- The positional index provides the token offset
+  of each matched term in each section.
+- The section content is read from cache (or disk
+  fallback).
+
+**Algorithm:**
+
+1. For each matched section, find the position of
+   the highest-scoring matched term (the term with
+   the highest BM25F contribution).
+2. Convert the token position to a byte offset in
+   the section content.
+3. Extract ~150 characters centered on that byte
+   offset. If the offset is near the start or end,
+   shift the window accordingly.
+4. Trim to word boundaries and add `...` ellipsis
+   at truncated boundaries.
+5. If the highest-scoring match is in the heading
+   (not body), use the first ~150 characters of
+   the body content instead.
+6. For image sections (`image/*`) or sections with
+   no body content, the snippet is empty string.

package/docs/modules/rust/kb-worker/functions/write.md

@@ -0,0 +1,172 @@
+# functions::write -- Write Orchestration
+
+*Add, update, and remove operations for text, code,
+and image sections.*
+
+## Source
+
+`engine/crates/kb-worker/src/orchestrate/write.rs`
+
+## Exported WASM Functions
+
+### `worker_add_section`
+
+```rust
+pub fn worker_add_section(
+   params_ptr: *const u8,
+   params_len: u32,
+) -> u64;
+```
+
+Params buffer contains Encode'd `AddSectionParams`.
+Returns `(ptr, len)` with the new `KbId`.
+
+### `worker_update_section`
+
+```rust
+pub fn worker_update_section(
+   params_ptr: *const u8,
+   params_len: u32,
+) -> u64;
+```
+
+### `worker_remove_section`
+
+```rust
+pub fn worker_remove_section(
+   kbid_ptr: *const u8,
+   kbid_len: u32,
+) -> u64;
+```
+
+### `worker_group_sections`
+
+```rust
+pub fn worker_group_sections(
+   params_ptr: *const u8,
+   params_len: u32,
+) -> u64;
+```
+
+Creates a document manifest grouping multiple
+sections.
+
+### `worker_remove_grouping`
+
+```rust
+pub fn worker_remove_grouping(
+   doc_id_ptr: *const u8,
+   doc_id_len: u32,
+) -> u64;
+```
+
+## Add Flow (Text)
+
+```
+1. Receive content + metadata via shared memory
+         ↓
+2. Per-field tokenization:
+   - title → qp_tokenize → heading tokens
+   - body + description → qp_tokenize → body tokens
+         ↓
+3. qp_extract_keywords  →  Vec<Keyphrase>
+         ↓
+4. qp_compute_field_tf  →  per-field TFs + token counts
+         ↓
+5. db_add_section  →  KbId (content-addressed)
+   (includes title, description, and 4 token counts
+    in section header)
+         ↓
+6. db_index_add  →  update inverted (per-field TFs)
+   + positional index
+         ↓
+7. Invalidate cache entries for affected terms
+         ↓
+   Return AddSectionResult
+```
+
+## Add Flow (Code)
+
+```
+1. Receive source code + language tag
+         ↓
+2. Per-field tokenization:
+   - title → qp_tokenize → heading tokens
+   - description → qp_tokenize → body tokens
+   - code content → qp_split_identifiers
+     + qp_tokenize → code tokens
+         ↓
+3. Merge identifier tokens + content tokens
+         ↓
+4. qp_extract_keywords  →  Vec<Keyphrase>
+         ↓
+5. qp_compute_field_tf  →  per-field TFs + token counts
+         ↓
+6-8. Same as text flow (store, index, cache)
+```
+
+Code sections use `qp_split_identifiers` to split
+camelCase and snake_case identifiers into searchable
+terms before merging with tokens from comments and
+string literals. Code content feeds the code field;
+title feeds the heading field; description feeds the
+body field.
+
+## Add Flow (Image)
+
+```
+1. Receive image binary + title (required)
+   + description (optional)
+         ↓
+2. Per-field tokenization:
+   - title → qp_tokenize → heading tokens (2.0x)
+   - description → qp_tokenize → body tokens (1.0x)
+         ↓
+3. qp_compute_field_tf  →  per-field TFs + token counts
+         ↓
+4. db_add_section  →  KbId (hash of binary)
+   (includes title, description, and 4 token counts
+    in section header)
+         ↓
+5. db_index_add  →  index title/description terms
+   with per-field TFs
+         ↓
+   Return AddSectionResult
+```
+
+Image binary is stored as-is and not indexed.
+Title (required) is tokenized into the heading
+field and description (optional) into the body
+field. Validation: image sections require a title.
+
+## Update Flow
+
+```
+1. Receive KbId + new content
+         ↓
+2. db_index_remove  →  remove old index entries
+         ↓
+3. Re-run the appropriate add flow (text/code/image)
+         ↓
+4. db_update_section  →  overwrite content
+         ↓
+5. Invalidate cache for old + new terms
+```
+
+Update is delete-old-indexes + re-add. The KbId may
+change if content changes (content-addressed), so the
+caller receives the new KbId.
+
+## Remove Flow
+
+```
+1. Receive KbId
+         ↓
+2. db_index_remove  →  remove all index entries
+         ↓
+3. db_remove_section  →  delete content file
+         ↓
+4. Invalidate cache for removed KbId
+         ↓
+   Return success/failure
+```

package/docs/modules/rust/kb-worker/module.md

@@ -0,0 +1,171 @@
+# kb-worker -- Worker Orchestration and Cache
+
+*Rust crate that imports query-parser and fs-database,
+orchestrates all database operations, and manages the
+in-memory LRU cache. Compiles to `kb-worker.wasm`.*
+
+## Crate
+
+| Field       | Value                              |
+|-------------|------------------------------------|
+| Name        | `kbdb-worker`                      |
+| Path        | `engine/crates/kb-worker/`         |
+| Crate type  | `cdylib` (WASM) + `rlib` (tests)  |
+| WASM output | `dist/wasm/kb-worker.wasm`         |
+| Depends on  | `kbdb-shared`, `kbdb-query-parser`, `kbdb-fs-database` |
+
+`kbdb-worker` is the **top-level** WASM module. It
+imports both `query-parser.wasm` and
+`fs-database.wasm` and sits between the TypeScript
+IPC layer and the two standalone modules.
+
+```
+query-parser.wasm    fs-database.wasm
+(standalone)         (standalone)
+      ↑                    ↑
+      └────────┬───────────┘
+               |
+        kb-worker.wasm
+   (imports both, orchestrates)
+               ↑
+               |
+        src/worker.ts
+   (TypeScript daemon, loads this module)
+```
+
+See [Worker Daemon](../../../goals/worker-daemon.md)
+for the full behavioral specification.
+
+## Workspace Position
+
+```
+engine/
+├── Cargo.toml                   # workspace root
+└── crates/
+    ├── shared/                  # kbdb-shared
+    ├── fs-database/             # kbdb-fs-database
+    ├── query-parser/            # kbdb-query-parser
+    └── kb-worker/               # this crate
+```
+
+## Directory Layout
+
+```
+engine/crates/kb-worker/
+├── Cargo.toml
+└── src/
+    ├── lib.rs                   # crate root, WASM exports
+    ├── mem.rs                   # shared-memory helpers
+    ├── context.rs               # database context init
+    ├── orchestrate/
+    │   ├── mod.rs               # dispatch facade
+    │   ├── search.rs            # search flow
+    │   ├── write.rs             # write flow
+    │   └── retrieve.rs          # read flow
+    ├── cache/
+    │   ├── mod.rs               # cache facade
+    │   ├── lru.rs               # LRU data structure
+    │   ├── entry.rs             # cache entry with TTL
+    │   └── config.rs            # cache configuration
+    ├── token.rs                 # result token management
+    └── status.rs                # status reporting
+```
+
+## Modules
+
+### [types/](types/module.md) -- Structs
+
+- **[context](types/context.md)** -- `Context` struct.
+  Database context initialization.
+- **[cache](types/cache.md)** -- `Cache`,
+  `CacheConfig`, `CacheEntry`, `CacheStats`,
+  `LruCache`.
+- **[token](types/token.md)** -- `TokenStore`,
+  `TokenEntry`, `TokenStatus`. Async result token
+  management.
+- **[status](types/status.md)** -- `StatusResult`.
+
+### [functions/](functions/module.md) -- Orchestration
+
+- **[search](functions/search.md)** -- Search flow:
+  parse → lookup → rank.
+- **[write](functions/write.md)** -- Write flow:
+  tokenize → store → index (text/code/image).
+- **[retrieve](functions/retrieve.md)** -- Read flow:
+  cache check → disk read. Status reporting.
+- **[compose](functions/compose.md)** -- Content
+  composition flow: resolve → fetch → compose.
+
+## WASM Exports
+
+| Export                    | Description              |
+|---------------------------|--------------------------|
+| `worker_init`             | Initialize context       |
+| `worker_search`           | Full search flow         |
+| `worker_add_section`      | Full add flow            |
+| `worker_update_section`   | Full update flow         |
+| `worker_remove_section`   | Full remove flow         |
+| `worker_read_sections`    | Read sections by kb-id   |
+| `worker_retrieve_doc`     | Read document + sections |
+| `worker_group_sections`   | Create document grouping |
+| `worker_remove_grouping`  | Remove document grouping |
+| `worker_status`           | Database + cache status  |
+| `worker_query`            | Async query (returns token) |
+| `worker_result_status`    | Poll token status        |
+| `worker_get_result`       | Fetch result by token    |
+| `worker_invalidate`       | Evict cache entries      |
+| `worker_invalidate_all`   | Evict entire cache       |
+| `worker_compose`          | Content composition flow |
+| `worker_check_integrity`  | Run integrity checks     |
+| `worker_garbage_collect`  | Remove orphaned sections |
+| `worker_rebuild_indexes`  | Full index rebuild       |
+| `worker_recall`           | Progressive context expansion (recall) |
+| `alloc` / `free`          | Memory management (from `kbdb-shared`) |
+
+## WASM Imports
+
+**From `query-parser.wasm`:** `qp_tokenize`,
+`qp_parse_query`, `qp_extract_keywords`,
+`qp_compute_tf`, `qp_compute_field_tf`,
+`qp_compute_tfidf`,
+`qp_score_bm25f`, `qp_score_proximity`,
+`qp_score_zone`, `qp_score_freshness`,
+`qp_split_identifiers`.
+
+**From `fs-database.wasm`:** `db_init`,
+`db_read_section`, `db_read_sections`,
+`db_add_section`, `db_update_section`,
+`db_remove_section`, `db_read_document`,
+`db_write_document`, `db_remove_document`,
+`db_update_section_ref`, `db_index_add`,
+`db_index_remove`, `db_index_rebuild`,
+`db_execute_plan`, `db_corpus_stats`, `db_status`,
+`db_check_integrity`, `db_garbage_collect`.
+
+## Dependencies
+
+```toml
+[dependencies]
+kbdb-shared = { path = "../shared" }
+kbdb-query-parser = { path = "../query-parser" }
+kbdb-fs-database = { path = "../fs-database" }
+```
+
+## Build
+
+```sh
+cd engine
+wasm-pack build crates/kb-worker \
+   --target no-modules \
+   --out-dir ../../../dist/wasm/kb-worker
+```
+
+Build order: `kbdb-shared` → `kbdb-query-parser` +
+`kbdb-fs-database` (parallel) → `kbdb-worker`.
+
+## Testing
+
+```sh
+cd engine && cargo test -p kbdb-worker
+cd engine && wasm-pack test --node crates/kb-worker
+```