@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/modules/rust/kb-worker/functions/write.md

@@ -1,172 +0,0 @@
-# functions::write -- Write Orchestration
-
-*Add, update, and remove operations for text, code,
-and image sections.*
-
-## Source
-
-`engine/crates/kb-worker/src/orchestrate/write.rs`
-
-## Exported WASM Functions
-
-### `worker_add_section`
-
-```rust
-pub fn worker_add_section(
-   params_ptr: *const u8,
-   params_len: u32,
-) -> u64;
-```
-
-Params buffer contains Encode'd `AddSectionParams`.
-Returns `(ptr, len)` with the new `KbId`.
-
-### `worker_update_section`
-
-```rust
-pub fn worker_update_section(
-   params_ptr: *const u8,
-   params_len: u32,
-) -> u64;
-```
-
-### `worker_remove_section`
-
-```rust
-pub fn worker_remove_section(
-   kbid_ptr: *const u8,
-   kbid_len: u32,
-) -> u64;
-```
-
-### `worker_group_sections`
-
-```rust
-pub fn worker_group_sections(
-   params_ptr: *const u8,
-   params_len: u32,
-) -> u64;
-```
-
-Creates a document manifest grouping multiple
-sections.
-
-### `worker_remove_grouping`
-
-```rust
-pub fn worker_remove_grouping(
-   doc_id_ptr: *const u8,
-   doc_id_len: u32,
-) -> u64;
-```
-
-## Add Flow (Text)
-
-```
-1. Receive content + metadata via shared memory
-         ↓
-2. Per-field tokenization:
-   - title → qp_tokenize → heading tokens
-   - body + description → qp_tokenize → body tokens
-         ↓
-3. qp_extract_keywords  →  Vec<Keyphrase>
-         ↓
-4. qp_compute_field_tf  →  per-field TFs + token counts
-         ↓
-5. db_add_section  →  KbId (content-addressed)
-   (includes title, description, and 4 token counts
-    in section header)
-         ↓
-6. db_index_add  →  update inverted (per-field TFs)
-   + positional index
-         ↓
-7. Invalidate cache entries for affected terms
-         ↓
-   Return AddSectionResult
-```
-
-## Add Flow (Code)
-
-```
-1. Receive source code + language tag
-         ↓
-2. Per-field tokenization:
-   - title → qp_tokenize → heading tokens
-   - description → qp_tokenize → body tokens
-   - code content → qp_split_identifiers
-     + qp_tokenize → code tokens
-         ↓
-3. Merge identifier tokens + content tokens
-         ↓
-4. qp_extract_keywords  →  Vec<Keyphrase>
-         ↓
-5. qp_compute_field_tf  →  per-field TFs + token counts
-         ↓
-6-8. Same as text flow (store, index, cache)
-```
-
-Code sections use `qp_split_identifiers` to split
-camelCase and snake_case identifiers into searchable
-terms before merging with tokens from comments and
-string literals. Code content feeds the code field;
-title feeds the heading field; description feeds the
-body field.
-
-## Add Flow (Image)
-
-```
-1. Receive image binary + title (required)
-   + description (optional)
-         ↓
-2. Per-field tokenization:
-   - title → qp_tokenize → heading tokens (2.0x)
-   - description → qp_tokenize → body tokens (1.0x)
-         ↓
-3. qp_compute_field_tf  →  per-field TFs + token counts
-         ↓
-4. db_add_section  →  KbId (hash of binary)
-   (includes title, description, and 4 token counts
-    in section header)
-         ↓
-5. db_index_add  →  index title/description terms
-   with per-field TFs
-         ↓
-   Return AddSectionResult
-```
-
-Image binary is stored as-is and not indexed.
-Title (required) is tokenized into the heading
-field and description (optional) into the body
-field. Validation: image sections require a title.
-
-## Update Flow
-
-```
-1. Receive KbId + new content
-         ↓
-2. db_index_remove  →  remove old index entries
-         ↓
-3. Re-run the appropriate add flow (text/code/image)
-         ↓
-4. db_update_section  →  overwrite content
-         ↓
-5. Invalidate cache for old + new terms
-```
-
-Update is delete-old-indexes + re-add. The KbId may
-change if content changes (content-addressed), so the
-caller receives the new KbId.
-
-## Remove Flow
-
-```
-1. Receive KbId
-         ↓
-2. db_index_remove  →  remove all index entries
-         ↓
-3. db_remove_section  →  delete content file
-         ↓
-4. Invalidate cache for removed KbId
-         ↓
-   Return success/failure
-```

package/docs/modules/rust/kb-worker/module.md

@@ -1,171 +0,0 @@
-# kb-worker -- Worker Orchestration and Cache
-
-*Rust crate that imports query-parser and fs-database,
-orchestrates all database operations, and manages the
-in-memory LRU cache. Compiles to `kb-worker.wasm`.*
-
-## Crate
-
-| Field       | Value                              |
-|-------------|------------------------------------|
-| Name        | `kbdb-worker`                      |
-| Path        | `engine/crates/kb-worker/`         |
-| Crate type  | `cdylib` (WASM) + `rlib` (tests)  |
-| WASM output | `dist/wasm/kb-worker.wasm`         |
-| Depends on  | `kbdb-shared`, `kbdb-query-parser`, `kbdb-fs-database` |
-
-`kbdb-worker` is the **top-level** WASM module. It
-imports both `query-parser.wasm` and
-`fs-database.wasm` and sits between the TypeScript
-IPC layer and the two standalone modules.
-
-```
-query-parser.wasm    fs-database.wasm
-(standalone)         (standalone)
-      ↑                    ↑
-      └────────┬───────────┘
-               |
-        kb-worker.wasm
-   (imports both, orchestrates)
-               ↑
-               |
-        src/worker.ts
-   (TypeScript daemon, loads this module)
-```
-
-See [Worker Daemon](../../../goals/worker-daemon.md)
-for the full behavioral specification.
-
-## Workspace Position
-
-```
-engine/
-├── Cargo.toml                   # workspace root
-└── crates/
-    ├── shared/                  # kbdb-shared
-    ├── fs-database/             # kbdb-fs-database
-    ├── query-parser/            # kbdb-query-parser
-    └── kb-worker/               # this crate
-```
-
-## Directory Layout
-
-```
-engine/crates/kb-worker/
-├── Cargo.toml
-└── src/
-    ├── lib.rs                   # crate root, WASM exports
-    ├── mem.rs                   # shared-memory helpers
-    ├── context.rs               # database context init
-    ├── orchestrate/
-    │   ├── mod.rs               # dispatch facade
-    │   ├── search.rs            # search flow
-    │   ├── write.rs             # write flow
-    │   └── retrieve.rs          # read flow
-    ├── cache/
-    │   ├── mod.rs               # cache facade
-    │   ├── lru.rs               # LRU data structure
-    │   ├── entry.rs             # cache entry with TTL
-    │   └── config.rs            # cache configuration
-    ├── token.rs                 # result token management
-    └── status.rs                # status reporting
-```
-
-## Modules
-
-### [types/](types/module.md) -- Structs
-
-- **[context](types/context.md)** -- `Context` struct.
-  Database context initialization.
-- **[cache](types/cache.md)** -- `Cache`,
-  `CacheConfig`, `CacheEntry`, `CacheStats`,
-  `LruCache`.
-- **[token](types/token.md)** -- `TokenStore`,
-  `TokenEntry`, `TokenStatus`. Async result token
-  management.
-- **[status](types/status.md)** -- `StatusResult`.
-
-### [functions/](functions/module.md) -- Orchestration
-
-- **[search](functions/search.md)** -- Search flow:
-  parse → lookup → rank.
-- **[write](functions/write.md)** -- Write flow:
-  tokenize → store → index (text/code/image).
-- **[retrieve](functions/retrieve.md)** -- Read flow:
-  cache check → disk read. Status reporting.
-- **[compose](functions/compose.md)** -- Content
-  composition flow: resolve → fetch → compose.
-
-## WASM Exports
-
-| Export                    | Description              |
-|---------------------------|--------------------------|
-| `worker_init`             | Initialize context       |
-| `worker_search`           | Full search flow         |
-| `worker_add_section`      | Full add flow            |
-| `worker_update_section`   | Full update flow         |
-| `worker_remove_section`   | Full remove flow         |
-| `worker_read_sections`    | Read sections by kb-id   |
-| `worker_retrieve_doc`     | Read document + sections |
-| `worker_group_sections`   | Create document grouping |
-| `worker_remove_grouping`  | Remove document grouping |
-| `worker_status`           | Database + cache status  |
-| `worker_query`            | Async query (returns token) |
-| `worker_result_status`    | Poll token status        |
-| `worker_get_result`       | Fetch result by token    |
-| `worker_invalidate`       | Evict cache entries      |
-| `worker_invalidate_all`   | Evict entire cache       |
-| `worker_compose`          | Content composition flow |
-| `worker_check_integrity`  | Run integrity checks     |
-| `worker_garbage_collect`  | Remove orphaned sections |
-| `worker_rebuild_indexes`  | Full index rebuild       |
-| `worker_recall`           | Progressive context expansion (recall) |
-| `alloc` / `free`          | Memory management (from `kbdb-shared`) |
-
-## WASM Imports
-
-**From `query-parser.wasm`:** `qp_tokenize`,
-`qp_parse_query`, `qp_extract_keywords`,
-`qp_compute_tf`, `qp_compute_field_tf`,
-`qp_compute_tfidf`,
-`qp_score_bm25f`, `qp_score_proximity`,
-`qp_score_zone`, `qp_score_freshness`,
-`qp_split_identifiers`.
-
-**From `fs-database.wasm`:** `db_init`,
-`db_read_section`, `db_read_sections`,
-`db_add_section`, `db_update_section`,
-`db_remove_section`, `db_read_document`,
-`db_write_document`, `db_remove_document`,
-`db_update_section_ref`, `db_index_add`,
-`db_index_remove`, `db_index_rebuild`,
-`db_execute_plan`, `db_corpus_stats`, `db_status`,
-`db_check_integrity`, `db_garbage_collect`.
-
-## Dependencies
-
-```toml
-[dependencies]
-kbdb-shared = { path = "../shared" }
-kbdb-query-parser = { path = "../query-parser" }
-kbdb-fs-database = { path = "../fs-database" }
-```
-
-## Build
-
-```sh
-cd engine
-wasm-pack build crates/kb-worker \
-   --target no-modules \
-   --out-dir ../../../dist/wasm/kb-worker
-```
-
-Build order: `kbdb-shared` → `kbdb-query-parser` +
-`kbdb-fs-database` (parallel) → `kbdb-worker`.
-
-## Testing
-
-```sh
-cd engine && cargo test -p kbdb-worker
-cd engine && wasm-pack test --node crates/kb-worker
-```

package/docs/modules/rust/kb-worker/types/cache.md

@@ -1,90 +0,0 @@
-# types::cache -- LRU Cache with TTL
-
-*In-memory cache between IPC request handling and
-disk I/O. Configuration from `worker.toml`.*
-
-## Source
-
-`engine/crates/kb-worker/src/cache/`
-
-## Type Definitions
-
-### `CacheConfig`
-
-```rust
-pub struct CacheConfig {
-   pub item_limit: u32,
-   pub index_limit: u32,
-   pub item_ttl_ms: u64,
-}
-```
-
-Defaults: `item_limit = 1000`,
-`index_limit = 100`, `item_ttl_ms = 60000`.
-
-### `CacheEntry<V>`
-
-```rust
-pub struct CacheEntry<V> {
-   pub value: V,
-   pub created_at: u64,
-   pub last_accessed: u64,
-   pub ttl_ms: u64,
-}
-
-impl<V> CacheEntry<V> {
-   pub fn is_expired(&self, now: u64) -> bool;
-}
-```
-
-### `LruCache<K, V>`
-
-```rust
-pub struct LruCache<K, V> { /* doubly-linked list + hashmap */ }
-
-impl<K: Hash + Eq, V> LruCache<K, V> {
-   pub fn new(capacity: u32) -> Self;
-   pub fn get(&mut self, key: &K) -> Option<&V>;
-   pub fn put(&mut self, key: K, value: V);
-   pub fn remove(&mut self, key: &K) -> bool;
-   pub fn clear(&mut self);
-   pub fn len(&self) -> u32;
-   pub fn stats(&self) -> CacheStats;
-}
-
-pub struct CacheStats {
-   pub items_cached: u32,
-   pub hits: u64,
-   pub misses: u64,
-   pub hit_rate: f32,
-}
-```
-
-O(1) access and O(1) eviction.
-
-### `Cache`
-
-```rust
-pub struct Cache {
-   content: LruCache<KbId, SectionRecord>,
-   indexes: LruCache<Term, PostingList>,
-   manifests: LruCache<String, DocumentManifest>,
-   config: CacheConfig,
-}
-
-impl Cache {
-   pub fn new(config: CacheConfig) -> Self;
-   pub fn get_section(&mut self, kbid: &KbId)
-      -> Option<&SectionRecord>;
-   pub fn put_section(
-      &mut self, kbid: KbId, record: SectionRecord,
-   );
-   pub fn invalidate(&mut self, keys: &[KbId])
-      -> u32;
-   pub fn invalidate_all(&mut self) -> u32;
-   pub fn stats(&self) -> CacheStats;
-}
-```
-
-Separate LRU instances for content objects, index
-segments, and manifests, each with its own capacity.

package/docs/modules/rust/kb-worker/types/context.md

@@ -1,26 +0,0 @@
-# types::context -- Database Context
-
-*The `Context` struct initialized at daemon startup.*
-
-## Source
-
-`engine/crates/kb-worker/src/context.rs`
-
-## Type Definition
-
-```rust
-pub struct Context {
-   pub ctx_id: String,
-   pub target_dir: String,
-   pub catalog: Catalog,
-}
-
-pub fn init_context(
-   target_dir: &str,
-) -> Result<Context, WorkerError>;
-```
-
-`init_context` calls `db_init` from
-`kbdb-fs-database` to validate the database
-directory and load the catalog. The context is held
-for the daemon's lifetime.

package/docs/modules/rust/kb-worker/types/module.md

@@ -1,16 +0,0 @@
-# kb-worker/types -- Type Definitions
-
-*Structs defined in the kb-worker crate for context,
-caching, result tokens, and status reporting.*
-
-## Modules
-
-- **[context](context.md)** -- `Context` struct.
-  Database context with catalog data.
-- **[cache](cache.md)** -- `Cache`, `CacheConfig`,
-  `CacheEntry`, `CacheStats`, `LruCache`. In-memory
-  LRU cache with TTL.
-- **[token](token.md)** -- `TokenStore`, `TokenEntry`,
-  `TokenStatus`. Async result token management.
-- **[status](status.md)** -- `StatusResult`. Combined
-  database and cache statistics.

package/docs/modules/rust/kb-worker/types/status.md

@@ -1,29 +0,0 @@
-# types::status -- Status Reporting
-
-*Combined database and cache statistics returned by
-`worker_status`.*
-
-## Source
-
-`engine/crates/kb-worker/src/status.rs`
-
-## Type Definition
-
-```rust
-pub struct StatusResult {
-   pub target_dir: String,
-   pub document_count: u32,
-   pub section_count: u32,
-   pub index_size_bytes: u64,
-   pub last_modified: String,
-   pub cache_stats: CacheStats,
-}
-
-pub fn get_status(
-   ctx: &Context,
-   cache: &Cache,
-) -> Result<StatusResult, WorkerError>;
-```
-
-Combines catalog data from `fs-database.wasm` with
-cache statistics from the local cache.

package/docs/modules/rust/kb-worker/types/token.md

@@ -1,50 +0,0 @@
-# types::token -- Result Token Management
-
-*Async result tokens for the submit → poll → fetch
-query flow.*
-
-## Source
-
-`engine/crates/kb-worker/src/token.rs`
-
-## Type Definitions
-
-```rust
-pub struct TokenStore {
-   tokens: HashMap<String, TokenEntry>,
-}
-
-pub struct TokenEntry {
-   pub status: TokenStatus,
-   pub data: Option<Vec<u8>>,
-   pub created_at: u64,
-}
-
-pub enum TokenStatus {
-   Busy,
-   Success,
-   Fail(String),
-}
-```
-
-## Methods
-
-```rust
-impl TokenStore {
-   pub fn create(&mut self) -> String;
-   pub fn complete(
-      &mut self, token: &str, data: Vec<u8>,
-   );
-   pub fn fail(
-      &mut self, token: &str, reason: String,
-   );
-   pub fn status(&self, token: &str)
-      -> Option<&TokenStatus>;
-   pub fn take_data(&mut self, token: &str)
-      -> Option<Vec<u8>>;
-}
-```
-
-Tokens are random 16-character hex strings. Tied to
-cached data -- when the cached entry is evicted, the
-token becomes invalid and returns `null` on fetch.