@dikolab/kbdb 0.1.6 → 0.2.0

package/docs/modules/rust/kb-worker/types/cache.md

@@ -0,0 +1,90 @@
+# types::cache -- LRU Cache with TTL
+
+*In-memory cache between IPC request handling and
+disk I/O. Configuration from `worker.toml`.*
+
+## Source
+
+`engine/crates/kb-worker/src/cache/`
+
+## Type Definitions
+
+### `CacheConfig`
+
+```rust
+pub struct CacheConfig {
+   pub item_limit: u32,
+   pub index_limit: u32,
+   pub item_ttl_ms: u64,
+}
+```
+
+Defaults: `item_limit = 1000`,
+`index_limit = 100`, `item_ttl_ms = 60000`.
+
+### `CacheEntry<V>`
+
+```rust
+pub struct CacheEntry<V> {
+   pub value: V,
+   pub created_at: u64,
+   pub last_accessed: u64,
+   pub ttl_ms: u64,
+}
+
+impl<V> CacheEntry<V> {
+   pub fn is_expired(&self, now: u64) -> bool;
+}
+```
+
+### `LruCache<K, V>`
+
+```rust
+pub struct LruCache<K, V> { /* doubly-linked list + hashmap */ }
+
+impl<K: Hash + Eq, V> LruCache<K, V> {
+   pub fn new(capacity: u32) -> Self;
+   pub fn get(&mut self, key: &K) -> Option<&V>;
+   pub fn put(&mut self, key: K, value: V);
+   pub fn remove(&mut self, key: &K) -> bool;
+   pub fn clear(&mut self);
+   pub fn len(&self) -> u32;
+   pub fn stats(&self) -> CacheStats;
+}
+
+pub struct CacheStats {
+   pub items_cached: u32,
+   pub hits: u64,
+   pub misses: u64,
+   pub hit_rate: f32,
+}
+```
+
+O(1) access and O(1) eviction.
+
+### `Cache`
+
+```rust
+pub struct Cache {
+   content: LruCache<KbId, SectionRecord>,
+   indexes: LruCache<Term, PostingList>,
+   manifests: LruCache<String, DocumentManifest>,
+   config: CacheConfig,
+}
+
+impl Cache {
+   pub fn new(config: CacheConfig) -> Self;
+   pub fn get_section(&mut self, kbid: &KbId)
+      -> Option<&SectionRecord>;
+   pub fn put_section(
+      &mut self, kbid: KbId, record: SectionRecord,
+   );
+   pub fn invalidate(&mut self, keys: &[KbId])
+      -> u32;
+   pub fn invalidate_all(&mut self) -> u32;
+   pub fn stats(&self) -> CacheStats;
+}
+```
+
+Separate LRU instances for content objects, index
+segments, and manifests, each with its own capacity.

package/docs/modules/rust/kb-worker/types/context.md

@@ -0,0 +1,26 @@
+# types::context -- Database Context
+
+*The `Context` struct initialized at daemon startup.*
+
+## Source
+
+`engine/crates/kb-worker/src/context.rs`
+
+## Type Definition
+
+```rust
+pub struct Context {
+   pub ctx_id: String,
+   pub target_dir: String,
+   pub catalog: Catalog,
+}
+
+pub fn init_context(
+   target_dir: &str,
+) -> Result<Context, WorkerError>;
+```
+
+`init_context` calls `db_init` from
+`kbdb-fs-database` to validate the database
+directory and load the catalog. The context is held
+for the daemon's lifetime.

package/docs/modules/rust/kb-worker/types/module.md

@@ -0,0 +1,16 @@
+# kb-worker/types -- Type Definitions
+
+*Structs defined in the kb-worker crate for context,
+caching, result tokens, and status reporting.*
+
+## Modules
+
+- **[context](context.md)** -- `Context` struct.
+  Database context with catalog data.
+- **[cache](cache.md)** -- `Cache`, `CacheConfig`,
+  `CacheEntry`, `CacheStats`, `LruCache`. In-memory
+  LRU cache with TTL.
+- **[token](token.md)** -- `TokenStore`, `TokenEntry`,
+  `TokenStatus`. Async result token management.
+- **[status](status.md)** -- `StatusResult`. Combined
+  database and cache statistics.

package/docs/modules/rust/kb-worker/types/status.md

@@ -0,0 +1,29 @@
+# types::status -- Status Reporting
+
+*Combined database and cache statistics returned by
+`worker_status`.*
+
+## Source
+
+`engine/crates/kb-worker/src/status.rs`
+
+## Type Definition
+
+```rust
+pub struct StatusResult {
+   pub target_dir: String,
+   pub document_count: u32,
+   pub section_count: u32,
+   pub index_size_bytes: u64,
+   pub last_modified: String,
+   pub cache_stats: CacheStats,
+}
+
+pub fn get_status(
+   ctx: &Context,
+   cache: &Cache,
+) -> Result<StatusResult, WorkerError>;
+```
+
+Combines catalog data from `fs-database.wasm` with
+cache statistics from the local cache.

package/docs/modules/rust/kb-worker/types/token.md

@@ -0,0 +1,50 @@
+# types::token -- Result Token Management
+
+*Async result tokens for the submit → poll → fetch
+query flow.*
+
+## Source
+
+`engine/crates/kb-worker/src/token.rs`
+
+## Type Definitions
+
+```rust
+pub struct TokenStore {
+   tokens: HashMap<String, TokenEntry>,
+}
+
+pub struct TokenEntry {
+   pub status: TokenStatus,
+   pub data: Option<Vec<u8>>,
+   pub created_at: u64,
+}
+
+pub enum TokenStatus {
+   Busy,
+   Success,
+   Fail(String),
+}
+```
+
+## Methods
+
+```rust
+impl TokenStore {
+   pub fn create(&mut self) -> String;
+   pub fn complete(
+      &mut self, token: &str, data: Vec<u8>,
+   );
+   pub fn fail(
+      &mut self, token: &str, reason: String,
+   );
+   pub fn status(&self, token: &str)
+      -> Option<&TokenStatus>;
+   pub fn take_data(&mut self, token: &str)
+      -> Option<Vec<u8>>;
+}
+```
+
+Tokens are random 16-character hex strings. Tied to
+cached data -- when the cached entry is evicted, the
+token becomes invalid and returns `null` on fetch.

package/docs/modules/rust/overview.md

@@ -0,0 +1,495 @@
+# Rust Engine -- Overview
+
+*Architecture, crate structure, and module layout
+for the Rust/WASM engine.*
+
+## Compilation Model
+
+Rust compiles by **crate** -- each crate is an
+independent compilation unit with its own
+`Cargo.toml`. Within a crate, code is organized into
+**modules** (namespaces) that compile together as one
+unit.
+
+```
+crate  = compilation unit  (has Cargo.toml)
+module = namespace         (has mod.rs)
+```
+
+A crate can only use code from:
+1. Its own modules (`crate::`)
+2. Dependencies declared in its `Cargo.toml`
+
+There are no path-based imports across crates --
+unlike TypeScript/ESM where any file can import any
+other file. Code sharing requires a crate dependency.
+
+## Workspace
+
+All Rust code lives in `engine/`. The workspace
+contains four crates under `engine/crates/`:
+
+```
+engine/
+├── Cargo.toml                 ← workspace root
+├── Cargo.lock                 ← shared lock file
+├── .cargo/
+│   └── config.toml            ← default: wasm32
+└── crates/
+    ├── shared/                ← rlib (library)
+    ├── fs-database/           ← cdylib (→ .wasm)
+    ├── query-parser/          ← cdylib (→ .wasm)
+    └── kb-worker/             ← cdylib (→ .wasm)
+```
+
+## Crate Types
+
+### `rlib` -- Rust Library
+
+Produces no output artifact. Compiled **into** each
+crate that depends on it. The compiler includes only
+the code each consumer actually uses (dead code
+elimination).
+
+- **`kbdb-shared`** is the only rlib crate.
+
+### `cdylib` -- WebAssembly Module
+
+Produces a standalone `.wasm` file. Each cdylib crate
+is an entry point that compiles to a self-contained
+WASM binary.
+
+- **`kbdb-fs-database`** → `fs-database.wasm`
+- **`kbdb-query-parser`** → `query-parser.wasm`
+- **`kbdb-worker`** → `kb-worker.wasm`
+
+## Crate Dependency Graph
+
+```
+kbdb-shared (rlib)
+    │
+    ├──→ kbdb-fs-database (cdylib)
+    │        produces: fs-database.wasm
+    │
+    ├──→ kbdb-query-parser (cdylib)
+    │        produces: query-parser.wasm
+    │
+    └──→ kbdb-worker (cdylib)
+             produces: kb-worker.wasm
+             also depends on:
+               kbdb-fs-database (Cargo)
+               kbdb-query-parser (Cargo)
+```
+
+### Cargo vs WASM Dependencies
+
+`kbdb-worker` depends on the other two crates at
+**two levels**:
+
+1. **Cargo dependency** -- `kbdb-worker` lists
+   `kbdb-fs-database` and `kbdb-query-parser` in its
+   `Cargo.toml`. This gives it compile-time access to
+   their type definitions and WASM import signatures.
+
+2. **WASM runtime imports** -- at runtime,
+   `kb-worker.wasm` imports functions from
+   `fs-database.wasm` and `query-parser.wasm` through
+   the shared `WebAssembly.Memory` interface.
+
+The standalone modules (`fs-database`, `query-parser`)
+have **no dependency** on each other or on `kb-worker`.
+
+## Build Order
+
+```
+1.  kbdb-shared         (no deps, builds first)
+2.  kbdb-fs-database    (depends on shared)
+    kbdb-query-parser   (depends on shared)
+    ↑ these two build in parallel
+3.  kbdb-worker         (depends on all three)
+```
+
+## WASM Output
+
+```
+dist/wasm/
+├── fs-database/
+│   ├── fs-database.wasm
+│   └── fs-database.js        ← JS glue
+├── query-parser/
+│   ├── query-parser.wasm
+│   └── query-parser.js
+└── kb-worker/
+    ├── kb-worker.wasm
+    └── kb-worker.js
+```
+
+All three modules share one `WebAssembly.Memory`
+instance at runtime. Data crosses module boundaries
+via `(pointer, length)` pairs encoded with the
+`Encode` trait from `kbdb-shared`.
+
+## WASM Size Optimization
+
+Each `.wasm` file is optimized for size:
+
+```toml
+# engine/Cargo.toml [profile.release]
+opt-level = "z"          # optimize for size
+lto = true               # link-time optimization
+codegen-units = 1        # better optimization
+strip = true             # remove debug symbols
+panic = "abort"          # no unwinding (~10KB saved)
+```
+
+The `shared` rlib is dead-code-eliminated per
+consumer -- each `.wasm` only includes the shared
+code it actually references.
+
+---
+
+## Crate Layouts
+
+### kbdb-shared (rlib)
+
+Shared types, traits, memory helpers, and error
+types. Not compiled to WASM on its own -- compiled
+into each consumer.
+
+Namespaces are first-level directories inside `src/`.
+Each namespace owns one concern.
+
+```
+engine/crates/shared/
+├── Cargo.toml
+└── src/
+    ├── lib.rs               ← pub mod declarations
+    │
+    ├── memory/              ← memory management
+    │   ├── mod.rs
+    │   └── alloc.rs
+    │
+    ├── encode/              ← binary serialization
+    │   ├── mod.rs
+    │   └── encode.rs
+    │
+    ├── kbid/                ← content-addressed ID
+    │   ├── mod.rs
+    │   └── kbid.rs
+    │
+    ├── term/                ← search term newtype
+    │   ├── mod.rs
+    │   └── term.rs
+    │
+    ├── section/             ← section types & records
+    │   ├── mod.rs
+    │   └── section.rs
+    │
+    ├── document/            ← document manifests
+    │   ├── mod.rs
+    │   └── document.rs
+    │
+    ├── query/               ← query plan & matches
+    │   ├── mod.rs
+    │   └── query.rs
+    │
+    ├── corpus/              ← corpus statistics
+    │   ├── mod.rs
+    │   └── corpus.rs
+    │
+    ├── content-parser/      ← content marker extraction
+    │   ├── mod.rs
+    │   ├── types.rs
+    │   ├── extract.rs
+    │   └── strip.rs
+    │
+    ├── content-composer/    ← content composition
+    │   ├── mod.rs
+    │   ├── types.rs
+    │   ├── compose.rs
+    │   └── render.rs
+    │
+    └── error/               ← error hierarchy
+        ├── mod.rs
+        └── error.rs
+```
+
+Detailed docs: [shared/module.md](shared/module.md)
+
+### kbdb-fs-database (cdylib → .wasm)
+
+Filesystem storage engine. Section CRUD, document
+manifests, B+ tree indexes, content-addressable
+storage.
+
+```
+engine/crates/fs-database/
+├── Cargo.toml
+└── src/
+    ├── lib.rs               ← WASM exports
+    ├── io.rs                ← host import wrappers
+    ├── mem.rs               ← shared-memory helpers
+    ├── catalog.rs           ← catalog.toml
+    │
+    ├── section/             ← section CRUD
+    │   ├── mod.rs
+    │   ├── hash.rs
+    │   ├── parse.rs
+    │   └── shard.rs
+    │
+    ├── document.rs          ← document manifests
+    │
+    ├── index/               ← index subsystem
+    │   ├── mod.rs
+    │   ├── btree.rs
+    │   ├── inverted.rs
+    │   ├── positional.rs
+    │   └── corpus.rs
+    │
+    ├── query_exec.rs        ← query plan execution
+    ├── integrity.rs         ← checksum, GC
+    └── atomic.rs            ← temp-file + rename
+```
+
+Detailed docs:
+[fs-database/module.md](fs-database/module.md)
+
+### kbdb-query-parser (cdylib → .wasm)
+
+Tokenization pipeline, query parsing, keyword
+extraction, and ranking.
+
+```
+engine/crates/query-parser/
+├── Cargo.toml
+└── src/
+    ├── lib.rs               ← WASM exports
+    ├── mem.rs               ← shared-memory helpers
+    │
+    ├── pipeline/            ← 8-stage tokenizer
+    │   ├── mod.rs
+    │   ├── markers.rs
+    │   ├── unicode.rs
+    │   ├── segment.rs
+    │   ├── identifiers.rs
+    │   ├── case.rs
+    │   ├── stopwords.rs
+    │   ├── stem.rs
+    │   └── ngram.rs
+    │
+    ├── parse.rs             ← query → QueryPlan
+    │
+    ├── extract/             ← keyword extraction
+    │   ├── mod.rs
+    │   ├── rake.rs
+    │   └── tfidf.rs
+    │
+    ├── rank/                ← scoring functions
+    │   ├── mod.rs
+    │   ├── bm25f.rs
+    │   ├── proximity.rs
+    │   ├── zone.rs
+    │   └── freshness.rs
+    │
+    └── cache.rs             ← query-scoped cache
+```
+
+Detailed docs:
+[query-parser/module.md](query-parser/module.md)
+
+### kbdb-worker (cdylib → .wasm)
+
+Orchestration layer. Imports the other two WASM
+modules, manages LRU cache and async result tokens.
+
+```
+engine/crates/kb-worker/
+├── Cargo.toml
+└── src/
+    ├── lib.rs               ← WASM exports
+    ├── mem.rs               ← shared-memory helpers
+    ├── context.rs           ← database context
+    │
+    ├── orchestrate/         ← operation flows
+    │   ├── mod.rs
+    │   ├── search.rs
+    │   ├── write.rs
+    │   └── retrieve.rs
+    │
+    ├── cache/               ← LRU cache
+    │   ├── mod.rs
+    │   ├── lru.rs
+    │   ├── entry.rs
+    │   └── config.rs
+    │
+    ├── token.rs             ← async result tokens
+    └── status.rs            ← status reporting
+```
+
+Detailed docs:
+[kb-worker/module.md](kb-worker/module.md)
+
+---
+
+## Module Conventions
+
+### File Roles
+
+| File          | Purpose                    |
+|---------------|----------------------------|
+| `lib.rs`      | Crate root. `pub mod` declarations, WASM exports |
+| `mod.rs`      | Module entry. Declares sub-modules, re-exports public API |
+| `*.rs`        | Implementation. Types, functions, or traits |
+
+### Naming
+
+| Rust convention  | File name            |
+|------------------|----------------------|
+| Module           | `snake_case/`        |
+| Source file       | `snake_case.rs`      |
+| Struct / Enum    | `PascalCase`         |
+| Function         | `snake_case`         |
+| Trait            | `PascalCase`         |
+| Constant         | `SCREAMING_SNAKE`    |
+
+### Visibility
+
+| Modifier       | Scope                     |
+|----------------|---------------------------|
+| (none)         | Private to current module |
+| `pub(super)`   | Parent module only        |
+| `pub(crate)`   | Anywhere in the crate     |
+| `pub`          | External consumers        |
+
+### `mod.rs` Pattern
+
+Every directory module has a `mod.rs` that declares
+its children and re-exports the public API:
+
+```rust
+// corpus/mod.rs
+mod types;
+mod functions;
+
+pub use types::CorpusStats;
+pub use functions::get_corpus_stats;
+```
+
+This is analogous to TypeScript's `index.ts` with
+re-exports -- but in Rust, if a file is not declared
+with `mod`, the compiler does not see it even if it
+exists on disk.
+
+### Shared Code Reuse
+
+Crates import shared namespaces via Cargo dependency:
+
+```toml
+# fs-database/Cargo.toml
+[dependencies]
+kbdb-shared = { path = "../shared" }
+```
+
+```rust
+// fs-database/src/lib.rs
+use kbdb_shared::corpus::CorpusStats;
+use kbdb_shared::encode::Encode;
+use kbdb_shared::kbid::KbId;
+```
+
+Only the imported code ends up in the final `.wasm`
+-- unused shared modules are stripped by dead code
+elimination.
+
+---
+
+## Testing Structure
+
+### Unit Tests (TDD)
+
+Inline `#[cfg(test)] mod tests` block at the bottom
+of each `.rs` file. Written before the implementation
+(spec-first).
+
+### BDD Module Tests
+
+`tests/` subdirectory inside a module for cross-unit
+scenarios:
+
+```
+section/
+├── mod.rs
+├── hash.rs
+├── parse.rs
+└── tests/
+    ├── mod.rs
+    └── hash_and_shard.rs
+```
+
+### Crate-Level Integration Tests
+
+`engine/tests/` directory for each crate's WASM API
+tests using `#[wasm_bindgen_test]`.
+
+### Running Tests
+
+```sh
+cd engine && cargo test                   # all
+cd engine && cargo test -p kbdb-shared    # one crate
+cd engine && wasm-pack test --node        # WASM tests
+```
+
+---
+
+## Doc Directory Structure
+
+Documentation mirrors the source layout:
+
+```
+docs/modules/rust/
+├── overview.md              ← this file
+│
+├── shared/                  ← kbdb-shared crate
+│   ├── module.md            ← crate overview
+│   ├── memory/
+│   │   ├── module.md        ← namespace overview
+│   │   └── functions.md     ← function docs
+│   ├── encode/
+│   │   ├── module.md
+│   │   └── traits.md
+│   ├── kbid/
+│   │   ├── module.md
+│   │   └── types.md
+│   └── ...                  ← one dir per namespace
+│
+├── fs-database/             ← kbdb-fs-database crate
+│   ├── module.md            ← crate overview
+│   ├── types/
+│   │   ├── module.md
+│   │   ├── catalog.md
+│   │   └── section.md
+│   ├── functions/
+│   │   ├── module.md
+│   │   ├── io.md
+│   │   └── ...
+│   └── indexes/
+│       ├── module.md
+│       └── ...
+│
+├── query-parser/            ← kbdb-query-parser crate
+│   ├── module.md
+│   ├── types/
+│   ├── functions/
+│   └── ...
+│
+└── kb-worker/               ← kbdb-worker crate
+    ├── module.md
+    ├── types/
+    ├── functions/
+    └── ...
+```
+
+`module.md` = `mod.rs` (namespace entry point).
+Category files (`types.md`, `functions.md`,
+`traits.md`) document the implementations within
+each namespace.