@dikolab/kbdb 0.1.6 → 0.2.0

package/docs/modules/overview.md

@@ -0,0 +1,298 @@
+# Modules Overview -- Build Pipeline
+
+*How the Rust engine compiles to WASM, lands in the
+TypeScript source tree, and gets bundled into the
+final `dist/` output.*
+
+## Build Pipeline
+
+```
+ 1. Rust/WASM                2. TypeScript
+ ─────────────               ────────────────
+
+ engine/crates/              src/shared/
+ ├── shared (rlib)           ├── worker-daemon/
+ ├── fs-database (cdylib)    │   ├── wasm/          ← 3 WASM modules
+ ├── query-parser (cdylib)   │   │   ├── fs-database.*
+ └── kb-worker (cdylib)      │   │   ├── query-parser.*
+       │                     │   │   └── kb-worker.*
+       │ wasm-pack build     │   └── functions/
+       └─────────────────────┘       └── load-wasm-modules
+                                          .function.ts
+                              │
+                              │ esbuild + tsc
+                              ▼
+                             dist/
+                             ├── cli.mjs
+                             ├── worker.mjs
+                             ├── mod.mjs
+                             └── ...
+```
+
+## Phase 1 -- Build Rust to WASM
+
+The Rust engine compiles to three standalone `.wasm`
+files. The `kbdb-shared` crate is an `rlib` -- it
+produces no artifact, compiled **into** each consumer
+via dead code elimination.
+
+### Crates
+
+| Crate              | Type     | Output                |
+|--------------------|----------|-----------------------|
+| `kbdb-shared`      | `rlib`   | none (compiled into consumers) |
+| `kbdb-fs-database` | `cdylib` | `fs-database.wasm`   |
+| `kbdb-query-parser`| `cdylib` | `query-parser.wasm`  |
+| `kbdb-worker`      | `cdylib` | `kb-worker.wasm`     |
+
+### Build Order
+
+```
+1.  kbdb-shared           (leaf, no deps)
+2.  kbdb-fs-database  ┐   (both depend on shared,
+    kbdb-query-parser ┘    built in parallel)
+3.  kbdb-worker           (depends on all three)
+```
+
+### Build Command
+
+```sh
+npm run build:wasm
+```
+
+This runs `wasm-pack build` for each cdylib crate.
+Output lands in the TypeScript source tree at
+`src/shared/worker-daemon/wasm/`:
+
+```sh
+wasm-pack build crates/fs-database \
+   --target no-modules \
+   --out-dir ../../../../src/shared/worker-daemon/wasm/fs-database
+
+wasm-pack build crates/query-parser \
+   --target no-modules \
+   --out-dir ../../../../src/shared/worker-daemon/wasm/query-parser
+
+wasm-pack build crates/kb-worker \
+   --target no-modules \
+   --out-dir ../../../../src/shared/worker-daemon/wasm/kb-worker
+```
+
+### Size Optimization
+
+```toml
+# engine/Cargo.toml [profile.release]
+opt-level = "z"          # optimize for size
+lto = true               # link-time optimization
+codegen-units = 1        # single codegen unit
+strip = true             # remove debug symbols
+panic = "abort"          # no unwind tables
+```
+
+Each `.wasm` includes only the `kbdb-shared` code it
+actually references -- LLVM eliminates unused code
+per consumer.
+
+**Full Rust architecture:**
+[rust/overview.md](rust/overview.md)
+
+---
+
+## Phase 2 -- WASM in TypeScript Source
+
+The WASM build output lands inside the TypeScript
+source tree, where the `worker-daemon` module loads
+it at runtime:
+
+```
+src/shared/worker-daemon/
+├── wasm/                          ← gitignored
+│   ├── fs-database/
+│   │   ├── fs-database.wasm
+│   │   ├── fs-database.js         ← JS glue
+│   │   └── fs-database.d.ts       ← type defs
+│   ├── query-parser/
+│   │   ├── query-parser.wasm
+│   │   ├── query-parser.js
+│   │   └── query-parser.d.ts
+│   └── kb-worker/
+│       ├── kb-worker.wasm
+│       ├── kb-worker.js
+│       └── kb-worker.d.ts
+│
+└── functions/
+    └── load-wasm-modules.function.ts
+```
+
+The `wasm/` directory is **gitignored** -- it is a
+build artifact, regenerated by `npm run build:wasm`.
+Never edit files inside it manually.
+
+### How TypeScript Loads WASM
+
+The `loadWasmModules` function in
+`shared/worker-daemon/` performs the loading sequence:
+
+```
+loadWasmModules(config)
+  │
+  ├─ 1. Allocate shared WebAssembly.Memory
+  │
+  ├─ 2. Compile + instantiate query-parser.wasm
+  │      (standalone, shared memory only)
+  │
+  ├─ 3. Compile + instantiate fs-database.wasm
+  │      (standalone, shared memory + host I/O)
+  │      ↑ steps 2 and 3 run in parallel
+  │
+  └─ 4. Compile + instantiate kb-worker.wasm
+         (imports query-parser + fs-database exports)
+```
+
+All three modules share one `WebAssembly.Memory`
+instance. Data crosses module boundaries via
+`(pointer, length)` pairs encoded with the `Encode`
+trait from `kbdb-shared`.
+
+Only the `worker-daemon` module imports WASM. No other
+TypeScript module directly accesses WASM files or the
+`WebAssembly` API.
+
+---
+
+## Phase 3 -- TypeScript Modules
+
+All TypeScript logic lives under `src/shared/` in
+8 sub-modules. Two thin entry points at the project
+root dispatch into them.
+
+### Entry Points
+
+| File               | Purpose                   |
+|--------------------|---------------------------|
+| `src/cli.ts`       | CLI + MCP executable      |
+| `src/worker.ts`    | Worker daemon executable  |
+| `mod.ts`           | Library API (npm / JSR)   |
+
+### Shared Modules
+
+| Module           | Responsibility             |
+|------------------|----------------------------|
+| `cli/`           | CLI arg parsing, command handlers, output formatting |
+| `mcp/`           | MCP server, tool + resource handlers |
+| `worker-client/` | IPC client -- discovers/spawns daemon, typed async API |
+| `worker-daemon/` | Daemon process -- loads WASM, IPC server, config |
+| `version/`       | Package version constant   |
+| `runtime/`       | Deno / Node.js detection   |
+| `platform/`      | OS abstractions (tmpdir, PID, IPC paths) |
+| `hash/`          | SHA-256 via Web Crypto API |
+
+### Module Dependencies
+
+```
+src/cli.ts
+  ├── shared/cli/ ──→ shared/worker-client/
+  └── shared/mcp/ ──→ shared/worker-client/
+                          ├── shared/platform/
+                          └── shared/hash/
+
+src/worker.ts
+  └── shared/worker-daemon/
+         ├── shared/platform/
+         └── wasm/ (fs-database, query-parser,
+                    kb-worker)
+
+mod.ts
+  └── shared/version/
+```
+
+**Full TypeScript architecture:**
+[typescript/overview.md](typescript/overview.md)
+
+---
+
+## Phase 4 -- Build to dist/
+
+esbuild bundles the TypeScript entry points into
+`dist/` as ESM (with code-splitting) and CJS. `tsc`
+emits declaration files separately.
+
+### Build Command
+
+```sh
+npm run build          # TypeScript only
+npm run build:all      # WASM + TypeScript
+```
+
+### Build Steps
+
+```
+1. esbuild bundles:
+   ├── src/cli.ts    → dist/cli.mjs + dist/cli.cjs
+   ├── src/worker.ts → dist/worker.mjs + dist/worker.cjs
+   ├── mod.ts        → dist/mod.mjs + dist/mod.cjs
+   └── shared code   → dist/chunk-*.mjs (code-split)
+
+2. tsc --emitDeclarationOnly:
+   └── *.d.ts files for all entry points
+```
+
+### Output
+
+```
+dist/
+├── cli.mjs              ← CLI + MCP (ESM, shebang)
+├── cli.cjs              ← CLI + MCP (CJS, shebang)
+├── worker.mjs           ← daemon (ESM, shebang)
+├── worker.cjs           ← daemon (CJS, shebang)
+├── mod.mjs              ← library API (ESM)
+├── mod.cjs              ← library API (CJS)
+├── mod.d.ts             ← declarations
+├── cli.d.ts
+├── worker.d.ts
+├── chunk-*.mjs          ← shared code chunks
+└── src/
+    └── shared/
+        └── **/*.d.ts    ← per-file declarations
+```
+
+WASM files are **not** bundled into `dist/`. They ship
+as separate `.wasm` binaries in the npm package and
+are loaded at runtime by the daemon via
+`WebAssembly.compile()`.
+
+---
+
+## Full Build Sequence
+
+```sh
+# 1. Build WASM (Rust → .wasm → src/shared/worker-daemon/wasm/)
+npm run build:wasm
+
+# 2. Build TypeScript (src/ → dist/)
+npm run build
+
+# Or both in one command:
+npm run build:all
+```
+
+### build:all Sequence
+
+```
+build:all
+  │
+  ├─ build:wasm
+  │    ├─ cargo build (4 crates)
+  │    └─ wasm-pack build (3 cdylib → wasm/)
+  │
+  └─ build
+       ├─ esbuild (2 entry points → dist/)
+       └─ tsc --emitDeclarationOnly (→ dist/*.d.ts)
+```
+
+## Module Documentation Index
+
+| Layer      | Overview                          | Modules |
+|------------|-----------------------------------|---------|
+| Rust/WASM  | [rust/overview.md](rust/overview.md) | 4 crates, 9 shared namespaces |
+| TypeScript | [typescript/overview.md](typescript/overview.md) | 2 entry points, 8 shared modules |

package/docs/modules/rust/fs-database/functions/document.md

@@ -0,0 +1,48 @@
+# functions::document -- Document Manifest CRUD
+
+*Document manifest operations: read, write, remove,
+and section reference updates.*
+
+## Source
+
+`engine/crates/fs-database/src/document.rs`
+
+## Functions
+
+```rust
+pub fn read_document(docid: &str)
+   -> Result<DocumentManifest, FsError>;
+
+pub fn write_document(manifest: &DocumentManifest)
+   -> Result<(), FsError>;
+
+pub fn remove_document(docid: &str)
+   -> Result<u32, FsError>;
+
+pub fn update_section_ref(
+   old_kbid: &KbId, new_kbid: &KbId,
+) -> Result<u32, FsError>;
+```
+
+### `write_document`
+
+Serializes the manifest to TOML and writes to
+`documents/{docid}.toml` atomically. Appends the
+docid to each referenced section's `docids` array
+in its header. Updates catalog.
+
+### `remove_document`
+
+Deletes the manifest file. Removes the docid from
+each section's `docids` array in its header.
+Sections themselves are not deleted -- only the
+grouping is removed. Returns the number of
+ungrouped sections.
+
+### `update_section_ref`
+
+Scans all document manifests and replaces occurrences
+of `old_kbid` with `new_kbid`, preserving position.
+Called during section updates (content-addressable
+storage produces new kb-ids on content change).
+Returns the number of updated documents.

package/docs/modules/rust/fs-database/functions/integrity.md

@@ -0,0 +1,68 @@
+# functions::integrity -- Integrity and Maintenance
+
+*Integrity checks, garbage collection, and full index
+rebuild.*
+
+## Source
+
+`engine/crates/fs-database/src/integrity.rs`
+
+## Functions
+
+```rust
+pub fn check_integrity()
+   -> Result<IntegrityReport, FsError>;
+
+pub fn garbage_collect()
+   -> Result<u32, FsError>;
+
+pub fn rebuild_indexes()
+   -> Result<(), FsError>;
+```
+
+### `check_integrity`
+
+- Verify section file checksums match content.
+- Detect orphaned sections (not referenced by any
+  document).
+- Detect orphaned documents (referencing missing
+  sections).
+- Find broken content marker references across all
+  six marker types:
+  - `$(image: {kbid})` -- referenced section must
+    exist and have type `image/*`.
+  - `$(code: {kbid})` -- referenced section must
+    exist and have type `text/*` or
+    `application/json`.
+  - `$(text: {kbid})` -- referenced section must
+    exist and have type `text/markdown`.
+  - `$(document: {docid})` -- referenced document
+    manifest must exist.
+  - `$(link: {kbid})` -- referenced section must
+    exist (any type).
+  - `$(doc-link: {docid})` -- referenced document
+    manifest must exist.
+- Detect circular references in `$(text:)` and
+  `$(document:)` marker chains.
+
+### `garbage_collect`
+
+Remove section files not referenced by any document
+manifest. Returns the number of removed files.
+
+### `rebuild_indexes`
+
+Reconstruct all index files (`terms.idx`,
+`positions.idx`, `corpus.dat`) from sections and
+document manifests. Recovers from index corruption.
+
+Rebuild recomputes per-field TFs (heading, body,
+code) from section content via per-field
+tokenization, recomputes per-field token counts
+(`heading_token_count`, `body_token_count`,
+`code_token_count`) in section headers, and
+recalculates per-field corpus averages
+(`avg_heading_length`, `avg_body_length`,
+`avg_code_length`) in `corpus.dat`. Image sections
+have their title and description re-tokenized into
+heading and body fields respectively.

package/docs/modules/rust/fs-database/functions/io.md

@@ -0,0 +1,87 @@
+# functions::io -- Host Import Wrappers
+
+*Raw extern FFI bindings for filesystem operations
+provided by the TypeScript host.*
+
+## Source
+
+`engine/crates/fs-database/src/io.rs`
+
+## Overview
+
+`fs-database.wasm` cannot access the filesystem
+directly. The TypeScript daemon provides file I/O
+functions as WASM imports when instantiating the
+module. All paths are relative to the `.kbdb` context
+directory -- the host resolves them to absolute paths
+and enforces sandbox boundaries.
+
+## Host Imports
+
+```rust
+#[link(wasm_import_module = "kbdb_host")]
+extern "C" {
+   fn host_read_file(
+      path_ptr: *const u8, path_len: u32,
+      buf_ptr: *mut u8, buf_len: u32,
+   ) -> i32;
+
+   fn host_write_file(
+      path_ptr: *const u8, path_len: u32,
+      data_ptr: *const u8, data_len: u32,
+   ) -> i32;
+
+   fn host_remove_file(
+      path_ptr: *const u8, path_len: u32,
+   ) -> i32;
+
+   fn host_list_dir(
+      path_ptr: *const u8, path_len: u32,
+      buf_ptr: *mut u8, buf_len: u32,
+   ) -> i32;
+
+   fn host_mkdir(
+      path_ptr: *const u8, path_len: u32,
+   ) -> i32;
+
+   fn host_file_exists(
+      path_ptr: *const u8, path_len: u32,
+   ) -> i32;
+
+   fn host_rename(
+      from_ptr: *const u8, from_len: u32,
+      to_ptr: *const u8, to_len: u32,
+   ) -> i32;
+}
+```
+
+## Return Conventions
+
+- Positive integer = bytes written or entry count.
+- `0` = success (for void operations).
+- `-1` = file not found.
+- `-2` = permission denied.
+- `-3` = I/O error.
+
+The `io.rs` module wraps these raw imports in safe
+Rust functions that return `Result<T, IoError>`.
+
+## Shared Memory
+
+All data exchange with the host uses the shared
+`WebAssembly.Memory` buffer. Strings and byte slices
+are passed as `(pointer, length)` pairs. The crate
+re-exports `alloc` and `free` from `kbdb-shared` for
+the host to manage buffer allocation.
+
+## Atomic Writes (`atomic.rs`)
+
+```rust
+pub fn atomic_write(
+   path: &str, data: &[u8],
+) -> Result<(), FsError>;
+```
+
+Writes data to a temporary file in the same
+directory, then renames it into place via
+`host_rename`. Prevents partial writes on crash.

package/docs/modules/rust/fs-database/functions/module.md

@@ -0,0 +1,22 @@
+# fs-database/functions -- Operations
+
+*Functions exported by the fs-database crate for
+section CRUD, document manifest management, query
+execution, and maintenance.*
+
+## Modules
+
+- **[io](io.md)** -- Host import wrappers. Raw extern
+  FFI bindings for filesystem operations, return
+  conventions, shared memory protocol.
+- **[section](section.md)** -- Section CRUD operations.
+  `read_section`, `add_section`, `update_section`,
+  `remove_section`.
+- **[document](document.md)** -- Document manifest
+  CRUD. `read_document`, `write_document`,
+  `remove_document`, `update_section_ref`.
+- **[query-exec](query-exec.md)** -- Query plan
+  execution against indexes. Returns raw matches with
+  term frequencies and positional data.
+- **[integrity](integrity.md)** -- Integrity checks,
+  garbage collection, full index rebuild.

package/docs/modules/rust/fs-database/functions/query-exec.md

@@ -0,0 +1,44 @@
+# functions::query-exec -- Query Plan Execution
+
+*Execute pre-parsed query plans against the indexes
+and return raw matches.*
+
+## Source
+
+`engine/crates/fs-database/src/query_exec.rs`
+
+## Function
+
+```rust
+pub fn execute_plan(
+   plan: &QueryPlan,
+   inverted: &InvertedIndex,
+   positional: &PositionalIndex,
+   corpus: &CorpusStats,
+) -> Vec<RawMatch>;
+```
+
+Does not parse queries or rank results -- those are
+`query-parser.wasm` responsibilities orchestrated by
+`kb-worker.wasm`.
+
+## Execution Steps
+
+1. Look up each term in `plan.terms` in the inverted
+   index.
+2. Combine posting lists based on `plan.mode`:
+   - `And` -- intersect all posting lists.
+   - `Or` -- union all posting lists.
+3. Remove sections matching any term in
+   `plan.exclusions`.
+4. For phrase queries (`plan.phrases`), verify
+   adjacent positions in the positional index.
+5. For each surviving section, collect per-field
+   term frequencies (heading_tf, body_tf, code_tf
+   from inverted index), positional data from the
+   positional index, `token_count` and
+   `field_token_counts` from the section header
+   into a `RawMatch`.
+
+Returns raw matches. `kb-worker.wasm` passes these to
+`query-parser.wasm`'s ranking functions for scoring.

package/docs/modules/rust/fs-database/functions/section.md

@@ -0,0 +1,59 @@
+# functions::section -- Section CRUD
+
+*Content-addressable section file operations: read,
+add, update, remove.*
+
+## Source
+
+`engine/crates/fs-database/src/section/mod.rs`
+
+## Functions
+
+```rust
+pub fn read_section(kbid: &KbId)
+   -> Result<SectionRecord, FsError>;
+
+pub fn read_sections(kbids: &[KbId])
+   -> Result<Vec<SectionRecord>, FsError>;
+
+pub fn add_section(params: &AddSectionParams)
+   -> Result<AddSectionResult, FsError>;
+
+pub fn update_section(
+   kbid: &KbId, new_content: &[u8],
+) -> Result<UpdateSectionResult, FsError>;
+
+pub fn remove_section(kbid: &KbId)
+   -> Result<bool, FsError>;
+```
+
+### `add_section`
+
+1. Compute kb-id from content via `compute_kbid`.
+2. Create shard directory if needed (`host_mkdir`).
+3. Serialize TOML header + `---` separator + body.
+   TOML header includes: kbid, docids, section_type,
+   title, description, size, created_at, checksum,
+   token_count, heading_token_count,
+   body_token_count, code_token_count.
+4. Write atomically via `atomic_write`.
+5. Update catalog via `increment_sections(1)`.
+
+Validation: image sections (`image/*` type) require
+a `title` in `AddSectionParams`.
+
+### `update_section`
+
+1. Read existing section to determine type.
+2. Create new section file (new kb-id).
+3. Call `update_section_ref` to update document
+   manifest references from old to new kb-id.
+4. Mark old section file for garbage collection.
+5. Update catalog.
+
+### `remove_section`
+
+1. Delete the section file.
+2. Remove kb-id from any document manifests.
+3. Update catalog via `increment_sections(-1)`.
+4. Trigger index update to remove terms.

package/docs/modules/rust/fs-database/indexes/btree.md

@@ -0,0 +1,35 @@
+# indexes::btree -- B+ Tree
+
+*Generic B+ tree implementation for ordered key-value
+storage in binary index files.*
+
+## Source
+
+`engine/crates/fs-database/src/index/btree.rs`
+
+## Type Definition
+
+```rust
+pub struct BPlusTree<K, V> {
+   // on-disk B+ tree with configurable page size
+}
+
+impl<K: Ord + Encode, V: Encode> BPlusTree<K, V> {
+   pub fn open(data: &[u8])
+      -> Result<Self, IndexError>;
+   pub fn lookup(&self, key: &K)
+      -> Option<&V>;
+   pub fn range(&self, start: &K, end: &K)
+      -> Vec<(&K, &V)>;
+   pub fn insert(&mut self, key: K, value: V);
+   pub fn remove(&mut self, key: &K) -> bool;
+   pub fn serialize(&self) -> Vec<u8>;
+}
+```
+
+Page size is tuned for WASM memory constraints.
+Interior nodes store keys and child pointers. Leaf
+nodes store keys and values with sibling links for
+range scans.
+
+Used by both `InvertedIndex` and `PositionalIndex`.