@dikolab/kbdb 0.5.0 → 0.6.0

package/docs/modules/overview.md

@@ -1,319 +0,0 @@
-# Modules Overview -- Build Pipeline
-
-*How the Rust engine compiles to WASM, lands in the
-TypeScript source tree, and gets bundled into the
-final `dist/` output.*
-
-## Build Pipeline
-
-```
- 1. Rust/WASM                2. TypeScript
- ─────────────               ────────────────
-
- engine/crates/              src/shared/
- ├── shared (rlib)           ├── worker-daemon/
- ├── fs-database (cdylib)    │   ├── wasm/          ← 4 WASM modules
- ├── query-parser (cdylib)   │   │   ├── fs-database.*
- ├── default-embedding (cdylib)│  │   ├── query-parser.*
- └── kb-worker (cdylib)      │   │   ├── default-embedding.*
-       │                     │   │   └── kb-worker.*
-       │ wasm-pack build     │   └── functions/
-       └─────────────────────┘       └── load-wasm-modules
-                                          .function.ts
-                              │
-                              │ esbuild + tsc
-                              ▼
-                             dist/
-                             ├── cli.mjs
-                             ├── worker.mjs
-                             ├── mod.mjs
-                             └── ...
-```
-
-## Phase 1 -- Build Rust to WASM
-
-The Rust engine compiles to four standalone `.wasm`
-files. The `kbdb-shared` crate is an `rlib` -- it
-produces no artifact, compiled **into** each consumer
-via dead code elimination.
-
-### Crates
-
-| Crate              | Type     | Output                |
-|--------------------|----------|-----------------------|
-| `kbdb-shared`      | `rlib`   | none (compiled into consumers) |
-| `kbdb-fs-database` | `cdylib` | `fs-database.wasm`   |
-| `kbdb-query-parser`| `cdylib` | `query-parser.wasm`  |
-| `kbdb-default-embedding` | `cdylib` | `default-embedding.wasm` |
-| `kbdb-worker`      | `cdylib` | `kb-worker.wasm`     |
-
-### Build Order
-
-```
-1.  kbdb-shared                (leaf, no deps)
-2.  kbdb-fs-database       ┐   (all depend on shared,
-    kbdb-query-parser      │    built in parallel)
-    kbdb-default-embedding ┘
-3.  kbdb-worker                (depends on all above)
-```
-
-### Build Command
-
-```sh
-npm run build:wasm
-```
-
-This runs `wasm-pack build` for each cdylib crate.
-Output lands in the TypeScript source tree at
-`src/wasm/`:
-
-```sh
-wasm-pack build crates/fs-database \
-   --target no-modules \
-   --out-dir ../../../src/wasm/fs-database
-
-wasm-pack build crates/query-parser \
-   --target no-modules \
-   --out-dir ../../../src/wasm/query-parser
-
-wasm-pack build crates/default-embedding \
-   --target no-modules \
-   --out-dir ../../../src/wasm/default-embedding
-
-wasm-pack build crates/kb-worker \
-   --target no-modules \
-   --out-dir ../../../src/wasm/kb-worker
-```
-
-### Size Optimization
-
-```toml
-# engine/Cargo.toml [profile.release]
-opt-level = "z"          # optimize for size
-lto = true               # link-time optimization
-codegen-units = 1        # single codegen unit
-strip = true             # remove debug symbols
-panic = "abort"          # no unwind tables
-```
-
-Each `.wasm` includes only the `kbdb-shared` code it
-actually references -- LLVM eliminates unused code
-per consumer.
-
-**Full Rust architecture:**
-[rust/overview.md](rust/overview.md)
-
----
-
-## Phase 2 -- WASM in TypeScript Source
-
-The WASM build output lands inside the TypeScript
-source tree, where the `worker-daemon` module loads
-it at runtime:
-
-```
-src/shared/worker-daemon/
-├── wasm/                          ← gitignored
-│   ├── fs-database/
-│   │   ├── fs-database.wasm
-│   │   ├── fs-database.js         ← JS glue
-│   │   └── fs-database.d.ts       ← type defs
-│   ├── query-parser/
-│   │   ├── query-parser.wasm
-│   │   ├── query-parser.js
-│   │   └── query-parser.d.ts
-│   ├── default-embedding/
-│   │   ├── default-embedding.wasm
-│   │   ├── default-embedding.js
-│   │   └── default-embedding.d.ts
-│   └── kb-worker/
-│       ├── kb-worker.wasm
-│       ├── kb-worker.js
-│       └── kb-worker.d.ts
-│
-└── functions/
-    └── load-wasm-modules.function.ts
-```
-
-The `wasm/` directory is **gitignored** -- it is a
-build artifact, regenerated by `npm run build:wasm`.
-Never edit files inside it manually.
-
-### How TypeScript Loads WASM
-
-The `loadWasmModules` function in
-`shared/worker-daemon/` performs the loading sequence:
-
-```
-loadWasmModules(config)
-  │
-  ├─ 1. Allocate shared WebAssembly.Memory
-  │
-  ├─ 2. Compile + instantiate query-parser.wasm
-  │      (standalone, shared memory only)
-  │
-  ├─ 3. Compile + instantiate fs-database.wasm
-  │      (standalone, shared memory + host I/O)
-  │      ↑ steps 2 and 3 run in parallel
-  │
-  ├─ 4. Compile + instantiate default-embedding.wasm
-  │      (standalone, no host imports -- pure computation)
-  │
-  └─ 5. Compile + instantiate kb-worker.wasm
-         (imports query-parser + fs-database exports)
-```
-
-All four WASM modules share one `WebAssembly.Memory`
-instance. Data crosses module boundaries via
-`(pointer, length)` pairs encoded with the `Encode`
-trait from `kbdb-shared`.
-
-Only the `worker-daemon` module imports WASM. No other
-TypeScript module directly accesses WASM files or the
-`WebAssembly` API.
-
----
-
-## Phase 3 -- TypeScript Modules
-
-All TypeScript logic lives under `src/shared/` in
-9 sub-modules. Two thin entry points at the project
-root dispatch into them.
-
-### Entry Points
-
-| File               | Purpose                   |
-|--------------------|---------------------------|
-| `src/cli.ts`       | CLI + MCP executable      |
-| `src/worker.ts`    | Worker daemon executable  |
-| `mod.ts`           | Library API (npm / JSR)   |
-
-### Shared Modules
-
-| Module           | Responsibility             |
-|------------------|----------------------------|
-| `cli/`           | CLI arg parsing, command handlers, output formatting |
-| `mcp/`           | MCP server, tool + resource handlers |
-| `worker-client/` | IPC client -- discovers/spawns daemon, typed async API |
-| `worker-daemon/` | Daemon process -- loads WASM, IPC server, config |
-| `version/`       | Package version constant   |
-| `runtime/`       | Deno / Node.js detection   |
-| `platform/`      | OS abstractions (tmpdir, PID, IPC paths) |
-| `hash/`          | SHA-256 via Web Crypto API |
-| `log/`           | Structured logging with configurable levels and ANSI formatting |
-
-### Module Dependencies
-
-```
-src/cli.ts
-  ├── shared/cli/ ──→ shared/worker-client/
-  │     └── shared/log/
-  └── shared/mcp/ ──→ shared/worker-client/
-        │                   ├── shared/platform/
-        │                   ├── shared/hash/
-        │                   └── shared/log/
-        └── shared/log/
-
-src/worker.ts
-  └── shared/worker-daemon/
-         ├── shared/platform/
-         ├── shared/log/
-         └── wasm/ (fs-database, query-parser,
-                    default-embedding, kb-worker)
-
-mod.ts
-  └── shared/version/
-
-shared/log/          (no dependencies -- standalone)
-```
-
-**Full TypeScript architecture:**
-[typescript/overview.md](typescript/overview.md)
-
----
-
-## Phase 4 -- Build to dist/
-
-esbuild bundles the TypeScript entry points into
-`dist/` as ESM (with code-splitting) and CJS. `tsc`
-emits declaration files separately.
-
-### Build Command
-
-```sh
-npm run build          # TypeScript only
-npm run build:all      # WASM + TypeScript
-```
-
-### Build Steps
-
-```
-1. esbuild bundles:
-   ├── src/cli.ts    → dist/cli.mjs + dist/cli.cjs
-   ├── src/worker.ts → dist/worker.mjs + dist/worker.cjs
-   ├── mod.ts        → dist/mod.mjs + dist/mod.cjs
-   └── shared code   → dist/chunk-*.mjs (code-split)
-
-2. tsc --emitDeclarationOnly:
-   └── *.d.ts files for all entry points
-```
-
-### Output
-
-```
-dist/
-├── cli.mjs              ← CLI + MCP (ESM, shebang)
-├── cli.cjs              ← CLI + MCP (CJS, shebang)
-├── worker.mjs           ← daemon (ESM, shebang)
-├── worker.cjs           ← daemon (CJS, shebang)
-├── mod.mjs              ← library API (ESM)
-├── mod.cjs              ← library API (CJS)
-├── mod.d.ts             ← declarations
-├── cli.d.ts
-├── worker.d.ts
-├── chunk-*.mjs          ← shared code chunks
-└── src/
-    └── shared/
-        └── **/*.d.ts    ← per-file declarations
-```
-
-WASM files are **not** bundled into `dist/`. They ship
-as separate `.wasm` binaries in the npm package and
-are loaded at runtime by the daemon via
-`WebAssembly.compile()`.
-
----
-
-## Full Build Sequence
-
-```sh
-# 1. Build WASM (Rust → .wasm → src/wasm/)
-npm run build:wasm
-
-# 2. Build TypeScript (src/ → dist/)
-npm run build
-
-# Or both in one command:
-npm run build:all
-```
-
-### build:all Sequence
-
-```
-build:all
-  │
-  ├─ build:wasm
-  │    ├─ cargo build (5 crates)
-  │    └─ wasm-pack build (4 cdylib → wasm/)
-  │
-  └─ build
-       ├─ esbuild (2 entry points → dist/)
-       └─ tsc --emitDeclarationOnly (→ dist/*.d.ts)
-```
-
-## Module Documentation Index
-
-| Layer      | Overview                          | Modules |
-|------------|-----------------------------------|---------|
-| Rust/WASM  | [rust/overview.md](rust/overview.md) | 5 crates, 9 shared namespaces |
-| TypeScript | [typescript/overview.md](typescript/overview.md) | 2 entry points, 9 shared modules |

package/docs/modules/rust/embedding/module.md

@@ -1,91 +0,0 @@
-# default-embedding -- TF-IDF Vector Computation
-
-**Source:** `engine/crates/default-embedding/`
-**Crate type:** `cdylib` + `rlib`
-**Output:** `default-embedding.wasm`
-
-## Responsibility
-
-Computes feature-hashed sublinear TF vectors for the
-default embedding provider. Reuses kbdb-shared's
-9-stage tokenization pipeline (NFC normalization,
-Unicode segmentation, camelCase splitting, stop-word
-removal, synonym expansion, Snowball stemming) so
-that TF-IDF vectors are token-consistent with the
-lexical index.
-
-No host imports -- pure computation with no file I/O.
-The WASM module is instantiated without an import
-object.
-
-## Algorithm
-
-1. Tokenize text via `kbdb_shared::pipeline::tokenize`
-2. Count term frequencies on stemmed tokens
-3. Hash each stemmed token to a bucket using FNV-1a:
-   `bucket = fnv1a(token) % 256`
-4. Apply sign trick to reduce collision bias:
-   `sign = if fnv1a(token + "_s") % 2 == 0 { 1 } else { -1 }`
-5. Accumulate `sign * (1.0 + tf.ln())` per bucket
-   (sublinear TF provides diminishing returns for
-   repeated terms)
-6. L2-normalize the vector
-
-Output: 256-dimensional `Float32Array`, suitable for
-cosine similarity via `cosineSimilarity()` in the
-TypeScript embedding module.
-
-## Directory Layout
-
-```
-engine/crates/default-embedding/
-├── Cargo.toml
-└── src/
-    ├── lib.rs               ← WASM exports
-    ├── mem.rs               ← shared-memory helpers
-    ├── fnv1a.rs             ← FNV-1a hash function
-    └── feature_hash.rs      ← feature hashing + L2 norm
-```
-
-## WASM Exports
-
-| Export                        | Signature              | Description |
-|-------------------------------|------------------------|-------------|
-| `embedding_tfidf`             | `(ptr, len) -> u32`    | Single text → 256×f32 vector |
-| `embedding_tfidf_batch`       | `(ptr, len) -> u32`    | Newline-delimited texts → count + count×256 f32s |
-| `embedding_dimensions`        | `() -> u32`            | Returns 256 |
-| `embedding_model_name`        | `(ptr, len) -> u32`    | Writes `tfidf-v1` to return slot |
-
-All exports use the standard return-slot protocol:
-result data is written via `mem::set_return(ptr, len)`
-and read by TypeScript via `get_return_ptr` /
-`get_return_len`.
-
-## Dependencies
-
-```toml
-[dependencies]
-kbdb-shared = { path = "../shared" }
-wasm-bindgen = "0.2"
-```
-
-- **kbdb-shared** -- memory management (`kbdb_alloc`,
-  `kbdb_free`, `set_return`, `read_str`) and
-  tokenization pipeline (`pipeline::tokenize`,
-  `pipeline::TokenResult`)
-
-## Dependency Direction
-
-```
-kbdb-shared (rlib)
-    │   provides: memory, pipeline::tokenize
-    │
-    └──→ kbdb-default-embedding (cdylib)
-             uses: shared memory + shared tokenization
-             consumed by: TypeScript TfidfEmbeddingProvider
-```
-
-The embedding crate is a leaf in the dependency graph.
-No other Rust crate depends on it. It is loaded as a
-standalone WASM instance by the TypeScript worker
-daemon.

package/docs/modules/rust/fs-database/functions/document.md

@@ -1,48 +0,0 @@
-# functions::document -- Document Manifest CRUD
-
-*Document manifest operations: read, write, remove,
-and section reference updates.*
-
-## Source
-
-`engine/crates/fs-database/src/document.rs`
-
-## Functions
-
-```rust
-pub fn read_document(docid: &str)
-   -> Result<DocumentManifest, FsError>;
-
-pub fn write_document(manifest: &DocumentManifest)
-   -> Result<(), FsError>;
-
-pub fn remove_document(docid: &str)
-   -> Result<u32, FsError>;
-
-pub fn update_section_ref(
-   old_kbid: &KbId, new_kbid: &KbId,
-) -> Result<u32, FsError>;
-```
-
-### `write_document`
-
-Serializes the manifest to TOML and writes to
-`documents/{docid}.toml` atomically. Appends the
-docid to each referenced section's `docids` array
-in its header. Updates catalog.
-
-### `remove_document`
-
-Deletes the manifest file. Removes the docid from
-each section's `docids` array in its header.
-Sections themselves are not deleted -- only the
-grouping is removed. Returns the number of
-ungrouped sections.
-
-### `update_section_ref`
-
-Scans all document manifests and replaces occurrences
-of `old_kbid` with `new_kbid`, preserving position.
-Called during section updates (content-addressable
-storage produces new kb-ids on content change).
-Returns the number of updated documents.

package/docs/modules/rust/fs-database/functions/integrity.md

@@ -1,79 +0,0 @@
-# functions::integrity -- Integrity and Maintenance
-
-*Integrity checks, garbage collection, and full index
-rebuild.*
-
-## Source
-
-`engine/crates/fs-database/src/integrity.rs`
-
-## Functions
-
-```rust
-pub fn check_integrity()
-   -> Result<IntegrityReport, FsError>;
-
-pub fn garbage_collect()
-   -> Result<u32, FsError>;
-
-pub fn rebuild_indexes()
-   -> Result<(), FsError>;
-```
-
-### `check_integrity`
-
-- Verify section file checksums match content.
-- Detect orphaned sections (not referenced by any
-  document).
-- Detect orphaned documents (referencing missing
-  sections).
-- Find broken content marker references across all
-  six marker types:
-  - `$(image: {kbid})` -- referenced section must
-    exist and have type `image/*`.
-  - `$(code: {kbid})` -- referenced section must
-    exist and have type `text/*` or
-    `application/json`.
-  - `$(text: {kbid})` -- referenced section must
-    exist and have type `text/markdown`.
-  - `$(document: {docid})` -- referenced document
-    manifest must exist.
-  - `$(link: {kbid})` -- referenced section must
-    exist (any type).
-  - `$(doc-link: {docid})` -- referenced document
-    manifest must exist.
-- Detect circular references in `$(text:)` and
-  `$(document:)` marker chains.
-- Detect **divergent content**: sections sharing the
-  same `source_key` that have different content.
-  Indicates contradictory versions that should be
-  resolved via supersession. Reported as
-  `divergent_content` entries in `IntegrityReport`.
-
-### `garbage_collect`
-
-Remove section files not referenced by any document
-manifest. Returns the number of removed files.
-
-Also reports `stale_sources` -- an advisory count of
-sections whose `source_path` points to a file that no
-longer contains their heading text. Stale sections
-are not automatically removed (they may still be
-referenced by documents); the count is informational.
-
-### `rebuild_indexes`
-
-Reconstruct all index files (`terms.idx`,
-`positions.idx`, `corpus.dat`) from sections and
-document manifests. Recovers from index corruption.
-
-Rebuild recomputes per-field TFs (heading, body,
-code) from section content via per-field
-tokenization, recomputes per-field token counts
-(`heading_token_count`, `body_token_count`,
-`code_token_count`) in section headers, and
-recalculates per-field corpus averages
-(`avg_heading_length`, `avg_body_length`,
-`avg_code_length`) in `corpus.dat`. Image sections
-have their title and description re-tokenized into
-heading and body fields respectively.

package/docs/modules/rust/fs-database/functions/io.md

@@ -1,87 +0,0 @@
-# functions::io -- Host Import Wrappers
-
-*Raw extern FFI bindings for filesystem operations
-provided by the TypeScript host.*
-
-## Source
-
-`engine/crates/fs-database/src/io.rs`
-
-## Overview
-
-`fs-database.wasm` cannot access the filesystem
-directly. The TypeScript daemon provides file I/O
-functions as WASM imports when instantiating the
-module. All paths are relative to the `.kbdb` context
-directory -- the host resolves them to absolute paths
-and enforces sandbox boundaries.
-
-## Host Imports
-
-```rust
-#[link(wasm_import_module = "kbdb_host")]
-extern "C" {
-   fn host_read_file(
-      path_ptr: *const u8, path_len: u32,
-      buf_ptr: *mut u8, buf_len: u32,
-   ) -> i32;
-
-   fn host_write_file(
-      path_ptr: *const u8, path_len: u32,
-      data_ptr: *const u8, data_len: u32,
-   ) -> i32;
-
-   fn host_remove_file(
-      path_ptr: *const u8, path_len: u32,
-   ) -> i32;
-
-   fn host_list_dir(
-      path_ptr: *const u8, path_len: u32,
-      buf_ptr: *mut u8, buf_len: u32,
-   ) -> i32;
-
-   fn host_mkdir(
-      path_ptr: *const u8, path_len: u32,
-   ) -> i32;
-
-   fn host_file_exists(
-      path_ptr: *const u8, path_len: u32,
-   ) -> i32;
-
-   fn host_rename(
-      from_ptr: *const u8, from_len: u32,
-      to_ptr: *const u8, to_len: u32,
-   ) -> i32;
-}
-```
-
-## Return Conventions
-
-- Positive integer = bytes written or entry count.
-- `0` = success (for void operations).
-- `-1` = file not found.
-- `-2` = permission denied.
-- `-3` = I/O error.
-
-The `io.rs` module wraps these raw imports in safe
-Rust functions that return `Result<T, IoError>`.
-
-## Shared Memory
-
-All data exchange with the host uses the shared
-`WebAssembly.Memory` buffer. Strings and byte slices
-are passed as `(pointer, length)` pairs. The crate
-re-exports `alloc` and `free` from `kbdb-shared` for
-the host to manage buffer allocation.
-
-## Atomic Writes (`atomic.rs`)
-
-```rust
-pub fn atomic_write(
-   path: &str, data: &[u8],
-) -> Result<(), FsError>;
-```
-
-Writes data to a temporary file in the same
-directory, then renames it into place via
-`host_rename`. Prevents partial writes on crash.

package/docs/modules/rust/fs-database/functions/module.md

@@ -1,22 +0,0 @@
-# fs-database/functions -- Operations
-
-*Functions exported by the fs-database crate for
-section CRUD, document manifest management, query
-execution, and maintenance.*
-
-## Modules
-
-- **[io](io.md)** -- Host import wrappers. Raw extern
-  FFI bindings for filesystem operations, return
-  conventions, shared memory protocol.
-- **[section](section.md)** -- Section CRUD operations.
-  `read_section`, `add_section`, `update_section`,
-  `remove_section`.
-- **[document](document.md)** -- Document manifest
-  CRUD. `read_document`, `write_document`,
-  `remove_document`, `update_section_ref`.
-- **[query-exec](query-exec.md)** -- Query plan
-  execution against indexes. Returns raw matches with
-  term frequencies and positional data.
-- **[integrity](integrity.md)** -- Integrity checks,
-  garbage collection, full index rebuild.

package/docs/modules/rust/fs-database/functions/query-exec.md

@@ -1,44 +0,0 @@
-# functions::query-exec -- Query Plan Execution
-
-*Execute pre-parsed query plans against the indexes
-and return raw matches.*
-
-## Source
-
-`engine/crates/fs-database/src/query_exec.rs`
-
-## Function
-
-```rust
-pub fn execute_plan(
-   plan: &QueryPlan,
-   inverted: &InvertedIndex,
-   positional: &PositionalIndex,
-   corpus: &CorpusStats,
-) -> Vec<RawMatch>;
-```
-
-Does not parse queries or rank results -- those are
-`query-parser.wasm` responsibilities orchestrated by
-`kb-worker.wasm`.
-
-## Execution Steps
-
-1. Look up each term in `plan.terms` in the inverted
-   index.
-2. Combine posting lists based on `plan.mode`:
-   - `And` -- intersect all posting lists.
-   - `Or` -- union all posting lists.
-3. Remove sections matching any term in
-   `plan.exclusions`.
-4. For phrase queries (`plan.phrases`), verify
-   adjacent positions in the positional index.
-5. For each surviving section, collect per-field
-   term frequencies (heading_tf, body_tf, code_tf
-   from inverted index), positional data from the
-   positional index, `token_count` and
-   `field_token_counts` from the section header
-   into a `RawMatch`.
-
-Returns raw matches. `kb-worker.wasm` passes these to
-`query-parser.wasm`'s ranking functions for scoring.