@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/modules/rust/overview.md

@@ -1,531 +0,0 @@
-# Rust Engine -- Overview
-
-*Architecture, crate structure, and module layout
-for the Rust/WASM engine.*
-
-## Compilation Model
-
-Rust compiles by **crate** -- each crate is an
-independent compilation unit with its own
-`Cargo.toml`. Within a crate, code is organized into
-**modules** (namespaces) that compile together as one
-unit.
-
-```
-crate  = compilation unit  (has Cargo.toml)
-module = namespace         (has mod.rs)
-```
-
-A crate can only use code from:
-1. Its own modules (`crate::`)
-2. Dependencies declared in its `Cargo.toml`
-
-There are no path-based imports across crates --
-unlike TypeScript/ESM where any file can import any
-other file. Code sharing requires a crate dependency.
-
-## Workspace
-
-All Rust code lives in `engine/`. The workspace
-contains six crates under `engine/crates/`:
-
-```
-engine/
-├── Cargo.toml                 ← workspace root
-├── Cargo.lock                 ← shared lock file
-├── .cargo/
-│   └── config.toml            ← default: wasm32
-└── crates/
-    ├── shared/                ← rlib (library)
-    ├── fs-database/           ← cdylib (→ .wasm)
-    ├── fs-migration/          ← cdylib (→ .wasm)
-    ├── default-embedding/     ← cdylib (→ .wasm)
-    ├── query-parser/          ← cdylib (→ .wasm)
-    └── kb-worker/             ← cdylib (→ .wasm)
-```
-
-## Crate Types
-
-### `rlib` -- Rust Library
-
-Produces no output artifact. Compiled **into** each
-crate that depends on it. The compiler includes only
-the code each consumer actually uses (dead code
-elimination).
-
-- **`kbdb-shared`** is the only rlib crate.
-
-### `cdylib` -- WebAssembly Module
-
-Produces a standalone `.wasm` file. Each cdylib crate
-is an entry point that compiles to a self-contained
-WASM binary.
-
-- **`kbdb-fs-database`** → `fs-database.wasm`
-- **`kbdb-query-parser`** → `query-parser.wasm`
-- **`kbdb-default-embedding`** → `default-embedding.wasm`
-- **`kbdb-worker`** → `kb-worker.wasm`
-
-## Crate Dependency Graph
-
-```
-kbdb-shared (rlib)
-    │
-    ├──→ kbdb-fs-database (cdylib)
-    │        produces: fs-database.wasm
-    │
-    ├──→ kbdb-query-parser (cdylib)
-    │        produces: query-parser.wasm
-    │
-    ├──→ kbdb-default-embedding (cdylib)
-    │        produces: default-embedding.wasm
-    │
-    └──→ kbdb-worker (cdylib)
-             produces: kb-worker.wasm
-             also depends on:
-               kbdb-fs-database (Cargo)
-               kbdb-query-parser (Cargo)
-```
-
-### Cargo vs WASM Dependencies
-
-`kbdb-worker` depends on the other two crates at
-**two levels**:
-
-1. **Cargo dependency** -- `kbdb-worker` lists
-   `kbdb-fs-database` and `kbdb-query-parser` in its
-   `Cargo.toml`. This gives it compile-time access to
-   their type definitions and WASM import signatures.
-
-2. **WASM runtime imports** -- at runtime,
-   `kb-worker.wasm` imports functions from
-   `fs-database.wasm` and `query-parser.wasm` through
-   the shared `WebAssembly.Memory` interface.
-
-The standalone modules (`fs-database`, `query-parser`)
-have **no dependency** on each other or on `kb-worker`.
-
-## Build Order
-
-```
-1.  kbdb-shared              (no deps, builds first)
-2.  kbdb-fs-database     ┐   (all depend on shared,
-    kbdb-query-parser    │    built in parallel)
-    kbdb-default-embedding┘
-3.  kbdb-worker              (depends on all above)
-```
-
-## WASM Output
-
-```
-dist/wasm/
-├── fs-database/
-│   ├── fs-database.wasm
-│   └── fs-database.js        ← JS glue
-├── query-parser/
-│   ├── query-parser.wasm
-│   └── query-parser.js
-├── embedding/
-│   ├── default-embedding.wasm
-│   └── default-embedding.js
-└── kb-worker/
-    ├── kb-worker.wasm
-    └── kb-worker.js
-```
-
-All modules share one `WebAssembly.Memory`
-instance at runtime. Data crosses module boundaries
-via `(pointer, length)` pairs encoded with the
-`Encode` trait from `kbdb-shared`.
-
-## WASM Size Optimization
-
-Each `.wasm` file is optimized for size:
-
-```toml
-# engine/Cargo.toml [profile.release]
-opt-level = "z"          # optimize for size
-lto = true               # link-time optimization
-codegen-units = 1        # better optimization
-strip = true             # remove debug symbols
-panic = "abort"          # no unwinding (~10KB saved)
-```
-
-The `shared` rlib is dead-code-eliminated per
-consumer -- each `.wasm` only includes the shared
-code it actually references.
-
----
-
-## Crate Layouts
-
-### kbdb-shared (rlib)
-
-Shared types, traits, memory helpers, and error
-types. Not compiled to WASM on its own -- compiled
-into each consumer.
-
-Namespaces are first-level directories inside `src/`.
-Each namespace owns one concern.
-
-```
-engine/crates/shared/
-├── Cargo.toml
-└── src/
-    ├── lib.rs               ← pub mod declarations
-    │
-    ├── memory/              ← memory management
-    │   ├── mod.rs
-    │   └── alloc.rs
-    │
-    ├── encode/              ← binary serialization
-    │   ├── mod.rs
-    │   └── encode.rs
-    │
-    ├── kbid/                ← content-addressed ID
-    │   ├── mod.rs
-    │   └── kbid.rs
-    │
-    ├── term/                ← search term newtype
-    │   ├── mod.rs
-    │   └── term.rs
-    │
-    ├── section/             ← section types & records
-    │   ├── mod.rs
-    │   └── section.rs
-    │
-    ├── document/            ← document manifests
-    │   ├── mod.rs
-    │   └── document.rs
-    │
-    ├── query/               ← query plan & matches
-    │   ├── mod.rs
-    │   └── query.rs
-    │
-    ├── corpus/              ← corpus statistics
-    │   ├── mod.rs
-    │   └── corpus.rs
-    │
-    ├── content-parser/      ← content marker extraction
-    │   ├── mod.rs
-    │   ├── types.rs
-    │   ├── extract.rs
-    │   └── strip.rs
-    │
-    ├── content-composer/    ← content composition
-    │   ├── mod.rs
-    │   ├── types.rs
-    │   ├── compose.rs
-    │   └── render.rs
-    │
-    ├── pipeline/            ← 9-stage tokenizer
-    │   ├── mod.rs
-    │   ├── markers.rs
-    │   ├── unicode.rs
-    │   ├── segment.rs
-    │   ├── identifiers.rs
-    │   ├── case.rs
-    │   ├── stopwords.rs
-    │   ├── synonyms.rs
-    │   ├── stem.rs
-    │   ├── ngram.rs
-    │   └── token_result.rs
-    │
-    └── error/               ← error hierarchy
-        ├── mod.rs
-        └── error.rs
-```
-
-Detailed docs: [shared/module.md](shared/module.md)
-
-### kbdb-fs-database (cdylib → .wasm)
-
-Filesystem storage engine. Section CRUD, document
-manifests, B+ tree indexes, content-addressable
-storage.
-
-```
-engine/crates/fs-database/
-├── Cargo.toml
-└── src/
-    ├── lib.rs               ← WASM exports
-    ├── io.rs                ← host import wrappers
-    ├── mem.rs               ← shared-memory helpers
-    ├── catalog.rs           ← catalog.toml
-    │
-    ├── section/             ← section CRUD
-    │   ├── mod.rs
-    │   ├── hash.rs
-    │   ├── parse.rs
-    │   └── shard.rs
-    │
-    ├── document.rs          ← document manifests
-    │
-    ├── index/               ← index subsystem
-    │   ├── mod.rs
-    │   ├── btree.rs
-    │   ├── inverted.rs
-    │   ├── positional.rs
-    │   └── corpus.rs
-    │
-    ├── query_exec.rs        ← query plan execution
-    ├── integrity.rs         ← checksum, GC
-    └── atomic.rs            ← temp-file + rename
-```
-
-Detailed docs:
-[fs-database/module.md](fs-database/module.md)
-
-### kbdb-query-parser (cdylib → .wasm)
-
-Query parsing, keyword extraction, and ranking.
-Tokenization is imported from `kbdb-shared`'s
-`pipeline` module.
-
-```
-engine/crates/query-parser/
-├── Cargo.toml
-└── src/
-    ├── lib.rs               ← WASM exports
-    ├── mem.rs               ← shared-memory helpers
-    │
-    ├── parse.rs             ← query → QueryPlan
-    │
-    ├── extract/             ← keyword extraction
-    │   ├── mod.rs
-    │   ├── rake.rs
-    │   └── tfidf.rs
-    │
-    ├── rank/                ← scoring functions
-    │   ├── mod.rs
-    │   ├── bm25f.rs
-    │   ├── proximity.rs
-    │   ├── zone.rs
-    │   └── freshness.rs
-    │
-    └── cache.rs             ← query-scoped cache
-```
-
-Detailed docs:
-[query-parser/module.md](query-parser/module.md)
-
-### kbdb-default-embedding (cdylib → .wasm)
-
-TF-IDF embedding computation. Reuses kbdb-shared's
-tokenization pipeline to produce feature-hashed
-sublinear TF vectors (256 dimensions, L2-normalized).
-Depends only on `kbdb-shared`. No host imports --
-pure computation with no file I/O.
-
-```
-engine/crates/default-embedding/
-├── Cargo.toml
-└── src/
-    ├── lib.rs               ← WASM exports
-    ├── mem.rs               ← shared-memory helpers
-    ├── fnv1a.rs             ← FNV-1a hash function
-    └── feature_hash.rs      ← feature hashing + L2 norm
-```
-
-Detailed docs:
-[embedding/module.md](embedding/module.md)
-
-### kbdb-worker (cdylib → .wasm)
-
-Orchestration layer. Imports the other two WASM
-modules, manages LRU cache and async result tokens.
-
-```
-engine/crates/kb-worker/
-├── Cargo.toml
-└── src/
-    ├── lib.rs               ← WASM exports
-    ├── mem.rs               ← shared-memory helpers
-    ├── context.rs           ← database context
-    │
-    ├── orchestrate/         ← operation flows
-    │   ├── mod.rs
-    │   ├── search.rs
-    │   ├── write.rs
-    │   └── retrieve.rs
-    │
-    ├── cache/               ← LRU cache
-    │   ├── mod.rs
-    │   ├── lru.rs
-    │   ├── entry.rs
-    │   └── config.rs
-    │
-    ├── token.rs             ← async result tokens
-    └── status.rs            ← status reporting
-```
-
-Detailed docs:
-[kb-worker/module.md](kb-worker/module.md)
-
----
-
-## Module Conventions
-
-### File Roles
-
-| File          | Purpose                    |
-|---------------|----------------------------|
-| `lib.rs`      | Crate root. `pub mod` declarations, WASM exports |
-| `mod.rs`      | Module entry. Declares sub-modules, re-exports public API |
-| `*.rs`        | Implementation. Types, functions, or traits |
-
-### Naming
-
-| Rust convention  | File name            |
-|------------------|----------------------|
-| Module           | `snake_case/`        |
-| Source file       | `snake_case.rs`      |
-| Struct / Enum    | `PascalCase`         |
-| Function         | `snake_case`         |
-| Trait            | `PascalCase`         |
-| Constant         | `SCREAMING_SNAKE`    |
-
-### Visibility
-
-| Modifier       | Scope                     |
-|----------------|---------------------------|
-| (none)         | Private to current module |
-| `pub(super)`   | Parent module only        |
-| `pub(crate)`   | Anywhere in the crate     |
-| `pub`          | External consumers        |
-
-### `mod.rs` Pattern
-
-Every directory module has a `mod.rs` that declares
-its children and re-exports the public API:
-
-```rust
-// corpus/mod.rs
-mod types;
-mod functions;
-
-pub use types::CorpusStats;
-pub use functions::get_corpus_stats;
-```
-
-This is analogous to TypeScript's `index.ts` with
-re-exports -- but in Rust, if a file is not declared
-with `mod`, the compiler does not see it even if it
-exists on disk.
-
-### Shared Code Reuse
-
-Crates import shared namespaces via Cargo dependency:
-
-```toml
-# fs-database/Cargo.toml
-[dependencies]
-kbdb-shared = { path = "../shared" }
-```
-
-```rust
-// fs-database/src/lib.rs
-use kbdb_shared::corpus::CorpusStats;
-use kbdb_shared::encode::Encode;
-use kbdb_shared::kbid::KbId;
-```
-
-Only the imported code ends up in the final `.wasm`
--- unused shared modules are stripped by dead code
-elimination.
-
----
-
-## Testing Structure
-
-### Unit Tests (TDD)
-
-Inline `#[cfg(test)] mod tests` block at the bottom
-of each `.rs` file. Written before the implementation
-(spec-first).
-
-### BDD Module Tests
-
-`tests/` subdirectory inside a module for cross-unit
-scenarios:
-
-```
-section/
-├── mod.rs
-├── hash.rs
-├── parse.rs
-└── tests/
-    ├── mod.rs
-    └── hash_and_shard.rs
-```
-
-### Crate-Level Integration Tests
-
-`engine/tests/` directory for each crate's WASM API
-tests using `#[wasm_bindgen_test]`.
-
-### Running Tests
-
-```sh
-cd engine && cargo test                   # all
-cd engine && cargo test -p kbdb-shared    # one crate
-cd engine && wasm-pack test --node        # WASM tests
-```
-
----
-
-## Doc Directory Structure
-
-Documentation mirrors the source layout:
-
-```
-docs/modules/rust/
-├── overview.md              ← this file
-│
-├── shared/                  ← kbdb-shared crate
-│   ├── module.md            ← crate overview
-│   ├── memory/
-│   │   ├── module.md        ← namespace overview
-│   │   └── functions.md     ← function docs
-│   ├── encode/
-│   │   ├── module.md
-│   │   └── traits.md
-│   ├── kbid/
-│   │   ├── module.md
-│   │   └── types.md
-│   └── ...                  ← one dir per namespace
-│
-├── fs-database/             ← kbdb-fs-database crate
-│   ├── module.md            ← crate overview
-│   ├── types/
-│   │   ├── module.md
-│   │   ├── catalog.md
-│   │   └── section.md
-│   ├── functions/
-│   │   ├── module.md
-│   │   ├── io.md
-│   │   └── ...
-│   └── indexes/
-│       ├── module.md
-│       └── ...
-│
-├── query-parser/            ← kbdb-query-parser crate
-│   ├── module.md
-│   ├── types/
-│   ├── functions/
-│   └── ...
-│
-├── embedding/               ← kbdb-default-embedding crate
-│   └── module.md
-│
-└── kb-worker/               ← kbdb-worker crate
-    ├── module.md
-    ├── types/
-    ├── functions/
-    └── ...
-```
-
-`module.md` = `mod.rs` (namespace entry point).
-Category files (`types.md`, `functions.md`,
-`traits.md`) document the implementations within
-each namespace.

package/docs/modules/rust/query-parser/functions/extract.md

@@ -1,56 +0,0 @@
-# functions::extract -- Keyword Extraction
-
-*RAKE keyphrase extraction and TF-IDF term weighting.*
-
-## Source
-
-`engine/crates/query-parser/src/extract/`
-
-## RAKE (`rake.rs`)
-
-```rust
-pub fn extract_keyphrases(
-   tokens: &[Token],
-   stop_words: &HashSet<String>,
-) -> Vec<Keyphrase>;
-```
-
-Rapid Automatic Keyword Extraction. Splits on stop
-words to identify candidate phrases. Scores each
-candidate by the ratio of word degree to word
-frequency. Unsupervised, no ML dependencies.
-
-## TF-IDF (`tfidf.rs`)
-
-```rust
-pub fn compute_tf(
-   tokens: &[Token],
-) -> Vec<(Term, f32)>;
-
-pub fn compute_field_tf(
-   heading_tokens: &[Token],
-   body_tokens: &[Token],
-   code_tokens: &[Token],
-) -> Vec<(Term, FieldTf)>;
-
-pub fn compute_tfidf(
-   tf: &[(Term, f32)],
-   idf: &dyn Fn(&Term) -> f32,
-) -> Vec<WeightedTerm>;
-```
-
-`compute_tf` calculates per-section term frequency
-for a single token stream.
-
-`compute_field_tf` computes per-field term
-frequencies from separately tokenized field streams.
-`kb-worker` calls this during indexing after running
-the tokenization pipeline separately for each field
-(title → heading, body + description → body,
-code → code). Returns `FieldTf` values with heading,
-body, and code frequencies per term.
-
-`compute_tfidf` combines TF with corpus-wide IDF
-(provided by `fs-database.wasm` via `kb-worker.wasm`)
-to produce final weights reflecting each term's
-discriminative value.

package/docs/modules/rust/query-parser/functions/module.md

@@ -1,18 +0,0 @@
-# query-parser/functions -- Operations
-
-*Functions for query parsing, keyword extraction,
-and ranking.*
-
-## Modules
-
-- **[parse](parse.md)** -- Query string parsing into
-  structured `QueryPlan`.
-- **[extract](extract.md)** -- RAKE keyphrase
-  extraction and TF-IDF term weighting.
-- **[rank](rank.md)** -- BM25F, proximity, zone, and
-  freshness scoring functions.
-
-Tokenization is provided by
-[kbdb-shared pipeline](../../shared/pipeline/functions.md).
-This crate re-exports it via the `qp_tokenize`
-WASM export.

package/docs/modules/rust/query-parser/functions/parse.md

@@ -1,33 +0,0 @@
-# functions::parse -- Query Parsing
-
-*Parse a search query string into a structured
-`QueryPlan`.*
-
-## Source
-
-`engine/crates/query-parser/src/parse.rs`
-
-## Function
-
-```rust
-pub fn parse_query(input: &str) -> QueryPlan;
-```
-
-## Parsing Stages
-
-1. **Tokenize** -- Run the query through the
-   tokenization pipeline.
-2. **Parse syntax** -- Recognize operators: quoted
-   phrases (`"..."`), boolean AND/OR, exclusions
-   (`-term`).
-3. **Build plan** -- Produce the `QueryPlan` with
-   terms, phrase groups, exclusions, and match mode.
-
-## Examples
-
-| Input                | terms        | phrases          | exclusions | mode |
-|----------------------|--------------|------------------|------------|------|
-| `rust wasm`          | [rust, wasm] | []               | []         | And  |
-| `"file database"`    | []           | [[file, databas]]| []         | And  |
-| `rust -draft`        | [rust]       | []               | [draft]    | And  |
-| `rust OR wasm`       | [rust, wasm] | []               | []         | Or   |

package/docs/modules/rust/query-parser/functions/pipeline.md

@@ -1,9 +0,0 @@
-# functions::pipeline -- Moved to kbdb-shared
-
-The tokenization pipeline has moved to `kbdb-shared`.
-
-See [shared/pipeline/functions.md](../../shared/pipeline/functions.md)
-for the full documentation.
-
-This crate re-exports the pipeline via the
-`qp_tokenize` WASM FFI export.