@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/modules/rust/fs-database/indexes/corpus.md

@@ -0,0 +1,61 @@
+# indexes::corpus -- Corpus Statistics
+
+*Corpus-wide term statistics for IDF computation.
+Stored in `indexes/corpus.dat`.*
+
+## Source
+
+`engine/crates/fs-database/src/index/corpus.rs`
+
+## Type Definition
+
+```rust
+pub struct CorpusStats {
+   pub total_sections: u32,
+   pub total_terms: u32,
+   pub term_doc_freq: HashMap<Term, u32>,
+   pub avg_section_length: f32,
+   pub avg_heading_length: f32,
+   pub avg_body_length: f32,
+   pub avg_code_length: f32,
+}
+```
+
+This is the **full** corpus stats struct with the
+per-term document frequency map. The lightweight
+`CorpusStats` in `kbdb-shared` (see
+[shared/corpus](../../shared/corpus/types.md))
+contains only the scalar fields that cross module
+boundaries.
+
+Per-field average lengths enable accurate BM25F
+per-field length normalization. Each field's
+average is computed from the corresponding
+per-field token counts stored in section headers.
+
+## Methods
+
+```rust
+impl CorpusStats {
+   pub fn idf(&self, term: &Term) -> f32;
+   pub fn update_add(
+      &mut self,
+      terms: &[Term],
+      field_token_counts: &FieldTokenCounts,
+   );
+   pub fn update_remove(
+      &mut self,
+      terms: &[Term],
+      field_token_counts: &FieldTokenCounts,
+   );
+}
+```
+
+`idf` computes inverse document frequency:
+`ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)`.
+
+`update_add` increments document frequency counters
+for each term and updates per-field average length
+accumulators using the section's field token counts.
+`update_remove` decrements them and adjusts the
+per-field averages accordingly.

package/docs/modules/rust/fs-database/indexes/inverted.md

@@ -0,0 +1,60 @@
+# indexes::inverted -- Inverted Index
+
+*Maps normalized terms to lists of
+(kb-id, heading_tf, body_tf, code_tf) posting
+entries. Stored in `indexes/terms.idx`.*
+
+## Source
+
+`engine/crates/fs-database/src/index/inverted.rs`
+
+## Type Definitions
+
+```rust
+pub struct InvertedIndex {
+   tree: BPlusTree<Term, PostingList>,
+}
+
+pub struct PostingList {
+   pub entries: Vec<Posting>,
+}
+
+pub struct Posting {
+   pub kbid: KbId,
+   pub heading_tf: f32,
+   pub body_tf: f32,
+   pub code_tf: f32,
+}
+```
+
+Each `Posting` stores per-field term frequencies
+instead of a single aggregate weight. Storing
+per-field TFs separately enables query-time field
+weight tuning (heading 2.0x, body 1.0x, code 1.5x)
+without rebuilding indexes. See
+[FieldTf](../../shared/query/types.md) for the
+shared struct used across module boundaries.
+
+## Methods
+
+```rust
+impl InvertedIndex {
+   pub fn lookup_term(&self, term: &Term)
+      -> Option<&PostingList>;
+
+   pub fn add_terms(
+      &mut self,
+      kbid: &KbId,
+      terms: &[(Term, FieldTf)],
+   );
+
+   pub fn remove_section(&mut self, kbid: &KbId);
+}
+```
+
+`add_terms` inserts or updates per-field TF
+entries for a section. Each `FieldTf` value
+contains the heading, body, and code term
+frequencies computed during per-field
+tokenization. `remove_section` removes all
+postings for a given kb-id across all terms.

package/docs/modules/rust/fs-database/indexes/module.md

@@ -0,0 +1,64 @@
+# fs-database/indexes -- B+ Tree Index Subsystem
+
+*Binary index files in `indexes/`. Optimized for read
+speed -- O(log n) term lookups.*
+
+## Source
+
+`engine/crates/fs-database/src/index/`
+
+## Modules
+
+- **[btree](btree.md)** -- Generic `BPlusTree<K, V>`
+  implementation. On-disk ordered key-value storage
+  with configurable page size.
+- **[inverted](inverted.md)** -- `InvertedIndex`.
+  Maps normalized terms to posting lists (kb-id +
+  weight). Stored in `indexes/terms.idx`.
+- **[positional](positional.md)** -- `PositionalIndex`.
+  Maps (term, kb-id) pairs to position lists for
+  phrase query support. Stored in
+  `indexes/positions.idx`.
+- **[corpus](corpus.md)** -- `CorpusStats`. Corpus-wide
+  term statistics for IDF computation. Stored in
+  `indexes/corpus.dat`.
+- **[references](references.md)** -- `ReferenceIndex`.
+  Bidirectional graph of content marker references
+  between sections. Stored in `indexes/references.idx`.
+
+---
+
+### Reference Graph Index (`references.idx`)
+
+Bidirectional index mapping content marker
+references between sections. Built at section
+insertion time by parsing content markers via
+`extract_content_markers()` from `kbdb-shared`.
+
+**Index entry format:**
+
+| Field             | Type   | Description           |
+|-------------------|--------|-----------------------|
+| `source_kbid`     | KbId   | Section with marker   |
+| `target_id`       | String | Referenced kbid/docid |
+| `marker_type`     | u8     | ContentMarkerType enum value |
+| `context_snippet` | String | ~150 chars around marker |
+| `direction`       | u8     | 0 = forward, 1 = back |
+
+**Operations:**
+
+- **Build**: On section add/update, extract markers,
+  generate context snippets (~150 chars of the
+  surrounding sentence/paragraph), write one forward
+  entry per marker plus one back-reference entry at
+  each target.
+- **Remove**: On section delete, remove all forward
+  entries where `source_kbid` matches and all back
+  entries where `target_id` matches the deleted
+  section's kbid.
+- **Lookup**: Given a kbid, return all forward
+  references (markers in this section's content) and
+  all back-references (markers in other sections
+  pointing to this section). O(1) via B+ tree index.
+- **Rebuild**: Regenerated from section content
+  alongside other index files during `rebuild`.

package/docs/modules/rust/fs-database/indexes/positional.md

@@ -0,0 +1,39 @@
+# indexes::positional -- Positional Index
+
+*Maps (term, kb-id) pairs to position lists for
+phrase query support. Stored in
+`indexes/positions.idx`.*
+
+## Source
+
+`engine/crates/fs-database/src/index/positional.rs`
+
+## Type Definition
+
+```rust
+pub struct PositionalIndex {
+   tree: BPlusTree<(Term, KbId), Vec<u32>>,
+}
+```
+
+## Methods
+
+```rust
+impl PositionalIndex {
+   pub fn positions(
+      &self, term: &Term, kbid: &KbId,
+   ) -> Option<&[u32]>;
+
+   pub fn add_positions(
+      &mut self,
+      kbid: &KbId,
+      term_positions: &[(Term, Vec<u32>)],
+   );
+
+   pub fn remove_section(&mut self, kbid: &KbId);
+}
+```
+
+Position values are zero-based token offsets. Used
+for phrase matching (verifying adjacent term
+positions) and proximity scoring.

package/docs/modules/rust/fs-database/module.md

@@ -0,0 +1,149 @@
+# fs-database -- Filesystem Storage Engine
+
+*Rust crate that manages the `.kbdb` directory:
+section CRUD, document manifests, B+ tree indexes,
+and content-addressable storage. Compiles to
+`fs-database.wasm`.*
+
+## Crate
+
+| Field       | Value                              |
+|-------------|------------------------------------|
+| Name        | `kbdb-fs-database`                 |
+| Path        | `engine/crates/fs-database/`       |
+| Crate type  | `cdylib` (WASM) + `rlib` (tests)  |
+| WASM output | `dist/wasm/fs-database.wasm`       |
+| Depends on  | `kbdb-shared`                      |
+
+`kbdb-fs-database` is a **standalone** WASM module. It
+does not import `kbdb-query-parser` or `kbdb-worker` at
+the WASM level. `kb-worker.wasm` imports this module's
+exports and orchestrates it alongside
+`query-parser.wasm`.
+
+See [Database](../../../goals/database.md) for the
+full behavioral specification.
+
+## Workspace Position
+
+```
+engine/
+├── Cargo.toml                   # workspace root
+└── crates/
+    ├── shared/                  # kbdb-shared
+    ├── fs-database/             # this crate
+    ├── query-parser/            # kbdb-query-parser
+    └── kb-worker/               # kbdb-worker
+```
+
+## Directory Layout
+
+```
+engine/crates/fs-database/
+├── Cargo.toml
+└── src/
+    ├── lib.rs                   # crate root, WASM exports
+    ├── io.rs                    # host import wrappers
+    ├── mem.rs                   # shared-memory helpers
+    ├── catalog.rs               # catalog.toml read/write
+    ├── section/
+    │   ├── mod.rs               # section CRUD facade
+    │   ├── hash.rs              # SHA-256 hashing, kb-id
+    │   ├── parse.rs             # TOML frontmatter parser
+    │   └── shard.rs             # hash-sharded path logic
+    ├── document.rs              # document manifest CRUD
+    ├── index/
+    │   ├── mod.rs               # index facade
+    │   ├── btree.rs             # B+ tree implementation
+    │   ├── inverted.rs          # term → kb-id mappings
+    │   ├── positional.rs        # term position data
+    │   └── corpus.rs            # corpus-wide statistics
+    ├── query_exec.rs            # execute query plans
+    ├── integrity.rs             # checksum verify, GC
+    └── atomic.rs                # temp-file + rename
+```
+
+## Modules
+
+### [types/](types/module.md) -- Structs
+
+- **[catalog](types/catalog.md)** -- `Catalog` struct.
+  Database table of contents.
+- **[section](types/section.md)** -- `SectionHeader`,
+  frontmatter parse/serialize, shard path resolution.
+
+### [functions/](functions/module.md) -- Operations
+
+- **[io](functions/io.md)** -- Host import wrappers,
+  return conventions, shared memory protocol.
+- **[section](functions/section.md)** -- Section CRUD
+  (`read`, `add`, `update`, `remove`), content
+  hashing, atomic writes.
+- **[document](functions/document.md)** -- Document
+  manifest CRUD, `update_section_ref`.
+- **[query-exec](functions/query-exec.md)** -- Execute
+  query plans against indexes, return raw matches.
+- **[integrity](functions/integrity.md)** -- Integrity
+  checks, garbage collection, index rebuild.
+
+### [indexes/](indexes/module.md) -- Index Subsystem
+
+- **[btree](indexes/btree.md)** -- Generic B+ tree
+  for binary index files.
+- **[inverted](indexes/inverted.md)** -- Term →
+  posting list mappings (`terms.idx`).
+- **[positional](indexes/positional.md)** -- Term
+  position data for phrase queries
+  (`positions.idx`).
+- **[corpus](indexes/corpus.md)** -- Corpus-wide
+  statistics for IDF (`corpus.dat`).
+
+## WASM Exports
+
+| Export                 | Description                |
+|------------------------|----------------------------|
+| `db_init`              | Open database, read catalog |
+| `db_read_section`      | Read section by kb-id      |
+| `db_read_sections`     | Read multiple sections     |
+| `db_add_section`       | Create section file        |
+| `db_update_section`    | Replace section content    |
+| `db_remove_section`    | Delete section file        |
+| `db_read_document`     | Read document manifest     |
+| `db_write_document`    | Create/update document     |
+| `db_remove_document`   | Delete document manifest   |
+| `db_update_section_ref`| Replace kb-id in manifests |
+| `db_index_add`         | Add terms to inverted index |
+| `db_index_remove`      | Remove section from indexes |
+| `db_index_rebuild`     | Full index rebuild         |
+| `db_execute_plan`      | Run query plan against indexes |
+| `db_corpus_stats`      | Return corpus statistics   |
+| `db_status`            | Return database metadata   |
+| `db_check_integrity`   | Run integrity checks       |
+| `db_garbage_collect`   | Remove orphaned sections   |
+| `alloc` / `free`       | Memory management (from `kbdb-shared`) |
+
+## Dependencies
+
+```toml
+[dependencies]
+kbdb-shared = { path = "../shared" }
+sha2 = "0.10"
+data-encoding = "2"
+toml = { version = "0.8", default-features = false }
+```
+
+## Build
+
+```sh
+cd engine
+wasm-pack build crates/fs-database \
+   --target no-modules \
+   --out-dir ../../../dist/wasm/fs-database
+```
+
+## Testing
+
+```sh
+cd engine && cargo test -p kbdb-fs-database
+cd engine && wasm-pack test --node crates/fs-database
+```

package/docs/modules/rust/fs-database/types/catalog.md

@@ -0,0 +1,53 @@
+# types::catalog -- Database Catalog
+
+*The `Catalog` struct representing `catalog.toml` --
+the database's table of contents.*
+
+## Source
+
+`engine/crates/fs-database/src/catalog.rs`
+
+## Type Definition
+
+```rust
+pub struct Catalog {
+   pub version: u32,
+   pub created_at: String,
+   pub last_modified: String,
+   pub section_count: u32,
+   pub document_count: u32,
+   pub checksum: String,
+}
+```
+
+| Field            | Description                     |
+|------------------|---------------------------------|
+| `version`        | Database format version         |
+| `created_at`     | ISO 8601 creation timestamp     |
+| `last_modified`  | ISO 8601 last write timestamp   |
+| `section_count`  | Total sections in the database  |
+| `document_count` | Total documents in the database |
+| `checksum`       | SHA-256 integrity checksum      |
+
+The catalog is the first file `fs-database.wasm`
+reads when opening a database. It is updated after
+every write operation using atomic temp-file + rename.
+
+## Functions
+
+```rust
+pub fn read_catalog() -> Result<Catalog, FsError>;
+
+pub fn write_catalog(cat: &Catalog)
+   -> Result<(), FsError>;
+
+pub fn increment_sections(delta: i32)
+   -> Result<Catalog, FsError>;
+
+pub fn increment_documents(delta: i32)
+   -> Result<Catalog, FsError>;
+```
+
+`increment_sections` and `increment_documents` read
+the catalog, adjust the count, update
+`last_modified`, and write back atomically.

package/docs/modules/rust/fs-database/types/module.md

@@ -0,0 +1,12 @@
+# fs-database/types -- Type Definitions
+
+*Structs defined in the fs-database crate for catalog
+metadata and section file parsing.*
+
+## Modules
+
+- **[catalog](catalog.md)** -- `Catalog` struct.
+  Database table of contents from `catalog.toml`.
+- **[section](section.md)** -- `SectionHeader` struct
+  and `IntegrityReport`. TOML frontmatter
+  parse/serialize, hash-sharded path resolution.

package/docs/modules/rust/fs-database/types/section.md

@@ -0,0 +1,101 @@
+# types::section -- Section Header and Parsing
+
+*The `SectionHeader` struct and `IntegrityReport`
+struct, plus TOML frontmatter parse/serialize and
+hash-sharded path resolution.*
+
+## Source
+
+`engine/crates/fs-database/src/section/`
+
+## Type Definitions
+
+### `SectionHeader`
+
+```rust
+pub struct SectionHeader {
+   pub kbid: KbId,
+   pub docids: Vec<String>,
+   pub section_type: SectionType,
+   pub title: Option<String>,
+   pub description: Option<String>,
+   pub size: u32,
+   pub created_at: String,
+   pub checksum: String,
+   pub token_count: u32,
+   pub heading_token_count: u32,
+   pub body_token_count: u32,
+   pub code_token_count: u32,
+}
+```
+
+TOML frontmatter fields from a section file. Parsed
+by `parse_section_file`, serialized by
+`serialize_section_file`.
+
+`docids` lists the document IDs this section belongs
+to (may be empty, one, or many -- many-to-many
+relationship). `title` is required for image sections
+and optional for text/code sections. `description` is
+always optional. Token counts are computed by
+`query-parser.wasm` during per-field tokenization
+and stored in the header for query-time retrieval.
+See [FieldTokenCounts](../../shared/query/types.md)
+for the shared struct.
+
+### `IntegrityReport`
+
+```rust
+pub struct IntegrityReport {
+   pub checksum_failures: Vec<KbId>,
+   pub orphaned_sections: Vec<KbId>,
+   pub orphaned_documents: Vec<String>,
+   pub broken_refs: Vec<(KbId, String, KbId)>,
+   pub type_mismatches: Vec<(KbId, String, KbId)>,
+   pub circular_refs: Vec<(KbId, Vec<KbId>)>,
+}
+```
+
+Returned by `check_integrity`. `broken_refs`
+entries are `(source_kbid, marker_type, target_kbid)`
+covering all six marker types (`text`, `code`,
+`image`, `document`, `link`, `doc-link`).
+`type_mismatches` are `(source_kbid, marker_type,
+target_kbid)` where the target section type does
+not match the marker type. `circular_refs` are
+`(start_kbid, cycle_path)` for reference chains
+that form loops.
+
+## Functions
+
+### Frontmatter Parsing (`section/parse.rs`)
+
+```rust
+pub fn parse_section_file(data: &[u8])
+   -> Result<(SectionHeader, &[u8]), ParseError>;
+
+pub fn serialize_section_file(
+   header: &SectionHeader,
+   body: &[u8],
+) -> Vec<u8>;
+```
+
+Splits on `---\n` separator. Header is TOML; body
+is raw bytes stored verbatim.
+
+### Content Hashing (`section/hash.rs`)
+
+```rust
+pub fn compute_kbid(content: &[u8]) -> KbId;
+```
+
+SHA-256 → truncate to 16 bytes → base32 (Crockford)
+→ 26-character kb-id.
+
+### Path Resolution (`section/shard.rs`)
+
+```rust
+pub fn section_path(kbid: &KbId) -> String;
+```
+
+Returns `sections/{first_two_chars}/{kbid}.sec`.

package/docs/modules/rust/kb-worker/functions/compose.md

@@ -0,0 +1,57 @@
+# functions::compose -- Content Composition Flow
+
+*Orchestrates content composition: resolve identifiers,
+fetch content, compose rendered Markdown.*
+
+## Source
+
+`engine/crates/kb-worker/src/orchestrate/compose.rs`
+
+## Exported WASM Function
+
+### `worker_compose`
+
+```rust
+pub fn worker_compose(
+   ids_ptr: *const u8,
+   ids_len: u32,
+) -> u64;
+```
+
+Composes a rendered Markdown document from one or
+more kbid or docid identifiers. Returns a result
+token (same async pattern as `worker_query`).
+
+## Compose Flow
+
+```
+1. Receive identifiers from IPC `compose` method
+         ↓
+2. Resolve identifiers via fs-database
+   (try kbid first, then docid)
+         ↓
+3. Expand documents to section kbids, deduplicate
+         ↓
+4. Fetch section/document content from fs-database
+         ↓
+5. Extract content markers recursively using
+   extract_content_markers() from content-parser
+         ↓
+6. Build ComposeRequest with all resolved data
+         ↓
+7. Call compose_content() from content-composer
+         ↓
+8. Return ComposeResult to daemon
+```
+
+The caller retrieves the composed document via
+`worker_get_result` with type `"content"`.
+
+Individual sections fetched during composition are
+cached. The composed result itself is NOT cached --
+it is computed fresh on each request because
+composition depends on the full set of input
+identifiers.
+
+See [Content Composer](../../shared/content-composer/module.md)
+for the full composition specification.

package/docs/modules/rust/kb-worker/functions/module.md

@@ -0,0 +1,20 @@
+# kb-worker/functions -- Orchestration Functions
+
+*Functions that coordinate calls between
+query-parser.wasm and fs-database.wasm to implement
+full database operations.*
+
+## Modules
+
+- **[search](search.md)** -- Search orchestration.
+  Parse query → build plan → execute → rank →
+  return results.
+- **[write](write.md)** -- Write orchestration.
+  Add/update/remove text, code, and image sections.
+- **[retrieve](retrieve.md)** -- Read orchestration.
+  Retrieve sections and documents, with cache.
+  Progressive context expansion (`worker_recall`).
+  Status reporting.
+- **[compose](compose.md)** -- Content composition.
+  Resolve identifiers, fetch content, compose
+  rendered Markdown via content-composer.