@dikolab/kbdb 0.5.0 → 0.6.0

package/docs/modules/rust/fs-database/functions/section.md

@@ -1,76 +0,0 @@
-# functions::section -- Section CRUD
-
-*Content-addressable section file operations: read,
-add, update, remove.*
-
-## Source
-
-`engine/crates/fs-database/src/section/mod.rs`
-
-## Functions
-
-```rust
-pub fn read_section(kbid: &KbId)
-   -> Result<SectionRecord, FsError>;
-
-pub fn read_sections(kbids: &[KbId])
-   -> Result<Vec<SectionRecord>, FsError>;
-
-pub fn add_section(params: &AddSectionParams)
-   -> Result<AddSectionResult, FsError>;
-
-pub fn update_section(
-   kbid: &KbId, new_content: &[u8],
-) -> Result<UpdateSectionResult, FsError>;
-
-pub fn remove_section(kbid: &KbId)
-   -> Result<bool, FsError>;
-```
-
-### `add_section`
-
-1. Compute kb-id from content via `compute_kbid`.
-2. **Supersession check**: if `source_key` is
-   provided and a section with the same `source_key`
-   already exists, set `supersedes` to the old
-   section's kbid, set `updated_at` to the current
-   timestamp, remove the old kbid from document
-   manifests, and mark the old section for GC. The
-   new section replaces the old one as the current
-   version.
-3. Create shard directory if needed (`host_mkdir`).
-4. Serialize TOML header + `---` separator + body.
-   TOML header includes: kbid, docids, section_type,
-   title, description, size, created_at, checksum,
-   token_count, heading_token_count,
-   body_token_count, code_token_count, source_key,
-   source_path, source_mtime, updated_at, tags,
-   supersedes.
-5. Write atomically via `atomic_write`.
-6. **Near-duplicate detection**: compute Jaccard
-   coefficient on stemmed term sets against existing
-   sections. Sections with similarity >= 0.85 are
-   reported as `near_duplicates` in the result
-   (advisory, not a block).
-7. Update catalog via `increment_sections(1)`.
-
-Validation: image sections (`image/*` type) require
-a `title` in `AddSectionParams`. Text sections
-without a `title` emit a `heading_missing` warning
-in the result.
-
-### `update_section`
-
-1. Read existing section to determine type.
-2. Create new section file (new kb-id).
-3. Call `update_section_ref` to update document
-   manifest references from old to new kb-id.
-4. Mark old section file for garbage collection.
-5. Update catalog.
-
-### `remove_section`
-
-1. Delete the section file.
-2. Remove kb-id from any document manifests.
-3. Update catalog via `increment_sections(-1)`.
-4. Trigger index update to remove terms.

package/docs/modules/rust/fs-database/indexes/btree.md

@@ -1,35 +0,0 @@
-# indexes::btree -- B+ Tree
-
-*Generic B+ tree implementation for ordered key-value
-storage in binary index files.*
-
-## Source
-
-`engine/crates/fs-database/src/index/btree.rs`
-
-## Type Definition
-
-```rust
-pub struct BPlusTree<K, V> {
-   // on-disk B+ tree with configurable page size
-}
-
-impl<K: Ord + Encode, V: Encode> BPlusTree<K, V> {
-   pub fn open(data: &[u8])
-      -> Result<Self, IndexError>;
-   pub fn lookup(&self, key: &K)
-      -> Option<&V>;
-   pub fn range(&self, start: &K, end: &K)
-      -> Vec<(&K, &V)>;
-   pub fn insert(&mut self, key: K, value: V);
-   pub fn remove(&mut self, key: &K) -> bool;
-   pub fn serialize(&self) -> Vec<u8>;
-}
-```
-
-Page size is tuned for WASM memory constraints.
-Interior nodes store keys and child pointers. Leaf
-nodes store keys and values with sibling links for
-range scans.
-
-Used by both `InvertedIndex` and `PositionalIndex`.

package/docs/modules/rust/fs-database/indexes/corpus.md

@@ -1,61 +0,0 @@
-# indexes::corpus -- Corpus Statistics
-
-*Corpus-wide term statistics for IDF computation.
-Stored in `indexes/corpus.dat`.*
-
-## Source
-
-`engine/crates/fs-database/src/index/corpus.rs`
-
-## Type Definition
-
-```rust
-pub struct CorpusStats {
-   pub total_sections: u32,
-   pub total_terms: u32,
-   pub term_doc_freq: HashMap<Term, u32>,
-   pub avg_section_length: f32,
-   pub avg_heading_length: f32,
-   pub avg_body_length: f32,
-   pub avg_code_length: f32,
-}
-```
-
-This is the **full** corpus stats struct with the
-per-term document frequency map. The lightweight
-`CorpusStats` in `kbdb-shared` (see
-[shared/corpus](../../shared/corpus/types.md))
-contains only the scalar fields that cross module
-boundaries.
-
-Per-field average lengths enable accurate BM25F
-per-field length normalization. Each field's
-average is computed from the corresponding
-per-field token counts stored in section headers.
-
-## Methods
-
-```rust
-impl CorpusStats {
-   pub fn idf(&self, term: &Term) -> f32;
-   pub fn update_add(
-      &mut self,
-      terms: &[Term],
-      field_token_counts: &FieldTokenCounts,
-   );
-   pub fn update_remove(
-      &mut self,
-      terms: &[Term],
-      field_token_counts: &FieldTokenCounts,
-   );
-}
-```
-
-`idf` computes inverse document frequency:
-`ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)`.
-
-`update_add` increments document frequency counters
-for each term and updates per-field average length
-accumulators using the section's field token counts.
-`update_remove` decrements them and adjusts the
-per-field averages accordingly.

package/docs/modules/rust/fs-database/indexes/inverted.md

@@ -1,60 +0,0 @@
-# indexes::inverted -- Inverted Index
-
-*Maps normalized terms to lists of
-(kb-id, heading_tf, body_tf, code_tf) posting
-entries. Stored in `indexes/terms.idx`.*
-
-## Source
-
-`engine/crates/fs-database/src/index/inverted.rs`
-
-## Type Definitions
-
-```rust
-pub struct InvertedIndex {
-   tree: BPlusTree<Term, PostingList>,
-}
-
-pub struct PostingList {
-   pub entries: Vec<Posting>,
-}
-
-pub struct Posting {
-   pub kbid: KbId,
-   pub heading_tf: f32,
-   pub body_tf: f32,
-   pub code_tf: f32,
-}
-```
-
-Each `Posting` stores per-field term frequencies
-instead of a single aggregate weight. Storing
-per-field TFs separately enables query-time field
-weight tuning (heading 2.0x, body 1.0x, code 1.5x)
-without rebuilding indexes. See
-[FieldTf](../../shared/query/types.md) for the
-shared struct used across module boundaries.
-
-## Methods
-
-```rust
-impl InvertedIndex {
-   pub fn lookup_term(&self, term: &Term)
-      -> Option<&PostingList>;
-
-   pub fn add_terms(
-      &mut self,
-      kbid: &KbId,
-      terms: &[(Term, FieldTf)],
-   );
-
-   pub fn remove_section(&mut self, kbid: &KbId);
-}
-```
-
-`add_terms` inserts or updates per-field TF
-entries for a section. Each `FieldTf` value
-contains the heading, body, and code term
-frequencies computed during per-field
-tokenization. `remove_section` removes all
-postings for a given kb-id across all terms.

package/docs/modules/rust/fs-database/indexes/module.md

@@ -1,64 +0,0 @@
-# fs-database/indexes -- B+ Tree Index Subsystem
-
-*Binary index files in `indexes/`. Optimized for read
-speed -- O(log n) term lookups.*
-
-## Source
-
-`engine/crates/fs-database/src/index/`
-
-## Modules
-
-- **[btree](btree.md)** -- Generic `BPlusTree<K, V>`
-  implementation. On-disk ordered key-value storage
-  with configurable page size.
-- **[inverted](inverted.md)** -- `InvertedIndex`.
-  Maps normalized terms to posting lists (kb-id +
-  weight). Stored in `indexes/terms.idx`.
-- **[positional](positional.md)** -- `PositionalIndex`.
-  Maps (term, kb-id) pairs to position lists for
-  phrase query support. Stored in
-  `indexes/positions.idx`.
-- **[corpus](corpus.md)** -- `CorpusStats`. Corpus-wide
-  term statistics for IDF computation. Stored in
-  `indexes/corpus.dat`.
-- **[references](references.md)** -- `ReferenceIndex`.
-  Bidirectional graph of content marker references
-  between sections. Stored in `indexes/references.idx`.
-
----
-
-### Reference Graph Index (`references.idx`)
-
-Bidirectional index mapping content marker
-references between sections. Built at section
-insertion time by parsing content markers via
-`extract_content_markers()` from `kbdb-shared`.
-
-**Index entry format:**
-
-| Field             | Type   | Description           |
-|-------------------|--------|-----------------------|
-| `source_kbid`     | KbId   | Section with marker   |
-| `target_id`       | String | Referenced kbid/docid |
-| `marker_type`     | u8     | ContentMarkerType enum value |
-| `context_snippet` | String | ~150 chars around marker |
-| `direction`       | u8     | 0 = forward, 1 = back |
-
-**Operations:**
-
-- **Build**: On section add/update, extract markers,
-  generate context snippets (~150 chars of the
-  surrounding sentence/paragraph), write one forward
-  entry per marker plus one back-reference entry at
-  each target.
-- **Remove**: On section delete, remove all forward
-  entries where `source_kbid` matches and all back
-  entries where `target_id` matches the deleted
-  section's kbid.
-- **Lookup**: Given a kbid, return all forward
-  references (markers in this section's content) and
-  all back-references (markers in other sections
-  pointing to this section). O(1) via B+ tree index.
-- **Rebuild**: Regenerated from section content
-  alongside other index files during `rebuild`.

package/docs/modules/rust/fs-database/indexes/positional.md

@@ -1,39 +0,0 @@
-# indexes::positional -- Positional Index
-
-*Maps (term, kb-id) pairs to position lists for
-phrase query support. Stored in
-`indexes/positions.idx`.*
-
-## Source
-
-`engine/crates/fs-database/src/index/positional.rs`
-
-## Type Definition
-
-```rust
-pub struct PositionalIndex {
-   tree: BPlusTree<(Term, KbId), Vec<u32>>,
-}
-```
-
-## Methods
-
-```rust
-impl PositionalIndex {
-   pub fn positions(
-      &self, term: &Term, kbid: &KbId,
-   ) -> Option<&[u32]>;
-
-   pub fn add_positions(
-      &mut self,
-      kbid: &KbId,
-      term_positions: &[(Term, Vec<u32>)],
-   );
-
-   pub fn remove_section(&mut self, kbid: &KbId);
-}
-```
-
-Position values are zero-based token offsets. Used
-for phrase matching (verifying adjacent term
-positions) and proximity scoring.

package/docs/modules/rust/fs-database/indexes/vectors.md

@@ -1,110 +0,0 @@
-# indexes::vectors -- Vector Embedding Index
-
-*Maps kb-ids to embedding vectors for similarity search. Stored in
-`indexes/vectors.idx` with metadata in `indexes/vectors-meta.toml`.*
-
-## Source
-
-`engine/crates/fs-database/src/index/vectors.rs`
-
-## Implementation Boundary
-
-Unlike other index files which are written by the Rust/WASM engine,
-the vector index is written and read by TypeScript. The Rust module
-defines the binary format specification and provides read/write
-utilities for potential future use, but the primary producer is the
-TypeScript embedding module. This separation exists because embedding
-computation requires network I/O to call external API providers, and
-the WASM target (`wasm32-unknown-unknown`) has no network access.
-ONNX models are also too large to fit in WASM linear memory.
-
-## Binary Format: vectors.idx
-
-Fixed-width binary file. A 32-byte header is followed by N records,
-one per section. Record size is fixed for a given file because all
-embeddings share the same `dimensions` value from the header.
-
-**Header (32 bytes):**
-
-| Offset | Size | Type      | Field                          |
-|--------|------|-----------|--------------------------------|
-| 0      | 4    | u32 LE    | magic (`0x56454358` = "VECX")  |
-| 4      | 4    | u32 LE    | version (1)                    |
-| 8      | 4    | u32 LE    | dimensions                     |
-| 12     | 4    | u32 LE    | record_count                   |
-| 16     | 16   | [u8; 16]  | reserved (zero-filled)         |
-
-**Record (variable, but fixed per file):**
-
-| Offset | Size          | Type           | Field                     |
-|--------|---------------|----------------|---------------------------|
-| 0      | 64            | [u8; 64]       | kbid (UTF-8, null-padded) |
-| 64     | dims × 4      | [f32 LE; dims] | embedding vector          |
-
-Record size = 64 + (dimensions × 4). For 384-dimensional embeddings,
-each record is 1,600 bytes.
-
-## Metadata: vectors-meta.toml
-
-```toml
-model = "all-MiniLM-L6-v2"
-dimensions = 384
-record_count = 1042
-version = 1
-```
-
-The metadata file is read before opening `vectors.idx`. If the
-`model` field differs from the current embedding provider's model
-name, the vector index is considered stale and must be rebuilt before
-it can be used for search.
-
-## Type Definitions
-
-```rust
-pub struct VectorIndex {
-    dimensions: u32,
-    entries: Vec<VectorEntry>,
-}
-
-pub struct VectorEntry {
-    pub kbid: KbId,
-    pub vector: Vec<f32>,
-}
-```
-
-`VectorIndex` owns all loaded entries in memory. `dimensions` is
-stored separately from the per-entry vector length so that the header
-value can be validated against each record during a read. Both structs
-are not exposed at the WASM boundary -- the index is consumed
-internally by `search` and `lookup`.
-
-## Methods
-
-```rust
-impl VectorIndex {
-    pub fn read_from(path: &str) -> Result<Self, FsError>;
-    pub fn write_to(&self, path: &str) -> Result<(), FsError>;
-    pub fn lookup(&self, kbid: &KbId) -> Option<&[f32]>;
-    pub fn search(
-        &self,
-        query_vector: &[f32],
-        limit: usize,
-    ) -> Vec<(KbId, f32)>;
-}
-```
-
-`read_from` parses the binary file header, validates the magic bytes
-and version, then reads all records into memory. `write_to` serializes
-the header followed by each entry in record order. `lookup` returns
-the stored embedding slice for a single kb-id, or `None` if the kb-id
-is not present. `search` computes cosine similarity between
-`query_vector` and every stored entry, then returns the top `limit`
-results as `(KbId, score)` pairs sorted by descending similarity.
-
-## Rebuild Semantics
-
-`vectors.idx` is derived data and can be regenerated from section
-content at any time. During rebuild, the TypeScript layer iterates all
-sections, computes embeddings via the configured `EmbeddingProvider`,
-and writes the binary file. If no embedding provider is configured,
-`vectors.idx` is not generated and similarity search is unavailable.

package/docs/modules/rust/fs-database/module.md

@@ -1,150 +0,0 @@
-# fs-database -- Filesystem Storage Engine
-
-*Rust crate that manages the `.kbdb` directory:
-section CRUD, document manifests, B+ tree indexes,
-and content-addressable storage. Compiles to
-`fs-database.wasm`.*
-
-## Crate
-
-| Field       | Value                              |
-|-------------|------------------------------------|
-| Name        | `kbdb-fs-database`                 |
-| Path        | `engine/crates/fs-database/`       |
-| Crate type  | `cdylib` (WASM) + `rlib` (tests)  |
-| WASM output | `dist/wasm/fs-database.wasm`       |
-| Depends on  | `kbdb-shared`                      |
-
-`kbdb-fs-database` is a **standalone** WASM module. It
-does not import `kbdb-query-parser` or `kbdb-worker` at
-the WASM level. `kb-worker.wasm` imports this module's
-exports and orchestrates it alongside
-`query-parser.wasm`.
-
-See [Database](../../../goals/database.md) for the
-full behavioral specification.
-
-## Workspace Position
-
-```
-engine/
-├── Cargo.toml                   # workspace root
-└── crates/
-    ├── shared/                  # kbdb-shared
-    ├── fs-database/             # this crate
-    ├── query-parser/            # kbdb-query-parser
-    └── kb-worker/               # kbdb-worker
-```
-
-## Directory Layout
-
-```
-engine/crates/fs-database/
-├── Cargo.toml
-└── src/
-    ├── lib.rs                   # crate root, WASM exports
-    ├── io.rs                    # host import wrappers
-    ├── mem.rs                   # shared-memory helpers
-    ├── catalog.rs               # catalog.toml read/write
-    ├── section/
-    │   ├── mod.rs               # section CRUD facade
-    │   ├── hash.rs              # SHA-256 hashing, kb-id
-    │   ├── parse.rs             # TOML frontmatter parser
-    │   └── shard.rs             # hash-sharded path logic
-    ├── document.rs              # document manifest CRUD
-    ├── index/
-    │   ├── mod.rs               # index facade
-    │   ├── btree.rs             # B+ tree implementation
-    │   ├── inverted.rs          # term → kb-id mappings
-    │   ├── positional.rs        # term position data
-    │   └── corpus.rs            # corpus-wide statistics
-    ├── query_exec.rs            # execute query plans
-    ├── integrity.rs             # checksum verify, GC
-    └── atomic.rs                # temp-file + rename
-```
-
-## Modules
-
-### [types/](types/module.md) -- Structs
-
-- **[catalog](types/catalog.md)** -- `Catalog` struct.
-  Database table of contents.
-- **[section](types/section.md)** -- `SectionHeader`,
-  frontmatter parse/serialize, shard path resolution.
-
-### [functions/](functions/module.md) -- Operations
-
-- **[io](functions/io.md)** -- Host import wrappers,
-  return conventions, shared memory protocol.
-- **[section](functions/section.md)** -- Section CRUD
-  (`read`, `add`, `update`, `remove`), content
-  hashing, atomic writes.
-- **[document](functions/document.md)** -- Document
-  manifest CRUD, `update_section_ref`.
-- **[query-exec](functions/query-exec.md)** -- Execute
-  query plans against indexes, return raw matches.
-- **[integrity](functions/integrity.md)** -- Integrity
-  checks, garbage collection, index rebuild.
-
-### [indexes/](indexes/module.md) -- Index Subsystem
-
-- **[btree](indexes/btree.md)** -- Generic B+ tree
-  for binary index files.
-- **[inverted](indexes/inverted.md)** -- Term →
-  posting list mappings (`terms.idx`).
-- **[positional](indexes/positional.md)** -- Term
-  position data for phrase queries
-  (`positions.idx`).
-- **[corpus](indexes/corpus.md)** -- Corpus-wide
-  statistics for IDF (`corpus.dat`).
-
-## WASM Exports
-
-| Export                 | Description                |
-|------------------------|----------------------------|
-| `db_init`              | Open database, read catalog |
-| `db_read_section`      | Read section by kb-id      |
-| `db_read_sections`     | Read multiple sections     |
-| `db_add_section`       | Create section file        |
-| `db_update_section`    | Replace section content    |
-| `db_remove_section`    | Delete section file        |
-| `db_read_document`     | Read document manifest     |
-| `db_write_document`    | Create/update document     |
-| `db_remove_document`   | Delete document manifest   |
-| `db_update_section_ref`| Replace kb-id in manifests |
-| `db_index_add`         | Add terms to inverted index |
-| `db_index_remove`      | Remove section from indexes |
-| `db_index_rebuild`     | Full index rebuild         |
-| `db_execute_plan`      | Run query plan against indexes |
-| `db_corpus_stats`      | Return corpus statistics   |
-| `db_status`            | Return database metadata   |
-| `db_check_integrity`   | Run integrity checks       |
-| `db_garbage_collect`   | Remove orphaned sections   |
-| `db_export`            | Export database snapshot    |
-| `alloc` / `free`       | Memory management (from `kbdb-shared`) |
-
-## Dependencies
-
-```toml
-[dependencies]
-kbdb-shared = { path = "../shared" }
-sha2 = "0.10"
-data-encoding = "2"
-toml = { version = "0.8", default-features = false }
-```
-
-## Build
-
-```sh
-cd engine
-wasm-pack build crates/fs-database \
-   --target no-modules \
-   --out-dir ../../../dist/wasm/fs-database
-```
-
-## Testing
-
-```sh
-cd engine && cargo test -p kbdb-fs-database
-cd engine && wasm-pack test --node crates/fs-database
-```

package/docs/modules/rust/fs-database/types/catalog.md

@@ -1,53 +0,0 @@
-# types::catalog -- Database Catalog
-
-*The `Catalog` struct representing `catalog.toml` --
-the database's table of contents.*
-
-## Source
-
-`engine/crates/fs-database/src/catalog.rs`
-
-## Type Definition
-
-```rust
-pub struct Catalog {
-   pub version: u32,
-   pub created_at: String,
-   pub last_modified: String,
-   pub section_count: u32,
-   pub document_count: u32,
-   pub checksum: String,
-}
-```
-
-| Field            | Description                     |
-|------------------|---------------------------------|
-| `version`        | Database format version         |
-| `created_at`     | ISO 8601 creation timestamp     |
-| `last_modified`  | ISO 8601 last write timestamp   |
-| `section_count`  | Total sections in the database  |
-| `document_count` | Total documents in the database |
-| `checksum`       | SHA-256 integrity checksum      |
-
-The catalog is the first file `fs-database.wasm`
-reads when opening a database. It is updated after
-every write operation using atomic temp-file + rename.
-
-## Functions
-
-```rust
-pub fn read_catalog() -> Result<Catalog, FsError>;
-
-pub fn write_catalog(cat: &Catalog)
-   -> Result<(), FsError>;
-
-pub fn increment_sections(delta: i32)
-   -> Result<Catalog, FsError>;
-
-pub fn increment_documents(delta: i32)
-   -> Result<Catalog, FsError>;
-```
-
-`increment_sections` and `increment_documents` read
-the catalog, adjust the count, update
-`last_modified`, and write back atomically.

package/docs/modules/rust/fs-database/types/module.md

@@ -1,12 +0,0 @@
-# fs-database/types -- Type Definitions
-
-*Structs defined in the fs-database crate for catalog
-metadata and section file parsing.*
-
-## Modules
-
-- **[catalog](catalog.md)** -- `Catalog` struct.
-  Database table of contents from `catalog.toml`.
-- **[section](section.md)** -- `SectionHeader` struct
-  and `IntegrityReport`. TOML frontmatter
-  parse/serialize, hash-sharded path resolution.