@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/modules/rust/shared/kbid/types.md

@@ -1,147 +0,0 @@
-# types::kbid -- Knowledge Base Section Identifier
-
-*Newtype wrapper around the 26-character,
-content-addressed section identifier used throughout
-the system.*
-
-## File
-
-`engine/crates/shared/src/kbid/kbid.rs`
-
-## Overview
-
-Every section in the knowledge base is identified by
-a **kb-id** -- a deterministic, collision-resistant,
-URL-safe string derived from the SHA-256 hash of the
-section's content. The `KbId` type encodes these
-invariants at the type level so that raw strings
-cannot be passed where a validated identifier is
-expected.
-
-See [Database -- Sections](../../../../../goals/database.md)
-for the behavioral specification of content-
-addressable storage.
-
-## Type Definition
-
-```rust
-#[derive(Clone, Debug, PartialEq, Eq, Hash)]
-pub struct KbId(String);
-```
-
-The inner `String` is private. Construction goes
-through `KbId::new` or `KbId::from_content`, both of
-which validate or derive the identifier. This
-prevents accidental construction from arbitrary
-strings.
-
-## Construction
-
-### `KbId::from_content`
-
-```rust
-impl KbId {
-   pub fn from_content(content: &[u8]) -> Self;
-}
-```
-
-Derives the kb-id from raw content bytes:
-
-1. Compute SHA-256 of `content`.
-2. Truncate the digest to 16 bytes (128 bits).
-3. Encode with base32 (Crockford variant, lowercase).
-4. Result: 26 alphanumeric characters.
-
-For text and code sections, `content` is the raw
-UTF-8 bytes. For image sections, `content` is the
-base64-encoded string bytes (not raw binary), so
-identical base64 input always produces the same
-kb-id.
-
-This function is infallible -- any byte slice
-produces a valid kb-id.
-
-### `KbId::new`
-
-```rust
-impl KbId {
-   pub fn new(s: &str) -> Result<Self, KbIdError>;
-}
-```
-
-Validates an existing identifier string:
-
-- Must be exactly 26 characters.
-- Must contain only lowercase alphanumeric characters
-  (`[a-z0-9]`).
-
-Returns `KbIdError::InvalidLength` or
-`KbIdError::InvalidCharacter` on failure.
-
-## Methods
-
-```rust
-impl KbId {
-   pub fn as_str(&self) -> &str;
-
-   pub fn shard_prefix(&self) -> &str;
-}
-```
-
-### `as_str`
-
-Returns the inner string slice. Used when passing
-the kb-id to I/O functions, serializers, or display.
-
-### `shard_prefix`
-
-Returns the first two characters of the kb-id. Used
-by `fs-database` to compute the hash-sharded
-directory path: `sections/{prefix}/{kbid}.sec`.
-
-```rust
-let id = KbId::from_content(b"hello");
-assert_eq!(id.shard_prefix().len(), 2);
-```
-
-## Trait Implementations
-
-| Trait        | Purpose                          |
-|--------------|----------------------------------|
-| `Clone`      | Cache entries need owned copies  |
-| `Debug`      | Diagnostic output                |
-| `PartialEq` / `Eq` | Identity comparison       |
-| `Hash`       | HashMap/HashSet keys (cache, index lookups) |
-| `Display`    | Formats as the raw 26-char string |
-| `Encode`     | Binary serialization for shared memory (see [encode](../encode/traits.md)) |
-
-`Ord` / `PartialOrd` are **not** derived. Kb-ids
-have no meaningful ordering -- they are content
-hashes. If sorting is needed (e.g. deterministic
-output), use `as_str()` for lexicographic comparison
-explicitly.
-
-## Error Type
-
-```rust
-#[derive(Debug)]
-pub enum KbIdError {
-   InvalidLength(usize),
-   InvalidCharacter(char, usize),
-}
-```
-
-## Testing
-
-Unit tests:
-
-- `from_content` produces 26-char lowercase
-  alphanumeric output for various inputs.
-- Identical content always produces the same kb-id.
-- Different content produces different kb-ids.
-- `shard_prefix` returns the first two characters.
-- `new` accepts valid 26-char strings.
-- `new` rejects strings that are too short, too long,
-  or contain uppercase/special characters.
-- `Encode` round-trip: encode then decode produces
-  an equal `KbId`.

package/docs/modules/rust/shared/memory/functions.md

@@ -1,209 +0,0 @@
-# functions::alloc -- Allocator and Read/Write Helpers
-
-*Allocator exports and shared-memory read/write
-helpers for passing data between the TypeScript host
-and WASM modules.*
-
-## File
-
-`engine/crates/shared/src/memory/alloc.rs`
-
-## Overview
-
-WASM modules and the TypeScript host share a single
-`WebAssembly.Memory` buffer. All data exchange --
-strings, byte slices, structured records -- passes
-through this buffer using `(pointer, length)` pairs.
-This module provides:
-
-1. **Allocator exports** (`kbdb_alloc`, `kbdb_free`)
-   that the TypeScript host calls to manage buffers.
-2. **Read helpers** that safely reconstruct Rust types
-   from raw memory pointers.
-3. **Write helpers** that serialize Rust types into
-   the shared buffer and return pointer/length pairs.
-
-Every WASM module re-exports `kbdb_alloc` and
-`kbdb_free` from this module so the host can allocate
-memory in any module's address space.
-
-## Allocator Exports
-
-```rust
-use std::alloc::{alloc, dealloc, Layout};
-
-#[no_mangle]
-pub extern "C" fn kbdb_alloc(size: u32) -> *mut u8 {
-   let layout = Layout::from_size_align(
-      size as usize, 1,
-   ).unwrap();
-   unsafe { alloc(layout) }
-}
-
-#[no_mangle]
-pub extern "C" fn kbdb_free(
-   ptr: *mut u8, size: u32,
-) {
-   let layout = Layout::from_size_align(
-      size as usize, 1,
-   ).unwrap();
-   unsafe { dealloc(ptr, layout) }
-}
-```
-
-### Usage from TypeScript
-
-The TypeScript daemon uses this protocol to pass
-data into WASM functions:
-
-```
-1. host calls kbdb_alloc(size) → ptr
-2. host writes bytes into memory at ptr
-3. host calls wasm_function(ptr, size)
-4. wasm reads from ptr
-5. host calls kbdb_free(ptr, size)
-```
-
-For return values:
-
-```
-1. wasm calls kbdb_alloc(size) internally → ptr
-2. wasm writes result into ptr
-3. wasm returns (ptr, size) to host
-4. host reads bytes from memory at ptr
-5. host calls kbdb_free(ptr, size)
-```
-
-### Alignment
-
-Alignment is `1` (byte-aligned). The data exchanged
-is always byte streams (UTF-8 strings, binary
-encoded structs). No multi-byte alignment is needed
-at the allocation level -- the `Encode` trait handles
-field alignment internally.
-
-### Null and Zero
-
-`kbdb_alloc(0)` returns a non-null, dangling pointer
-(per Rust's allocator contract). Callers must not
-dereference zero-length allocations. `kbdb_free` with
-size `0` is a no-op.
-
-## Read Helpers
-
-Safe functions that reconstruct Rust types from raw
-shared-memory pointers.
-
-```rust
-pub unsafe fn read_str<'a>(
-   ptr: *const u8, len: u32,
-) -> &'a str;
-
-pub unsafe fn read_bytes<'a>(
-   ptr: *const u8, len: u32,
-) -> &'a [u8];
-```
-
-### `read_str`
-
-Interprets the byte range `[ptr, ptr + len)` as
-UTF-8. Panics if the bytes are not valid UTF-8.
-
-This is used for string arguments from the host
-(file paths, query strings, MIME types). The host
-always writes valid UTF-8 -- a panic here indicates
-a host bug.
-
-### `read_bytes`
-
-Returns the byte range `[ptr, ptr + len)` as a
-slice. No UTF-8 validation. Used for binary data
-(section content, encoded structs).
-
-## Write Helpers
-
-Functions that serialize Rust types into the shared
-buffer and return `(pointer, length)` pairs for the
-caller to pass back to the host or another module.
-
-```rust
-pub fn write_str(s: &str) -> (*const u8, u32);
-
-pub fn write_bytes(data: &[u8]) -> (*const u8, u32);
-
-pub fn write_encoded<T: Encode>(
-   value: &T,
-) -> (*const u8, u32);
-```
-
-### `write_str`
-
-Allocates `s.len()` bytes via `kbdb_alloc`, copies
-the UTF-8 bytes, and returns the pointer and length.
-The caller (or host) is responsible for freeing the
-allocation.
-
-### `write_bytes`
-
-Same as `write_str` but for arbitrary byte slices.
-
-### `write_encoded`
-
-Encodes `value` using the `Encode` trait (see
-[encode](../encode/traits.md)), allocates a
-buffer for the result, copies the encoded bytes, and
-returns the pointer and length. Convenience wrapper
-for returning structured data from WASM exports.
-
-## Return Value Convention
-
-WASM functions can only return scalar values. For
-functions that return variable-length data, the
-convention is a **two-slot return buffer**:
-
-```rust
-pub fn set_return(ptr: *const u8, len: u32);
-pub fn get_return_ptr() -> *const u8;
-pub fn get_return_len() -> u32;
-```
-
-A pair of module-global slots holds the most recent
-return value. The WASM export writes `(ptr, len)`
-into the slots via `set_return`, returns a status
-code (`0` for success, negative for error), and the
-host reads the actual data via `get_return_ptr` and
-`get_return_len`.
-
-This avoids multi-value returns, which have uneven
-support across WASM runtimes.
-
-## Safety
-
-All read functions are `unsafe` because they
-dereference raw pointers. The safety contract:
-
-- `ptr` must point into the shared
-  `WebAssembly.Memory` buffer.
-- `ptr + len` must not exceed the buffer's current
-  size.
-- The memory region must not be freed or
-  reallocated while the returned reference is alive.
-
-The write functions are safe -- they allocate fresh
-buffers and copy data in.
-
-## Testing
-
-Unit tests (native Rust, not WASM):
-
-- `kbdb_alloc` + `kbdb_free` round-trip without
-  panic for various sizes (1, 256, 65536).
-- `kbdb_alloc(0)` returns non-null.
-- `write_str` + `read_str` round-trip preserves
-  content for ASCII, Unicode, and empty strings.
-- `write_bytes` + `read_bytes` round-trip preserves
-  arbitrary byte sequences.
-- `write_encoded` + `Encode::decode` round-trip for
-  sample types.
-- Return value slots: `set_return` → `get_return_ptr`
-  / `get_return_len` retrieves the same values.

package/docs/modules/rust/shared/memory/module.md

@@ -1,24 +0,0 @@
-# memory -- Memory Management
-
-*Allocator exports and shared-memory read/write
-helpers for passing data between the TypeScript host
-and WASM modules.*
-
-## Source
-
-`engine/crates/shared/src/memory/`
-
-## Responsibility
-
-Provides the shared-memory allocation protocol used
-by all four WASM modules. Every WASM export that
-exchanges variable-length data (strings, byte
-slices, encoded structs) goes through this module's
-allocator and read/write helpers.
-
-## Exports
-
-- **[functions.md](functions.md)** -- `kbdb_alloc`,
-  `kbdb_free`, `read_str`, `read_bytes`, `write_str`,
-  `write_bytes`, `write_encoded`, `set_return`,
-  `get_return_ptr`, `get_return_len`.

package/docs/modules/rust/shared/module.md

@@ -1,196 +0,0 @@
-# shared -- Shared Types and Utilities
-
-*Rust crate that provides types, memory helpers, error
-types, the tokenization pipeline, and serialization
-utilities shared across all four WASM modules.*
-
-## Crate
-
-| Field       | Value                              |
-|-------------|------------------------------------|
-| Name        | `kbdb-shared`                      |
-| Path        | `engine/crates/shared/`            |
-| Crate type  | `rlib` (library only, not WASM)    |
-| Depends on  | nothing (leaf crate)               |
-
-`kbdb-shared` is a pure Rust library. It is **not**
-compiled to WASM on its own -- it is compiled into
-each of the four WASM modules that depend on it
-(`kbdb-fs-database`, `kbdb-query-parser`,
-`kbdb-default-embedding`, `kbdb-worker`).
-
-## Workspace Position
-
-```
-engine/
-├── Cargo.toml                   # workspace root
-└── crates/
-    ├── shared/                  # this crate (leaf)
-    ├── fs-database/             # depends on shared
-    ├── query-parser/            # depends on shared
-    ├── default-embedding/       # depends on shared
-    └── kb-worker/               # depends on shared
-```
-
-## Directory Layout
-
-```
-engine/crates/shared/
-├── Cargo.toml
-└── src/
-    ├── lib.rs                   # crate root, re-exports
-    ├── memory/
-    │   ├── mod.rs               # memory module facade
-    │   └── alloc.rs             # kbdb_alloc, kbdb_free
-    ├── encode/
-    │   ├── mod.rs               # encode module facade
-    │   └── encode.rs            # Encode trait + impls
-    ├── kbid/
-    │   ├── mod.rs               # kbid module facade
-    │   └── kbid.rs              # KbId newtype
-    ├── term/
-    │   ├── mod.rs               # term module facade
-    │   └── term.rs              # Term newtype
-    ├── section/
-    │   ├── mod.rs               # section module facade
-    │   └── section.rs           # SectionType, records
-    ├── document/
-    │   ├── mod.rs               # document module facade
-    │   └── document.rs          # DocumentManifest, records
-    ├── query/
-    │   ├── mod.rs               # query module facade
-    │   └── query.rs             # QueryPlan, RawMatch
-    ├── corpus/
-    │   ├── mod.rs               # corpus module facade
-    │   └── corpus.rs            # CorpusStats
-    ├── content-parser/
-    │   ├── mod.rs               # content-parser facade
-    │   ├── types.rs             # ContentMarkerType,
-    │   │                        # ExtractedContentMarker
-    │   ├── extract.rs           # extract_content_markers()
-    │   └── strip.rs             # strip_content_markers()
-    ├── content-composer/
-    │   ├── mod.rs               # content-composer facade
-    │   ├── types.rs             # ComposeRequest, etc.
-    │   ├── compose.rs           # compose_content()
-    │   └── render.rs            # marker rendering
-    ├── pipeline/
-    │   ├── mod.rs               # pipeline orchestrator
-    │   ├── markers.rs           # stage 1: strip markers
-    │   ├── unicode.rs           # stage 2: NFC normalize
-    │   ├── segment.rs           # stage 3: word boundaries
-    │   ├── identifiers.rs       # stage 4: camelCase split
-    │   ├── case.rs              # stage 5: case folding
-    │   ├── stopwords.rs         # stage 6: stop words
-    │   ├── synonyms.rs          # stage 7: synonym expand
-    │   ├── stem.rs              # stage 8: Snowball stemmer
-    │   ├── ngram.rs             # stage 9: n-gram extract
-    │   └── token_result.rs      # TokenResult struct
-    └── error/
-        ├── mod.rs               # error module facade
-        └── error.rs             # FsError, IndexError, etc.
-```
-
-## Modules
-
-Each sub-module has a single responsibility. Files
-within each module mirror their Rust source
-counterparts: `module.md` = `mod.rs`,
-`types.md` = type definitions,
-`functions.md` = function definitions,
-`traits.md` = trait definitions.
-
-### [memory/](memory/module.md) -- Memory Management
-
-- [functions.md](memory/functions.md) --
-  `kbdb_alloc`, `kbdb_free`, read/write helpers,
-  return value convention.
-
-### [encode/](encode/module.md) -- Binary Encoding
-
-- [traits.md](encode/traits.md) -- `Encode` trait,
-  wire format, per-type implementations.
-
-### [kbid/](kbid/module.md) -- Content-Addressed ID
-
-- [types.md](kbid/types.md) -- `KbId` newtype,
-  `from_content`, `shard_prefix`, `KbIdError`.
-
-### [term/](term/module.md) -- Search Term
-
-- [types.md](term/types.md) -- `Term` newtype,
-  `Ord` for B+ tree keys, `TermError`.
-
-### [section/](section/module.md) -- Section Types
-
-- [types.md](section/types.md) -- `SectionType`,
-  `MimeType`, `SectionRecord`, CRUD params/results.
-
-### [document/](document/module.md) -- Document Records
-
-- [types.md](document/types.md) --
-  `DocumentManifest`, `DocumentSection`,
-  `DocumentRecord`.
-
-### [query/](query/module.md) -- Query Plan
-
-- [types.md](query/types.md) -- `QueryPlan`,
-  `RawMatch`, `MatchMode`.
-
-### [corpus/](corpus/module.md) -- Corpus Statistics
-
-- [types.md](corpus/types.md) -- `CorpusStats`.
-
-### [content-parser/](content-parser/module.md) -- Content Parser
-
-- [types.md](content-parser/types.md) --
-  `ContentMarkerType` enum,
-  `ExtractedContentMarker` struct.
-- [functions.md](content-parser/functions.md) --
-  `extract_content_markers()`,
-  `strip_content_markers()`.
-
-### [content-composer/](content-composer/module.md) -- Content Composer
-
-- [types.md](content-composer/types.md) --
-  `ComposeRequest`, `ComposeResult`,
-  `RenderedSection`, `ResolvedId`, `SectionData`,
-  `DocumentData`.
-- [functions.md](content-composer/functions.md) --
-  `compose_content()`.
-
-### [pipeline/](pipeline/module.md) -- Tokenization Pipeline
-
-- [functions.md](pipeline/functions.md) --
-  `tokenize()`, `bigrams()`, `trigrams()`. 9-stage
-  pipeline: marker stripping, NFC, segmentation,
-  identifier splitting, case folding, stop words,
-  synonym expansion, stemming, n-grams.
-- [types.md](pipeline/types.md) -- `TokenResult`,
-  `TokenStream`, `Ngram`, `NgramType`, `StemResult`.
-
-Consumers: `kbdb-query-parser` (re-exports via WASM
-FFI), `kbdb-default-embedding` (TF-IDF vectors).
-
-### [error/](error/module.md) -- Error Types
-
-- [types.md](error/types.md) -- `FsError`,
-  `IndexError`, `ParseError`, `WorkerError`.
-
-## Dependencies
-
-```toml
-[dependencies]
-wasm-bindgen = "0.2"
-unicode-segmentation = "1"
-unicode-normalization = "0.1"
-rust-stemmers = "1"
-```
-
-## Testing
-
-```sh
-cd engine && cargo test -p kbdb-shared
-```
-
-See each module doc for detailed test descriptions.

package/docs/modules/rust/shared/pipeline/functions.md

@@ -1,141 +0,0 @@
-# pipeline -- Tokenization Pipeline Functions
-
-*9-stage pipeline that converts raw text into
-normalized, stemmed tokens with synonym expansion.
-Used for both indexing and query processing.*
-
-## Source
-
-`engine/crates/shared/src/pipeline/`
-
-## Entry Point
-
-```rust
-pub fn tokenize(input: &str) -> TokenStream;
-```
-
-Runs all nine stages in sequence. Each stage is a
-pure function.
-
-## Stages
-
-### 1. Content Marker Stripping
-
-```rust
-use kbdb_shared::content_parser::strip_content_markers;
-```
-
-Removes all six content marker types from the input
-text before tokenization:
-`$(image:)`, `$(code:)`, `$(text:)`, `$(document:)`,
-`$(link:)`, `$(doc-link:)`.
-
-Delegates to `strip_content_markers()` from the
-sibling `content_parser` module (see
-[Content Parser](../content-parser/module.md)).
-These are structural references, not searchable
-content.
-
-### 2. Unicode Normalization (`unicode.rs`)
-
-```rust
-pub fn normalize_nfc(input: &str) -> String;
-```
-
-NFC normalization for consistent codepoints.
-
-### 3. Word Boundary Detection (`segment.rs`)
-
-```rust
-pub fn segment_words(input: &str) -> Vec<&str>;
-```
-
-Unicode Segmentation (UAX #29).
-
-### 4. Identifier Splitting (`identifiers.rs`)
-
-```rust
-pub fn split_identifiers(word: &str) -> Vec<String>;
-```
-
-- `camelCase` → `["camel", "case"]`
-- `snake_case` → `["snake", "case"]`
-- `PascalCase` → `["pascal", "case"]`
-- `HTMLParser` → `["html", "parser"]`
-
-### 5. Case Folding (`case.rs`)
-
-```rust
-pub fn fold_case(word: &str) -> String;
-```
-
-Unicode case folding (lowercase).
-
-### 6. Stop Word Removal (`stopwords.rs`)
-
-```rust
-pub fn is_stop_word(word: &str) -> bool;
-pub fn remove_stop_words(
-   words: &[String],
-) -> Vec<String>;
-```
-
-Filters common function words. Single-character
-tokens also removed.
-
-### 7. Synonym Expansion (`synonyms.rs`)
-
-```rust
-pub fn expand_synonyms(
-   word: &str,
-   map: &SynonymMap,
-) -> Vec<String>;
-```
-
-Looks up the lowercased word in the built-in synonym
-map. Returns a vector containing the original word
-plus any synonyms. When no synonyms exist, returns a
-single-element vector containing only the original.
-
-The synonym map is a bidirectional
-`HashMap<String, Vec<String>>` built at compile time.
-Expansion is applied after stop-word removal and
-before stemming, so both original and synonym tokens
-are stemmed consistently.
-
-The map is small (~30-50 entries) and focused on
-common IT/development vocabulary mismatches:
-login/sign-in, auth/authentication, remove/delete,
-config/configuration, credential/password, etc.
-
-### 8. Stemming (`stem.rs`)
-
-```rust
-pub fn stem_word(word: &str) -> StemResult;
-```
-
-Snowball / Porter2. Preserves both original and
-stemmed forms.
-
-### 9. N-gram Extraction (`ngram.rs`)
-
-```rust
-pub fn extract_ngrams(
-   tokens: &[Token],
-) -> Vec<Ngram>;
-```
-
-Bigrams and trigrams for phrase indexing.
-
-## Relaxed Query Mode
-
-The tokenization pipeline is used for both the
-primary AND-mode query plan and the relaxed OR-mode
-fallback plan. `qp_parse_query` produces a
-`QueryPlan` with a `match_mode` field (`All` | `Any`).
-When the AND plan returns zero matches,
-`kb-worker.wasm` retries with the OR plan using the
-same tokenized terms. No additional pipeline stage or
-WASM export is needed. See
-[Query Parser -- Relaxed Query
-Mode](../../../../goals/query-parser.md).