@dikolab/kbdb 0.5.0 → 0.6.0

package/docs/modules/rust/query-parser/functions/rank.md

@@ -1,85 +0,0 @@
-# functions::rank -- Ranking Functions
-
-*Scoring functions that `kb-worker.wasm` calls after
-retrieving raw index results from `fs-database.wasm`.*
-
-## Source
-
-`engine/crates/query-parser/src/rank/`
-
-## BM25F (`bm25f.rs`)
-
-```rust
-pub fn score_bm25f(
-   params: &BM25FParams,
-   raw_match: &RawMatch,
-   corpus: &CorpusStats,
-) -> f32;
-```
-
-Primary ranking function. Field-weighted Okapi BM25
-with configurable saturation (`k1`), length
-normalization (`b`), and per-field boost weights.
-
-Per-field BM25F formula:
-
-```
-score_f(t, d) = idf(t)
-   * (tf_f(t,d) * (k1 + 1))
-   / (tf_f(t,d) + k1
-      * (1 - b + b * |d_f| / avgdl_f))
-
-bm25f(t, d) = w_heading * score_heading(t, d)
-            + w_body * score_body(t, d)
-            + w_code * score_code(t, d)
-```
-
-`raw_match.field_tfs` provides per-field term
-frequencies (`tf_f`). `raw_match.field_token_counts`
-provides per-field section lengths (`|d_f|`).
-`corpus` provides per-field average lengths
-(`avgdl_f`). Field weights are applied at query
-time, making them tunable without index rebuilds.
-See [Query Result](../../../../goals/query-result.md)
-for the full scoring specification.
-
-## Proximity (`proximity.rs`)
-
-```rust
-pub fn score_proximity(
-   positions: &[(Term, Vec<u32>)],
-   query_terms: &[Term],
-) -> f32;
-```
-
-Boosts sections where query terms appear close
-together. Computes minimum span across all query
-terms, inversely weighted by span length.
-
-## Zone (`zone.rs`)
-
-```rust
-pub fn score_zone(
-   positions: &[(Term, Vec<u32>)],
-   section_length: u32,
-) -> f32;
-```
-
-Structural weighting. Terms in the first ~20% of a
-section (heading zone) receive higher weight. Uses
-`section_length` from `RawMatch.token_count`
-(pre-computed during indexing) to determine the
-zone boundary.
-
-## Freshness (`freshness.rs`)
-
-```rust
-pub fn score_freshness(
-   created_at: u64,
-   now: u64,
-   half_life_days: f32,
-) -> f32;
-```
-
-Optional boost based on insertion timestamps.
-Exponential decay with configurable half-life.

package/docs/modules/rust/query-parser/module.md

@@ -1,131 +0,0 @@
-# query-parser -- Query Processing and Tokenization
-
-*Rust crate that provides query parsing, keyword
-extraction, and ranking functions. Tokenization is
-imported from `kbdb-shared`'s `pipeline` module.
-Compiles to `query-parser.wasm`.*
-
-## Crate
-
-| Field       | Value                              |
-|-------------|------------------------------------|
-| Name        | `kbdb-query-parser`                |
-| Path        | `engine/crates/query-parser/`      |
-| Crate type  | `cdylib` (WASM) + `rlib` (tests)  |
-| WASM output | `dist/wasm/query-parser.wasm`      |
-| Depends on  | `kbdb-shared`                      |
-
-`kbdb-query-parser` is a **standalone** WASM module.
-Pure computation -- no file I/O, no network. The only
-host import is `WebAssembly.Memory`.
-
-See [Query Parser](../../../goals/query-parser.md)
-for the full behavioral specification.
-
-## Workspace Position
-
-```
-engine/
-├── Cargo.toml                   # workspace root
-└── crates/
-    ├── shared/                  # kbdb-shared
-    ├── fs-database/             # kbdb-fs-database
-    ├── query-parser/            # this crate
-    └── kb-worker/               # kbdb-worker
-```
-
-## Directory Layout
-
-```
-engine/crates/query-parser/
-├── Cargo.toml
-└── src/
-    ├── lib.rs                   # crate root, WASM exports
-    ├── mem.rs                   # shared-memory helpers
-    ├── parse.rs                 # query string → QueryPlan
-    ├── extract/
-    │   ├── mod.rs               # extraction facade
-    │   ├── rake.rs              # RAKE algorithm
-    │   └── tfidf.rs             # TF-IDF weighting
-    ├── rank/
-    │   ├── mod.rs               # ranking facade
-    │   ├── bm25f.rs             # BM25F scoring
-    │   ├── proximity.rs         # proximity scoring
-    │   ├── zone.rs              # zone scoring
-    │   └── freshness.rs         # freshness weighting
-    └── cache.rs                 # query-scoped cache
-```
-
-The tokenization pipeline lives in `kbdb-shared`'s
-[pipeline](../shared/pipeline/module.md) module.
-This crate re-exports it via WASM FFI
-(`qp_tokenize`).
-
-## Modules
-
-### [types/](types/module.md) -- Structs
-
-- **[token](types/token.md)** -- `Token`,
-  `TokenStream`, `Ngram`, `NgramType`, `StemResult`.
-- **[keyphrase](types/keyphrase.md)** -- `Keyphrase`,
-  `WeightedTerm`, `BM25FParams`, `FieldWeights`.
-- **[cache](types/cache.md)** -- `QueryCache`.
-  Short-lived query-scoped cache.
-
-### [functions/](functions/module.md) -- Operations
-
-- **[parse](functions/parse.md)** -- Query string
-  parsing into `QueryPlan`.
-- **[extract](functions/extract.md)** -- RAKE
-  keyphrase extraction and TF-IDF term weighting.
-- **[rank](functions/rank.md)** -- BM25F, proximity,
-  zone, and freshness scoring functions.
-
-Tokenization is provided by
-[kbdb-shared pipeline](../shared/pipeline/module.md).
-
-## WASM Exports
-
-| Export                   | Description             |
-|--------------------------|-------------------------|
-| `qp_tokenize`            | Full tokenization pipeline |
-| `qp_parse_query`         | Parse query → QueryPlan |
-| `qp_normalize_term`      | Normalize a single term |
-| `qp_stem`                | Stem a single term      |
-| `qp_extract_keywords`    | RAKE keyphrase extraction |
-| `qp_compute_tf`          | Single-stream term frequency |
-| `qp_compute_field_tf`    | Per-field term frequency (heading, body, code) |
-| `qp_compute_tfidf`       | TF-IDF with supplied IDF |
-| `qp_score_bm25f`         | BM25F ranking score     |
-| `qp_score_proximity`     | Proximity ranking score |
-| `qp_score_zone`          | Zone ranking score      |
-| `qp_score_freshness`     | Freshness ranking score |
-| `qp_split_identifiers`   | camelCase/snake_case split |
-| `alloc` / `free`         | Memory management (from `kbdb-shared`) |
-
-## Dependencies
-
-```toml
-[dependencies]
-kbdb-shared = { path = "../shared" }
-```
-
-Tokenization dependencies (`unicode-segmentation`,
-`unicode-normalization`, `rust-stemmers`) are now in
-`kbdb-shared` alongside the `pipeline` module.
-
-## Build
-
-```sh
-cd engine
-wasm-pack build crates/query-parser \
-   --target no-modules \
-   --out-dir ../../../dist/wasm/query-parser
-```
-
-## Testing
-
-```sh
-cd engine && cargo test -p kbdb-query-parser
-cd engine && wasm-pack test --node crates/query-parser
-```

package/docs/modules/rust/query-parser/types/cache.md

@@ -1,39 +0,0 @@
-# types::cache -- Query-Scoped Cache
-
-*Short-lived cache scoped to a single query or
-indexing operation.*
-
-## Source
-
-`engine/crates/query-parser/src/cache.rs`
-
-## Type Definition
-
-```rust
-pub struct QueryCache {
-   tokenized: HashMap<String, Vec<Token>>,
-   stemmed: HashMap<String, String>,
-}
-
-impl QueryCache {
-   pub fn new() -> Self;
-   pub fn get_tokens(&self, input: &str)
-      -> Option<&[Token]>;
-   pub fn put_tokens(
-      &mut self, input: String, tokens: Vec<Token>,
-   );
-   pub fn get_stem(&self, word: &str)
-      -> Option<&str>;
-   pub fn put_stem(
-      &mut self, word: String, stem: String,
-   );
-}
-```
-
-Allocated when processing begins, discarded
-immediately after the operation completes. Prevents
-redundant tokenization and stemming when the same
-term appears multiple times in a query or content.
-
-Not persisted, not shared across queries, not
-visible to other modules.

package/docs/modules/rust/query-parser/types/keyphrase.md

@@ -1,71 +0,0 @@
-# types::keyphrase -- Extraction and Ranking Types
-
-*Types for keyword extraction results and ranking
-function configuration.*
-
-## Source
-
-`engine/crates/query-parser/src/extract/mod.rs`,
-`engine/crates/query-parser/src/rank/bm25f.rs`
-
-## Type Definitions
-
-```rust
-pub struct Keyphrase {
-   pub terms: Vec<String>,
-   pub score: f32,
-}
-
-pub struct WeightedTerm {
-   pub term: Term,
-   pub tf: f32,
-   pub weight: f32,
-}
-
-pub struct BM25FParams {
-   pub k1: f32,           // term saturation (1.2)
-   pub b: f32,            // length normalization (0.75)
-   pub field_weights: FieldWeights,
-}
-
-pub struct FieldWeights {
-   pub heading: f32,      // default 2.0
-   pub body: f32,         // default 1.0
-   pub code: f32,         // default 1.5
-}
-```
-
-`Keyphrase` is the output of RAKE extraction --
-multi-word candidate phrases scored by word degree /
-frequency ratio.
-
-`WeightedTerm` combines a term with its per-section
-TF and final TF-IDF weight. This is an
-extraction-time type used during keyword extraction.
-At ranking time, the per-field `FieldTf` from
-`RawMatch` (see
-[shared/query/types](../../shared/query/types.md))
-is used instead -- it contains separate heading,
-body, and code frequencies pre-computed during
-indexing.
-
-`BM25FParams` configures the primary ranking function
-with field-specific boost weights. Each field weight
-in `FieldWeights` multiplies the per-field BM25
-score computed from the corresponding `FieldTf`
-component and per-field corpus average from
-`CorpusStats`. See
-[FieldTokenCounts](../../shared/query/types.md)
-for the per-field token counts used in length
-normalization.
-
-```rust
-pub type SynonymMap = HashMap<String, Vec<String>>;
-```
-
-`SynonymMap` is a type alias for the bidirectional
-synonym lookup used in tokenization pipeline stage 7.
-It is constructed once via `build_synonym_map()` from
-a curated list of synonym pairs and reused across all
-tokenization calls. The map is immutable after
-construction.

package/docs/modules/rust/query-parser/types/module.md

@@ -1,18 +0,0 @@
-# query-parser/types -- Type Definitions
-
-*Structs defined in the query-parser crate for
-keyword extraction and ranking.*
-
-## Modules
-
-- **[keyphrase](keyphrase.md)** -- `Keyphrase`,
-  `WeightedTerm`, `BM25FParams`, `FieldWeights`,
-  `SynonymMap`. Types for keyword extraction, ranking
-  configuration, and synonym expansion.
-- **[cache](cache.md)** -- `QueryCache`. Short-lived
-  cache scoped to a single query or indexing
-  operation.
-
-Token types (`Token`, `TokenStream`, `Ngram`,
-`NgramType`, `StemResult`) are defined in
-[kbdb-shared pipeline](../../shared/pipeline/types.md).

package/docs/modules/rust/query-parser/types/token.md

@@ -1,46 +0,0 @@
-# types::token -- Tokenization Types
-
-*Output types from the tokenization pipeline.*
-
-## Source
-
-`engine/crates/shared/src/pipeline/mod.rs`
-
-**Note:** These types are now defined in `kbdb-shared`.
-See [shared/pipeline/types](../../shared/pipeline/types.md)
-for the canonical documentation.
-
-## Type Definitions
-
-```rust
-pub struct Token {
-   pub original: String,
-   pub stemmed: String,
-   pub position: u32,
-}
-
-pub struct TokenStream {
-   pub tokens: Vec<Token>,
-   pub ngrams: Vec<Ngram>,
-}
-
-pub struct Ngram {
-   pub terms: Vec<String>,
-   pub gram_type: NgramType,
-}
-
-pub enum NgramType { Bigram, Trigram }
-
-pub struct StemResult {
-   pub original: String,
-   pub stemmed: String,
-}
-```
-
-`Token` preserves both original and stemmed forms so
-the index can support exact-match and stem-match
-queries. `position` is the zero-based token offset.
-
-`TokenStream` is the output of `tokenize()` -- the
-full set of tokens and extracted n-grams from a
-single input text.

package/docs/modules/rust/shared/content-composer/functions.md

@@ -1,114 +0,0 @@
-# content-composer -- Functions
-
-## `compose_content`
-
-```rust
-pub fn compose_content(
-   request: &ComposeRequest,
-) -> ComposeResult
-```
-
-Takes a fully resolved request (all sections and
-documents pre-fetched, content markers pre-extracted)
-and returns the composed Markdown string. The
-function is pure -- no I/O, no database access.
-Resolution and fetching are the caller's
-responsibility (`kb-worker`).
-
-### Composition Flow
-
-1. Initialize `renderedSections` and
-   `renderedDocuments` tracking sets.
-2. For each `ResolvedId` in `main_ids` (in order):
-   - If `Section(kbid)`: render as a section (anchor
-     + heading + content with marker processing), add
-     to `renderedSections`.
-   - If `Document(docid)`: render as a document
-     section (see Document Rendering below), add to
-     `renderedDocuments`.
-3. Traverse the full references set and render any
-   entities not yet in `renderedSections` or
-   `renderedDocuments` (remaining references, placed
-   at the bottom of the output).
-4. Return `ComposeResult` with the composed Markdown
-   and character count.
-
-### Content Marker Rendering
-
-Each `$(type: value)` content marker in a section's
-body is replaced by its rendered form:
-
-| Marker             | Rendered output                   |
-|--------------------|-----------------------------------|
-| `$(image: {kbid})` | Inline image (base64-decoded content from referenced `image/*` section) |
-| `$(code: {kbid})`  | Fenced code block with language tag derived from MIME type |
-| `$(text: {kbid})`  | Inline expansion of referenced section content (recursive, with cycle detection) |
-| `$(document: {docid})` | Document section rendering (heading + subsections, recursive) |
-| `$(link: {kbid})`  | Markdown link `[{kbid}](#{kbid})` (no inline expansion) |
-| `$(doc-link: {docid})` | Markdown link `[{docid}](#{docid})` (no inline expansion) |
-
-Cyclic `$(text:)` and `$(document:)` embeds (entity
-already in the render stack) are rendered as Markdown
-links to the entity's anchor instead of expanding
-inline.
-
-### MIME-to-Language Mapping
-
-For `$(code:)` markers, the fenced code block
-language tag is derived from the referenced section's
-MIME type:
-
-| MIME type              | Language tag |
-|------------------------|-------------|
-| `text/typescript`      | `ts`        |
-| `text/javascript`      | `js`        |
-| `text/html`            | `html`      |
-| `text/css`             | `css`       |
-| `text/xml`             | `xml`       |
-| `text/plain`           | *(none)*    |
-| `application/json`     | `json`      |
-| other `text/*`         | subtype     |
-
-### Document Rendering
-
-When a document is rendered (from `$(document:)`
-marker, `$(doc-link:)` target, or main docid input):
-
-1. Emit document anchor: `<a id="{docid}"></a>`
-2. Emit heading with document title.
-3. For each section in the document (position order):
-   - Emit section anchor: `<a id="{kbid}"></a>`
-   - Emit sub-heading.
-   - Render section content with marker processing.
-   - Add kbid to `renderedSections`.
-4. Add docid to `renderedDocuments`.
-
-### Tests
-
-Unit tests in
-`engine/crates/shared/src/content-composer/compose.rs`
-and `render.rs`:
-
-- Empty request returns empty markdown.
-- Single text section with no markers.
-- Single section with `$(image:)` marker.
-- Single section with `$(code:)` marker and correct
-  language tag.
-- Recursive `$(text:)` embedding.
-- `$(document:)` renders document heading with
-  section subsections.
-- Cyclic `$(text:)` embed becomes link to anchor.
-- Cyclic `$(document:)` embed becomes link to
-  document anchor.
-- `$(link:)` renders as Markdown link, target
-  rendered at bottom.
-- `$(doc-link:)` renders as Markdown link pointing
-  to document anchor.
-- Mixed kbids and docids in main list.
-- Named anchors assigned correctly to all entities.
-- `renderedSections` prevents duplicate rendering.
-- `renderedDocuments` prevents duplicate document
-  rendering.
-- Remaining references rendered at bottom only for
-  unrendered entities.
-- Unicode content preserved through composition.

package/docs/modules/rust/shared/content-composer/module.md

@@ -1,147 +0,0 @@
-# content-composer -- Content Composition
-
-*Composition engine that resolves sections and
-documents from identifiers, processes content markers
-recursively, and produces a single rendered Markdown
-document.*
-
-## Module
-
-| Field       | Value                                        |
-|-------------|----------------------------------------------|
-| Rust module | `content_composer`                           |
-| Source path | `engine/crates/shared/src/content-composer/` |
-| Crate       | `kbdb-shared` (rlib)                         |
-
-The module is declared in `lib.rs` with a path
-attribute because the source directory uses a
-hyphenated name:
-
-```rust
-#[path = "content-composer/mod.rs"]
-pub mod content_composer;
-```
-
-## Directory Layout
-
-```
-engine/crates/shared/src/content-composer/
-   mod.rs          -- re-exports
-   types.rs        -- ComposeRequest, ComposeResult,
-                      RenderedSection
-   compose.rs      -- compose_content()
-   render.rs       -- section and marker rendering
-```
-
-## Purpose
-
-The content creator takes one or more identifiers
-(kbids or docids, mixed) and produces a composed
-Markdown document. It resolves each identifier to its
-sections, follows embedding and link content markers
-recursively, and renders everything into a single
-UTF-8 Markdown string with named anchors for
-cross-referencing.
-
-The rendering logic is **pure composition** -- no
-I/O, no database access. Resolution and fetching of
-sections from the database is handled by `kb-worker`
-before calling into this module.
-
-## Two-Phase Composition
-
-### Phase 1 -- Resolve and Gather (kb-worker)
-
-1. Resolve identifiers (try kbid, then docid)
-2. Expand documents to section kbids
-3. Deduplicate while preserving order
-4. Fetch main sections from fs-database
-5. Extract content markers recursively using
-   `extract_content_markers()` from content-parser
-6. Fetch reference content
-7. Assign HTML named anchors
-
-### Phase 2 -- Content Composition (this module)
-
-Takes the pre-fetched `ComposeRequest` and produces
-a Markdown string. Processes content markers in
-section bodies, rendering each by type (see
-[functions.md](functions.md) for rendering rules).
-
-## Cycle Detection
-
-Two distinct mechanisms prevent infinite loops:
-
-- **Resolution cycle detection** (Phase 1): a visited
-  set tracks which kbids and docids have been
-  processed during recursive marker extraction.
-- **Render cycle detection** (Phase 2): a render
-  stack tracks which entities are currently being
-  expanded. If a `$(text:)` or `$(document:)` marker
-  references an entity already in the render stack,
-  it is a cyclic embed -- rendered as a Markdown link
-  to the entity's anchor instead of expanding inline.
-
-The render stack is distinct from the
-`renderedSections` / `renderedDocuments` tracking
-sets. An entity in the render stack is currently
-being expanded (not yet complete). An entity in
-`renderedSections` has been fully rendered and will
-not be rendered again.
-
-## Dependencies
-
-Sibling modules within `kbdb-shared`:
-
-- `content_parser::extract_content_markers()` -- for
-  processing markers within referenced content during
-  recursive rendering.
-- `content_parser::ContentMarkerType` -- for
-  dispatching marker rendering by type.
-- `section::SectionType` -- for type-specific
-  rendering (text vs code vs image).
-
-## Integration Points
-
-### Worker Orchestration
-
-The `kb-worker` crate orchestrates the full flow:
-receives a `compose` IPC request, resolves
-identifiers via `fs-database`, fetches content,
-extracts content markers recursively, builds a
-`ComposeRequest`, and calls `compose_content()`.
-See [compose](../../kb-worker/functions/compose.md).
-
-### Worker Client
-
-`WorkerClient.content(ids)` sends the `compose` IPC
-method to the daemon and returns a `ContentResult`.
-
-### CLI
-
-`runContent(client, options)` calls
-`client.content(ids)` and writes the composed
-Markdown to stdout. Command: `kbdb content <id>...`
-
-### MCP
-
-The `content` MCP tool routes to
-`client.content(ids)` and wraps the result in the
-MCP tool response format.
-
-## Sub-documents
-
-- [types.md](types.md) -- `ComposeRequest`,
-  `ComposeResult`, `ResolvedId`, `SectionData`,
-  `DocumentData`, `RenderedSection`
-- [functions.md](functions.md) --
-  `compose_content()`
-
-## Testing
-
-Tests live inline in
-`engine/crates/shared/src/content-composer/compose.rs`
-and
-`engine/crates/shared/src/content-composer/render.rs`
-(`#[cfg(test)]` modules). See
-[functions.md](functions.md) for test descriptions.