@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/modules/rust/query-parser/functions/extract.md

@@ -0,0 +1,56 @@
+# functions::extract -- Keyword Extraction
+
+*RAKE keyphrase extraction and TF-IDF term weighting.*
+
+## Source
+
+`engine/crates/query-parser/src/extract/`
+
+## RAKE (`rake.rs`)
+
+```rust
+pub fn extract_keyphrases(
+   tokens: &[Token],
+   stop_words: &HashSet<String>,
+) -> Vec<Keyphrase>;
+```
+
+Rapid Automatic Keyword Extraction. Splits on stop
+words to identify candidate phrases. Scores each
+candidate by the ratio of word degree to word
+frequency. Unsupervised, no ML dependencies.
+
+## TF-IDF (`tfidf.rs`)
+
+```rust
+pub fn compute_tf(
+   tokens: &[Token],
+) -> Vec<(Term, f32)>;
+
+pub fn compute_field_tf(
+   heading_tokens: &[Token],
+   body_tokens: &[Token],
+   code_tokens: &[Token],
+) -> Vec<(Term, FieldTf)>;
+
+pub fn compute_tfidf(
+   tf: &[(Term, f32)],
+   idf: &dyn Fn(&Term) -> f32,
+) -> Vec<WeightedTerm>;
+```
+
+`compute_tf` calculates per-section term frequency
+for a single token stream.
+
+`compute_field_tf` computes per-field term
+frequencies from separately tokenized field streams.
+`kb-worker` calls this during indexing after running
+the tokenization pipeline separately for each field
+(title → heading, body + description → body,
+code → code). Returns `FieldTf` values with heading,
+body, and code frequencies per term.
+
+`compute_tfidf` combines TF with corpus-wide IDF
+(provided by `fs-database.wasm` via `kb-worker.wasm`)
+to produce final weights reflecting each term's
+discriminative value.

package/docs/modules/rust/query-parser/functions/module.md

@@ -0,0 +1,16 @@
+# query-parser/functions -- Operations
+
+*Functions for tokenization, query parsing, keyword
+extraction, and ranking.*
+
+## Modules
+
+- **[pipeline](pipeline.md)** -- 8-stage tokenization
+  pipeline. Converts raw text into normalized, stemmed
+  tokens with n-grams.
+- **[parse](parse.md)** -- Query string parsing into
+  structured `QueryPlan`.
+- **[extract](extract.md)** -- RAKE keyphrase
+  extraction and TF-IDF term weighting.
+- **[rank](rank.md)** -- BM25F, proximity, zone, and
+  freshness scoring functions.

package/docs/modules/rust/query-parser/functions/parse.md

@@ -0,0 +1,33 @@
+# functions::parse -- Query Parsing
+
+*Parse a search query string into a structured
+`QueryPlan`.*
+
+## Source
+
+`engine/crates/query-parser/src/parse.rs`
+
+## Function
+
+```rust
+pub fn parse_query(input: &str) -> QueryPlan;
+```
+
+## Parsing Stages
+
+1. **Tokenize** -- Run the query through the
+   tokenization pipeline.
+2. **Parse syntax** -- Recognize operators: quoted
+   phrases (`"..."`), boolean AND/OR, exclusions
+   (`-term`).
+3. **Build plan** -- Produce the `QueryPlan` with
+   terms, phrase groups, exclusions, and match mode.
+
+## Examples
+
+| Input                | terms        | phrases          | exclusions | mode |
+|----------------------|--------------|------------------|------------|------|
+| `rust wasm`          | [rust, wasm] | []               | []         | And  |
+| `"file database"`    | []           | [[file, databas]]| []         | And  |
+| `rust -draft`        | [rust]       | []               | [draft]    | And  |
+| `rust OR wasm`       | [rust, wasm] | []               | []         | Or   |

package/docs/modules/rust/query-parser/functions/pipeline.md

@@ -0,0 +1,103 @@
+# functions::pipeline -- Tokenization Pipeline
+
+*8-stage pipeline that converts raw text into
+normalized, stemmed tokens. Used for both indexing
+and query processing.*
+
+## Source
+
+`engine/crates/query-parser/src/pipeline/`
+
+## Entry Point
+
+```rust
+pub fn tokenize(input: &str) -> TokenStream;
+```
+
+Runs all eight stages in sequence. Each stage is a
+pure function.
+
+## Stages
+
+### 1. Content Marker Stripping
+
+```rust
+use kbdb_shared::content_parser::strip_content_markers;
+```
+
+Removes all six content marker types from the input
+text before tokenization:
+`$(image:)`, `$(code:)`, `$(text:)`, `$(document:)`,
+`$(link:)`, `$(doc-link:)`.
+
+Delegates to `strip_content_markers()` imported from
+`kbdb_shared::content_parser` (see
+[Content Parser](../../shared/content-parser/module.md)).
+These are structural references, not searchable
+content.
+
+### 2. Unicode Normalization (`unicode.rs`)
+
+```rust
+pub fn normalize_nfc(input: &str) -> String;
+```
+
+NFC normalization for consistent codepoints.
+
+### 3. Word Boundary Detection (`segment.rs`)
+
+```rust
+pub fn segment_words(input: &str) -> Vec<&str>;
+```
+
+Unicode Segmentation (UAX #29).
+
+### 4. Identifier Splitting (`identifiers.rs`)
+
+```rust
+pub fn split_identifiers(word: &str) -> Vec<String>;
+```
+
+- `camelCase` → `["camel", "case"]`
+- `snake_case` → `["snake", "case"]`
+- `PascalCase` → `["pascal", "case"]`
+- `HTMLParser` → `["html", "parser"]`
+
+### 5. Case Folding (`case.rs`)
+
+```rust
+pub fn fold_case(word: &str) -> String;
+```
+
+Unicode case folding (lowercase).
+
+### 6. Stop Word Removal (`stopwords.rs`)
+
+```rust
+pub fn is_stop_word(word: &str) -> bool;
+pub fn remove_stop_words(
+   words: &[String],
+) -> Vec<String>;
+```
+
+Filters common function words. Single-character
+tokens also removed.
+
+### 7. Stemming (`stem.rs`)
+
+```rust
+pub fn stem_word(word: &str) -> StemResult;
+```
+
+Snowball / Porter2. Preserves both original and
+stemmed forms.
+
+### 8. N-gram Extraction (`ngram.rs`)
+
+```rust
+pub fn extract_ngrams(
+   tokens: &[Token],
+) -> Vec<Ngram>;
+```
+
+Bigrams and trigrams for phrase indexing.

package/docs/modules/rust/query-parser/functions/rank.md

@@ -0,0 +1,85 @@
+# functions::rank -- Ranking Functions
+
+*Scoring functions that `kb-worker.wasm` calls after
+retrieving raw index results from `fs-database.wasm`.*
+
+## Source
+
+`engine/crates/query-parser/src/rank/`
+
+## BM25F (`bm25f.rs`)
+
+```rust
+pub fn score_bm25f(
+   params: &BM25FParams,
+   raw_match: &RawMatch,
+   corpus: &CorpusStats,
+) -> f32;
+```
+
+Primary ranking function. Field-weighted Okapi BM25
+with configurable saturation (`k1`), length
+normalization (`b`), and per-field boost weights.
+
+Per-field BM25F formula:
+
+```
+score_f(t, d) = idf(t)
+   * (tf_f(t,d) * (k1 + 1))
+   / (tf_f(t,d) + k1
+      * (1 - b + b * |d_f| / avgdl_f))
+
+bm25f(t, d) = w_heading * score_heading(t, d)
+            + w_body * score_body(t, d)
+            + w_code * score_code(t, d)
+```
+
+`raw_match.field_tfs` provides per-field term
+frequencies (`tf_f`). `raw_match.field_token_counts`
+provides per-field section lengths (`|d_f|`).
+`corpus` provides per-field average lengths
+(`avgdl_f`). Field weights are applied at query
+time, making them tunable without index rebuilds.
+See [Query Result](../../../../goals/query-result.md)
+for the full scoring specification.
+
+## Proximity (`proximity.rs`)
+
+```rust
+pub fn score_proximity(
+   positions: &[(Term, Vec<u32>)],
+   query_terms: &[Term],
+) -> f32;
+```
+
+Boosts sections where query terms appear close
+together. Computes minimum span across all query
+terms, inversely weighted by span length.
+
+## Zone (`zone.rs`)
+
+```rust
+pub fn score_zone(
+   positions: &[(Term, Vec<u32>)],
+   section_length: u32,
+) -> f32;
+```
+
+Structural weighting. Terms in the first ~20% of a
+section (heading zone) receive higher weight. Uses
+`section_length` from `RawMatch.token_count`
+(pre-computed during indexing) to determine the
+zone boundary.
+
+## Freshness (`freshness.rs`)
+
+```rust
+pub fn score_freshness(
+   created_at: u64,
+   now: u64,
+   half_life_days: f32,
+) -> f32;
+```
+
+Optional boost based on insertion timestamps.
+Exponential decay with configurable half-life.

package/docs/modules/rust/query-parser/module.md

@@ -0,0 +1,136 @@
+# query-parser -- Query Processing and Tokenization
+
+*Rust crate that provides the tokenization pipeline,
+query parsing, keyword extraction, and ranking
+functions. Compiles to `query-parser.wasm`.*
+
+## Crate
+
+| Field       | Value                              |
+|-------------|------------------------------------|
+| Name        | `kbdb-query-parser`                |
+| Path        | `engine/crates/query-parser/`      |
+| Crate type  | `cdylib` (WASM) + `rlib` (tests)  |
+| WASM output | `dist/wasm/query-parser.wasm`      |
+| Depends on  | `kbdb-shared`                      |
+
+`kbdb-query-parser` is a **standalone** WASM module.
+Pure computation -- no file I/O, no network. The only
+host import is `WebAssembly.Memory`.
+
+See [Query Parser](../../../goals/query-parser.md)
+for the full behavioral specification.
+
+## Workspace Position
+
+```
+engine/
+├── Cargo.toml                   # workspace root
+└── crates/
+    ├── shared/                  # kbdb-shared
+    ├── fs-database/             # kbdb-fs-database
+    ├── query-parser/            # this crate
+    └── kb-worker/               # kbdb-worker
+```
+
+## Directory Layout
+
+```
+engine/crates/query-parser/
+├── Cargo.toml
+└── src/
+    ├── lib.rs                   # crate root, WASM exports
+    ├── mem.rs                   # shared-memory helpers
+    ├── pipeline/
+    │   ├── mod.rs               # pipeline orchestrator
+    │   ├── markers.rs           # stage 1: strip markers
+    │   ├── unicode.rs           # stage 2: NFC normalize
+    │   ├── segment.rs           # stage 3: word boundaries
+    │   ├── identifiers.rs       # stage 4: camelCase split
+    │   ├── case.rs              # stage 5: case folding
+    │   ├── stopwords.rs         # stage 6: stop words
+    │   ├── stem.rs              # stage 7: Snowball stemmer
+    │   └── ngram.rs             # stage 8: n-gram extract
+    ├── parse.rs                 # query string → QueryPlan
+    ├── extract/
+    │   ├── mod.rs               # extraction facade
+    │   ├── rake.rs              # RAKE algorithm
+    │   └── tfidf.rs             # TF-IDF weighting
+    ├── rank/
+    │   ├── mod.rs               # ranking facade
+    │   ├── bm25f.rs             # BM25F scoring
+    │   ├── proximity.rs         # proximity scoring
+    │   ├── zone.rs              # zone scoring
+    │   └── freshness.rs         # freshness weighting
+    └── cache.rs                 # query-scoped cache
+```
+
+## Modules
+
+### [types/](types/module.md) -- Structs
+
+- **[token](types/token.md)** -- `Token`,
+  `TokenStream`, `Ngram`, `NgramType`, `StemResult`.
+- **[keyphrase](types/keyphrase.md)** -- `Keyphrase`,
+  `WeightedTerm`, `BM25FParams`, `FieldWeights`.
+- **[cache](types/cache.md)** -- `QueryCache`.
+  Short-lived query-scoped cache.
+
+### [functions/](functions/module.md) -- Operations
+
+- **[pipeline](functions/pipeline.md)** -- 8-stage
+  tokenization pipeline. Marker stripping, NFC,
+  segmentation, identifier splitting, case folding,
+  stop words, stemming, n-grams.
+- **[parse](functions/parse.md)** -- Query string
+  parsing into `QueryPlan`.
+- **[extract](functions/extract.md)** -- RAKE
+  keyphrase extraction and TF-IDF term weighting.
+- **[rank](functions/rank.md)** -- BM25F, proximity,
+  zone, and freshness scoring functions.
+
+## WASM Exports
+
+| Export                   | Description             |
+|--------------------------|-------------------------|
+| `qp_tokenize`            | Full tokenization pipeline |
+| `qp_parse_query`         | Parse query → QueryPlan |
+| `qp_normalize_term`      | Normalize a single term |
+| `qp_stem`                | Stem a single term      |
+| `qp_extract_keywords`    | RAKE keyphrase extraction |
+| `qp_compute_tf`          | Single-stream term frequency |
+| `qp_compute_field_tf`    | Per-field term frequency (heading, body, code) |
+| `qp_compute_tfidf`       | TF-IDF with supplied IDF |
+| `qp_score_bm25f`         | BM25F ranking score     |
+| `qp_score_proximity`     | Proximity ranking score |
+| `qp_score_zone`          | Zone ranking score      |
+| `qp_score_freshness`     | Freshness ranking score |
+| `qp_split_identifiers`   | camelCase/snake_case split |
+| `alloc` / `free`         | Memory management (from `kbdb-shared`) |
+
+## Dependencies
+
+```toml
+[dependencies]
+kbdb-shared = { path = "../shared" }
+unicode-segmentation = "1"
+unicode-normalization = "0.1"
+rust-stemmers = "1"
+regex = { version = "1", default-features = false }
+```
+
+## Build
+
+```sh
+cd engine
+wasm-pack build crates/query-parser \
+   --target no-modules \
+   --out-dir ../../../dist/wasm/query-parser
+```
+
+## Testing
+
+```sh
+cd engine && cargo test -p kbdb-query-parser
+cd engine && wasm-pack test --node crates/query-parser
+```

package/docs/modules/rust/query-parser/types/cache.md

@@ -0,0 +1,39 @@
+# types::cache -- Query-Scoped Cache
+
+*Short-lived cache scoped to a single query or
+indexing operation.*
+
+## Source
+
+`engine/crates/query-parser/src/cache.rs`
+
+## Type Definition
+
+```rust
+pub struct QueryCache {
+   tokenized: HashMap<String, Vec<Token>>,
+   stemmed: HashMap<String, String>,
+}
+
+impl QueryCache {
+   pub fn new() -> Self;
+   pub fn get_tokens(&self, input: &str)
+      -> Option<&[Token]>;
+   pub fn put_tokens(
+      &mut self, input: String, tokens: Vec<Token>,
+   );
+   pub fn get_stem(&self, word: &str)
+      -> Option<&str>;
+   pub fn put_stem(
+      &mut self, word: String, stem: String,
+   );
+}
+```
+
+Allocated when processing begins, discarded
+immediately after the operation completes. Prevents
+redundant tokenization and stemming when the same
+term appears multiple times in a query or content.
+
+Not persisted, not shared across queries, not
+visible to other modules.

package/docs/modules/rust/query-parser/types/keyphrase.md

@@ -0,0 +1,60 @@
+# types::keyphrase -- Extraction and Ranking Types
+
+*Types for keyword extraction results and ranking
+function configuration.*
+
+## Source
+
+`engine/crates/query-parser/src/extract/mod.rs`,
+`engine/crates/query-parser/src/rank/bm25f.rs`
+
+## Type Definitions
+
+```rust
+pub struct Keyphrase {
+   pub terms: Vec<String>,
+   pub score: f32,
+}
+
+pub struct WeightedTerm {
+   pub term: Term,
+   pub tf: f32,
+   pub weight: f32,
+}
+
+pub struct BM25FParams {
+   pub k1: f32,           // term saturation (1.2)
+   pub b: f32,            // length normalization (0.75)
+   pub field_weights: FieldWeights,
+}
+
+pub struct FieldWeights {
+   pub heading: f32,      // default 2.0
+   pub body: f32,         // default 1.0
+   pub code: f32,         // default 1.5
+}
+```
+
+`Keyphrase` is the output of RAKE extraction --
+multi-word candidate phrases scored by word degree /
+frequency ratio.
+
+`WeightedTerm` combines a term with its per-section
+TF and final TF-IDF weight. This is an
+extraction-time type used during keyword extraction.
+At ranking time, the per-field `FieldTf` from
+`RawMatch` (see
+[shared/query/types](../../shared/query/types.md))
+is used instead -- it contains separate heading,
+body, and code frequencies pre-computed during
+indexing.
+
+`BM25FParams` configures the primary ranking function
+with field-specific boost weights. Each field weight
+in `FieldWeights` multiplies the per-field BM25
+score computed from the corresponding `FieldTf`
+component and per-field corpus average from
+`CorpusStats`. See
+[FieldTokenCounts](../../shared/query/types.md)
+for the per-field token counts used in length
+normalization.

package/docs/modules/rust/query-parser/types/module.md

@@ -0,0 +1,17 @@
+# query-parser/types -- Type Definitions
+
+*Structs defined in the query-parser crate for
+tokenization, keyword extraction, and ranking.*
+
+## Modules
+
+- **[token](token.md)** -- `Token`, `TokenStream`,
+  `Ngram`, `NgramType`, `StemResult`. Output types
+  from the tokenization pipeline.
+- **[keyphrase](keyphrase.md)** -- `Keyphrase`,
+  `WeightedTerm`, `BM25FParams`, `FieldWeights`.
+  Types for keyword extraction and ranking
+  configuration.
+- **[cache](cache.md)** -- `QueryCache`. Short-lived
+  cache scoped to a single query or indexing
+  operation.

package/docs/modules/rust/query-parser/types/token.md

@@ -0,0 +1,42 @@
+# types::token -- Tokenization Types
+
+*Output types from the tokenization pipeline.*
+
+## Source
+
+`engine/crates/query-parser/src/pipeline/mod.rs`
+
+## Type Definitions
+
+```rust
+pub struct Token {
+   pub original: String,
+   pub stemmed: String,
+   pub position: u32,
+}
+
+pub struct TokenStream {
+   pub tokens: Vec<Token>,
+   pub ngrams: Vec<Ngram>,
+}
+
+pub struct Ngram {
+   pub terms: Vec<String>,
+   pub gram_type: NgramType,
+}
+
+pub enum NgramType { Bigram, Trigram }
+
+pub struct StemResult {
+   pub original: String,
+   pub stemmed: String,
+}
+```
+
+`Token` preserves both original and stemmed forms so
+the index can support exact-match and stem-match
+queries. `position` is the zero-based token offset.
+
+`TokenStream` is the output of `tokenize()` -- the
+full set of tokens and extracted n-grams from a
+single input text.