@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/modules/rust/shared/pipeline/module.md

@@ -1,62 +0,0 @@
-# pipeline -- Tokenization Pipeline
-
-*9-stage pipeline that converts raw text into
-normalized, stemmed tokens with synonym expansion.
-Shared across all crates that need text
-tokenization.*
-
-## Source
-
-`engine/crates/shared/src/pipeline/`
-
-## Responsibility
-
-The pipeline module provides the canonical
-tokenization function used for both indexing and
-query processing. All crates that need to tokenize
-text import from this single source of truth.
-
-## Consumers
-
-- **kbdb-query-parser** -- re-exports via the
-  `qp_tokenize` WASM FFI export. Uses internally
-  for query parsing (`parse`), keyword extraction
-  (`extract/rake`), and n-gram generation.
-- **kbdb-default-embedding** -- tokenizes text
-  before feature hashing to produce TF-IDF vectors
-  that are token-consistent with the lexical index.
-
-## Directory Layout
-
-```
-engine/crates/shared/src/pipeline/
-├── mod.rs               # pipeline orchestrator
-├── markers.rs           # stage 1: strip markers
-├── unicode.rs           # stage 2: NFC normalize
-├── segment.rs           # stage 3: word boundaries
-├── identifiers.rs       # stage 4: camelCase split
-├── case.rs              # stage 5: case folding
-├── stopwords.rs         # stage 6: stop words
-├── synonyms.rs          # stage 7: synonym expansion
-├── stem.rs              # stage 8: Snowball stemmer
-├── ngram.rs             # stage 9: n-gram extract
-└── token_result.rs      # TokenResult struct
-```
-
-## Exports
-
-- **[functions.md](functions.md)** -- `tokenize()`,
-  `bigrams()`, `trigrams()`. Full 9-stage pipeline.
-- **[types.md](types.md)** -- `TokenResult`,
-  `TokenStream`, `Ngram`, `NgramType`, `StemResult`.
-
-## Dependencies
-
-The pipeline brings three external crates into
-`kbdb-shared`:
-
-```toml
-unicode-segmentation = "1"     # stage 3
-unicode-normalization = "0.1"  # stage 2
-rust-stemmers = "1"            # stage 8
-```

package/docs/modules/rust/shared/pipeline/types.md

@@ -1,43 +0,0 @@
-# pipeline -- Tokenization Types
-
-*Output types from the tokenization pipeline.*
-
-## Source
-
-`engine/crates/shared/src/pipeline/`
-
-## Type Definitions
-
-```rust
-pub struct TokenResult {
-   pub original: String,
-   pub stemmed: String,
-   pub position: u32,
-}
-
-pub struct TokenStream {
-   pub tokens: Vec<TokenResult>,
-   pub ngrams: Vec<Ngram>,
-}
-
-pub struct Ngram {
-   pub terms: Vec<String>,
-   pub gram_type: NgramType,
-}
-
-pub enum NgramType { Bigram, Trigram }
-
-pub struct StemResult {
-   pub original: String,
-   pub stemmed: String,
-}
-```
-
-`TokenResult` preserves both original and stemmed
-forms so the index can support exact-match and
-stem-match queries. `position` is the zero-based
-token offset.
-
-`TokenStream` is the output of `tokenize()` -- the
-full set of tokens and extracted n-grams from a
-single input text.

package/docs/modules/rust/shared/query/module.md

@@ -1,27 +0,0 @@
-# query -- Query Plan and Match Results
-
-*Data structures that flow between query-parser and
-fs-database via kb-worker: the structured query plan
-and the raw index match results.*
-
-## Source
-
-`engine/crates/shared/src/query/`
-
-## Responsibility
-
-Defines the query plan and raw match types that cross
-WASM module boundaries during the search lifecycle.
-`query-parser` produces `QueryPlan`, `fs-database`
-executes it and returns `Vec<RawMatch>`, then
-`query-parser` scores the matches.
-
-See [Query Parser -- Query Parsing](
-../../../../goals/query-parser.md) for parsing stages.
-See [Database -- Search](
-../../../../goals/database.md) for query plan execution.
-
-## Exports
-
-- **[types.md](types.md)** -- `QueryPlan`,
-  `RawMatch`, `MatchMode`.

package/docs/modules/rust/shared/query/types.md

@@ -1,236 +0,0 @@
-# types::query -- Query Plan and Raw Match Results
-
-*Data structures that flow between query-parser and
-fs-database via kb-worker: the structured query plan
-and the raw index match results.*
-
-## File
-
-`engine/crates/shared/src/query/query.rs`
-
-## Overview
-
-The query lifecycle splits across two standalone WASM
-modules:
-
-1. `query-parser.wasm` parses a search string into a
-   **QueryPlan**.
-2. `fs-database.wasm` executes the plan against
-   indexes and returns **RawMatch** results.
-3. `query-parser.wasm` scores the raw matches into
-   ranked results.
-
-`kb-worker.wasm` orchestrates these steps. The types
-in this module define the data shapes that cross
-module boundaries at steps 1->2 and 2->3.
-
-See [Query Parser -- Query Parsing](
-../../../../../goals/query-parser.md) for parsing stages.
-See [Database -- Search](
-../../../../../goals/database.md) for query plan execution.
-
-## Type Definitions
-
-### `QueryPlan`
-
-```rust
-pub struct QueryPlan {
-   pub terms: Vec<Term>,
-   pub phrases: Vec<Vec<Term>>,
-   pub exclusions: Vec<Term>,
-   pub mode: MatchMode,
-}
-```
-
-The output of `qp_parse_query`. A data retrieval
-strategy that describes which terms to look up, how
-to combine them, and what to exclude.
-
-| Field        | Description                         |
-|--------------|-------------------------------------|
-| `terms`      | Individual terms to match           |
-| `phrases`    | Ordered term sequences for exact phrase matching |
-| `exclusions` | Terms to exclude from results       |
-| `mode`       | How to combine term matches         |
-
-#### `terms`
-
-The tokenized, stemmed terms from the query string.
-Each term is looked up in the inverted index. A query
-like `"rust wasm engine"` produces three terms.
-
-#### `phrases`
-
-Quoted phrases from the query. `"file database"`
-produces `vec![vec![Term("file"), Term("databas")]]`
-(stemmed). Phrase matching uses the positional index
-to verify that terms appear adjacent and in order.
-
-#### `exclusions`
-
-Terms prefixed with `-` in the query. `-draft`
-produces `vec![Term("draft")]`. Sections matching
-excluded terms are removed from results.
-
-#### `mode`
-
-```rust
-pub enum MatchMode {
-   And,
-   Or,
-}
-```
-
-- `And` -- All non-excluded terms must appear
-  (default).
-- `Or` -- Any non-excluded term may appear.
-
-`fs-database` uses this to decide whether to
-intersect or union posting lists.
-
-### `RawMatch`
-
-```rust
-pub struct RawMatch {
-   pub kbid: KbId,
-   pub field_tfs: Vec<(Term, FieldTf)>,
-   pub positions: Vec<(Term, Vec<u32>)>,
-   pub token_count: u32,
-   pub field_token_counts: FieldTokenCounts,
-}
-```
-
-The output of `db_execute_plan`. One `RawMatch` per
-section that passed the query plan's term and
-exclusion filters.
-
-| Field                | Description                    |
-|----------------------|--------------------------------|
-| `kbid`               | The matching section           |
-| `field_tfs`          | Per-term per-field TF values   |
-| `positions`          | Per-term position lists for proximity/phrase scoring |
-| `token_count`        | Total token count of the section |
-| `field_token_counts` | Per-field token counts         |
-
-#### `field_tfs`
-
-A list of (term, `FieldTf`) pairs. Each `FieldTf`
-contains the term's frequency in the heading, body,
-and code fields separately. Pre-computed during
-indexing and stored in the inverted index. Used by
-`qp_score_bm25f` for per-field BM25 scoring with
-field-specific weights.
-
-#### `positions`
-
-A list of (term, positions) pairs. Each position is
-the zero-based token offset where the term occurs.
-Used by:
-
-- `qp_score_proximity` -- to measure how close query
-  terms appear to each other.
-- `qp_score_zone` -- to determine whether terms fall
-  in heading zones (early positions).
-- Phrase verification -- to confirm that phrase terms
-  appear adjacent and in order.
-
-#### `token_count`
-
-Total token count of the matching section. Read
-from the section header (pre-computed during
-indexing). Used by `qp_score_zone` to determine
-the heading zone boundary (~20% of section length).
-
-#### `field_token_counts`
-
-Per-field token counts read from the section header.
-Used by `qp_score_bm25f` for per-field length
-normalization: each field's token count is compared
-to the corpus average for that field.
-
-### `FieldTf`
-
-```rust
-pub struct FieldTf {
-   pub heading: f32,
-   pub body: f32,
-   pub code: f32,
-}
-```
-
-Per-field term frequency. Each field stores the
-count of how many times a term appears in that
-field of a section, computed during per-field
-tokenization at indexing time. See
-[Inverted Index](../../fs-database/indexes/inverted.md)
-for storage format.
-
-| Field     | Source content              | Default weight |
-|-----------|-----------------------------|----------------|
-| `heading` | Section title               | 2.0x           |
-| `body`    | Body text + description     | 1.0x           |
-| `code`    | Code / source content       | 1.5x           |
-
-### `FieldTokenCounts`
-
-```rust
-pub struct FieldTokenCounts {
-   pub heading: u32,
-   pub body: u32,
-   pub code: u32,
-}
-```
-
-Per-field token counts stored in section headers.
-Pre-computed during indexing and used at query time
-for per-field BM25 length normalization.
-
-## Relationships
-
-```
-         qp_parse_query
-query string ──────────→ QueryPlan
-                              │
-                   db_execute_plan
-                              │
-                              ▼
-                      Vec<RawMatch>
-                              │
-              qp_score_bm25f / proximity / zone
-                              │
-                              ▼
-                      ranked results
-```
-
-Both `QueryPlan` and `RawMatch` are serialized
-through shared `WebAssembly.Memory` via the `Encode`
-trait (see [encode](../encode/traits.md))
-when crossing module boundaries. `kb-worker` handles
-the serialization and deserialization at each step.
-
-## Trait Implementations
-
-| Trait    | Types                    |
-|----------|--------------------------|
-| `Clone`  | `QueryPlan`, `MatchMode`, `RawMatch`, `FieldTf`, `FieldTokenCounts` |
-| `Debug`  | All types                |
-| `Encode` | `QueryPlan`, `MatchMode`, `RawMatch`, `FieldTf`, `FieldTokenCounts` |
-| `Copy`   | `MatchMode`, `FieldTf`, `FieldTokenCounts` |
-
-`MatchMode` also derives `PartialEq` and `Eq`.
-
-## Testing
-
-Unit tests:
-
-- `QueryPlan` construction with terms, phrases,
-  exclusions, and both match modes.
-- `Encode` round-trip for `QueryPlan` with all
-  fields populated.
-- `Encode` round-trip for `RawMatch` with
-  `field_tfs`, positions, `token_count`, and
-  `field_token_counts`.
-- `FieldTf` and `FieldTokenCounts` construction
-  and `Encode` round-trip.
-- Empty edge cases: plan with no terms, match with
-  no positions, zero field counts.

package/docs/modules/rust/shared/section/module.md

@@ -1,25 +0,0 @@
-# section -- Section Types and Records
-
-*Section type enum, MIME type wrapper, and data
-structures for section CRUD operations.*
-
-## Source
-
-`engine/crates/shared/src/section/`
-
-## Responsibility
-
-Defines the content type system (`SectionType`,
-`MimeType`) and the data structures for section CRUD
-operations (`SectionRecord`, `AddSectionParams`,
-etc.) shared across all four WASM modules.
-
-See [Database -- Sections](../../../../goals/database.md)
-for the behavioral specification of section types.
-
-## Exports
-
-- **[types.md](types.md)** -- `SectionType`,
-  `MimeType`, `SectionRecord`, `AddSectionParams`,
-  `AddSectionResult`, `UpdateSectionResult`,
-  `RemoveSectionResult`, `MimeError`.

package/docs/modules/rust/shared/section/types.md

@@ -1,294 +0,0 @@
-# types::section -- Section Types and Records
-
-*Section type enum, MIME type wrapper, and the data
-structures for section CRUD operations across all
-four WASM modules.*
-
-## File
-
-`engine/crates/shared/src/section/section.rs`
-
-## Overview
-
-Sections are the atomic content units in the
-knowledge base. Each section has a **type** that
-determines how it is stored, whether it is indexed,
-and how it is rendered. The types in this module are
-shared across `fs-database` (storage), `query-parser`
-(indexing decisions), and `kb-worker` (orchestration).
-
-See [Database -- Sections](../../../../../goals/database.md)
-for the behavioral specification of section types,
-embedding markers, and MIME-to-language mapping.
-
-## Type Definitions
-
-### `MimeType`
-
-```rust
-#[derive(Clone, Debug, PartialEq, Eq)]
-pub struct MimeType(String);
-
-impl MimeType {
-   pub fn new(mime: &str) -> Result<Self, MimeError>;
-   pub fn as_str(&self) -> &str;
-   pub fn subtype(&self) -> &str;
-}
-```
-
-Newtype wrapper for MIME type strings. Validates
-that the string contains a `/` separator (e.g.
-`"text/typescript"`, `"image/png"`,
-`"application/json"`).
-
-`subtype()` returns the part after the `/`:
-`"text/typescript"` -> `"typescript"`,
-`"application/json"` -> `"json"`.
-
-### `SectionType`
-
-```rust
-#[derive(Clone, Debug, PartialEq, Eq)]
-pub enum SectionType {
-   Text,
-   Code(MimeType),
-   Image(MimeType),
-}
-```
-
-Three variants corresponding to the three content
-kinds in the knowledge base:
-
-| Variant       | Content      | Indexed | Embeddable via    |
-|---------------|--------------|---------|-------------------|
-| `Text`        | Raw text     | Yes     | `$(text: {kbid})` |
-| `Code(mime)`  | Raw source   | Yes     | `$(code: {kbid})` |
-| `Image(mime)` | Base64 bytes | title/description only | `$(image: {kbid})`|
-
-### `SectionType` Methods
-
-```rust
-impl SectionType {
-   pub fn is_indexable(&self) -> bool;
-   pub fn as_mime_str(&self) -> &str;
-   pub fn from_mime(mime: &str) -> Self;
-   pub fn language_tag(&self) -> Option<&str>;
-}
-```
-
-#### `is_indexable`
-
-```rust
-pub fn is_indexable(&self) -> bool {
-   matches!(self, Self::Text | Self::Code(_))
-}
-```
-
-Returns `true` for `Text` and `Code`, `false` for
-`Image`. Used by `kb-worker` to decide whether to
-tokenize the section body content. Note:
-`kb-worker` independently indexes image sections'
-`title` (required, heading field 2.0x) and
-`description` (optional, body field 1.0x)
-regardless of `is_indexable()` returning `false`.
-
-#### `as_mime_str`
-
-Returns the MIME string representation:
-
-- `Text` -> `"text/markdown"`
-- `Code(mime)` -> the inner MIME string
-  (e.g. `"text/typescript"`)
-- `Image(mime)` -> the inner MIME string
-  (e.g. `"image/png"`)
-
-#### `from_mime`
-
-Parses a MIME string into the appropriate variant:
-
-| Input                | Result                              |
-|----------------------|-------------------------------------|
-| `"text/markdown"`    | `SectionType::Text`                 |
-| `"text/typescript"`  | `SectionType::Code("text/typescript")` |
-| `"text/javascript"`  | `SectionType::Code("text/javascript")` |
-| `"text/html"`        | `SectionType::Code("text/html")`    |
-| `"text/css"`         | `SectionType::Code("text/css")`     |
-| `"text/xml"`         | `SectionType::Code("text/xml")`     |
-| `"text/plain"`       | `SectionType::Code("text/plain")`   |
-| `"text/*"` (other)   | `SectionType::Code("text/{subtype}")` |
-| `"application/json"` | `SectionType::Code("application/json")` |
-| `"image/png"`        | `SectionType::Image("image/png")`   |
-| `"image/jpeg"`       | `SectionType::Image("image/jpeg")`  |
-| `"image/*"` (other)  | `SectionType::Image("image/{subtype}")` |
-
-`"text/markdown"` is the canonical type for
-natural-language text sections. Any other `text/*`
-subtype maps to `Code`.
-
-#### `language_tag`
-
-Returns the fenced code block language tag for code
-sections. Returns `None` for `Text` and `Image`.
-
-| MIME Type            | Language Tag |
-|----------------------|--------------|
-| `text/typescript`    | `"ts"`       |
-| `text/javascript`    | `"js"`       |
-| `text/html`          | `"html"`     |
-| `text/css`           | `"css"`      |
-| `text/xml`           | `"xml"`      |
-| `text/plain`         | `None`       |
-| `application/json`   | `"json"`     |
-
-Other `text/*` subtypes return the subtype as the
-language tag (e.g. `"text/python"` -> `"python"`,
-`"text/rust"` -> `"rust"`).
-
-## Section Records
-
-Data structures for section CRUD operations. Used in
-IPC request/response payloads and as cache entries.
-
-### `SectionRecord`
-
-```rust
-pub struct SectionRecord {
-   pub kbid: KbId,
-   pub docids: Vec<String>,
-   pub section_type: SectionType,
-   pub title: Option<String>,
-   pub description: Option<String>,
-   pub content: Vec<u8>,
-   pub size: u32,
-   pub created_at: String,
-   pub checksum: String,
-   pub token_count: u32,
-   pub heading_token_count: u32,
-   pub body_token_count: u32,
-   pub code_token_count: u32,
-}
-```
-
-Full section data as read from disk. `docids` lists
-the document IDs this section belongs to (may be
-empty, one, or many -- many-to-many relationship).
-`content` is raw UTF-8 bytes for text and code
-sections, or base64-encoded bytes for image sections.
-`size` is the byte length of the content body.
-`title` is required for image sections and optional
-for text/code. `description` is always optional.
-Token counts are pre-computed during per-field
-tokenization at indexing time.
-
-### `AddSectionParams`
-
-```rust
-pub struct AddSectionParams {
-   pub content: Vec<u8>,
-   pub section_type: SectionType,
-   pub title: Option<String>,
-   pub description: Option<String>,
-   pub docid: Option<String>,
-}
-```
-
-Input for creating a new section. The caller passes
-raw content; `fs-database.wasm` applies the
-preprocessing pipeline internally to text and code
-sections (BOM removal, line ending normalization,
-Unicode NFC, whitespace trimming) before computing
-the kbid. Image content is stored verbatim. The
-kbid is derived from the preprocessed `content` by
-`fs-database` during creation -- it is not provided
-by the caller.
-
-`title` is required when `section_type` is
-`Image(*)` -- validation fails without it.
-`description` is always optional. Both are
-tokenized during indexing: title into the heading
-field (2.0x weight), description into the body
-field (1.0x weight).
-
-### `AddSectionResult`
-
-```rust
-pub struct AddSectionResult {
-   pub kbid: KbId,
-   pub section_type: SectionType,
-   pub title: Option<String>,
-   pub description: Option<String>,
-   pub indexed_terms_count: u32,
-   pub token_count: u32,
-   pub elapsed_ms: u64,
-   pub attached: bool,
-}
-```
-
-`indexed_terms_count` reflects the number of
-distinct terms indexed across all fields. For
-image sections, this is the count from
-title/description tokenization (zero only when
-both produce no terms). `token_count` is the total
-token count across all fields. `attached` is `true`
-when the content already existed and a new docid
-was added to the section's `docids` list instead of
-creating a new section.
-
-### `UpdateSectionResult`
-
-```rust
-pub struct UpdateSectionResult {
-   pub old_kbid: KbId,
-   pub new_kbid: KbId,
-   pub reindexed_terms_count: u32,
-   pub elapsed_ms: u64,
-}
-```
-
-Updates produce a new section file with a new kb-id
-(content-addressable). The old kb-id is returned so
-the caller and cache can invalidate stale references.
-
-### `RemoveSectionResult`
-
-```rust
-pub struct RemoveSectionResult {
-   pub removed: bool,
-   pub elapsed_ms: u64,
-}
-```
-
-`removed` is `false` if the kb-id was not found.
-
-## Trait Implementations
-
-All structs implement `Clone`, `Debug`, and `Encode`
-(see [encode](../encode/traits.md)).
-`SectionType` also implements `PartialEq` and `Eq`
-for type comparison.
-
-## Error Type
-
-```rust
-#[derive(Debug)]
-pub enum MimeError {
-   MissingSeparator(String),
-   EmptyType(String),
-   EmptySubtype(String),
-}
-```
-
-## Testing
-
-Unit tests:
-
-- `SectionType::from_mime` mapping for all documented
-  MIME types.
-- `is_indexable` returns `true` for `Text` and `Code`,
-  `false` for `Image`.
-- `language_tag` returns correct tags for known MIME
-  types and subtype-as-fallback for unknown ones.
-- `from_mime` with `"text/markdown"` produces `Text`,
-  not `Code`.
-- `MimeType::subtype` extracts correctly.
-- `Encode` round-trip for all record types.