@dikolab/kbdb 0.1.6 → 0.2.0

package/docs/modules/rust/shared/module.md

@@ -0,0 +1,168 @@
+# shared -- Shared Types and Utilities
+
+*Rust crate that provides types, memory helpers, error
+types, and serialization utilities shared across all
+three WASM modules.*
+
+## Crate
+
+| Field       | Value                              |
+|-------------|------------------------------------|
+| Name        | `kbdb-shared`                      |
+| Path        | `engine/crates/shared/`            |
+| Crate type  | `rlib` (library only, not WASM)    |
+| Depends on  | nothing (leaf crate)               |
+
+`kbdb-shared` is a pure Rust library. It is **not**
+compiled to WASM on its own -- it is compiled into
+each of the three WASM modules that depend on it
+(`kbdb-fs-database`, `kbdb-query-parser`,
+`kbdb-worker`). It has no external dependencies beyond
+`wasm-bindgen` for the `alloc`/`free` exports.
+
+## Workspace Position
+
+```
+engine/
+├── Cargo.toml                   # workspace root
+└── crates/
+    ├── shared/                  # this crate (leaf)
+    ├── fs-database/             # depends on shared
+    ├── query-parser/            # depends on shared
+    └── kb-worker/               # depends on shared
+```
+
+## Directory Layout
+
+```
+engine/crates/shared/
+├── Cargo.toml
+└── src/
+    ├── lib.rs                   # crate root, re-exports
+    ├── memory/
+    │   ├── mod.rs               # memory module facade
+    │   └── alloc.rs             # kbdb_alloc, kbdb_free
+    ├── encode/
+    │   ├── mod.rs               # encode module facade
+    │   └── encode.rs            # Encode trait + impls
+    ├── kbid/
+    │   ├── mod.rs               # kbid module facade
+    │   └── kbid.rs              # KbId newtype
+    ├── term/
+    │   ├── mod.rs               # term module facade
+    │   └── term.rs              # Term newtype
+    ├── section/
+    │   ├── mod.rs               # section module facade
+    │   └── section.rs           # SectionType, records
+    ├── document/
+    │   ├── mod.rs               # document module facade
+    │   └── document.rs          # DocumentManifest, records
+    ├── query/
+    │   ├── mod.rs               # query module facade
+    │   └── query.rs             # QueryPlan, RawMatch
+    ├── corpus/
+    │   ├── mod.rs               # corpus module facade
+    │   └── corpus.rs            # CorpusStats
+    ├── content-parser/
+    │   ├── mod.rs               # content-parser facade
+    │   ├── types.rs             # ContentMarkerType,
+    │   │                        # ExtractedContentMarker
+    │   ├── extract.rs           # extract_content_markers()
+    │   └── strip.rs             # strip_content_markers()
+    ├── content-composer/
+    │   ├── mod.rs               # content-composer facade
+    │   ├── types.rs             # ComposeRequest, etc.
+    │   ├── compose.rs           # compose_content()
+    │   └── render.rs            # marker rendering
+    └── error/
+        ├── mod.rs               # error module facade
+        └── error.rs             # FsError, IndexError, etc.
+```
+
+## Modules
+
+Each sub-module has a single responsibility. Files
+within each module mirror their Rust source
+counterparts: `module.md` = `mod.rs`,
+`types.md` = type definitions,
+`functions.md` = function definitions,
+`traits.md` = trait definitions.
+
+### [memory/](memory/module.md) -- Memory Management
+
+- [functions.md](memory/functions.md) --
+  `kbdb_alloc`, `kbdb_free`, read/write helpers,
+  return value convention.
+
+### [encode/](encode/module.md) -- Binary Encoding
+
+- [traits.md](encode/traits.md) -- `Encode` trait,
+  wire format, per-type implementations.
+
+### [kbid/](kbid/module.md) -- Content-Addressed ID
+
+- [types.md](kbid/types.md) -- `KbId` newtype,
+  `from_content`, `shard_prefix`, `KbIdError`.
+
+### [term/](term/module.md) -- Search Term
+
+- [types.md](term/types.md) -- `Term` newtype,
+  `Ord` for B+ tree keys, `TermError`.
+
+### [section/](section/module.md) -- Section Types
+
+- [types.md](section/types.md) -- `SectionType`,
+  `MimeType`, `SectionRecord`, CRUD params/results.
+
+### [document/](document/module.md) -- Document Records
+
+- [types.md](document/types.md) --
+  `DocumentManifest`, `DocumentSection`,
+  `DocumentRecord`.
+
+### [query/](query/module.md) -- Query Plan
+
+- [types.md](query/types.md) -- `QueryPlan`,
+  `RawMatch`, `MatchMode`.
+
+### [corpus/](corpus/module.md) -- Corpus Statistics
+
+- [types.md](corpus/types.md) -- `CorpusStats`.
+
+### [content-parser/](content-parser/module.md) -- Content Parser
+
+- [types.md](content-parser/types.md) --
+  `ContentMarkerType` enum,
+  `ExtractedContentMarker` struct.
+- [functions.md](content-parser/functions.md) --
+  `extract_content_markers()`,
+  `strip_content_markers()`.
+
+### [content-composer/](content-composer/module.md) -- Content Composer
+
+- [types.md](content-composer/types.md) --
+  `ComposeRequest`, `ComposeResult`,
+  `RenderedSection`, `ResolvedId`, `SectionData`,
+  `DocumentData`.
+- [functions.md](content-composer/functions.md) --
+  `compose_content()`.
+
+### [error/](error/module.md) -- Error Types
+
+- [types.md](error/types.md) -- `FsError`,
+  `IndexError`, `ParseError`, `WorkerError`.
+
+## Dependencies
+
+```toml
+[dependencies]
+wasm-bindgen = "0.2"
+```
+
+## Testing
+
+```sh
+cd engine && cargo test -p kbdb-shared
+```
+
+See each module doc for detailed test descriptions.

package/docs/modules/rust/shared/query/module.md

@@ -0,0 +1,27 @@
+# query -- Query Plan and Match Results
+
+*Data structures that flow between query-parser and
+fs-database via kb-worker: the structured query plan
+and the raw index match results.*
+
+## Source
+
+`engine/crates/shared/src/query/`
+
+## Responsibility
+
+Defines the query plan and raw match types that cross
+WASM module boundaries during the search lifecycle.
+`query-parser` produces `QueryPlan`, `fs-database`
+executes it and returns `Vec<RawMatch>`, then
+`query-parser` scores the matches.
+
+See [Query Parser -- Query Parsing](
+../../../../goals/query-parser.md) for parsing stages.
+See [Database -- Search](
+../../../../goals/database.md) for query plan execution.
+
+## Exports
+
+- **[types.md](types.md)** -- `QueryPlan`,
+  `RawMatch`, `MatchMode`.

package/docs/modules/rust/shared/query/types.md

@@ -0,0 +1,236 @@
+# types::query -- Query Plan and Raw Match Results
+
+*Data structures that flow between query-parser and
+fs-database via kb-worker: the structured query plan
+and the raw index match results.*
+
+## File
+
+`engine/crates/shared/src/query/query.rs`
+
+## Overview
+
+The query lifecycle splits across two standalone WASM
+modules:
+
+1. `query-parser.wasm` parses a search string into a
+   **QueryPlan**.
+2. `fs-database.wasm` executes the plan against
+   indexes and returns **RawMatch** results.
+3. `query-parser.wasm` scores the raw matches into
+   ranked results.
+
+`kb-worker.wasm` orchestrates these steps. The types
+in this module define the data shapes that cross
+module boundaries at steps 1->2 and 2->3.
+
+See [Query Parser -- Query Parsing](
+../../../../../goals/query-parser.md) for parsing stages.
+See [Database -- Search](
+../../../../../goals/database.md) for query plan execution.
+
+## Type Definitions
+
+### `QueryPlan`
+
+```rust
+pub struct QueryPlan {
+   pub terms: Vec<Term>,
+   pub phrases: Vec<Vec<Term>>,
+   pub exclusions: Vec<Term>,
+   pub mode: MatchMode,
+}
+```
+
+The output of `qp_parse_query`. A data retrieval
+strategy that describes which terms to look up, how
+to combine them, and what to exclude.
+
+| Field        | Description                         |
+|--------------|-------------------------------------|
+| `terms`      | Individual terms to match           |
+| `phrases`    | Ordered term sequences for exact phrase matching |
+| `exclusions` | Terms to exclude from results       |
+| `mode`       | How to combine term matches         |
+
+#### `terms`
+
+The tokenized, stemmed terms from the query string.
+Each term is looked up in the inverted index. A query
+like `"rust wasm engine"` produces three terms.
+
+#### `phrases`
+
+Quoted phrases from the query. `"file database"`
+produces `vec![vec![Term("file"), Term("databas")]]`
+(stemmed). Phrase matching uses the positional index
+to verify that terms appear adjacent and in order.
+
+#### `exclusions`
+
+Terms prefixed with `-` in the query. `-draft`
+produces `vec![Term("draft")]`. Sections matching
+excluded terms are removed from results.
+
+#### `mode`
+
+```rust
+pub enum MatchMode {
+   And,
+   Or,
+}
+```
+
+- `And` -- All non-excluded terms must appear
+  (default).
+- `Or` -- Any non-excluded term may appear.
+
+`fs-database` uses this to decide whether to
+intersect or union posting lists.
+
+### `RawMatch`
+
+```rust
+pub struct RawMatch {
+   pub kbid: KbId,
+   pub field_tfs: Vec<(Term, FieldTf)>,
+   pub positions: Vec<(Term, Vec<u32>)>,
+   pub token_count: u32,
+   pub field_token_counts: FieldTokenCounts,
+}
+```
+
+The output of `db_execute_plan`. One `RawMatch` per
+section that passed the query plan's term and
+exclusion filters.
+
+| Field                | Description                    |
+|----------------------|--------------------------------|
+| `kbid`               | The matching section           |
+| `field_tfs`          | Per-term per-field TF values   |
+| `positions`          | Per-term position lists for proximity/phrase scoring |
+| `token_count`        | Total token count of the section |
+| `field_token_counts` | Per-field token counts         |
+
+#### `field_tfs`
+
+A list of (term, `FieldTf`) pairs. Each `FieldTf`
+contains the term's frequency in the heading, body,
+and code fields separately. Pre-computed during
+indexing and stored in the inverted index. Used by
+`qp_score_bm25f` for per-field BM25 scoring with
+field-specific weights.
+
+#### `positions`
+
+A list of (term, positions) pairs. Each position is
+the zero-based token offset where the term occurs.
+Used by:
+
+- `qp_score_proximity` -- to measure how close query
+  terms appear to each other.
+- `qp_score_zone` -- to determine whether terms fall
+  in heading zones (early positions).
+- Phrase verification -- to confirm that phrase terms
+  appear adjacent and in order.
+
+#### `token_count`
+
+Total token count of the matching section. Read
+from the section header (pre-computed during
+indexing). Used by `qp_score_zone` to determine
+the heading zone boundary (~20% of section length).
+
+#### `field_token_counts`
+
+Per-field token counts read from the section header.
+Used by `qp_score_bm25f` for per-field length
+normalization: each field's token count is compared
+to the corpus average for that field.
+
+### `FieldTf`
+
+```rust
+pub struct FieldTf {
+   pub heading: f32,
+   pub body: f32,
+   pub code: f32,
+}
+```
+
+Per-field term frequency. Each field stores the
+count of how many times a term appears in that
+field of a section, computed during per-field
+tokenization at indexing time. See
+[Inverted Index](../../fs-database/indexes/inverted.md)
+for storage format.
+
+| Field     | Source content              | Default weight |
+|-----------|-----------------------------|----------------|
+| `heading` | Section title               | 2.0x           |
+| `body`    | Body text + description     | 1.0x           |
+| `code`    | Code / source content       | 1.5x           |
+
+### `FieldTokenCounts`
+
+```rust
+pub struct FieldTokenCounts {
+   pub heading: u32,
+   pub body: u32,
+   pub code: u32,
+}
+```
+
+Per-field token counts stored in section headers.
+Pre-computed during indexing and used at query time
+for per-field BM25 length normalization.
+
+## Relationships
+
+```
+         qp_parse_query
+query string ──────────→ QueryPlan
+                              │
+                   db_execute_plan
+                              │
+                              ▼
+                      Vec<RawMatch>
+                              │
+              qp_score_bm25f / proximity / zone
+                              │
+                              ▼
+                      ranked results
+```
+
+Both `QueryPlan` and `RawMatch` are serialized
+through shared `WebAssembly.Memory` via the `Encode`
+trait (see [encode](../encode/traits.md))
+when crossing module boundaries. `kb-worker` handles
+the serialization and deserialization at each step.
+
+## Trait Implementations
+
+| Trait    | Types                    |
+|----------|--------------------------|
+| `Clone`  | `QueryPlan`, `MatchMode`, `RawMatch`, `FieldTf`, `FieldTokenCounts` |
+| `Debug`  | All types                |
+| `Encode` | `QueryPlan`, `MatchMode`, `RawMatch`, `FieldTf`, `FieldTokenCounts` |
+| `Copy`   | `MatchMode`, `FieldTf`, `FieldTokenCounts` |
+
+`MatchMode` also derives `PartialEq` and `Eq`.
+
+## Testing
+
+Unit tests:
+
+- `QueryPlan` construction with terms, phrases,
+  exclusions, and both match modes.
+- `Encode` round-trip for `QueryPlan` with all
+  fields populated.
+- `Encode` round-trip for `RawMatch` with
+  `field_tfs`, positions, `token_count`, and
+  `field_token_counts`.
+- `FieldTf` and `FieldTokenCounts` construction
+  and `Encode` round-trip.
+- Empty edge cases: plan with no terms, match with
+  no positions, zero field counts.

package/docs/modules/rust/shared/section/module.md

@@ -0,0 +1,25 @@
+# section -- Section Types and Records
+
+*Section type enum, MIME type wrapper, and data
+structures for section CRUD operations.*
+
+## Source
+
+`engine/crates/shared/src/section/`
+
+## Responsibility
+
+Defines the content type system (`SectionType`,
+`MimeType`) and the data structures for section CRUD
+operations (`SectionRecord`, `AddSectionParams`,
+etc.) shared across all three WASM modules.
+
+See [Database -- Sections](../../../../goals/database.md)
+for the behavioral specification of section types.
+
+## Exports
+
+- **[types.md](types.md)** -- `SectionType`,
+  `MimeType`, `SectionRecord`, `AddSectionParams`,
+  `AddSectionResult`, `UpdateSectionResult`,
+  `RemoveSectionResult`, `MimeError`.