@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/details/search-and-ranking.md

@@ -0,0 +1,457 @@
+# How Search Works
+
+kbdb uses field-weighted BM25F ranking to score search
+results by relevance. Sections are indexed across
+three fields -- heading, body, and code -- each with a
+different weight so that a match in a title counts
+more than a match buried in body text. Queries
+support simple keyword searches, exact phrases,
+boolean operators, and term exclusions.
+
+## Query Syntax
+
+Queries are plain text with optional operators.
+The query parser tokenizes and normalizes query
+terms through the same pipeline used for indexing,
+so search terms match consistently regardless of
+case or inflection.
+
+### Examples
+
+| Query               | Meaning                          |
+|---------------------|----------------------------------|
+| `auth flow`         | Sections containing both terms   |
+| `"error handling"`  | Exact phrase match               |
+| `auth -oauth`       | "auth" but not "oauth"           |
+| `auth AND token`    | Both terms required              |
+| `cache OR redis`    | Either term matches              |
+
+### Syntax Reference
+
+| Syntax          | Effect                             |
+|-----------------|------------------------------------|
+| `word`          | Match normalized/stemmed form      |
+| `"two words"`   | Match exact phrase in order        |
+| `-word`         | Exclude sections containing term   |
+| `AND`           | Both operands required             |
+| `OR`            | Either operand sufficient          |
+
+Operators can be combined. Unquoted multi-word
+queries are implicitly ANDed -- `auth flow` matches
+sections containing both "auth" and "flow".
+
+## Tokenization Pipeline
+
+Both indexing and search queries pass through the
+same eight-stage pipeline. Using identical processing
+for both sides guarantees that stored terms match
+query terms.
+
+```
+  Raw text
+    |
+    v
+  1. Strip content markers
+    |
+    v
+  2. Unicode NFC normalization
+    |
+    v
+  3. Word boundary detection
+    |
+    v
+  4. Identifier splitting
+    |
+    v
+  5. Case folding
+    |
+    v
+  6. Stop word removal
+    |
+    v
+  7. Stemming
+    |
+    v
+  8. N-gram extraction
+    |
+    v
+  Normalized tokens
+```
+
+1. **Marker stripping** -- Remove all six content
+   marker types (`$(image:)`, `$(code:)`, `$(text:)`,
+   `$(document:)`, `$(link:)`, `$(doc-link:)`) before
+   processing. Markers are structural references, not
+   searchable content.
+
+2. **Unicode NFC normalization** -- Compose decomposed
+   Unicode sequences into their canonical form. For
+   example, `e` + combining acute accent becomes the
+   single codepoint `e` (U+00E9). Ensures equivalent
+   Unicode representations produce the same tokens.
+
+3. **Word boundary detection** -- Split text at
+   Unicode-defined word boundaries using the Unicode
+   Segmentation algorithm. Handles punctuation,
+   whitespace, and mixed-script text correctly.
+
+4. **Identifier splitting** -- Decompose programming
+   identifiers into constituent words.
+   `camelCase` becomes `["camel", "case"]`;
+   `snake_case` becomes `["snake", "case"]`;
+   `PascalCase` becomes `["pascal", "case"]`.
+   This stage makes code searchable by individual
+   words within identifiers.
+
+5. **Case folding** -- Convert all tokens to
+   lowercase. `HTTP`, `Http`, and `http` all become
+   `http`.
+
+6. **Stop word removal** -- Remove common words with
+   no discriminative value (`the`, `is`, `and`,
+   etc.). The stop word list is configurable.
+
+7. **Stemming** -- Reduce words to root forms using
+   the Snowball/Porter2 stemmer. `running` and `runs`
+   both stem to `run`. Both the stemmed form and the
+   original are stored so exact-match queries still
+   work.
+
+8. **N-gram extraction** -- Generate bigrams and
+   trigrams for phrase indexing and multi-word term
+   matching. Supports phrase queries without requiring
+   a separate phrase index.
+
+### Per-Field Processing
+
+During indexing, the pipeline runs once per field for
+each section:
+
+| Content source          | Field     |
+|-------------------------|-----------|
+| `title` attribute       | heading   |
+| Body text + description | body      |
+| Code content            | code      |
+
+Per-field term frequencies are stored separately in
+the inverted index. This separation is what enables
+BM25F field weights to be tuned at query time without
+rebuilding the indexes.
+
+## BM25F Ranking
+
+BM25F (Field-Weighted Okapi BM25) is the primary
+ranking function. It is a probabilistic scoring model
+from information retrieval that balances three
+factors: how often a term appears in a section, how
+rare the term is across the corpus, and how long the
+section is relative to average.
+
+### The Three Fields
+
+Each section is indexed across three fields, each
+with a default weight:
+
+| Field     | Weight | Content source             |
+|-----------|--------|----------------------------|
+| `heading` | 2.0x   | Section `title` attribute  |
+| `body`    | 1.0x   | Body text + `description`  |
+| `code`    | 1.5x   | Code section content       |
+
+A term match in a heading contributes twice as much
+to the score as the same match in body text. A match
+in code contributes 1.5 times as much. This reflects
+the intuition that titles are the strongest signal of
+what a section is about.
+
+### How Scoring Works
+
+For each query term `t` matched in section `d`:
+
+1. **IDF (Inverse Document Frequency)** -- Measures
+   how rare the term is across all sections. Common
+   terms like "function" get a low IDF; rare terms
+   like "kademlia" get a high IDF.
+
+   ```
+   idf(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)
+   ```
+
+   `N` = total indexed sections, `df(t)` = number of
+   sections containing `t`.
+
+2. **Per-field score** -- For each field `f` (heading,
+   body, code), compute a BM25 score using that
+   field's term frequency and average length:
+
+   ```
+   score_f(t, d) = idf(t)
+      * (tf_f(t,d) * (k1 + 1))
+      / (tf_f(t,d) + k1 * (1 - b + b * |d_f| / avgdl_f))
+   ```
+
+   Where:
+   - `tf_f(t,d)` = term frequency in field `f`
+   - `|d_f|` = token count of field `f` in section `d`
+   - `avgdl_f` = average token count of field `f`
+     across the corpus
+   - `k1` = 1.2 (term saturation parameter)
+   - `b` = 0.75 (length normalization parameter)
+
+3. **Weighted combination** -- Multiply each field
+   score by its weight and sum:
+
+   ```
+   bm25f(t, d) = 2.0 * score_heading(t, d)
+               + 1.0 * score_body(t, d)
+               + 1.5 * score_code(t, d)
+   ```
+
+4. **Total section score** -- Sum `bm25f(t, d)`
+   across all matched query terms.
+
+### Key Parameters
+
+| Parameter        | Default | Effect                    |
+|------------------|---------|---------------------------|
+| `k1`             | 1.2     | Term saturation. Higher = more weight to repeated terms |
+| `b`              | 0.75    | Length normalization. 0 = none, 1 = full |
+| `heading` weight | 2.0     | Title match influence     |
+| `body` weight    | 1.0     | Body text match influence |
+| `code` weight    | 1.5     | Code match influence      |
+
+### Why Per-Field Storage Matters
+
+Per-field term frequencies (`heading_tf`, `body_tf`,
+`code_tf`) are stored separately in the inverted
+index rather than as a single aggregated value. This
+means field weights can be changed at query time --
+for example, boosting code matches to 3.0x for a
+code-focused search -- without rebuilding any index
+files.
+
+### Per-Field Length Normalization
+
+Corpus statistics store per-field average lengths
+(`avg_heading_length`, `avg_body_length`,
+`avg_code_length`) alongside overall averages. This
+prevents unfair comparisons: a 3-word title is
+normalized against the average title length, not
+against the average body length of 500 words.
+
+## Supplementary Scoring Signals
+
+Three additional signals are added to the BM25F base
+score:
+
+### Proximity Boost
+
+Boosts sections where query terms appear close
+together. For multi-term queries, the proximity
+scorer computes the minimum distance between each
+pair of query terms using their stored positions. A
+smaller span produces a higher boost. Single-term
+queries receive no proximity boost.
+
+```
+final proximity = inverse of minimum span
+                  across all query term pairs
+```
+
+### Zone Scoring
+
+Structural weighting based on position within a
+section. Terms in the first ~20% of a section (the
+"heading zone") receive a higher weight than terms in
+the rest of the body. The zone boundary is computed
+as `0.2 * token_count` using the pre-computed total
+token count. This approximates the idea that
+headings and opening paragraphs carry more topical
+signal.
+
+### Freshness Weighting (Optional)
+
+Time-decay boost that favors recently added sections.
+Uses exponential decay with a configurable half-life:
+after one half-life period, the freshness score is
+halved. Can be disabled entirely by omitting the
+freshness weight from score combination.
+
+### Score Combination
+
+```
+final_score = bm25f_score
+            + proximity_weight * proximity_score
+            + zone_weight * zone_score
+            + freshness_weight * freshness_score
+```
+
+BM25F is always the dominant signal. Proximity, zone,
+and freshness are additive boosts with configurable
+weights. Results are sorted by `final_score`
+descending; ties broken by kbid lexicographic order.
+
+## Keyword Extraction
+
+When content is added to the knowledge base, the
+system extracts searchable terms and computes
+weights using statistical methods that require no
+training data or external models.
+
+### RAKE (Rapid Automatic Keyword Extraction)
+
+Splits text on stop words to identify candidate
+keyphrases. Scores candidates by word degree divided
+by frequency ratio -- multi-word phrases with
+infrequent component words score highest. Runs on
+both text and code content.
+
+### N-Gram Scoring
+
+Bigrams and trigrams are scored by co-occurrence
+frequency. High-frequency pairs and triples become
+indexed terms, supporting multi-word searches without
+exact phrase syntax.
+
+### TF-IDF Term Weighting
+
+For each section, per-field term frequency (TF) is
+computed separately for heading, body, and code
+fields. These per-field TFs are combined with
+corpus-wide inverse document frequency (IDF) to
+produce weights reflecting each term's
+discriminative value. The per-field TFs are stored in
+the inverted index; IDF is computed from corpus
+statistics at query time.
+
+### What Gets Indexed
+
+| Section type   | Body indexed? | Title indexed? | Description indexed? |
+|----------------|---------------|----------------|----------------------|
+| `text/*`       | Yes (body)    | Yes (heading)  | Yes (body)           |
+| `image/*`      | No            | Yes (heading)  | Yes (body)           |
+
+Image binary content is skipped entirely. Image
+sections are searchable only through their `title`
+(required) and `description` (optional) attributes.
+
+## Index Structure
+
+All index files live in `.kbdb/indexes/`. They are
+binary, derived, and rebuildable from source data.
+The `.gitignore` at the database root excludes them
+from version control.
+
+```
+indexes/
++-- meta.toml        human-readable metadata
++-- terms.idx        inverted index (B+ tree)
++-- positions.idx    positional index (B+ tree)
++-- corpus.dat       corpus statistics (binary)
+```
+
+### `terms.idx` -- Inverted Index
+
+B+ tree mapping normalized terms to posting lists.
+Each posting entry contains:
+
+| Field        | Type  | Description                |
+|--------------|-------|----------------------------|
+| `kbid`       | bytes | Section identifier         |
+| `heading_tf` | f32   | Term frequency in heading  |
+| `body_tf`    | f32   | Term frequency in body     |
+| `code_tf`    | f32   | Term frequency in code     |
+
+Supports term lookup, IDF computation (via document
+frequency counts), and per-field term frequency
+retrieval. O(log n) lookup via B+ tree structure.
+
+### `positions.idx` -- Positional Index
+
+B+ tree mapping `(term, kbid)` pairs to arrays of
+zero-based token offsets. Supports three functions:
+
+- **Phrase queries** -- Verify that terms appear in
+  the correct order at consecutive positions.
+- **Proximity scoring** -- Compute minimum span
+  between query terms.
+- **Zone scoring** -- Determine whether a term falls
+  in the heading zone (first 20% of the section).
+
+### `corpus.dat` -- Corpus Statistics
+
+Binary file storing corpus-wide aggregates:
+
+| Field                 | Type | Description              |
+|-----------------------|------|--------------------------|
+| `total_sections`      | u32  | Indexed section count    |
+| `total_terms`         | u32  | Distinct term count      |
+| `avg_section_length`  | f32  | Average tokens/section   |
+| `avg_heading_length`  | f32  | Average heading tokens   |
+| `avg_body_length`     | f32  | Average body tokens      |
+| `avg_code_length`     | f32  | Average code tokens      |
+| per-term doc freqs    | --   | Per-term section counts  |
+
+Used for IDF computation and per-field BM25 length
+normalization.
+
+### `meta.toml` -- Index Metadata
+
+Human-readable TOML file with informational metadata
+about the indexes:
+
+- Last rebuild timestamp
+- Total term count
+- Corpus statistics summary
+
+This file is informational only -- the binary index
+files are the source of truth for search operations.
+
+## Ranking Pipeline (End to End)
+
+```
+  "auth token" (query string)
+       |
+       v
+  Tokenize + parse query       (query-parser.wasm)
+       |
+       v
+  QueryPlan: terms=["auth","token"], mode=AND
+       |
+       v
+  Execute plan against indexes  (fs-database.wasm)
+       |
+       v
+  Vec<RawMatch>: kbids + per-field TFs + positions
+       |
+       v
+  Fetch corpus statistics       (fs-database.wasm)
+       |
+       v
+  Score: BM25F                  (query-parser.wasm)
+  Score: proximity              (query-parser.wasm)
+  Score: zone                   (query-parser.wasm)
+  Score: freshness (optional)   (query-parser.wasm)
+       |
+       v
+  Combine scores, sort descending, truncate
+       |
+       v
+  Shape into result mode (sections/documents/stats)
+       |
+       v
+  Ranked results
+```
+
+`kb-worker.wasm` orchestrates the entire flow,
+calling `query-parser.wasm` for text processing and
+scoring, and `fs-database.wasm` for storage lookups.
+Neither standalone module calls the other directly.
+
+## Result Modes
+
+| Mode        | Returns                             |
+|-------------|-------------------------------------|
+| `sections`  | Individual matching sections with scores, snippets, and matched terms |
+| `documents` | Documents containing matches, with aggregated scores and per-section details |
+| `stats`     | Match counts and top terms without content bodies |