@dikolab/kbdb 0.5.0 → 0.6.1

package/docs/details/search-and-ranking.md

@@ -1,575 +0,0 @@
-# How Search Works
-
-When you search, kbdb finds the sections that best
-match your query and ranks them by relevance. By
-default it uses keyword search -- matching terms with
-stemming, synonym expansion, and BM25F ranking. Use
-`--algo hybrid` to combine keyword matching with
-AI-powered similarity for broader recall. The rest
-of this page explains how ranking
-works under the hood -- you don't need to read it
-to use kbdb.
-
----
-
-kbdb uses field-weighted BM25F ranking to score search
-results by relevance. Sections are indexed across
-three fields -- heading, body, and code -- each with a
-different weight so that a match in a title counts
-more than a match buried in body text. Queries
-support simple keyword searches, exact phrases,
-boolean operators, and term exclusions.
-
-## Query Syntax
-
-Queries are plain text with optional operators.
-The query parser tokenizes and normalizes query
-terms through the same pipeline used for indexing,
-so search terms match consistently regardless of
-case or inflection.
-
-### Examples
-
-| Query               | Meaning                          |
-|---------------------|----------------------------------|
-| `auth flow`         | Sections containing both terms   |
-| `"error handling"`  | Exact phrase match               |
-| `auth -oauth`       | "auth" but not "oauth"           |
-| `auth AND token`    | Both terms required              |
-| `cache OR redis`    | Either term matches              |
-
-### Syntax Reference
-
-| Syntax          | Effect                             |
-|-----------------|------------------------------------|
-| `word`          | Match normalized/stemmed form      |
-| `"two words"`   | Match exact phrase in order        |
-| `-word`         | Exclude sections containing term   |
-| `AND`           | Both operands required             |
-| `OR`            | Either operand sufficient          |
-
-Operators can be combined. Unquoted multi-word
-queries are implicitly ANDed -- `auth flow` matches
-sections containing both "auth" and "flow".
-
-## Tokenization Pipeline
-
-Both indexing and search queries pass through the
-same nine-stage pipeline. Using identical processing
-for both sides guarantees that stored terms match
-query terms.
-
-```
-  Raw text
-    |
-    v
-  1. Strip content markers
-    |
-    v
-  2. Unicode NFC normalization
-    |
-    v
-  3. Word boundary detection
-    |
-    v
-  4. Identifier splitting
-    |
-    v
-  5. Case folding
-    |
-    v
-  6. Stop word removal
-    |
-    v
-  7. Synonym expansion
-    |
-    v
-  8. Stemming
-    |
-    v
-  9. N-gram extraction
-    |
-    v
-  Normalized tokens
-```
-
-1. **Marker stripping** -- Remove all six content
-   marker types (`$(image:)`, `$(code:)`, `$(text:)`,
-   `$(document:)`, `$(link:)`, `$(doc-link:)`) before
-   processing. Markers are structural references, not
-   searchable content.
-
-2. **Unicode NFC normalization** -- Compose decomposed
-   Unicode sequences into their canonical form. For
-   example, `e` + combining acute accent becomes the
-   single codepoint `e` (U+00E9). Ensures equivalent
-   Unicode representations produce the same tokens.
-
-3. **Word boundary detection** -- Split text at
-   Unicode-defined word boundaries using the Unicode
-   Segmentation algorithm. Handles punctuation,
-   whitespace, and mixed-script text correctly.
-
-4. **Identifier splitting** -- Decompose programming
-   identifiers into constituent words.
-   `camelCase` becomes `["camel", "case"]`;
-   `snake_case` becomes `["snake", "case"]`;
-   `PascalCase` becomes `["pascal", "case"]`.
-   This stage makes code searchable by individual
-   words within identifiers.
-
-5. **Case folding** -- Convert all tokens to
-   lowercase. `HTTP`, `Http`, and `http` all become
-   `http`.
-
-6. **Stop word removal** -- Remove common words with
-   no discriminative value (`the`, `is`, `and`,
-   etc.). The stop word list is configurable.
-
-7. **Synonym expansion** -- Look up each token in a
-   built-in bidirectional synonym map and insert
-   synonym tokens alongside the original. Bridges
-   vocabulary gaps: a document using "sign-in" becomes
-   findable via "login". The map covers common
-   IT/development vocabulary (~30-50 entries:
-   login/sign-in, auth/authentication,
-   remove/delete, config/configuration, etc.).
-   Applied symmetrically during both indexing and
-   querying.
-
-8. **Stemming** -- Reduce words to root forms using
-   the Snowball/Porter2 stemmer. `running` and `runs`
-   both stem to `run`. Both the stemmed form and the
-   original are stored so exact-match queries still
-   work.
-
-9. **N-gram extraction** -- Generate bigrams and
-   trigrams for phrase indexing and multi-word term
-   matching. Supports phrase queries without requiring
-   a separate phrase index.
-
-### Per-Field Processing
-
-During indexing, the pipeline runs once per field for
-each section:
-
-| Content source          | Field     |
-|-------------------------|-----------|
-| `title` attribute       | heading   |
-| Body text + description | body      |
-| Code content            | code      |
-
-Per-field term frequencies are stored separately in
-the inverted index. This separation is what enables
-BM25F field weights to be tuned at query time without
-rebuilding the indexes.
-
-## BM25F Ranking
-
-BM25F (Field-Weighted Okapi BM25) is the primary
-ranking function. It is a probabilistic scoring model
-from information retrieval that balances three
-factors: how often a term appears in a section, how
-rare the term is across the corpus, and how long the
-section is relative to average.
-
-### The Three Fields
-
-Each section is indexed across three fields, each
-with a default weight:
-
-| Field     | Weight | Content source             |
-|-----------|--------|----------------------------|
-| `heading` | 2.0x   | Section `title` attribute  |
-| `body`    | 1.0x   | Body text + `description`  |
-| `code`    | 1.5x   | Code section content       |
-
-A term match in a heading contributes twice as much
-to the score as the same match in body text. A match
-in code contributes 1.5 times as much. This reflects
-the intuition that titles are the strongest signal of
-what a section is about.
-
-### How Scoring Works
-
-For each query term `t` matched in section `d`:
-
-1. **IDF (Inverse Document Frequency)** -- Measures
-   how rare the term is across all sections. Common
-   terms like "function" get a low IDF; rare terms
-   like "kademlia" get a high IDF.
-
-   ```
-   idf(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)
-   ```
-
-   `N` = total indexed sections, `df(t)` = number of
-   sections containing `t`.
-
-2. **Per-field score** -- For each field `f` (heading,
-   body, code), compute a BM25 score using that
-   field's term frequency and average length:
-
-   ```
-   score_f(t, d) = idf(t)
-      * (tf_f(t,d) * (k1 + 1))
-      / (tf_f(t,d) + k1 * (1 - b + b * |d_f| / avgdl_f))
-   ```
-
-   Where:
-   - `tf_f(t,d)` = term frequency in field `f`
-   - `|d_f|` = token count of field `f` in section `d`
-   - `avgdl_f` = average token count of field `f`
-     across the corpus
-   - `k1` = 1.2 (term saturation parameter)
-   - `b` = 0.75 (length normalization parameter)
-
-3. **Weighted combination** -- Multiply each field
-   score by its weight and sum:
-
-   ```
-   bm25f(t, d) = 2.0 * score_heading(t, d)
-               + 1.0 * score_body(t, d)
-               + 1.5 * score_code(t, d)
-   ```
-
-4. **Total section score** -- Sum `bm25f(t, d)`
-   across all matched query terms.
-
-### Key Parameters
-
-| Parameter        | Default | Effect                    |
-|------------------|---------|---------------------------|
-| `k1`             | 1.2     | Term saturation. Higher = more weight to repeated terms |
-| `b`              | 0.75    | Length normalization. 0 = none, 1 = full |
-| `heading` weight | 2.0     | Title match influence     |
-| `body` weight    | 1.0     | Body text match influence |
-| `code` weight    | 1.5     | Code match influence      |
-
-### Why Per-Field Storage Matters
-
-Per-field term frequencies (`heading_tf`, `body_tf`,
-`code_tf`) are stored separately in the inverted
-index rather than as a single aggregated value. This
-means field weights can be changed at query time --
-for example, boosting code matches to 3.0x for a
-code-focused search -- without rebuilding any index
-files.
-
-### Per-Field Length Normalization
-
-Corpus statistics store per-field average lengths
-(`avg_heading_length`, `avg_body_length`,
-`avg_code_length`) alongside overall averages. This
-prevents unfair comparisons: a 3-word title is
-normalized against the average title length, not
-against the average body length of 500 words.
-
-## Supplementary Scoring Signals
-
-Three additional signals are added to the BM25F base
-score:
-
-### Proximity Boost
-
-Boosts sections where query terms appear close
-together. For multi-term queries, the proximity
-scorer computes the minimum distance between each
-pair of query terms using their stored positions. A
-smaller span produces a higher boost. Single-term
-queries receive no proximity boost.
-
-```
-final proximity = inverse of minimum span
-                  across all query term pairs
-```
-
-### Zone Scoring
-
-Structural weighting based on position within a
-section. Terms in the first ~20% of a section (the
-"heading zone") receive a higher weight than terms in
-the rest of the body. The zone boundary is computed
-as `0.2 * token_count` using the pre-computed total
-token count. This approximates the idea that
-headings and opening paragraphs carry more topical
-signal.
-
-### Freshness Weighting (Optional)
-
-Time-decay boost that favors recently added sections.
-Uses exponential decay with a configurable half-life:
-after one half-life period, the freshness score is
-halved. Can be disabled entirely by omitting the
-freshness weight from score combination.
-
-### Score Combination
-
-```
-final_score = bm25f_score
-            + proximity_weight * proximity_score
-            + zone_weight * zone_score
-            + freshness_weight * freshness_score
-```
-
-BM25F is always the dominant signal. Proximity, zone,
-and freshness are additive boosts with configurable
-weights. Results are sorted by `final_score`
-descending; ties broken by kbid lexicographic order.
-
-### Score Normalization
-
-Raw BM25F scores are normalized to a 0.0--1.0
-`confidence` value within each result set. Each
-score is divided by the maximum score in the result
-set, so the top result always has confidence 1.0.
-This lets agents distinguish strong matches from
-weak tail results without interpreting raw score
-magnitudes.
-
-Confidence is exposed in CLI output but omitted from
-MCP results (ordering is the signal for agents).
-
-## Relaxed-Match Fallback
-
-When a multi-term AND query returns zero results,
-the engine automatically retries with OR semantics
--- any term matches instead of all required. This
-prevents silent misses when the query is too
-specific for the corpus.
-
-The fallback is transparent: no special syntax
-needed. Results from relaxed matching are flagged
-with `relaxed: true` in the response envelope so
-the caller knows the match is weaker. Single-term
-queries are unaffected (AND and OR are equivalent
-for one term). Phrase queries and exclusions are
-preserved in both modes.
-
-## Keyword Extraction
-
-When content is added to the knowledge base, the
-system extracts searchable terms and computes
-weights using statistical methods that require no
-training data or external models.
-
-### RAKE (Rapid Automatic Keyword Extraction)
-
-Splits text on stop words to identify candidate
-keyphrases. Scores candidates by word degree divided
-by frequency ratio -- multi-word phrases with
-infrequent component words score highest. Runs on
-both text and code content.
-
-### N-Gram Scoring
-
-Bigrams and trigrams are scored by co-occurrence
-frequency. High-frequency pairs and triples become
-indexed terms, supporting multi-word searches without
-exact phrase syntax.
-
-### TF-IDF Term Weighting
-
-For each section, per-field term frequency (TF) is
-computed separately for heading, body, and code
-fields. These per-field TFs are combined with
-corpus-wide inverse document frequency (IDF) to
-produce weights reflecting each term's
-discriminative value. The per-field TFs are stored in
-the inverted index; IDF is computed from corpus
-statistics at query time.
-
-### What Gets Indexed
-
-| Section type   | Body indexed? | Title indexed? | Description indexed? |
-|----------------|---------------|----------------|----------------------|
-| `text/*`       | Yes (body)    | Yes (heading)  | Yes (body)           |
-| `image/*`      | No            | Yes (heading)  | Yes (body)           |
-
-Image binary content is skipped entirely. Image
-sections are searchable only through their `title`
-(required) and `description` (optional) attributes.
-
-## Index Structure
-
-All index files live in `.kbdb/indexes/`. They are
-binary, derived, and rebuildable from source data.
-The `.gitignore` at the database root excludes them
-from version control.
-
-```
-indexes/
-+-- meta.toml        human-readable metadata
-+-- terms.idx        inverted index (B+ tree)
-+-- positions.idx    positional index (B+ tree)
-+-- references.idx   reference graph (B+ tree)
-+-- corpus.dat       corpus statistics (binary)
-```
-
-### `terms.idx` -- Inverted Index
-
-B+ tree mapping normalized terms to posting lists.
-Each posting entry contains:
-
-| Field        | Type  | Description                |
-|--------------|-------|----------------------------|
-| `kbid`       | bytes | Section identifier         |
-| `heading_tf` | f32   | Term frequency in heading  |
-| `body_tf`    | f32   | Term frequency in body     |
-| `code_tf`    | f32   | Term frequency in code     |
-
-Supports term lookup, IDF computation (via document
-frequency counts), and per-field term frequency
-retrieval. O(log n) lookup via B+ tree structure.
-
-### `positions.idx` -- Positional Index
-
-B+ tree mapping `(term, kbid)` pairs to arrays of
-zero-based token offsets. Supports three functions:
-
-- **Phrase queries** -- Verify that terms appear in
-  the correct order at consecutive positions.
-- **Proximity scoring** -- Compute minimum span
-  between query terms.
-- **Zone scoring** -- Determine whether a term falls
-  in the heading zone (first 20% of the section).
-
-### `references.idx` -- Reference Graph Index
-
-B+ tree mapping section kbids to their content
-marker cross-references. Stores bidirectional
-edges: for each `$(text:)`, `$(code:)`, `$(image:)`,
-`$(link:)`, `$(document:)`, and `$(doc-link:)`
-marker, both a forward entry (source -> target) and
-a back entry (target -> source) are recorded. This
-enables the `recall` command's progressive context
-expansion -- depth 1+ retrieves back-references
-(sections that point to this one) and depth 2+
-retrieves forward references (sections this one
-points to).
-
-### `corpus.dat` -- Corpus Statistics
-
-Binary file storing corpus-wide aggregates:
-
-| Field                 | Type | Description              |
-|-----------------------|------|--------------------------|
-| `total_sections`      | u32  | Indexed section count    |
-| `total_terms`         | u32  | Distinct term count      |
-| `avg_section_length`  | f32  | Average tokens/section   |
-| `avg_heading_length`  | f32  | Average heading tokens   |
-| `avg_body_length`     | f32  | Average body tokens      |
-| `avg_code_length`     | f32  | Average code tokens      |
-| per-term doc freqs    | --   | Per-term section counts  |
-
-Used for IDF computation and per-field BM25 length
-normalization.
-
-### `meta.toml` -- Index Metadata
-
-Human-readable TOML file with informational metadata
-about the indexes:
-
-- Last rebuild timestamp
-- Total term count
-- Corpus statistics summary
-
-This file is informational only -- the binary index
-files are the source of truth for search operations.
-
-## Ranking Pipeline (End to End)
-
-```
-  "auth token" (query string)
-       |
-       v
-  Tokenize + parse query       (query-parser.wasm)
-       |
-       v
-  QueryPlan: terms=["auth","token"], mode=AND
-       |
-       v
-  Execute plan against indexes  (fs-database.wasm)
-       |
-       v
-  Vec<RawMatch>: kbids + per-field TFs + positions
-       |
-       v
-  Fetch corpus statistics       (fs-database.wasm)
-       |
-       v
-  Score: BM25F                  (query-parser.wasm)
-  Score: proximity              (query-parser.wasm)
-  Score: zone                   (query-parser.wasm)
-  Score: freshness (optional)   (query-parser.wasm)
-       |
-       v
-  Combine scores, sort descending, truncate
-       |
-       v
-  Shape into result mode (sections/documents/stats)
-       |
-       v
-  Ranked results
-```
-
-`kb-worker.wasm` orchestrates the entire flow,
-calling `query-parser.wasm` for text processing and
-scoring, and `fs-database.wasm` for storage lookups.
-Neither standalone module calls the other directly.
-
-## Result Modes
-
-| Mode        | Returns                             |
-|-------------|-------------------------------------|
-| `sections`  | Individual matching sections with scores, snippets, and matched terms |
-| `documents` | Documents containing matches, with aggregated scores and per-section details |
-| `stats`     | Match counts and top terms without content bodies |
-
-## Search Algorithms
-
-kbdb supports three search algorithms, selected
-with `--algo` (CLI) or the `algo` parameter (MCP).
-
-### Lexical
-
-Keyword matching with stemming and synonym
-expansion, scored by BM25F as described above.
-Fast, works offline, and gives consistent results.
-
-### Vector
-
-Embedding-based similarity search. Each section's
-content is converted into a numeric vector (an
-"embedding") by an AI model. Search finds sections
-whose vectors are closest to the query's vector
-using cosine similarity.
-
-Vector search catches matches that keyword search
-misses -- for example, searching "database
-rationale" finds a section about "we chose Postgres
-because of JSONB" even though none of the words
-overlap.
-
-Requires an embedding provider to be configured.
-Without one, vector mode returns an error.
-
-### Hybrid
-
-Runs both lexical and vector searches in parallel,
-then merges the two ranked lists using Reciprocal
-Rank Fusion (RRF). This combines the precision of
-keyword matching with the associative recall of
-embeddings. Use `--algo hybrid` to enable.
-
-RRF works by converting each result's rank position
-into a score (`1 / (k + rank)` with k=60), then
-summing scores across both lists. Results that rank
-well in both lists get the highest combined scores.
-
-Hybrid mode uses a built-in TF-IDF embedding provider
-by default. Configure `provider = "onnx"` or
-`provider = "api"` in `worker.toml` for higher
-quality embeddings. Setting `provider = "none"`
-explicitly disables embeddings and returns an error
-for hybrid and vector searches.