@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/goals/query-result.md

@@ -0,0 +1,693 @@
+# Query Result -- Ranking and Ordering
+
+*How search results are scored, combined, ordered,
+and shaped into the three result modes.*
+
+This document is the authoritative specification for
+the ranking pipeline -- the path from raw index
+matches to ordered, scored results. It consolidates
+the algorithms, formulas, parameters, and data
+structures that determine result relevance. For the
+tokenization pipeline that produces the indexed terms,
+see [Query Parser](query-parser.md). For the storage
+engine that holds the indexes, see
+[Database](database.md).
+
+## Ranking Pipeline Overview
+
+The full ranking path, from query string to final
+result set:
+
+```
+query string
+  -> tokenize (query-parser.wasm)
+  -> parse query -> QueryPlan (query-parser.wasm)
+  -> execute plan -> Vec<RawMatch> (fs-database.wasm)
+  -> fetch CorpusStats (fs-database.wasm)
+  -> score: BM25F (query-parser.wasm)
+  -> score: proximity (query-parser.wasm)
+  -> score: zone (query-parser.wasm)
+  -> score: freshness (query-parser.wasm, optional)
+  -> combine scores
+  -> sort descending
+  -> truncate to limit
+  -> shape into result mode
+  -> return
+```
+
+`kb-worker.wasm` orchestrates the entire flow. It
+calls `query-parser.wasm` for tokenization, parsing,
+and scoring, and `fs-database.wasm` for index lookups
+and corpus statistics. Neither standalone module calls
+the other directly.
+
+## Searchable Content Sources
+
+Not all section data contributes equally to search
+and ranking. Content is tokenized and pre-computed
+at section insertion time (not at query time) and
+stored in the section header and inverted index for
+fast retrieval. See [Database](database.md) for the
+storage format of token counts and per-field term
+frequencies.
+
+### Text Content (`text/*` Sections)
+
+Sections with MIME type `text/*` (e.g.
+`text/markdown`, `text/typescript`,
+`text/javascript`) are the primary searchable
+content. Their text body is tokenized, stemmed, and
+indexed. During indexing, the system extracts and
+stores the per-field term frequencies, positions,
+and token counts needed for later search and
+ranking.
+
+Text content maps to the **body** field in BM25F
+(`FieldWeights.body`, default 1.0).
+
+### Title Attribute (All Sections)
+
+Any section with a `title` attribute has its title
+tokenized and indexed alongside (or instead of) the
+body content. Title terms receive the **heading**
+field weight in BM25F (`FieldWeights.heading`,
+default 2.0) -- giving title terms **2x the ranking
+influence** compared to body text.
+
+For binary sections (e.g. `image/*`,
+`application/octet-stream`) that have no searchable
+text body, the `title` is **required** and serves as
+the primary searchable content. Without a title,
+binary sections are not searchable and cannot appear
+in ranked results.
+
+### Description Attribute (Optional)
+
+Any section with a `description` attribute has its
+description tokenized and indexed. Description terms
+receive the **body** field weight in BM25F
+(`FieldWeights.body`, default 1.0) -- the same
+ranking weight as `text/*` section body content.
+
+Description is optional for all section types. When
+present, it provides additional searchable text that
+supplements the body content (for text sections) or
+the title (for binary sections).
+
+### Binary Sections
+
+Sections with binary MIME types (e.g. `image/*`,
+`application/octet-stream`) contain non-textual data
+(base64-encoded) that cannot be tokenized or indexed
+directly. Their searchability relies entirely on
+metadata attributes:
+
+1. **`title`** (required for searchability) -- indexed
+   with heading weight (2.0x). A binary section
+   without a title is stored but invisible to search.
+2. **`description`** (optional) -- indexed with body
+   weight (1.0x). Provides additional search surface
+   for binary content.
+
+A well-titled image section can appear in search
+results ranked by its title relevance, even though
+its pixel data is not searchable.
+
+### Code Sections
+
+Sections with code-related MIME types (e.g.
+`text/typescript`, `text/javascript`) have their
+source code tokenized with identifier splitting
+(camelCase, snake_case, PascalCase). Code content
+maps to the **code** field weight in BM25F
+(`FieldWeights.code`, default 1.5) -- giving code
+terms **1.5x the ranking influence** compared to
+plain body text.
+
+If a code section also has `title` and/or
+`description`, those are indexed with their
+respective weights (heading 2.0x, body 1.0x) in
+addition to the code body.
+
+### Field Weight Mapping
+
+| Content source          | BM25F field | Default weight |
+|-------------------------|-------------|----------------|
+| `title` attribute       | `heading`   | 2.0            |
+| `text/*` body content   | `body`      | 1.0            |
+| `description` attribute | `body`      | 1.0            |
+| Code section body       | `code`      | 1.5            |
+| Binary content (raw)    | --          | not indexed    |
+
+### Pre-Computed Data at Insertion Time
+
+When a section is added or updated, `kb-worker.wasm`
+calls `query-parser.wasm` to tokenize the content
+and compute the following data, which is then stored
+by `fs-database.wasm` for use at query time:
+
+- **Per-field term frequencies** -- for each term,
+  the term frequency in each field (heading, body,
+  code) is stored in the inverted index as separate
+  values rather than a single aggregated weight. This
+  lets BM25F apply field weights at query time,
+  making them tunable without rebuilding indexes.
+- **Token counts** -- total and per-field token
+  counts are stored in the section header. Used for
+  BM25F length normalization and zone scoring.
+- **Term positions** -- zero-based token offsets
+  stored in the positional index. Used for proximity
+  and zone scoring.
+- **Corpus statistics updates** -- total sections,
+  total terms, per-term document frequencies, and
+  per-field average lengths are updated in
+  `corpus.dat`.
+
+## Raw Match (Pre-Ranking Data)
+
+A `RawMatch` is the data that comes out of
+`fs-database.wasm` after index lookups, before any
+scoring is applied:
+
+```rust
+pub struct RawMatch {
+   pub kbid: KbId,
+   pub field_tfs: Vec<(Term, FieldTf)>,
+   pub positions: Vec<(Term, Vec<u32>)>,
+   pub token_count: u32,
+   pub field_token_counts: FieldTokenCounts,
+}
+
+pub struct FieldTf {
+   pub heading: f32,
+   pub body: f32,
+   pub code: f32,
+}
+
+pub struct FieldTokenCounts {
+   pub heading: u32,
+   pub body: u32,
+   pub code: u32,
+}
+```
+
+- `field_tfs` -- per-term, per-field term frequency
+  values retrieved from the inverted index. Each
+  entry contains the TF for the heading, body, and
+  code fields separately.
+- `positions` -- zero-based token offsets per term,
+  retrieved from the positional index. Fed to
+  proximity and zone scoring.
+- `token_count` -- total tokens in the section,
+  retrieved from the section header. Used for BM25F
+  length normalization (`|d|`) and zone scoring (20%
+  heading zone boundary).
+- `field_token_counts` -- per-field token counts,
+  retrieved from the section header. Used for
+  per-field BM25 length normalization.
+
+## Corpus Statistics
+
+`CorpusStats` provides corpus-wide aggregates needed
+for IDF computation and length normalization:
+
+```rust
+pub struct CorpusStats {
+   pub total_sections: u32,
+   pub total_terms: u32,
+   pub avg_section_length: f32,
+   pub avg_heading_length: f32,
+   pub avg_body_length: f32,
+   pub avg_code_length: f32,
+}
+```
+
+- `total_sections` -- `N` in the IDF formula. Counts
+  only sections with at least one indexed term
+  (excludes binary sections with no title).
+- `avg_section_length` -- average total token count
+  per indexed section. Used for overall BM25F length
+  normalization.
+- `avg_heading_length` -- average heading field token
+  count. Used for per-field BM25 normalization of
+  title terms.
+- `avg_body_length` -- average body field token
+  count. Used for per-field BM25 normalization of
+  body and description terms.
+- `avg_code_length` -- average code field token
+  count. Used for per-field BM25 normalization of
+  code terms.
+
+Per-field averages enable accurate per-field BM25
+normalization -- a 3-word title is not penalized
+against a 500-word average body length.
+
+## Scoring Functions
+
+### BM25F (Primary Ranking)
+
+Field-Weighted Okapi BM25. The primary ranking
+function that produces the base relevance score.
+
+**Function signature:**
+
+```rust
+pub fn score_bm25f(
+   params: &BM25FParams,
+   raw_match: &RawMatch,
+   corpus: &CorpusStats,
+) -> f32;
+```
+
+**Parameters:**
+
+```rust
+pub struct BM25FParams {
+   pub k1: f32,              // default 1.2
+   pub b: f32,               // default 0.75
+   pub field_weights: FieldWeights,
+}
+
+pub struct FieldWeights {
+   pub heading: f32,         // default 2.0
+   pub body: f32,            // default 1.0
+   pub code: f32,            // default 1.5
+}
+```
+
+**IDF formula:**
+
+```
+idf(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)
+```
+
+Where `N` = `corpus.total_sections` and `df(t)` =
+number of sections containing term `t`.
+
+**Per-field BM25 score:**
+
+For each field `f` (heading, body, code), the
+per-term score uses the field-specific TF and the
+field-specific average length:
+
+```
+score_f(t, d) = idf(t)
+   * (tf_f(t,d) * (k1 + 1))
+   / (tf_f(t,d) + k1 * (1 - b + b * |d_f| / avgdl_f))
+```
+
+Where:
+- `tf_f(t,d)` = term frequency of `t` in field `f`
+  of section `d` (from `FieldTf`)
+- `|d_f|` = token count of field `f` in section `d`
+  (from `FieldTokenCounts`)
+- `avgdl_f` = average token count of field `f`
+  across the corpus (from `CorpusStats`)
+
+**Field weighting:**
+
+The final BM25F score for a term is the weighted sum
+of per-field scores:
+
+```
+bm25f(t, d) = w_heading * score_heading(t, d)
+            + w_body * score_body(t, d)
+            + w_code * score_code(t, d)
+```
+
+The total BM25F score for a section is the sum of
+`bm25f(t, d)` across all matched query terms.
+
+**Parameter explanations:**
+
+- `k1` (1.2) -- term saturation. Higher values let
+  term frequency continue to increase the score;
+  lower values flatten the curve faster.
+- `b` (0.75) -- length normalization. 0 = no length
+  normalization; 1 = full normalization. At 0.75,
+  longer sections are moderately penalized relative
+  to average-length sections.
+- `field_weights` -- per-field multipliers. Because
+  per-field TFs are stored separately in the inverted
+  index, field weights can be tuned at query time
+  without rebuilding indexes.
+
+### Proximity Scoring (Supplementary)
+
+Boosts sections where query terms appear close
+together.
+
+**Function signature:**
+
+```rust
+pub fn score_proximity(
+   positions: &[(Term, Vec<u32>)],
+   query_terms: &[Term],
+) -> f32;
+```
+
+**Algorithm:**
+
+1. For each pair of query terms that both appear in
+   the section, compute the minimum distance between
+   any of their positions.
+2. The proximity score is the inverse of the minimum
+   span across all query terms.
+3. Sections where all query terms appear within a
+   small window get the highest boost.
+4. Single-term queries receive a proximity score of 0
+   (no boost).
+
+### Zone Scoring (Supplementary)
+
+Structural weighting. Terms appearing in high-value
+positions within a section get boosted.
+
+**Function signature:**
+
+```rust
+pub fn score_zone(
+   positions: &[(Term, Vec<u32>)],
+   section_length: u32,
+) -> f32;
+```
+
+**Algorithm:**
+
+- Terms in the first ~20% of a section (the "heading
+  zone") receive higher weight than terms in the
+  remaining body.
+- The heading zone boundary is computed as
+  `0.2 * token_count`, using the pre-computed
+  `token_count` from the section header.
+- The heading zone approximates headings and first
+  paragraphs, which typically carry more topical
+  signal.
+- The zone score is added to the combined score as a
+  supplementary boost.
+
+### Freshness Weighting (Optional)
+
+Time-decay boost based on section insertion
+timestamp.
+
+**Function signature:**
+
+```rust
+pub fn score_freshness(
+   created_at: u64,
+   now: u64,
+   half_life_days: f32,
+) -> f32;
+```
+
+**Algorithm:**
+
+- Exponential decay: newer sections score higher.
+- `half_life_days` controls how fast the boost
+  decays. After one half-life, the freshness score
+  is halved.
+- This is an **optional** signal -- it can be
+  disabled by omitting the freshness weight from
+  score combination.
+
+## Score Combination
+
+The four individual scores are combined into a final
+ranking score:
+
+```
+final_score = bm25f_score
+            + proximity_weight * proximity_score
+            + zone_weight * zone_score
+            + freshness_weight * freshness_score
+```
+
+BM25F is the primary signal. Proximity, zone, and
+freshness are supplementary signals added to the
+BM25F base. Their weights control how much influence
+each supplementary signal has relative to the primary
+BM25F score.
+
+Results are sorted by `final_score` in **descending
+order** (highest relevance first). Ties are broken
+by kbid lexicographic order for deterministic
+ordering.
+
+## Result Truncation and Pagination
+
+After sorting, results are windowed by `offset` and
+`limit`. The ranking pipeline scores and sorts the
+full result set, then skips the first `offset`
+entries and takes the next `limit` entries. Both
+parameters are optional: `offset` defaults to `0`,
+`limit` defaults to `20`.
+
+The result is a paged envelope, not a bare array:
+
+```ts
+interface PagedSearchResult {
+   items: SectionMatch[] | DocumentMatch[];
+   total: number;
+   offset: number;
+   limit: number;
+   has_more: boolean;
+}
+```
+
+| Field      | Description                       |
+|------------|-----------------------------------|
+| `items`    | The windowed result records       |
+| `total`    | Full match count before windowing |
+| `offset`   | Echo of requested offset (or 0)   |
+| `limit`    | Echo of effective limit (or 20)   |
+| `has_more` | `true` when `offset + items.length < total` |
+
+`has_more` is a convenience boolean so the caller
+does not need to compute whether more pages exist.
+The AI agent can page forward by calling `search`
+again with `offset` incremented by `limit`.
+
+Stats mode uses the same envelope with
+`items: []` and aggregate data in a separate
+`stats` field alongside the paging fields.
+
+## Result Modes
+
+Three result modes shape the ranked results
+differently.
+
+### Sections Mode
+
+Returns individual `SectionMatch` records, each with
+its own score:
+
+```ts
+interface SectionMatch {
+   kbid: string;
+   docids: string[];
+   type: string;
+   heading: string;
+   score: number;
+   snippet: string;
+   matched_terms: string[];
+}
+```
+
+`heading` is `string | null` -- `null` when the
+section has no title. `snippet` is always populated
+for text and code sections using KWIC
+(keyword-in-context): ~150 characters centered on
+the highest-scoring matched term position, with
+`...` ellipsis at boundaries. If the best match is
+in the heading (not body), the snippet is the first
+~150 characters of the body content. For image
+sections and sections with no body content,
+`snippet` is an empty string.
+
+Sections are ordered by `score` descending. Each
+section appears at most once.
+
+### Documents Mode
+
+Returns `DocumentMatch` records. Aggregates
+section-level scores into a document-level score:
+
+```ts
+interface DocumentMatch {
+   docid: string;
+   title: string;
+   score: number;
+   matching_sections: Array<{
+      kbid: string;
+      type: string;
+      heading: string;
+      score: number;
+      snippet: string;
+   }>;
+}
+```
+
+**Document score aggregation:** A document's score is
+the sum (or max, depending on config) of its matching
+sections' scores. Documents are ordered by their
+aggregated score descending. Within each document,
+matching sections are ordered by their individual
+scores descending.
+
+Sections not belonging to any document are returned
+as single-section documents.
+
+### Stats Mode
+
+Returns aggregate statistics without content bodies:
+
+```ts
+interface StatsSearchResult {
+   total_matches: number;
+   query_time_ms: number;
+   top_terms: TermStat[];
+   query: string;
+}
+
+interface TermStat {
+   term: string;
+   frequency: number;
+   sections_matched: number;
+}
+```
+
+No individual scores. `top_terms` are ordered by
+`frequency` descending.
+
+## MCP Search Projection
+
+When search results are returned via the MCP
+`search` tool, the ranking engine produces full
+`SectionMatch` records but the MCP layer projects
+a subset of fields optimized for AI agent
+consumption:
+
+```ts
+interface McpSearchResult {
+   kbid: string;
+   heading: string | null;
+   type: string;
+   docids: string[];
+   snippet: string;
+   matched_terms: string[];
+   content?: string;
+}
+```
+
+The `score` field from `SectionMatch` is
+intentionally omitted. Results are ordered by
+relevance descending; the position in the array
+is the ranking signal. Exposing uncalibrated BM25F
+scores provides no actionable information to the
+agent and risks confusing score magnitudes across
+different queries.
+
+`matched_terms` is a flat `string[]` of the query
+terms that matched -- no per-term frequency counts.
+The agent needs to know *which* terms matched, not
+*how often*.
+
+`heading` is `string | null`. When a section has
+no title, `heading` is `null` (not empty string).
+This lets the AI agent distinguish "no title" from
+"empty title."
+
+The `content` field is present only when the MCP
+search tool's `content` parameter is `true`. When
+present, it contains the full section text (or
+base64-encoded data for image sections).
+
+The CLI `search` command continues to expose
+`score` via the `SearchDisplay` type for human
+inspection and debugging.
+
+The MCP result uses the same `PagedSearchResult`
+envelope with `items`, `total`, `offset`, `limit`,
+and `has_more`.
+
+## References Resolution
+
+After ranking and truncation, the search flow
+resolves content marker references for the returned
+sections. Each result set includes a `references`
+array sourced from the bidirectional reference
+graph index (`references.idx`). Each reference
+entry includes:
+
+- `kbid` or `docid` of the referenced entity.
+- `marker_type` (Image, Code, Text, Document,
+  Link, or DocLink).
+- `context_snippet` (~150 chars of the surrounding
+  text where the marker appears).
+- `direction` (`forward` or `back`).
+
+Forward references come from markers in the
+returned sections' content. Back-references come
+from markers in *other* sections that point to
+the returned sections. Both directions are looked
+up from the reference graph index in O(1) rather
+than re-parsing content at query time. See
+[Database -- Reference Graph](database.md) for the
+index structure and [Content Parser](content-parser.md)
+for how markers are extracted at indexing time.
+
+## Index Structures Supporting Ranking
+
+The ranking pipeline consumes data from three index
+structures maintained by `fs-database.wasm`:
+
+- **Inverted index** (`indexes/terms.idx`) -- maps
+  normalized terms to posting lists of
+  `(kbid, heading_tf, body_tf, code_tf)` tuples.
+  Supports term lookup, IDF computation, and
+  per-field term frequency retrieval.
+- **Positional index** (`indexes/positions.idx`) --
+  maps `(term, kbid)` pairs to position arrays
+  (zero-based token offsets). Supports proximity
+  scoring, zone scoring, and phrase verification.
+- **Corpus statistics** (`indexes/corpus.dat`) --
+  stores `total_sections`, `total_terms`,
+  `avg_section_length`, `avg_heading_length`,
+  `avg_body_length`, `avg_code_length`, and per-term
+  document frequencies. Supports IDF and per-field
+  length normalization.
+
+See [Database](database.md) for full index
+specifications.
+
+## WASM Exports
+
+The query-parser WASM exports used for ranking:
+
+| Export                 | Role                     |
+|------------------------|--------------------------|
+| `qp_score_bm25f`      | BM25F primary scoring    |
+| `qp_score_proximity`  | Proximity boost          |
+| `qp_score_zone`       | Zone/structural boost    |
+| `qp_score_freshness`  | Freshness decay boost    |
+
+These are called by `kb-worker.wasm` after
+`fs-database.wasm` returns raw matches.
+
+## Keyword Extraction and Term Weighting
+
+Terms are weighted during indexing -- the other side
+of the ranking coin:
+
+- **RAKE** (Rapid Automatic Keyword Extraction) --
+  identifies candidate keyphrases by splitting on
+  stop words, scores by word degree / frequency
+  ratio.
+- **TF-IDF** -- per-section, per-field term frequency
+  combined with corpus-wide inverse document
+  frequency. Per-field TFs are stored in the inverted
+  index as separate values per posting entry.
+
+See [Query Parser](query-parser.md) for the full
+tokenization pipeline and extraction algorithms.