@dikolab/kbdb 0.1.6 → 0.2.1

package/docs/goals/overview.md

@@ -0,0 +1,124 @@
+# Goals Overview
+
+*Design goals and behavioral specifications for every
+major subsystem in @dikolab/kbdb.*
+
+## Purpose
+
+The goal documents describe the **intended behavior**
+of each subsystem -- what it should do, how it should
+respond, and what constraints it operates under. They
+serve as implementation specifications, not
+descriptions of current functionality.
+
+## AI-Agent-First Design Principle
+
+kbdb exists to provide useful context to AI agents.
+Every interface decision is evaluated by a single
+criterion: can an AI agent, seeing this response
+for the first time, make a relevance judgment and
+answer the user's question without additional
+round trips?
+
+This principle drives three concrete rules:
+
+1. **Search results are self-describing.** Every
+   result includes a snippet, heading, section type,
+   and source document identifiers -- enough for the
+   agent to judge relevance from the result list
+   alone, without fetching each section individually.
+
+2. **Progressive drill-down over batch fetching.**
+   The `recall` tool lets the agent expand context
+   incrementally -- first the section, then its
+   parent document, then siblings and references --
+   rather than forcing a choice between "fetch
+   nothing" and "fetch everything."
+
+3. **Rank order is the signal.** Internal scoring
+   details (BM25F coefficients, raw term
+   frequencies) are implementation concerns, not
+   agent-facing data. Results are ordered by
+   relevance; the agent reads them in order.
+
+## Architecture
+
+The system is organized in five layers. Data flows
+downward from user commands or agent requests through
+the worker client and daemon into the WASM engine,
+which performs heavy processing and manages the file
+database.
+
+![System Architecture](architecture.svg)
+
+## Goal Documents
+
+### WASM Engine
+
+- **[Database](database.md)** -- Filesystem storage
+  engine (`fs-database.wasm`). The `.kbdb` directory
+  structure, section and document CRUD operations,
+  index management, and storage design.
+- **[Query Parser](query-parser.md)** -- Query
+  processing and tokenization (`query-parser.wasm`).
+  Standalone WASM module for tokenization, query
+  parsing, keyword extraction, and ranking.
+- **[Document](document.md)** -- Text content format
+  specification. Markdown body text with embedding
+  markers and cross-reference link markers.
+- **[Content Parser](content-parser.md)** -- Marker
+  extraction engine (`kbdb-shared` rlib). Scans
+  section content for `$(type: value)` markers and
+  returns structured reference lists.
+- **[Content Composer](content-composer.md)** --
+  Content composition engine (`kbdb-shared` rlib).
+  Resolves sections and markers, produces rendered
+  Markdown documents from kbid and docid identifiers.
+- **[Query Result](query-result.md)** -- Ranking and
+  result ordering specification. BM25F scoring with
+  per-field term frequencies, proximity and zone
+  boosts, score combination, result modes, and
+  pre-computed indexing data.
+
+### TypeScript Layer
+
+- **[CLI](cli.md)** -- The `kbdb` command and its two
+  run modes. CLI mode commands (`learn`, `unlearn`,
+  `search`, `content`, `check`, `gc`, `rebuild`),
+  MCP mode dispatch, output formats, and the
+  pass-through execution model.
+- **[MCP](mcp.md)** -- MCP server mode (`kbdb mcp`).
+  Invocation, database path configuration, stdio
+  transport, MCP tools (search, recall, retrieve,
+  content, learn, unlearn, and more), resources,
+  agent wiring examples, and relationship to CLI
+  mode.
+- **[Worker Daemon](worker-daemon.md)** -- Cache
+  daemon (`kb-worker.wasm` + `src/worker.ts`).
+  The detached TypeScript daemon, WASM module loading,
+  caching, IPC protocol, lifecycle, and configuration.
+- **[Worker Client](worker-client.md)** -- Shared
+  TypeScript client module (`src/shared/worker-client/`).
+  Daemon discovery, spawning, IPC connection, typed
+  API, and the sole entry point used by `src/cli.ts`.
+
+## Implementation Guide
+
+The modules documentation describes **how** the goals
+above are implemented -- crate and module structure,
+file layout, build pipeline, and coding conventions
+for both the Rust engine and the TypeScript layer.
+
+- **[Modules Overview](../modules/overview.md)** --
+  Full build pipeline. How Rust compiles to WASM,
+  lands in the TypeScript source tree, and gets
+  bundled into `dist/`. Start here.
+- **[Rust/WASM Modules](../modules/rust/overview.md)**
+  -- Cargo workspace, 4 crates, module namespaces,
+  WASM exports/imports, size optimization, and
+  testing.
+- **[TypeScript Modules](../modules/typescript/overview.md)**
+  -- 2 entry points (`src/cli.ts`,
+  `src/worker.ts`), 8 shared sub-modules, file
+  conventions, dual-runtime constraints, and esbuild
+  bundling.

package/docs/goals/query-parser.md

@@ -0,0 +1,271 @@
+# Query Parser -- Query Processing and Tokenization
+
+*A shared WASM module that provides tokenization, query
+parsing, and term normalization for the knowledge base
+system.*
+
+## Goal
+
+The query parser module (`query-parser.wasm`) provides the core text
+processing capabilities shared across the knowledge base
+system. It handles:
+
+- Tokenizing input text into normalized terms.
+- Parsing search queries into structured query plans.
+- Normalizing and stemming terms for consistent indexing
+  and matching.
+- Maintaining a query-scoped cache for intermediate
+  results during query processing.
+
+The query parser is a pure computation module with no file I/O
+and no host dependencies beyond shared memory. It is
+imported by `kb-worker.wasm`, which orchestrates query
+processing and content indexing by combining the
+query parser's text analysis with `fs-database.wasm`'s
+storage operations.
+
+## WASM Module: query-parser.wasm
+
+### Dependency Position
+
+```
+query-parser.wasm         fs-database.wasm
+(standalone)        (standalone)
+      ↑                   ↑
+      └───────┬───────────┘
+              |
+       kb-worker.wasm
+  (imports both, orchestrates)
+```
+
+`query-parser.wasm` is a standalone module. It imports
+nothing from other WASM modules and depends only on
+host-provided memory. `kb-worker.wasm` imports its
+exports to drive tokenization, query parsing, and
+ranking.
+
+### Host Imports
+
+| Import   | Description                         |
+|----------|-------------------------------------|
+| `memory` | Shared `WebAssembly.Memory` for passing strings and buffers between modules |
+
+No file I/O, no network, no timers. The module is pure
+computation.
+
+### Exports
+
+Functions exported by `query-parser.wasm` for use by other
+WASM modules and the TypeScript host:
+
+| Export           | Description                        |
+|------------------|------------------------------------|
+| `tokenize`       | Tokenize input text into an array of normalized tokens |
+| `parse_query`    | Parse a query string into a structured query plan      |
+| `normalize_term` | Normalize a single term (case fold, stem) |
+| `stem`           | Stem a single term to its root form |
+| `alloc` / `free` | Memory management for shared buffer allocation         |
+
+All functions operate on the shared
+`WebAssembly.Memory`. String arguments and return values
+are passed as `(pointer, length)` pairs into the shared
+memory buffer.
+
+## Tokenization Pipeline
+
+The query parser provides a multi-stage tokenization
+pipeline used for both indexing (when content is added
+via `kbdb learn`) and query processing (when
+searching). Using the same pipeline for both ensures
+that indexed terms match query terms consistently.
+
+The pipeline processes **text and code content**.
+Image sections (`image/*` type) have their binary
+content skipped, but their `title` and `description`
+attributes are tokenized through this pipeline.
+
+Code sections (`text/*` and `application/json`) are
+processed through the same pipeline as text sections.
+The identifier splitting stage (stage 4) handles
+camelCase and snake_case decomposition common in
+source code.
+
+During indexing, the pipeline runs **per field** for
+each section: title content produces heading-field
+terms, body text and description produce body-field
+terms, and code content produces code-field terms.
+Per-field term frequencies are stored separately in
+the inverted index, enabling BM25F field weights to
+be tuned at query time without rebuilding indexes.
+
+Stages:
+
+1. **Marker stripping** -- Remove all six marker
+   types from the input text before tokenization:
+   `$(image:)`, `$(code:)`, `$(text:)`,
+   `$(document:)`, `$(link:)`, and `$(doc-link:)`.
+   These are structural references, not searchable
+   content. The `strip_content_markers()` function is
+   imported from `kbdb-shared` (see
+   [Content Parser](content-parser.md) and
+   [Document](document.md)).
+2. **Unicode normalization** (NFC).
+3. **Word boundary detection** (Unicode Segmentation).
+4. **Identifier splitting** (camelCase, snake_case
+   decomposition).
+5. **Case folding** (lowercase).
+6. **Stop word removal** (configurable list).
+7. **Stemming** (Snowball / Porter2 -- stores both
+   stemmed and original forms).
+8. **N-gram extraction** (bigrams, trigrams for phrase
+   indexing and multi-word term matching).
+
+`kb-worker.wasm` calls the tokenization pipeline for
+both indexing (when content is added via `kbdb learn`)
+and search (when a query is submitted). It feeds the
+tokenized output to `fs-database.wasm` for index
+storage or index lookups as needed.
+
+## Query Parsing
+
+Search queries are parsed into a structured query plan
+-- a data retrieval strategy that describes which terms
+to look up, how to combine them, and what to exclude.
+`kb-worker.wasm` feeds this plan to `fs-database.wasm`
+to execute index lookups.
+
+Parsing stages:
+
+1. **Tokenize** -- Run the query text through the
+   tokenization pipeline.
+2. **Parse syntax** -- Recognize query operators: exact
+   phrases (`"..."`), boolean AND/OR, and term exclusion
+   (`-term`).
+3. **Build query plan** -- Produce a structured plan
+   containing: a list of terms to match, phrase groups,
+   exclusions, and match mode.
+
+The query plan is the query parser's output --
+`kb-worker.wasm` receives it and passes it to
+`fs-database.wasm` for index lookups, then calls the
+query parser's ranking functions to score and order the raw
+results.
+
+## Query Cache
+
+During query processing, the query parser maintains a
+short-lived cache scoped to a single query's lifetime.
+This cache holds:
+
+- **Tokenized terms** -- Avoids re-tokenizing duplicate
+  terms within the same query.
+- **Stemmed forms** -- Avoids re-stemming the same root
+  across query terms.
+- **Parsed sub-expressions** -- Reuses parsed structures
+  for repeated patterns.
+
+The cache exists only for the duration of one query
+processing cycle. It is allocated when query processing
+begins and discarded immediately after the query plan is
+returned. It is not persisted, not shared across
+queries, and not visible to other modules.
+
+This prevents redundant computation when a query
+contains repeated terms or when the same term appears in
+different syntactic positions.
+
+## Keyword Extraction
+
+When text content is added to the knowledge base, the
+query parser extracts searchable terms from the raw
+text. `kb-worker.wasm` calls the query parser's
+tokenization exports and then applies the statistical
+extraction pipeline to produce terms and weights,
+which it passes to `fs-database.wasm` for index
+storage.
+
+Code sections (`text/*` and `application/json`) are
+processed through the keyword extraction pipeline.
+Their content is tokenized and weighted the same way
+as text sections -- the identifier splitting stage
+handles code-specific patterns (camelCase,
+snake_case).
+
+Image sections (`image/*` type) have their binary
+content skipped by the keyword extraction pipeline --
+base64-encoded data produces no searchable terms.
+However, their `title` (required) and `description`
+(optional) attributes are tokenized and indexed
+through the same per-field pipeline, making image
+sections searchable via their metadata. See
+[Query Result](query-result.md) for field weight
+details.
+
+### Statistical Keyword Extraction
+
+Extract candidate keywords and keyphrases using
+statistical methods that do not require training data or
+external models:
+
+- **RAKE** (Rapid Automatic Keyword Extraction) --
+  Splits on stop words, scores candidates by word
+  degree / frequency ratio.
+- **N-gram scoring** -- Bigrams and trigrams scored by
+  co-occurrence frequency.
+
+The extraction method is unsupervised and implemented
+in pure Rust without ML dependencies.
+
+### Term Weighting
+
+- Compute **per-field term frequency** (TF) -- term
+  frequencies are computed separately for each BM25F
+  field: heading (from title), body (from body text +
+  description), and code (from code content).
+- Compute corpus-wide **term rarity** (IDF -- how
+  uncommon a term is across all stored sections).
+- Combine frequency and rarity into per-field weights
+  that reflect each term's discriminative value for
+  search (TF-IDF).
+
+`kb-worker.wasm` passes the per-field term
+frequencies to `fs-database.wasm` for storage in the
+inverted index as separate `heading_tf`, `body_tf`,
+and `code_tf` values per posting entry. It also
+passes per-field token counts for storage in the
+section header. These pre-computed values are
+retrieved during search and fed to the query parser's
+ranking functions. See [Database](database.md) for
+the storage format.
+
+## Ranking
+
+The query parser provides ranking functions that
+`kb-worker.wasm` calls after retrieving raw index
+results from `fs-database.wasm` to order results by
+relevance. See [Query Result](query-result.md) for
+the full ranking specification including scoring
+formulas, field weight mapping, and pre-computed data.
+
+- **BM25F** (Field-Weighted Okapi BM25) -- Primary
+  ranking function. Probabilistic scoring with
+  per-field term frequencies and field weights
+  (heading 2.0x, body 1.0x, code 1.5x). Field
+  weights are applied at query time using the stored
+  per-field TFs, making them tunable without index
+  rebuilds.
+- **Proximity scoring** -- Supplementary signal. Boosts
+  sections where query terms appear close together.
+- **Zone scoring** -- Structural weighting. Terms in
+  the first ~20% of a section (heading zone) weighted
+  higher than mid-body text. Uses the pre-computed
+  `token_count` to determine the zone boundary.
+- **Freshness weighting** -- Optional boost based on
+  insertion timestamps.
+
+## Rust Crates
+
+| Crate                  | Role                          |
+|------------------------|-------------------------------|
+| `unicode-segmentation` | Word boundary detection       |
+| `rust-stemmers`        | Snowball / Porter2 stemmer    |