@dikolab/kbdb 0.5.0 → 0.6.1

package/docs/goals/overview.md

@@ -1,124 +0,0 @@
-# Goals Overview
-
-*Design goals and behavioral specifications for every
-major subsystem in @dikolab/kbdb.*
-
-## Purpose
-
-The goal documents describe the **intended behavior**
-of each subsystem -- what it should do, how it should
-respond, and what constraints it operates under. They
-serve as implementation specifications, not
-descriptions of current functionality.
-
-## AI-Agent-First Design Principle
-
-kbdb exists to provide useful context to AI agents.
-Every interface decision is evaluated by a single
-criterion: can an AI agent, seeing this response
-for the first time, make a relevance judgment and
-answer the user's question without additional
-round trips?
-
-This principle drives three concrete rules:
-
-1. **Search results are self-describing.** Every
-   result includes a snippet, heading, section type,
-   and source document identifiers -- enough for the
-   agent to judge relevance from the result list
-   alone, without fetching each section individually.
-
-2. **Progressive drill-down over batch fetching.**
-   The `recall` tool lets the agent expand context
-   incrementally -- first the section, then its
-   parent document, then siblings and references --
-   rather than forcing a choice between "fetch
-   nothing" and "fetch everything."
-
-3. **Rank order is the signal.** Internal scoring
-   details (BM25F coefficients, raw term
-   frequencies) are implementation concerns, not
-   agent-facing data. Results are ordered by
-   relevance; the agent reads them in order.
-
-## Architecture
-
-The system is organized in five layers. Data flows
-downward from user commands or agent requests through
-the worker client and daemon into the WASM engine,
-which performs heavy processing and manages the file
-database.
-
-![System Architecture](architecture.svg)
-
-## Goal Documents
-
-### WASM Engine
-
-- **[Database](database.md)** -- Filesystem storage
-  engine (`fs-database.wasm`). The `.kbdb` directory
-  structure, section and document CRUD operations,
-  index management, and storage design.
-- **[Query Parser](query-parser.md)** -- Query
-  processing and tokenization (`query-parser.wasm`).
-  Standalone WASM module for tokenization, query
-  parsing, keyword extraction, and ranking.
-- **[Document](document.md)** -- Text content format
-  specification. Markdown body text with embedding
-  markers and cross-reference link markers.
-- **[Content Parser](content-parser.md)** -- Marker
-  extraction engine (`kbdb-shared` rlib). Scans
-  section content for `$(type: value)` markers and
-  returns structured reference lists.
-- **[Content Composer](content-composer.md)** --
-  Content composition engine (`kbdb-shared` rlib).
-  Resolves sections and markers, produces rendered
-  Markdown documents from kbid and docid identifiers.
-- **[Query Result](query-result.md)** -- Ranking and
-  result ordering specification. BM25F scoring with
-  per-field term frequencies, proximity and zone
-  boosts, score combination, result modes, and
-  pre-computed indexing data.
-
-### TypeScript Layer
-
-- **[CLI](cli.md)** -- The `kbdb` command and its two
-  run modes. CLI mode commands (`learn`, `unlearn`,
-  `search`, `content`, `check`, `gc`, `rebuild`),
-  MCP mode dispatch, output formats, and the
-  pass-through execution model.
-- **[MCP](mcp.md)** -- MCP server mode (`kbdb mcp`).
-  Invocation, database path configuration, stdio
-  transport, MCP tools (search, recall, retrieve,
-  content, learn, unlearn, and more), resources,
-  agent wiring examples, and relationship to CLI
-  mode.
-- **[Worker Daemon](worker-daemon.md)** -- Cache
-  daemon (`kb-worker.wasm` + `src/worker.ts`).
-  The detached TypeScript daemon, WASM module loading,
-  caching, IPC protocol, lifecycle, and configuration.
-- **[Worker Client](worker-client.md)** -- Shared
-  TypeScript client module (`src/shared/worker-client/`).
-  Daemon discovery, spawning, IPC connection, typed
-  API, and the sole entry point used by `src/cli.ts`.
-
-## Implementation Guide
-
-The modules documentation describes **how** the goals
-above are implemented -- crate and module structure,
-file layout, build pipeline, and coding conventions
-for both the Rust engine and the TypeScript layer.
-
-- **[Modules Overview](../modules/overview.md)** --
-  Full build pipeline. How Rust compiles to WASM,
-  lands in the TypeScript source tree, and gets
-  bundled into `dist/`. Start here.
-- **[Rust/WASM Modules](../modules/rust/overview.md)**
-  -- Cargo workspace, 5 crates, module namespaces,
-  WASM exports/imports, size optimization, and
-  testing.
-- **[TypeScript Modules](../modules/typescript/overview.md)**
-  -- 2 entry points (`src/cli.ts`,
-  `src/worker.ts`), 8 shared sub-modules, file
-  conventions, dual-runtime constraints, and esbuild
-  bundling.

package/docs/goals/query-parser.md

@@ -1,373 +0,0 @@
-# Query Parser -- Query Processing and Tokenization
-
-*A shared WASM module that provides tokenization, query
-parsing, and term normalization for the knowledge base
-system.*
-
-## Goal
-
-The query parser module (`query-parser.wasm`) provides the core text
-processing capabilities shared across the knowledge base
-system. It handles:
-
-- Tokenizing input text into normalized terms.
-- Parsing search queries into structured query plans.
-- Normalizing and stemming terms for consistent indexing
-  and matching.
-- Maintaining a query-scoped cache for intermediate
-  results during query processing.
-
-The query parser is a pure computation module with no file I/O
-and no host dependencies beyond shared memory. It is
-imported by `kb-worker.wasm`, which orchestrates query
-processing and content indexing by combining the
-query parser's text analysis with `fs-database.wasm`'s
-storage operations.
-
-## WASM Module: query-parser.wasm
-
-### Dependency Position
-
-```
-query-parser.wasm         fs-database.wasm
-(standalone)        (standalone)
-      ↑                   ↑
-      └───────┬───────────┘
-              |
-       kb-worker.wasm
-  (imports both, orchestrates)
-```
-
-`query-parser.wasm` is a standalone module. It imports
-nothing from other WASM modules and depends only on
-host-provided memory. `kb-worker.wasm` imports its
-exports to drive tokenization, query parsing, and
-ranking.
-
-### Host Imports
-
-| Import   | Description                         |
-|----------|-------------------------------------|
-| `memory` | Shared `WebAssembly.Memory` for passing strings and buffers between modules |
-
-No file I/O, no network, no timers. The module is pure
-computation.
-
-### Exports
-
-Functions exported by `query-parser.wasm` for use by other
-WASM modules and the TypeScript host:
-
-| Export           | Description                        |
-|------------------|------------------------------------|
-| `tokenize`       | Tokenize input text into an array of normalized tokens |
-| `parse_query`    | Parse a query string into a structured query plan      |
-| `normalize_term` | Normalize a single term (case fold, stem) |
-| `stem`           | Stem a single term to its root form |
-| `alloc` / `free` | Memory management for shared buffer allocation         |
-
-All functions operate on the shared
-`WebAssembly.Memory`. String arguments and return values
-are passed as `(pointer, length)` pairs into the shared
-memory buffer.
-
-## Tokenization Pipeline
-
-The query parser provides a multi-stage tokenization
-pipeline used for both indexing (when content is added
-via `kbdb learn`) and query processing (when
-searching). Using the same pipeline for both ensures
-that indexed terms match query terms consistently.
-
-The pipeline processes **text and code content**.
-Image sections (`image/*` type) have their binary
-content skipped, but their `title` and `description`
-attributes are tokenized through this pipeline.
-
-Code sections (`text/*` and `application/json`) are
-processed through the same pipeline as text sections.
-The identifier splitting stage (stage 4) handles
-camelCase and snake_case decomposition common in
-source code.
-
-During indexing, the pipeline runs **per field** for
-each section: title content produces heading-field
-terms, body text and description produce body-field
-terms, and code content produces code-field terms.
-Per-field term frequencies are stored separately in
-the inverted index, enabling BM25F field weights to
-be tuned at query time without rebuilding indexes.
-
-Stages:
-
-1. **Marker stripping** -- Remove all six marker
-   types from the input text before tokenization:
-   `$(image:)`, `$(code:)`, `$(text:)`,
-   `$(document:)`, `$(link:)`, and `$(doc-link:)`.
-   These are structural references, not searchable
-   content. The `strip_content_markers()` function is
-   imported from `kbdb-shared` (see
-   [Content Parser](content-parser.md) and
-   [Document](document.md)).
-2. **Unicode normalization** (NFC).
-3. **Word boundary detection** (Unicode Segmentation).
-4. **Identifier splitting** (camelCase, snake_case
-   decomposition).
-5. **Case folding** (lowercase).
-6. **Stop word removal** (configurable list).
-7. **Synonym expansion** (bidirectional synonym map --
-   inserts synonym tokens alongside originals).
-8. **Stemming** (Snowball / Porter2 -- stores both
-   stemmed and original forms).
-9. **N-gram extraction** (bigrams, trigrams for phrase
-   indexing and multi-word term matching).
-
-`kb-worker.wasm` calls the tokenization pipeline for
-both indexing (when content is added via `kbdb learn`)
-and search (when a query is submitted). It feeds the
-tokenized output to `fs-database.wasm` for index
-storage or index lookups as needed.
-
-### Synonym Expansion (Stage 7)
-
-Bridges the lexical gap between user vocabulary and
-indexed content. Without synonym expansion, a search
-for `"login credentials"` returns zero results when
-the indexed document uses "signs in" and "identity"
-instead -- different root words with the same meaning.
-
-**Placement rationale:** Applied after stop-word
-removal (stage 6) and before stemming (stage 8). This
-ensures synonyms are expanded on lowercased,
-stop-word-filtered tokens, and both the original and
-synonym tokens are then stemmed consistently.
-
-**Built-in synonym map:** A small, curated
-`HashMap<String, Vec<String>>` of bidirectional
-synonym pairs. Bidirectional means if `login` maps to
-`sign-in`, then `sign-in` also maps to `login`.
-
-**Example map entries:**
-
-| Term         | Synonyms                           |
-|--------------|------------------------------------|
-| login        | sign-in, sign_in, signin           |
-| auth         | authentication, authenticate       |
-| remove       | delete, erase                      |
-| create       | add, insert                        |
-| update       | modify, change, edit               |
-| error        | exception, fault, failure          |
-| config       | configuration, settings            |
-| credential   | password, identity                 |
-| user         | account, member                    |
-| deploy       | release, publish, ship             |
-| search       | query, find, lookup                |
-| doc          | document, documentation            |
-| api          | endpoint, interface                |
-| db           | database                           |
-| env          | environment                        |
-| repo         | repository                         |
-| param        | parameter, argument                |
-| dir          | directory, folder                  |
-| msg          | message                            |
-| info         | information                        |
-
-**Expansion behaviour:** For each token in the stream,
-look up the synonym map. If found, insert the synonym
-tokens at the same position as the original. Both the
-original and its synonyms proceed to the stemming
-stage. This preserves the original token's position
-for proximity/zone scoring while adding synonym
-coverage.
-
-**Applied symmetrically:** The same synonym expansion
-applies during both `learn` (indexing) and `search`
-(querying). If a document uses "login" and the user
-searches "sign-in", the expansion during indexing
-produced "sign-in" tokens from "login", and the
-expansion during querying produced "login" tokens
-from "sign-in".
-
-**Not user-configurable in v1.** The built-in map is
-hardcoded. Future versions may allow user-defined
-synonym maps via a `synonyms.toml` configuration file
-in the `.kbdb` directory.
-
-**Implementation status:** Synonym expansion is
-implemented in the Rust pipeline
-(`shared/src/pipeline/synonyms.rs`). The
-built-in map covers common technical synonyms.
-The `query-guidance` default skill complements this
-by instructing AI agents to rephrase queries with
-alternative terms before searching.
-
-## Query Parsing
-
-Search queries are parsed into a structured query plan
--- a data retrieval strategy that describes which terms
-to look up, how to combine them, and what to exclude.
-`kb-worker.wasm` feeds this plan to `fs-database.wasm`
-to execute index lookups.
-
-Parsing stages:
-
-1. **Tokenize** -- Run the query text through the
-   tokenization pipeline.
-2. **Parse syntax** -- Recognize query operators: exact
-   phrases (`"..."`), boolean AND/OR, and term exclusion
-   (`-term`).
-3. **Build query plan** -- Produce a structured plan
-   containing: a list of terms to match, phrase groups,
-   exclusions, and match mode.
-
-The query plan is the query parser's output --
-`kb-worker.wasm` receives it and passes it to
-`fs-database.wasm` for index lookups, then calls the
-query parser's ranking functions to score and order
-the raw results.
-
-### Relaxed Query Mode
-
-When a multi-term AND query returns zero results,
-`kb-worker.wasm` retries the query in **relaxed
-mode** -- OR semantics where any term matches, not
-all required.
-
-The query parser generates both the primary (AND)
-plan and a relaxed (OR) plan from the same parsed
-query. `kb-worker.wasm` executes the AND plan first.
-If the AND plan produces zero matches, it falls back
-to the OR plan. No new WASM export is needed --
-`qp_parse_query` returns a `QueryPlan` that supports
-mode switching via a `match_mode` field
-(`All` | `Any`).
-
-Relaxed results are flagged with `relaxed: true` in
-the `PagedSearchResult` envelope (see
-[Query Result -- Relaxed-Match
-Fallback](query-result.md)). The fallback is
-transparent to the caller.
-
-Single-term queries are unaffected (AND and OR are
-equivalent for one term). Phrase queries and
-exclusions are preserved in both modes -- only the
-inter-term combination changes.
-
-## Query Cache
-
-During query processing, the query parser maintains a
-short-lived cache scoped to a single query's lifetime.
-This cache holds:
-
-- **Tokenized terms** -- Avoids re-tokenizing duplicate
-  terms within the same query.
-- **Stemmed forms** -- Avoids re-stemming the same root
-  across query terms.
-- **Parsed sub-expressions** -- Reuses parsed structures
-  for repeated patterns.
-
-The cache exists only for the duration of one query
-processing cycle. It is allocated when query processing
-begins and discarded immediately after the query plan is
-returned. It is not persisted, not shared across
-queries, and not visible to other modules.
-
-This prevents redundant computation when a query
-contains repeated terms or when the same term appears in
-different syntactic positions.
-
-## Keyword Extraction
-
-When text content is added to the knowledge base, the
-query parser extracts searchable terms from the raw
-text. `kb-worker.wasm` calls the query parser's
-tokenization exports and then applies the statistical
-extraction pipeline to produce terms and weights,
-which it passes to `fs-database.wasm` for index
-storage.
-
-Code sections (`text/*` and `application/json`) are
-processed through the keyword extraction pipeline.
-Their content is tokenized and weighted the same way
-as text sections -- the identifier splitting stage
-handles code-specific patterns (camelCase,
-snake_case).
-
-Image sections (`image/*` type) have their binary
-content skipped by the keyword extraction pipeline --
-base64-encoded data produces no searchable terms.
-However, their `title` (required) and `description`
-(optional) attributes are tokenized and indexed
-through the same per-field pipeline, making image
-sections searchable via their metadata. See
-[Query Result](query-result.md) for field weight
-details.
-
-### Statistical Keyword Extraction
-
-Extract candidate keywords and keyphrases using
-statistical methods that do not require training data or
-external models:
-
-- **RAKE** (Rapid Automatic Keyword Extraction) --
-  Splits on stop words, scores candidates by word
-  degree / frequency ratio.
-- **N-gram scoring** -- Bigrams and trigrams scored by
-  co-occurrence frequency.
-
-The extraction method is unsupervised and implemented
-in pure Rust without ML dependencies.
-
-### Term Weighting
-
-- Compute **per-field term frequency** (TF) -- term
-  frequencies are computed separately for each BM25F
-  field: heading (from title), body (from body text +
-  description), and code (from code content).
-- Compute corpus-wide **term rarity** (IDF -- how
-  uncommon a term is across all stored sections).
-- Combine frequency and rarity into per-field weights
-  that reflect each term's discriminative value for
-  search (TF-IDF).
-
-`kb-worker.wasm` passes the per-field term
-frequencies to `fs-database.wasm` for storage in the
-inverted index as separate `heading_tf`, `body_tf`,
-and `code_tf` values per posting entry. It also
-passes per-field token counts for storage in the
-section header. These pre-computed values are
-retrieved during search and fed to the query parser's
-ranking functions. See [Database](database.md) for
-the storage format.
-
-## Ranking
-
-The query parser provides ranking functions that
-`kb-worker.wasm` calls after retrieving raw index
-results from `fs-database.wasm` to order results by
-relevance. See [Query Result](query-result.md) for
-the full ranking specification including scoring
-formulas, field weight mapping, and pre-computed data.
-
-- **BM25F** (Field-Weighted Okapi BM25) -- Primary
-  ranking function. Probabilistic scoring with
-  per-field term frequencies and field weights
-  (heading 2.0x, body 1.0x, code 1.5x). Field
-  weights are applied at query time using the stored
-  per-field TFs, making them tunable without index
-  rebuilds.
-- **Proximity scoring** -- Supplementary signal. Boosts
-  sections where query terms appear close together.
-- **Zone scoring** -- Structural weighting. Terms in
-  the first ~20% of a section (heading zone) weighted
-  higher than mid-body text. Uses the pre-computed
-  `token_count` to determine the zone boundary.
-- **Freshness weighting** -- Optional boost based on
-  insertion timestamps.
-
-## Rust Crates
-
-| Crate                  | Role                          |
-|------------------------|-------------------------------|
-| `unicode-segmentation` | Word boundary detection       |
-| `rust-stemmers`        | Snowball / Porter2 stemmer    |