@dikolab/kbdb 0.2.2 → 0.3.1

package/docs/goals/hybrid-search.md

@@ -0,0 +1,436 @@
+# Hybrid Search -- Lexical + Vector Retrieval
+
+*Optional vector retrieval path alongside BM25F, fused
+with RRF for best-of-both ranking.*
+
+## Goal
+
+The current retrieval mode is lexical search: BM25F with
+stemming and synonym expansion. It works well for
+structured documentation where terminology is consistent
+-- the agent searches "authentication" and the indexed
+content uses the same word.
+
+Lexical search has a ceiling, however. It misses
+associative recall when terminology diverges between the
+time content was stored and the time it is retrieved.
+Example: an agent stores "we chose Postgres because of
+JSONB" and later searches "database rationale" -- lexical
+search misses it because none of the indexed tokens
+(`postgres`, `chose`, `jsonb`) overlap with the query
+tokens (`database`, `rationale`). Vector search finds
+it because the embedding of the query is geometrically
+close to the embedding of the stored sentence.
+
+This gap is identified as the critical lexical-only
+ceiling in
+[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md)
+(gap #1). Hybrid search closes it by adding an optional
+vector retrieval path alongside the existing lexical
+path, fusing both ranked lists using Reciprocal Rank
+Fusion (RRF) for a final result set that benefits from
+both modes.
+
+## Architecture
+
+Two retrieval paths run in parallel. The vector path
+runs entirely in TypeScript. No WASM code is added or
+modified.
+
+```
+Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
+        │                                              ├─→ RRF Fusion → Final results
+        └─ Vector path (TS)   ─→ Cosine ranked list ──┘
+```
+
+The WASM boundary is untouched. `query-parser.wasm`
+remains a pure-computation lexical module. `kb-worker.wasm`
+continues to orchestrate lexical retrieval via
+`fs-database.wasm` exactly as today. No new WASM crate,
+no new WASM export, and no new host import is needed for
+the vector path.
+
+The TypeScript daemon layer adds the vector path as an
+independent retrieval step that runs after the lexical
+path completes (or concurrently if the runtime allows).
+The fused list is assembled in TypeScript before the
+response is returned to the worker client.
+
+## EmbeddingProvider Interface
+
+### Why TypeScript, Not WASM
+
+Three reasons embeddings are computed in TypeScript
+rather than in WASM:
+
+1. `wasm32-unknown-unknown` has no network I/O. Calling
+   an embedding API from WASM is impossible without a
+   host bridge that would defeat the purpose of running
+   in WASM.
+2. Embedding models are too large for WASM. A typical
+   all-MiniLM-L6-v2 model is ~90 MB. Loading that inside
+   the WASM sandbox on every daemon startup is
+   impractical.
+3. TypeScript allows pluggable backends. An interface-based
+   design lets the daemon swap providers (local ONNX,
+   remote API) without changing any WASM code.
+
+### Interface Specification
+
+```typescript
+interface EmbeddingProvider {
+   embed(texts: string[]): Promise<Float32Array[]>;
+   dimensions(): number;
+   modelName(): string;
+}
+```
+
+| Member       | Description                                  |
+|--------------|----------------------------------------------|
+| `embed`      | Compute embeddings for a batch of input texts. Returns one `Float32Array` per input, each of length `dimensions()`. |
+| `dimensions` | Fixed embedding dimensionality for this provider. All vectors stored under the same model must have this length. |
+| `modelName`  | Stable identifier for the model. Used for stale-detection when the configured model changes. |
+
+### Implementations
+
+Three concrete implementations are specified.
+
+**`NullEmbeddingProvider`** -- no-op, returns empty
+arrays. The default when no `[embedding]` section is
+present in `worker.toml`. Requesting `vector` or
+`hybrid` search mode with this provider returns a
+structured error rather than silently returning empty
+results. Satisfies POLA: the absence of configuration
+does not silently degrade results.
+
+**`OnnxEmbeddingProvider`** -- local offline inference
+using ONNX Runtime. Default model: `all-MiniLM-L6-v2`
+(384 dimensions). The model file is downloaded on first
+use and cached in the OS user-data directory. Works on
+both Deno and Node.js >= 20. No network dependency
+after the initial model download. Suitable for
+air-gapped or cost-sensitive deployments.
+
+**`ApiEmbeddingProvider`** -- calls an external HTTP
+embedding API. Compatible with OpenAI, Anthropic, and
+any endpoint that accepts a `POST /embeddings` request
+with a JSON body `{ "input": string[] }` and returns
+`{ "data": [{ "embedding": number[] }] }`. Configured
+via `worker.toml`. The API key is read from an
+environment variable named by `api_key_env`.
+
+## Vector Storage
+
+### New Index Files
+
+Two new files are added to `indexes/` alongside the
+existing index files. They follow the same rebuild
+semantics as `terms.idx`, `positions.idx`, and
+`corpus.dat`: derived data, rebuildable from section
+content, excluded from version control.
+
+**`vectors.idx`** -- binary, fixed-width records.
+
+Header layout:
+
+| Bytes | Field          | Type   | Description                   |
+|-------|----------------|--------|-------------------------------|
+| 0-3   | magic          | u8[4]  | `b"KBVX"` -- format sentinel  |
+| 4     | version        | u8     | Format version, currently `1` |
+| 5-6   | dimensions     | u16    | Vector length per record       |
+| 7-10  | record_count   | u32    | Total number of records        |
+
+Each record immediately follows the header:
+
+| Bytes          | Field          | Type           | Description       |
+|----------------|----------------|----------------|-------------------|
+| 0-15           | kbid_hash      | u8[16]         | First 16 bytes of the section kbid, used as a compact lookup key |
+| 16 onwards     | embedding      | f32[dimensions]| Raw embedding vector |
+
+Record size = `16 + dimensions * 4` bytes. All records
+are the same size, enabling O(1) offset computation by
+record index and binary search by kbid hash.
+
+**`vectors-meta.toml`** -- human-readable metadata.
+
+```toml
+[vectors]
+model_name  = "all-MiniLM-L6-v2"
+dimensions  = 384
+version     = 1
+last_built  = "2026-06-23T10:00:00Z"
+section_count = 2847
+```
+
+| Field           | Description                               |
+|-----------------|-------------------------------------------|
+| `model_name`    | Model name from `EmbeddingProvider.modelName()`. Used for stale detection. |
+| `dimensions`    | Must match the `dimensions` header in `vectors.idx`. |
+| `version`       | `vectors.idx` format version.             |
+| `last_built`    | ISO 8601 timestamp of the last rebuild.   |
+| `section_count` | Number of sections embedded.              |
+
+### Updated Directory Layout
+
+```
+.kbdb/
+└── indexes/
+    ├── meta.toml
+    ├── terms.idx
+    ├── positions.idx
+    ├── corpus.dat
+    ├── references.idx
+    ├── vectors.idx          # new -- binary embedding store
+    └── vectors-meta.toml   # new -- embedding metadata
+```
+
+## Embedding Lifecycle
+
+### Learn Time
+
+When a section is added via `kbdb learn`, the lexical
+indexing pipeline runs first (unchanged). After lexical
+indexing completes, the daemon asynchronously computes
+an embedding for the section using the configured
+`EmbeddingProvider`.
+
+The embedding input is the concatenation of the
+section's title and body content:
+
+```
+{title}\n{content}
+```
+
+The title is prepended because it carries strong
+semantic weight (it receives a 2.0x BM25F field weight
+for the same reason). Concatenating both fields gives
+the embedding model the same semantic signal that BM25F
+heading-weight gives to lexical scoring.
+
+Sections with no body content (binary sections where
+content is base64) embed only the title and
+description. Sections with neither title nor body
+content are not embedded.
+
+Embedding is best-effort at learn time: if the provider
+fails for a section, the section is still stored
+lexically. A warning is emitted in the learn response
+(`vector_embedding_failed: true`).
+
+### Rebuild
+
+`kbdb rebuild` regenerates all vector embeddings from
+section content alongside the existing lexical index
+rebuild. Sections are embedded in batches of up to 64
+to respect provider rate limits. The `vectors.idx` and
+`vectors-meta.toml` files are written atomically after
+all embeddings are computed.
+
+### Stale Detection
+
+When the daemon starts, it reads `vectors-meta.toml`
+and compares `model_name` to the currently configured
+provider's `modelName()`. If they differ, the vector
+index is marked stale. Stale vector indexes are not
+used for retrieval. Search falls back to lexical-only
+with a warning field in the response:
+`vector_index_stale: true`.
+
+The agent or operator must run `kbdb rebuild` to
+regenerate vectors with the new model before hybrid or
+vector search resumes.
+
+## Score Fusion
+
+### Reciprocal Rank Fusion (RRF)
+
+RRF combines the lexical ranked list and the vector
+ranked list into a single fused list without requiring
+score normalization. BM25F scores and cosine similarity
+scores live on different scales and cannot be directly
+added -- RRF sidesteps this by working on ranks rather
+than scores.
+
+Formula:
+
+```
+score(d) = 1/(k + rank_lexical(d)) + 1/(k + rank_vector(d))
+```
+
+Where:
+
+- `k = 60` (standard default, configurable via
+  `worker.toml` as `rrf_k`)
+- `rank_lexical(d)` is the 1-based rank of document `d`
+  in the BM25F result list, or infinity if `d` does not
+  appear in the lexical results
+- `rank_vector(d)` is the 1-based rank of document `d`
+  in the cosine similarity result list, or infinity if
+  `d` does not appear in the vector results
+- A rank of infinity contributes 0 to the sum (since
+  `1/(k + ∞) = 0`)
+
+Documents appearing in only one result set receive full
+contribution from that set and zero from the other.
+Documents appearing in both receive a higher combined
+score than either path could produce alone, naturally
+surfacing cross-path agreement.
+
+The final result list is sorted by RRF score descending.
+
+### Why RRF
+
+- Parameter-light: `k = 60` is the well-established
+  default from the original RRF paper (Cormack et al.,
+  2009). It rarely needs tuning.
+- No normalization: BM25F scores and cosine similarity
+  are not comparable in magnitude. RRF requires only
+  rank order, not the scores themselves.
+- Robust: adding a low-quality second path cannot hurt
+  the top results badly because RRF dampens rank
+  contributions as `k` is approached.
+
+## Search Mode Parameter
+
+A new `search_mode` parameter controls which retrieval
+paths are active for a given search request.
+
+| Value      | Retrieval behaviour                         | Embedding provider required |
+|------------|---------------------------------------------|-----------------------------|
+| `lexical`  | Lexical path only. Current behaviour. Default. | No                        |
+| `vector`   | Vector path only. Cosine similarity ranking.   | Yes                       |
+| `hybrid`   | Both paths, RRF fusion.                        | Yes                        |
+
+The default is `lexical`. Existing integrations that do
+not supply `search_mode` are unchanged. This is a
+strict backward-compatibility guarantee.
+
+Requesting `vector` or `hybrid` when no embedding
+provider is configured (i.e. `NullEmbeddingProvider` is
+active) returns a structured error:
+
+```json
+{
+  "error": "Vector search requires a configured embedding provider",
+  "code": "EMBEDDING_PROVIDER_NOT_CONFIGURED"
+}
+```
+
+## CLI and MCP Surface
+
+### CLI
+
+The `search` subcommand gains an optional `--mode` flag:
+
+```
+kbdb search "database rationale"
+kbdb search --mode hybrid "database rationale"
+kbdb search --mode vector "database rationale"
+```
+
+Omitting `--mode` defaults to `lexical`. The flag is
+not shown in the short help unless an embedding provider
+is configured.
+
+### MCP
+
+The `search` MCP tool gains an optional `mode` parameter
+(string enum: `"lexical" | "vector" | "hybrid"`). The
+parameter is documented as optional in the tool schema
+with a default of `"lexical"`.
+
+### Response Envelope
+
+`PagedSearchResult` gains two new fields:
+
+| Field              | Type   | Description                     |
+|--------------------|--------|---------------------------------|
+| `search_mode`      | string | The mode that was requested: `"lexical"`, `"vector"`, or `"hybrid"` |
+| `retrieval_method` | string | The mode that was actually used: `"lexical"`, `"vector"`, or `"hybrid"`. Differs from `search_mode` when a fallback occurred (e.g. stale vector index forced `"lexical"`). |
+
+Both fields are always present in the response,
+including when `search_mode` was not supplied (in which
+case both fields are `"lexical"`).
+
+## Configuration
+
+`worker.toml` gains an `[embedding]` section:
+
+```toml
+[embedding]
+provider    = "onnx"           # "onnx" | "api" | "none"
+model       = "all-MiniLM-L6-v2"
+dimensions  = 384
+rrf_k       = 60               # RRF parameter k
+
+# API provider only:
+api_url     = ""
+api_key_env = "KBDB_API_KEY"
+```
+
+| Key           | Default              | Description                  |
+|---------------|----------------------|------------------------------|
+| `provider`    | `"none"`             | Which `EmbeddingProvider` implementation to use |
+| `model`       | `"all-MiniLM-L6-v2"` | Model name, passed to the provider |
+| `dimensions`  | `384`                | Embedding vector length      |
+| `rrf_k`       | `60`                 | RRF constant `k`             |
+| `api_url`     | `""`                 | Base URL for API provider    |
+| `api_key_env` | `"KBDB_API_KEY"`     | Environment variable holding the API key |
+
+When `[embedding]` is absent from `worker.toml`,
+`provider` defaults to `"none"` and `NullEmbeddingProvider`
+is used.
+
+## Fallback Behaviour
+
+Three fallback conditions are defined. Each is surfaced
+in the search response with a specific warning field
+rather than silently degrading.
+
+**No embedding provider configured.**
+`search_mode` defaults to `lexical`. Requesting `hybrid`
+or `vector` returns a structured error with code
+`EMBEDDING_PROVIDER_NOT_CONFIGURED`. No silent fallback
+occurs -- the caller receives an actionable error
+message.
+
+**Stale vector index (model mismatch).**
+When `vectors-meta.toml` model name does not match the
+configured provider's `modelName()`, the daemon marks
+the vector index stale. Any search request that would
+use the vector path falls back to `lexical`. The
+response includes `retrieval_method: "lexical"` and
+`vector_index_stale: true`. The correct remediation is
+`kbdb rebuild`.
+
+**Embedding failure during learn.**
+If the embedding provider returns an error for a section
+during learn, the section is stored lexically without an
+embedding. The learn response includes
+`vector_embedding_failed: true`. The section will
+appear in lexical results but not in vector or hybrid
+results until a rebuild is run.
+
+## Provider Tradeoffs
+
+| Aspect        | ONNX (local)              | API (cloud)                  |
+|---------------|---------------------------|------------------------------|
+| Latency       | ~10 ms/embed              | ~100 ms/embed                |
+| Offline       | Yes                       | No                           |
+| Cost          | Free                      | Per-token                    |
+| Quality       | Good (384 dimensions)     | Excellent (1536+ dimensions) |
+| Setup         | Model download on first use | API key required            |
+| Dual runtime  | Deno + Node.js >= 20      | Deno + Node.js >= 20         |
+
+Both providers implement the same `EmbeddingProvider`
+interface. The daemon interacts only with the interface.
+
+## Design Constraints
+
+| Constraint            | Details                                        |
+|-----------------------|------------------------------------------------|
+| WASM purity           | `query-parser.wasm` remains pure lexical computation with no I/O. No new WASM crate or export is added. |
+| Backward compatibility | Default `search_mode` is `lexical`. Callers that do not supply the parameter receive identical behaviour to today. |
+| File-based storage    | `vectors.idx` and `vectors-meta.toml` follow the same derived-and-rebuildable pattern as all other index files. |
+| Dual runtime          | `EmbeddingProvider` and all embedding logic must work on Deno and Node.js >= 20. No Node-specific or Deno-specific APIs in the implementation. |
+| No new WASM deps      | All new dependencies are TypeScript-side only. The WASM modules (`fs-database.wasm`, `query-parser.wasm`, `kb-worker.wasm`) are not recompiled for this feature. |

package/docs/goals/mcp.md

@@ -50,6 +50,17 @@ the user.
 See [CLI -- Run Modes](cli.md#run-modes) for how
 the `kbdb` executable dispatches to MCP mode.
 
+## Startup Health Check
+
+When the MCP server starts, it runs
+`client.integrityCheck()` automatically after the
+worker daemon connection is established. If the
+check returns `ok: false`, the server emits a
+`notifications/message` log with level `warning`
+listing the error summary. The server continues to
+operate -- health issues are not fatal. This ensures
+corruption surfaces before it produces wrong answers.
+
 ## Installation and Wiring
 
 Installation is the same as the CLI:
@@ -332,6 +343,7 @@ daemon via the worker client.
 
 Search the knowledge base. Returns ranked results
 with headings and snippets for relevance evaluation.
+Use multiple short keyword queries for best coverage.
 Use `recall` to get full content of specific results.
 
 **Parameters:**
@@ -358,13 +370,16 @@ additional calls:
          "type": "text/markdown",
          "docids": ["doc-5678"],
          "snippet": "The system is organized...",
-         "matched_terms": ["architecture", "system"]
+         "matched_terms": ["architecture", "system"],
+         "matched_fields": ["heading", "body"],
+         "created_at": "2026-06-20T14:30:00Z"
       }
    ],
    "total": 42,
    "offset": 0,
    "limit": 20,
-   "has_more": true
+   "has_more": true,
+   "relaxed": false
 }
 ```
 
@@ -379,6 +394,15 @@ additional calls:
   matched term. Empty string for image sections.
 - `matched_terms` -- query terms that matched, as a
   flat `string[]`. No frequency counts.
+- `matched_fields` -- subset of `["heading", "body",
+  "code"]` indicating which fields contained matches.
+- `created_at` -- ISO 8601 timestamp of when the
+  section was first indexed.
+- `relaxed` -- `true` when the result came from an
+  automatic AND-to-OR fallback (see
+  [Query Result -- Relaxed-Match
+  Fallback](query-result.md)). `false` for normal
+  results.
 
 When `content` is `true`, each item also includes
 a `content` field with the full section text.
@@ -393,20 +417,24 @@ Maps to `client.search()`.
 
 ### learn
 
-Import content into the knowledge base. The daemon
-decomposes the content into sections, extracts
-keywords, builds indexes, and assigns ranking
-weights.
+Import content into the knowledge base. Use for
+durable facts, decisions, and corrections. Use
+`--replace` when updating existing knowledge. Attach
+tags for scoping. The daemon decomposes the content
+into sections, extracts keywords, builds indexes,
+and assigns ranking weights.
 
 **Parameters:**
 
 | Name          | Type   | Required | Description      |
 |---------------|--------|----------|------------------|
-| `content`     | string | yes      | Text content or base64-encoded image data |
-| `type`        | string | no       | `"text/markdown"` (default), `text/*` MIME type, `"application/json"`, or `image/*` MIME type |
-| `title`       | string | no*      | Section title. ***Required** for `image/*` types.* Optional for text and code types. |
-| `description` | string | no       | Optional section description |
-| `docid`       | string | no       | Group under this document ID |
+| `content`     | string   | yes      | Text content or base64-encoded image data |
+| `type`        | string   | no       | `"text/markdown"` (default), `text/*` MIME type, `"application/json"`, or `image/*` MIME type |
+| `title`       | string   | no*      | Section title. ***Required** for `image/*` types.* Optional for text and code types. |
+| `description` | string   | no       | Optional section description |
+| `docid`       | string   | no       | Group under this document ID |
+| `tags`        | string[] | no       | Tags for scoping and filtering |
+| `replace`     | boolean  | no       | Enable supersession by source key. When `true` and a section with the same source key exists, the old section is replaced. |
 
 **Content type rules:**
 
@@ -441,6 +469,16 @@ Text and code sections return a positive
 `indexed_terms_count`. Image sections return
 `indexed_terms_count: 0`.
 
+Additional result fields when applicable:
+- `superseded` -- kbid of the section this one
+  replaced via `replace: true`, or `null`.
+- `warnings` -- string array of advisory warnings
+  (e.g. `"heading_missing"` when a text section has
+  no title).
+- `near_duplicates` -- array of
+  `{ kbid, similarity }` objects for existing
+  sections with Jaccard similarity >= 0.85.
+
 If the content already exists in the database (same
 kb-id) and `docid` is provided, the section is not
 duplicated. Instead, the docid is added to the
@@ -662,8 +700,12 @@ checks pass, `ok` is `true` and `errors` is empty.
 
 Error types: `checksum_mismatch`, `orphan_section`,
 `orphan_document`, `broken_reference`,
-`type_mismatch`, `circular_reference`. See
-[Database -- Integrity and
+`type_mismatch`, `circular_reference`,
+`divergent_content`. The `divergent_content` error
+flags sections sharing the same `source_key` that
+have different content -- indicates contradictory
+versions that should be resolved via supersession.
+See [Database -- Integrity and
 Maintenance](database.md).
 
 Maps to `client.integrityCheck()`.
@@ -696,6 +738,32 @@ this to recover from index corruption.
 
 Maps to `client.rebuildIndexes()`.
 
+### export
+
+Export the knowledge base as a self-contained
+directory snapshot for backup or versioning. Copies
+sections, documents, and catalog (no indexes -- they
+are rebuildable via `rebuild`).
+
+**Parameters:**
+
+| Name   | Type   | Required | Description               |
+|--------|--------|----------|---------------------------|
+| `path` | string | no       | Export destination path (default: `./kbdb-export/`) |
+
+**Returns:** Compact JSON with export summary:
+
+```json
+{
+   "path": "/home/user/kbdb-export",
+   "sections_exported": 142,
+   "documents_exported": 23,
+   "elapsed_ms": 350
+}
+```
+
+Maps to `client.export()`.
+
 ### skill-learn
 
 Create a new skill definition and store it in the
@@ -1057,7 +1125,7 @@ interface McpToolAnnotations {
 }
 ```
 
-Annotations for all 18 kbdb tools:
+Annotations for all 19 kbdb tools:
 
 | Tool           | title             | readOnly | destructive | idempotent | openWorld |
 |----------------|-------------------|----------|-------------|------------|-----------|
@@ -1071,6 +1139,7 @@ Annotations for all 18 kbdb tools:
 | `check`        | Integrity Check   | true     | false       | true       | false     |
 | `gc`           | Garbage Collect   | false    | true        | false      | false     |
 | `rebuild`      | Rebuild Indexes   | false    | false       | false      | false     |
+| `export`       | Export Snapshot    | false    | false       | true       | false     |
 | `skill-learn`  | Create Skill      | false    | false       | true       | false     |
 | `skill-list`   | List Skills       | true     | false       | true       | false     |
 | `skill-get`    | Get Skill         | true     | false       | true       | false     |