@dikolab/kbdb 0.5.0 → 0.6.0

package/docs/goals/hybrid-search.md

@@ -1,465 +0,0 @@
-# Hybrid Search -- Lexical + Vector Retrieval
-
-*Optional vector retrieval path alongside BM25F, fused
-with RRF for best-of-both ranking.*
-
-## Goal
-
-The current retrieval mode is lexical search: BM25F with
-stemming and synonym expansion. It works well for
-structured documentation where terminology is consistent
--- the agent searches "authentication" and the indexed
-content uses the same word.
-
-Lexical search has a ceiling, however. It misses
-associative recall when terminology diverges between the
-time content was stored and the time it is retrieved.
-Example: an agent stores "we chose Postgres because of
-JSONB" and later searches "database rationale" -- lexical
-search misses it because none of the indexed tokens
-(`postgres`, `chose`, `jsonb`) overlap with the query
-tokens (`database`, `rationale`). Vector search finds
-it because the embedding of the query is geometrically
-close to the embedding of the stored sentence.
-
-This gap is identified as the critical lexical-only
-ceiling in
-[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md)
-(gap #1). Hybrid search closes it by adding an optional
-vector retrieval path alongside the existing lexical
-path, fusing both ranked lists using Reciprocal Rank
-Fusion (RRF) for a final result set that benefits from
-both modes.
-
-## Architecture
-
-Two retrieval paths run in parallel. The vector path
-runs in TypeScript, backed by a lightweight WASM crate
-(`kbdb-default-embedding`) for the built-in TF-IDF
-provider.
-
-```
-Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
-        │                                              ├─→ RRF Fusion → Final results
-        └─ Vector path (TS)   ─→ Cosine ranked list ──┘
-```
-
-The existing WASM modules are untouched.
-`query-parser.wasm` remains a pure-computation lexical
-module. `kb-worker.wasm` continues to orchestrate
-lexical retrieval via `fs-database.wasm` exactly as
-today. No new exports or host imports are added to
-any existing WASM module.
-
-A new standalone WASM crate, `kbdb-default-embedding`,
-provides the built-in TF-IDF embedding computation.
-It depends only on `kbdb-shared` and requires no host
-imports -- pure computation. It is loaded as an
-independent WASM instance by the worker daemon
-alongside the existing three modules.
-
-The TypeScript daemon layer adds the vector path as an
-independent retrieval step that runs after the lexical
-path completes (or concurrently if the runtime allows).
-The fused list is assembled in TypeScript before the
-response is returned to the worker client.
-
-## EmbeddingProvider Interface
-
-### Why TypeScript, Not WASM
-
-Three reasons embeddings are computed in TypeScript
-rather than in WASM:
-
-1. `wasm32-unknown-unknown` has no network I/O. Calling
-   an embedding API from WASM is impossible without a
-   host bridge that would defeat the purpose of running
-   in WASM.
-2. Embedding models are too large for WASM. A typical
-   all-MiniLM-L6-v2 model is ~90 MB. Loading that inside
-   the WASM sandbox on every daemon startup is
-   impractical.
-3. TypeScript allows pluggable backends. An interface-based
-   design lets the daemon swap providers (local ONNX,
-   remote API) without changing any WASM code.
-
-### Interface Specification
-
-```typescript
-interface EmbeddingProvider {
-   embed(texts: string[]): Promise<Float32Array[]>;
-   dimensions(): number;
-   modelName(): string;
-}
-```
-
-| Member       | Description                                  |
-|--------------|----------------------------------------------|
-| `embed`      | Compute embeddings for a batch of input texts. Returns one `Float32Array` per input, each of length `dimensions()`. |
-| `dimensions` | Fixed embedding dimensionality for this provider. All vectors stored under the same model must have this length. |
-| `modelName`  | Stable identifier for the model. Used for stale-detection when the configured model changes. |
-
-### Implementations
-
-Four concrete implementations are specified.
-
-**`TfidfEmbeddingProvider`** -- lightweight built-in
-provider backed by the `kbdb-default-embedding` WASM
-crate. Computes feature-hashed sublinear TF vectors
-(256 dimensions, L2-normalized) using kbdb-shared's
-tokenization pipeline (NFC normalization, Unicode
-segmentation, camelCase splitting, stop-word removal,
-synonym expansion, Snowball stemming). This ensures
-token consistency between the lexical index and the
-TF-IDF vectors -- hybrid search fuses two views of
-the same vocabulary. The default when no `[embedding]`
-section is present in `worker.toml`. Enables hybrid
-search out of the box with zero configuration and
-zero external dependencies. Uses FNV-1a feature
-hashing with the sign trick (Weinberger et al., 2009).
-Model name: `tfidf-v1`.
-
-**`NullEmbeddingProvider`** -- no-op, returns empty
-arrays. Available for explicit use but no longer the
-default. Returns 0 dimensions, which causes
-`assertProvider` to reject hybrid/vector search
-requests with a structured error.
-
-**`OnnxEmbeddingProvider`** -- local offline inference
-using ONNX Runtime. Default model: `all-MiniLM-L6-v2`
-(384 dimensions). The model file is downloaded on first
-use and cached in the OS user-data directory. Works on
-both Deno and Node.js >= 20. No network dependency
-after the initial model download. Suitable for
-air-gapped or cost-sensitive deployments.
-
-**`ApiEmbeddingProvider`** -- calls an external HTTP
-embedding API. Compatible with OpenAI, Anthropic, and
-any endpoint that accepts a `POST /embeddings` request
-with a JSON body `{ "input": string[] }` and returns
-`{ "data": [{ "embedding": number[] }] }`. Configured
-via `worker.toml`. The API key is read from an
-environment variable named by `api_key_env`.
-
-## Vector Storage
-
-### New Index Files
-
-Two new files are added to `indexes/` alongside the
-existing index files. They follow the same rebuild
-semantics as `terms.idx`, `positions.idx`, and
-`corpus.dat`: derived data, rebuildable from section
-content, excluded from version control.
-
-**`vectors.idx`** -- binary, fixed-width records.
-
-Header layout:
-
-| Bytes | Field          | Type   | Description                   |
-|-------|----------------|--------|-------------------------------|
-| 0-3   | magic          | u8[4]  | `b"KBVX"` -- format sentinel  |
-| 4     | version        | u8     | Format version, currently `1` |
-| 5-6   | dimensions     | u16    | Vector length per record       |
-| 7-10  | record_count   | u32    | Total number of records        |
-
-Each record immediately follows the header:
-
-| Bytes          | Field          | Type           | Description       |
-|----------------|----------------|----------------|-------------------|
-| 0-15           | kbid_hash      | u8[16]         | First 16 bytes of the section kbid, used as a compact lookup key |
-| 16 onwards     | embedding      | f32[dimensions]| Raw embedding vector |
-
-Record size = `16 + dimensions * 4` bytes. All records
-are the same size, enabling O(1) offset computation by
-record index and binary search by kbid hash.
-
-**`vectors-meta.toml`** -- human-readable metadata.
-
-```toml
-[vectors]
-model_name  = "all-MiniLM-L6-v2"
-dimensions  = 384
-version     = 1
-last_built  = "2026-06-23T10:00:00Z"
-section_count = 2847
-```
-
-| Field           | Description                               |
-|-----------------|-------------------------------------------|
-| `model_name`    | Model name from `EmbeddingProvider.modelName()`. Used for stale detection. |
-| `dimensions`    | Must match the `dimensions` header in `vectors.idx`. |
-| `version`       | `vectors.idx` format version.             |
-| `last_built`    | ISO 8601 timestamp of the last rebuild.   |
-| `section_count` | Number of sections embedded.              |
-
-### Updated Directory Layout
-
-```
-.kbdb/
-└── indexes/
-    ├── meta.toml
-    ├── terms.idx
-    ├── positions.idx
-    ├── corpus.dat
-    ├── references.idx
-    ├── vectors.idx          # new -- binary embedding store
-    └── vectors-meta.toml   # new -- embedding metadata
-```
-
-## Embedding Lifecycle
-
-### Learn Time
-
-When a section is added via `kbdb learn`, the lexical
-indexing pipeline runs first (unchanged). After lexical
-indexing completes, the daemon asynchronously computes
-an embedding for the section using the configured
-`EmbeddingProvider`.
-
-The embedding input is the concatenation of the
-section's title and body content:
-
-```
-{title}\n{content}
-```
-
-The title is prepended because it carries strong
-semantic weight (it receives a 2.0x BM25F field weight
-for the same reason). Concatenating both fields gives
-the embedding model the same semantic signal that BM25F
-heading-weight gives to lexical scoring.
-
-Sections with no body content (binary sections where
-content is base64) embed only the title and
-description. Sections with neither title nor body
-content are not embedded.
-
-Embedding is best-effort at learn time: if the provider
-fails for a section, the section is still stored
-lexically. A warning is emitted in the learn response
-(`vector_embedding_failed: true`).
-
-### Rebuild
-
-`kbdb rebuild` regenerates all vector embeddings from
-section content alongside the existing lexical index
-rebuild. Sections are embedded in batches of up to 64
-to respect provider rate limits. The `vectors.idx` and
-`vectors-meta.toml` files are written atomically after
-all embeddings are computed.
-
-### Stale Detection
-
-When the daemon starts, it reads `vectors-meta.toml`
-and compares `model_name` to the currently configured
-provider's `modelName()`. If they differ, the vector
-index is marked stale. Stale vector indexes are not
-used for retrieval. Search falls back to lexical-only
-with a warning field in the response:
-`vector_index_stale: true`.
-
-The agent or operator must run `kbdb rebuild` to
-regenerate vectors with the new model before hybrid or
-vector search resumes.
-
-## Score Fusion
-
-### Reciprocal Rank Fusion (RRF)
-
-RRF combines the lexical ranked list and the vector
-ranked list into a single fused list without requiring
-score normalization. BM25F scores and cosine similarity
-scores live on different scales and cannot be directly
-added -- RRF sidesteps this by working on ranks rather
-than scores.
-
-Formula:
-
-```
-score(d) = 1/(k + rank_lexical(d)) + 1/(k + rank_vector(d))
-```
-
-Where:
-
-- `k = 60` (standard default, configurable via
-  `worker.toml` as `rrf_k`)
-- `rank_lexical(d)` is the 1-based rank of document `d`
-  in the BM25F result list, or infinity if `d` does not
-  appear in the lexical results
-- `rank_vector(d)` is the 1-based rank of document `d`
-  in the cosine similarity result list, or infinity if
-  `d` does not appear in the vector results
-- A rank of infinity contributes 0 to the sum (since
-  `1/(k + ∞) = 0`)
-
-Documents appearing in only one result set receive full
-contribution from that set and zero from the other.
-Documents appearing in both receive a higher combined
-score than either path could produce alone, naturally
-surfacing cross-path agreement.
-
-The final result list is sorted by RRF score descending.
-
-### Why RRF
-
-- Parameter-light: `k = 60` is the well-established
-  default from the original RRF paper (Cormack et al.,
-  2009). It rarely needs tuning.
-- No normalization: BM25F scores and cosine similarity
-  are not comparable in magnitude. RRF requires only
-  rank order, not the scores themselves.
-- Robust: adding a low-quality second path cannot hurt
-  the top results badly because RRF dampens rank
-  contributions as `k` is approached.
-
-## Search Mode Parameter
-
-A new `search_mode` parameter controls which retrieval
-paths are active for a given search request.
-
-| Value      | Retrieval behaviour                         | Embedding provider required |
-|------------|---------------------------------------------|-----------------------------|
-| `lexical`  | Lexical path only. Current behaviour. Default. | No                        |
-| `vector`   | Vector path only. Cosine similarity ranking.   | Yes                       |
-| `hybrid`   | Both paths, RRF fusion.                        | Yes                        |
-
-The default is `lexical`. Existing integrations that do
-not supply `search_mode` are unchanged. This is a
-strict backward-compatibility guarantee.
-
-When `[embedding]` is absent from `worker.toml`, the
-`TfidfEmbeddingProvider` is active as the default.
-Hybrid and vector search work out of the box using
-feature-hashed TF vectors. If the user explicitly sets
-`provider = "none"`, `NullEmbeddingProvider` is used
-and hybrid/vector search returns a structured error:
-
-```json
-{
-  "error": "Vector search requires a configured embedding provider",
-  "code": "EMBEDDING_PROVIDER_NOT_CONFIGURED"
-}
-```
-
-## CLI and MCP Surface
-
-### CLI
-
-The `search` subcommand gains an optional `--algo` flag:
-
-```
-kbdb search "database rationale"
-kbdb search --algo hybrid "database rationale"
-kbdb search --algo vector "database rationale"
-```
-
-Omitting `--algo` defaults to `lexical`. The flag is
-not shown in the short help unless an embedding provider
-is configured.
-
-### MCP
-
-The `search` MCP tool gains an optional `algo` parameter
-(string enum: `"lexical" | "vector" | "hybrid"`). The
-parameter is documented as optional in the tool schema
-with a default of `"lexical"`.
-
-### Response Envelope
-
-`PagedSearchResult` gains two new fields:
-
-| Field              | Type   | Description                     |
-|--------------------|--------|---------------------------------|
-| `search_mode`      | string | The mode that was requested: `"lexical"`, `"vector"`, or `"hybrid"` |
-| `retrieval_method` | string | The mode that was actually used: `"lexical"`, `"vector"`, or `"hybrid"`. Differs from `search_mode` when a fallback occurred (e.g. stale vector index forced `"lexical"`). |
-
-Both fields are always present in the response,
-including when `search_mode` was not supplied (in which
-case both fields are `"lexical"`).
-
-## Configuration
-
-`worker.toml` gains an `[embedding]` section:
-
-```toml
-[embedding]
-provider    = "onnx"           # "onnx" | "api" | "none"
-model       = "all-MiniLM-L6-v2"
-dimensions  = 384
-rrf_k       = 60               # RRF parameter k
-
-# API provider only:
-api_url     = ""
-api_key_env = "KBDB_API_KEY"
-```
-
-| Key           | Default              | Description                  |
-|---------------|----------------------|------------------------------|
-| `provider`    | `"none"`             | Which `EmbeddingProvider` implementation to use |
-| `model`       | `"all-MiniLM-L6-v2"` | Model name, passed to the provider |
-| `dimensions`  | `384`                | Embedding vector length      |
-| `rrf_k`       | `60`                 | RRF constant `k`             |
-| `api_url`     | `""`                 | Base URL for API provider    |
-| `api_key_env` | `"KBDB_API_KEY"`     | Environment variable holding the API key |
-
-When `[embedding]` is absent from `worker.toml`,
-`provider` defaults to `"none"` and the built-in
-`TfidfEmbeddingProvider` is used (256-dimensional
-feature-hashed TF vectors). To fully disable
-embeddings, set `provider = "none"` explicitly.
-
-## Fallback Behaviour
-
-Three fallback conditions are defined. Each is surfaced
-in the search response with a specific warning field
-rather than silently degrading.
-
-**No embedding provider configured.**
-The built-in `TfidfEmbeddingProvider` is active by
-default, so hybrid and vector search work without
-configuration. If the user explicitly sets
-`provider = "none"`, requesting `hybrid` or `vector`
-returns a structured error with code
-`EMBEDDING_PROVIDER_NOT_CONFIGURED`.
-
-**Stale vector index (model mismatch).**
-When `vectors-meta.toml` model name does not match the
-configured provider's `modelName()`, the daemon marks
-the vector index stale. Any search request that would
-use the vector path falls back to `lexical`. The
-response includes `retrieval_method: "lexical"` and
-`vector_index_stale: true`. The correct remediation is
-`kbdb rebuild`.
-
-**Embedding failure during learn.**
-If the embedding provider returns an error for a section
-during learn, the section is stored lexically without an
-embedding. The learn response includes
-`vector_embedding_failed: true`. The section will
-appear in lexical results but not in vector or hybrid
-results until a rebuild is run.
-
-## Provider Tradeoffs
-
-| Aspect        | TF-IDF (built-in)         | ONNX (local)              | API (cloud)                  |
-|---------------|---------------------------|---------------------------|------------------------------|
-| Latency       | < 1 ms/embed              | ~10 ms/embed              | ~100 ms/embed                |
-| Offline       | Yes                       | Yes                       | No                           |
-| Cost          | Free                      | Free                      | Per-token                    |
-| Quality       | Adequate (256 dimensions) | Good (384 dimensions)     | Excellent (1536+ dimensions) |
-| Setup         | None (default)            | Model download on first use | API key required            |
-| Dependencies  | None                      | onnxruntime-node          | None                         |
-| Dual runtime  | Deno + Node.js >= 20      | Deno + Node.js >= 20      | Deno + Node.js >= 20         |
-
-All providers implement the same `EmbeddingProvider`
-interface. The daemon interacts only with the interface.
-
-## Design Constraints
-
-| Constraint            | Details                                        |
-|-----------------------|------------------------------------------------|
-| WASM purity           | `query-parser.wasm` remains pure lexical computation with no I/O. A new standalone crate, `kbdb-default-embedding`, is added for TF-IDF embedding but has no host imports. Existing WASM modules are unchanged. |
-| Backward compatibility | Default `search_mode` is `lexical`. Callers that do not supply the parameter receive identical behaviour to today. |
-| File-based storage    | `vectors.idx` and `vectors-meta.toml` follow the same derived-and-rebuildable pattern as all other index files. |
-| Dual runtime          | `EmbeddingProvider` and all embedding logic must work on Deno and Node.js >= 20. No Node-specific or Deno-specific APIs in the implementation. |
-| No new deps on existing WASM | The existing WASM modules (`fs-database.wasm`, `query-parser.wasm`, `kb-worker.wasm`) are not modified or recompiled. A new standalone module (`default-embedding.wasm`) is added, bringing the total to four WASM modules. |