@dikolab/kbdb 0.2.2 → 0.3.1

package/docs/modules/rust/query-parser/types/keyphrase.md

@@ -58,3 +58,14 @@ component and per-field corpus average from
 [FieldTokenCounts](../../shared/query/types.md)
 for the per-field token counts used in length
 normalization.
+
+```rust
+pub type SynonymMap = HashMap<String, Vec<String>>;
+```
+
+`SynonymMap` is a type alias for the bidirectional
+synonym lookup used in tokenization pipeline stage 7.
+It is constructed once via `build_synonym_map()` from
+a curated list of synonym pairs and reused across all
+tokenization calls. The map is immutable after
+construction.

package/docs/modules/rust/query-parser/types/module.md

@@ -9,9 +9,9 @@ tokenization, keyword extraction, and ranking.*
   `Ngram`, `NgramType`, `StemResult`. Output types
   from the tokenization pipeline.
 - **[keyphrase](keyphrase.md)** -- `Keyphrase`,
-  `WeightedTerm`, `BM25FParams`, `FieldWeights`.
-  Types for keyword extraction and ranking
-  configuration.
+  `WeightedTerm`, `BM25FParams`, `FieldWeights`,
+  `SynonymMap`. Types for keyword extraction, ranking
+  configuration, and synonym expansion.
 - **[cache](cache.md)** -- `QueryCache`. Short-lived
   cache scoped to a single query or indexing
   operation.

package/docs/modules/typescript/shared/auto-capture/module.md

@@ -0,0 +1,73 @@
+# auto-capture -- MCP Sampling-Based Automatic Knowledge Capture
+
+**Source:** `src/shared/auto-capture/`
+
+## Responsibility
+
+Automatic knowledge capture from AI agent conversations. Uses MCP
+`sampling/createMessage` to ask the host LLM to identify knowledge
+worth storing, then feeds extracted knowledge into kbdb via `learn`.
+Operates as an MCP server-side feature -- the server proactively
+requests captures based on configurable triggers.
+
+## Directory Layout
+
+```
+src/shared/auto-capture/
+   index.ts
+   auto-capture.controller.ts
+   trigger.ts
+   sampling-prompt.ts
+   rate-limiter.ts
+```
+
+## Components
+
+- **AutoCaptureController** (`auto-capture.controller.ts`) --
+  Orchestrates the capture flow. Tracks tool call count, checks
+  trigger conditions, invokes sampling, and feeds results to
+  `learn`. Stateful per MCP session.
+- **Trigger logic** (`trigger.ts`) -- Evaluates whether a capture
+  check should fire. Conditions: (a) after N tool calls (default:
+  10), (b) configurable time interval, (c) explicit
+  `auto-capture-review` tool invocation. Multiple triggers can be
+  active simultaneously.
+- **Sampling prompts** (`sampling-prompt.ts`) -- Prompt templates
+  sent via `sampling/createMessage`. The prompt asks the host LLM
+  to extract knowledge from the conversation that matches the
+  `memory-protocol` skill's storage criteria: facts, decisions,
+  rationale, procedures.
+- **Rate limiter** (`rate-limiter.ts`) -- Prevents excessive
+  capture attempts. Enforces minimum interval between captures
+  (default: 60 s) and maximum captures per session (default: 20).
+
+## Configuration
+
+Configured via the `[auto_capture]` section of `worker.toml`:
+
+```toml
+[auto_capture]
+enabled = false
+trigger_after_tool_calls = 10
+min_interval_seconds = 60
+max_captures_per_session = 20
+dry_run = false
+```
+
+When `dry_run = true`, the controller logs what would be captured
+without actually writing to the database. Useful for tuning
+triggers.
+
+## Dependency Direction
+
+```
+shared/auto-capture/
+  └── (no shared module deps -- uses MCP sampling API)
+
+shared/mcp/
+  └── shared/auto-capture/   (controller lifecycle)
+```
+
+The auto-capture module is instantiated by the MCP server module.
+It uses the MCP protocol's sampling capability -- not direct imports
+of other shared modules.

package/docs/modules/typescript/shared/cli/functions.md

@@ -68,14 +68,17 @@ async function dispatchCommand(
 
 `DispatchClient` is the intersection of all per-command
 client interfaces (`SearchClient & LearnClient &
-UnlearnClient & ContentClient & CheckClient &
-GcClient & RebuildClient & SkillClient & AgentClient`).
+UnlearnClient & RecallClient & ContentClient &
+CheckClient & GcClient & RebuildClient & ExportClient &
+SkillClient & AgentClient`).
 
 | Command        | Handler                          |
 |----------------|----------------------------------|
 | `learn`        | `runLearn(client, options, io)`  |
+| `export`       | `runExport(client, options)`     |
 | `unlearn`      | `runUnlearn(client, options)`    |
 | `search`       | `runSearch(client, options)`     |
+| `recall`       | `runRecall(client, options)`     |
 | `content`      | `runContent(client, options)`    |
 | `check`        | `runCheck(client, options)`      |
 | `gc`           | `runGc(client, options)`         |
@@ -222,8 +225,15 @@ async function runLearn(
 - If input is a **text file**: adds that single file.
   `--title` and `--description` are optional.
 - If input is **stdin** (`-`): reads `text/markdown`.
+- `--tags <t,...>` attaches comma-separated tags for
+  scoping and filtering.
+- `--replace` enables supersession by source key:
+  if a section with the same source key exists, the
+  old section is replaced.
 - Returns a `LearnResult[]` with one record per file
-  processed.
+  processed. Each result includes `superseded` (kbid
+  of replaced section or `null`), `warnings` (e.g.
+  `"heading_missing"`), and `near_duplicates`.
 
 ### `runUnlearn`
 
@@ -275,6 +285,36 @@ async function runContent(
 Takes identifiers from `options.args`. Calls
 `client.content(ids)`.
 
+### `runRecall`
+
+**File:** `run-recall.function.ts`
+
+Retrieves full content and progressive context for one
+or more sections by kbid.
+
+```ts
+async function runRecall(
+   client: RecallClient,
+   options: CliOptions,
+): Promise<CommandResult>;
+```
+
+`RecallClient` interface:
+
+```ts
+interface RecallClient {
+   recall(
+      params: RecallParams,
+   ): Promise<RecallResult | RecallResult[]>;
+}
+```
+
+Takes kbid(s) from `options.args` and depth from
+`options.depth` (default 0, clamped 0-3). When a
+single kbid is provided, returns a `RecallResult`
+JSON object. When multiple kbids are provided, returns
+a `RecallResult[]` JSON array.
+
 ### `runCheck`
 
 **File:** `run-check.function.ts`
@@ -320,6 +360,30 @@ async function runRebuild(
 
 Calls `client.rebuildIndexes()`.
 
+### `runExport`
+
+**File:** `run-export.function.ts`
+
+Exports the knowledge base as a self-contained
+directory snapshot for backup or versioning.
+
+```ts
+async function runExport(
+   client: ExportClient,
+   options: CliOptions,
+): Promise<CommandResult>;
+```
+
+Copies sections, documents, and catalog (no indexes
+-- they are rebuildable via `rebuild`). The export
+destination defaults to `./kbdb-export/` and can be
+overridden via `options.args[0]`.
+
+Output: JSON `{ "path": "...",
+"sections_exported": number,
+"documents_exported": number,
+"elapsed_ms": number }`.
+
 ### `ensureDb`
 
 **File:** `ensure-db.function.ts`
@@ -608,6 +672,7 @@ src/shared/cli/
 ├── run-db-init.spec.ts
 ├── run-db-migrate.spec.ts
 ├── run-learn.spec.ts
+├── run-export.spec.ts
 ├── run-unlearn.spec.ts
 ├── run-search.spec.ts
 ├── run-content.spec.ts

package/docs/modules/typescript/shared/embedding/functions.md

@@ -0,0 +1,94 @@
+# embedding/functions -- Embedding Functions
+
+**Source:** `src/shared/embedding/functions/`
+
+## embed
+
+```ts
+embed(
+   provider: EmbeddingProvider,
+   text: string,
+): Promise<Float32Array>
+```
+
+Computes a single embedding vector for the given text.
+Delegates to the provider's `embed()` method. Returns a
+`Float32Array` of dimensions matching `provider.dimensions()`.
+
+Used during `learn` to embed section content and during
+`search` to embed the query string.
+
+## batchEmbed
+
+```ts
+batchEmbed(
+   provider: EmbeddingProvider,
+   texts: string[],
+): Promise<Float32Array[]>
+```
+
+Computes embeddings for multiple texts in a single batch.
+More efficient than calling `embed()` in a loop -- providers
+can batch API calls or parallelize ONNX inference. Used
+during `rebuild` to re-embed all sections.
+
+## cosineSimilarity
+
+```ts
+cosineSimilarity(
+   a: Float32Array,
+   b: Float32Array,
+): number
+```
+
+Computes the cosine similarity between two vectors. Returns a
+value in [-1, 1], where 1 means identical direction. Used by
+semantic near-duplicate detection (threshold >= 0.92) and by
+vector search ranking.
+
+Both vectors must have the same length (same embedding
+dimensions). Throws if lengths differ.
+
+## buildVectorIndex
+
+```ts
+buildVectorIndex(
+   provider: EmbeddingProvider,
+   sections: SectionRecord[],
+   outPath: string,
+): Promise<VectorIndexMeta>
+```
+
+Builds the vector index file (`vectors.idx`) from all
+sections. Embeds each section's content via `batchEmbed`,
+writes fixed-width records (kbid + vector) to `outPath`, and
+writes `vectors-meta.toml` alongside it with model name,
+dimensions, and record count.
+
+Called by `rebuild` when an embedding provider is configured.
+Returns metadata about the built index.
+
+## searchVectors
+
+```ts
+searchVectors(
+   provider: EmbeddingProvider,
+   query: string,
+   indexPath: string,
+   limit: number,
+): Promise<VectorSearchResult[]>
+```
+
+Performs vector similarity search. Embeds the query, reads
+`vectors.idx`, computes cosine similarity against all stored
+vectors, and returns the top `limit` results sorted by
+similarity descending.
+
+Each result includes the kbid and similarity score. The
+results are passed to the RRF fusion step alongside lexical
+search results.
+
+Note: This is a brute-force linear scan. For corpus sizes
+under 100k sections, this is fast enough (< 50ms). A future
+version may add approximate nearest neighbor (ANN) indexing
+if scale demands it.

package/docs/modules/typescript/shared/embedding/module.md

@@ -0,0 +1,71 @@
+# embedding -- Pluggable Embedding Computation
+
+**Source:** `src/shared/embedding/`
+
+## Responsibility
+
+Pluggable embedding computation for hybrid lexical+vector search.
+Provides the `EmbeddingProvider` interface and implementations for
+different backends. Used by the worker daemon to embed sections at
+learn time and to compute query embeddings at search time.
+
+## Directory Layout
+
+```
+src/shared/embedding/
+   index.ts
+   functions/
+      embed.function.ts
+      batch-embed.function.ts
+      cosine-similarity.function.ts
+      build-vector-index.function.ts
+      search-vectors.function.ts
+   providers/
+      null-embedding.provider.ts
+      onnx-embedding.provider.ts
+      api-embedding.provider.ts
+   typings/
+      embedding-provider.typing.ts
+      embedding-config.typing.ts
+      vector-search-result.typing.ts
+      vector-index-meta.typing.ts
+```
+
+## Exports
+
+- [Functions](functions.md) -- `embed`, `batchEmbed`,
+  `cosineSimilarity`, `buildVectorIndex`, `searchVectors`
+- [Typings](typings.md) -- `EmbeddingProvider`, `EmbeddingConfig`,
+  `VectorSearchResult`, `VectorIndexMeta`
+
+## Providers
+
+Three implementations of the `EmbeddingProvider` interface:
+
+- **NullEmbeddingProvider** -- No-op provider used when no embedding
+  backend is configured. Returns empty vectors. Allows all
+  lexical-only paths to work unchanged.
+- **OnnxEmbeddingProvider** -- Local inference using ONNX Runtime.
+  Loads a model from disk (e.g., `all-MiniLM-L6-v2`). Zero network
+  calls, fully offline.
+- **ApiEmbeddingProvider** -- Remote inference via HTTP API (OpenAI,
+  Anthropic, or compatible endpoints). Configured via `worker.toml`
+  with endpoint URL and API key.
+
+## Dependency Direction
+
+```
+shared/embedding/
+  ├── shared/hash/        (content hashing for cache keys)
+  └── shared/platform/    (file paths for local models)
+
+shared/worker-daemon/
+  └── shared/embedding/   (init provider at startup)
+
+shared/mcp/
+  └── (indirect via worker-client)
+```
+
+The embedding module is a utility module -- it depends only on other
+utility modules. The worker daemon initializes the provider at startup
+based on `worker.toml` config.

package/docs/modules/typescript/shared/embedding/typings.md

@@ -0,0 +1,122 @@
+# embedding/typings -- Embedding Type Definitions
+
+*Interfaces and type aliases for the embedding
+module's providers, configuration, and search
+results.*
+
+## Source
+
+`src/shared/embedding/typings/`
+
+## Typings
+
+### `EmbeddingProvider`
+
+**File:** `embedding-provider.typing.ts`
+
+Core abstraction for text embedding. Accepts a batch
+of texts and returns a vector for each. Implemented
+by `NullEmbeddingProvider`, `OnnxEmbeddingProvider`,
+and `ApiEmbeddingProvider`.
+
+```ts
+interface EmbeddingProvider {
+   embed(texts: string[]): Promise<Float32Array[]>;
+   dimensions(): number;
+   modelName(): string;
+}
+```
+
+| Member       | Description                             |
+|--------------|-----------------------------------------|
+| `embed`      | Produces one embedding vector per input |
+|              | text; accepts an array for batching     |
+|              | efficiency                              |
+| `dimensions` | Returns the length of each vector       |
+|              | (e.g. 384 for MiniLM)                  |
+| `modelName`  | Returns a stable identifier stored in   |
+|              | `vectors-meta.toml`; used to detect     |
+|              | model changes that require a rebuild    |
+
+### `EmbeddingConfig`
+
+**File:** `embedding-config.typing.ts`
+
+Configuration read from the `[embedding]` section
+of `worker.toml`. Determines which `EmbeddingProvider`
+implementation to instantiate at daemon startup.
+
+```ts
+interface EmbeddingConfig {
+   provider: 'none' | 'onnx' | 'api';
+   model?: string;
+   apiEndpoint?: string;
+   apiKey?: string;
+   dimensions?: number;
+}
+```
+
+| Property      | Description                              |
+|---------------|------------------------------------------|
+| `provider`    | Selects the backend: `'none'` →          |
+|               | `NullEmbeddingProvider`; `'onnx'` →      |
+|               | `OnnxEmbeddingProvider`; `'api'` →       |
+|               | `ApiEmbeddingProvider`                   |
+| `model`       | Path to the local ONNX model file;       |
+|               | required when `provider` is `'onnx'`     |
+| `apiEndpoint` | Remote HTTP API URL; required when       |
+|               | `provider` is `'api'`                    |
+| `apiKey`      | Bearer token for the remote API;         |
+|               | required when `provider` is `'api'`      |
+| `dimensions`  | Vector length; inferred from the model   |
+|               | when possible; required for API          |
+|               | providers that do not report dimensions  |
+
+### `VectorSearchResult`
+
+**File:** `vector-search-result.typing.ts`
+
+A single result from a vector similarity search.
+Used as input to the RRF fusion step alongside
+lexical search results.
+
+```ts
+interface VectorSearchResult {
+   kbid: string;
+   similarity: number;
+}
+```
+
+| Property     | Description                             |
+|--------------|-----------------------------------------|
+| `kbid`       | Knowledge-base identifier of the        |
+|              | matched section                         |
+| `similarity` | Cosine similarity score in [-1, 1]      |
+
+### `VectorIndexMeta`
+
+**File:** `vector-index-meta.typing.ts`
+
+Metadata stored in `vectors-meta.toml` alongside
+`vectors.idx`. Used to detect stale indexes: when
+`model` differs from the current provider's
+`modelName()`, vectors must be rebuilt.
+
+```ts
+interface VectorIndexMeta {
+   model: string;
+   dimensions: number;
+   recordCount: number;
+   version: number;
+}
+```
+
+| Property      | Description                             |
+|---------------|-----------------------------------------|
+| `model`       | `modelName()` value at index build time |
+| `dimensions`  | Vector length used when building the    |
+|               | index                                   |
+| `recordCount` | Number of vectors stored in the index   |
+| `version`     | Format version; currently `1`           |
+
+No spec -- typings are compile-time only.