@dikolab/kbdb 0.3.0 → 0.3.2

package/docs/modules/typescript/shared/embedding/functions.md

@@ -0,0 +1,94 @@
+# embedding/functions -- Embedding Functions
+
+**Source:** `src/shared/embedding/functions/`
+
+## embed
+
+```ts
+embed(
+   provider: EmbeddingProvider,
+   text: string,
+): Promise<Float32Array>
+```
+
+Computes a single embedding vector for the given text.
+Delegates to the provider's `embed()` method. Returns a
+`Float32Array` of dimensions matching `provider.dimensions()`.
+
+Used during `learn` to embed section content and during
+`search` to embed the query string.
+
+## batchEmbed
+
+```ts
+batchEmbed(
+   provider: EmbeddingProvider,
+   texts: string[],
+): Promise<Float32Array[]>
+```
+
+Computes embeddings for multiple texts in a single batch.
+More efficient than calling `embed()` in a loop -- providers
+can batch API calls or parallelize ONNX inference. Used
+during `rebuild` to re-embed all sections.
+
+## cosineSimilarity
+
+```ts
+cosineSimilarity(
+   a: Float32Array,
+   b: Float32Array,
+): number
+```
+
+Computes the cosine similarity between two vectors. Returns a
+value in [-1, 1], where 1 means identical direction. Used by
+semantic near-duplicate detection (threshold >= 0.92) and by
+vector search ranking.
+
+Both vectors must have the same length (same embedding
+dimensions). Throws if lengths differ.
+
+## buildVectorIndex
+
+```ts
+buildVectorIndex(
+   provider: EmbeddingProvider,
+   sections: SectionRecord[],
+   outPath: string,
+): Promise<VectorIndexMeta>
+```
+
+Builds the vector index file (`vectors.idx`) from all
+sections. Embeds each section's content via `batchEmbed`,
+writes fixed-width records (kbid + vector) to `outPath`, and
+writes `vectors-meta.toml` alongside it with model name,
+dimensions, and record count.
+
+Called by `rebuild` when an embedding provider is configured.
+Returns metadata about the built index.
+
+## searchVectors
+
+```ts
+searchVectors(
+   provider: EmbeddingProvider,
+   query: string,
+   indexPath: string,
+   limit: number,
+): Promise<VectorSearchResult[]>
+```
+
+Performs vector similarity search. Embeds the query, reads
+`vectors.idx`, computes cosine similarity against all stored
+vectors, and returns the top `limit` results sorted by
+similarity descending.
+
+Each result includes the kbid and similarity score. The
+results are passed to the RRF fusion step alongside lexical
+search results.
+
+Note: This is a brute-force linear scan. For corpus sizes
+under 100k sections, this is fast enough (< 50ms). A future
+version may add approximate nearest neighbor (ANN) indexing
+if scale demands it.

package/docs/modules/typescript/shared/embedding/module.md

@@ -0,0 +1,71 @@
+# embedding -- Pluggable Embedding Computation
+
+**Source:** `src/shared/embedding/`
+
+## Responsibility
+
+Pluggable embedding computation for hybrid lexical+vector search.
+Provides the `EmbeddingProvider` interface and implementations for
+different backends. Used by the worker daemon to embed sections at
+learn time and to compute query embeddings at search time.
+
+## Directory Layout
+
+```
+src/shared/embedding/
+   index.ts
+   functions/
+      embed.function.ts
+      batch-embed.function.ts
+      cosine-similarity.function.ts
+      build-vector-index.function.ts
+      search-vectors.function.ts
+   providers/
+      null-embedding.provider.ts
+      onnx-embedding.provider.ts
+      api-embedding.provider.ts
+   typings/
+      embedding-provider.typing.ts
+      embedding-config.typing.ts
+      vector-search-result.typing.ts
+      vector-index-meta.typing.ts
+```
+
+## Exports
+
+- [Functions](functions.md) -- `embed`, `batchEmbed`,
+  `cosineSimilarity`, `buildVectorIndex`, `searchVectors`
+- [Typings](typings.md) -- `EmbeddingProvider`, `EmbeddingConfig`,
+  `VectorSearchResult`, `VectorIndexMeta`
+
+## Providers
+
+Three implementations of the `EmbeddingProvider` interface:
+
+- **NullEmbeddingProvider** -- No-op provider used when no embedding
+  backend is configured. Returns empty vectors. Allows all
+  lexical-only paths to work unchanged.
+- **OnnxEmbeddingProvider** -- Local inference using ONNX Runtime.
+  Loads a model from disk (e.g., `all-MiniLM-L6-v2`). Zero network
+  calls, fully offline.
+- **ApiEmbeddingProvider** -- Remote inference via HTTP API (OpenAI,
+  Anthropic, or compatible endpoints). Configured via `worker.toml`
+  with endpoint URL and API key.
+
+## Dependency Direction
+
+```
+shared/embedding/
+  ├── shared/hash/        (content hashing for cache keys)
+  └── shared/platform/    (file paths for local models)
+
+shared/worker-daemon/
+  └── shared/embedding/   (init provider at startup)
+
+shared/mcp/
+  └── (indirect via worker-client)
+```
+
+The embedding module is a utility module -- it depends only on other
+utility modules. The worker daemon initializes the provider at startup
+based on `worker.toml` config.

package/docs/modules/typescript/shared/embedding/typings.md

@@ -0,0 +1,122 @@
+# embedding/typings -- Embedding Type Definitions
+
+*Interfaces and type aliases for the embedding
+module's providers, configuration, and search
+results.*
+
+## Source
+
+`src/shared/embedding/typings/`
+
+## Typings
+
+### `EmbeddingProvider`
+
+**File:** `embedding-provider.typing.ts`
+
+Core abstraction for text embedding. Accepts a batch
+of texts and returns a vector for each. Implemented
+by `NullEmbeddingProvider`, `OnnxEmbeddingProvider`,
+and `ApiEmbeddingProvider`.
+
+```ts
+interface EmbeddingProvider {
+   embed(texts: string[]): Promise<Float32Array[]>;
+   dimensions(): number;
+   modelName(): string;
+}
+```
+
+| Member       | Description                             |
+|--------------|-----------------------------------------|
+| `embed`      | Produces one embedding vector per input |
+|              | text; accepts an array for batching     |
+|              | efficiency                              |
+| `dimensions` | Returns the length of each vector       |
+|              | (e.g. 384 for MiniLM)                  |
+| `modelName`  | Returns a stable identifier stored in   |
+|              | `vectors-meta.toml`; used to detect     |
+|              | model changes that require a rebuild    |
+
+### `EmbeddingConfig`
+
+**File:** `embedding-config.typing.ts`
+
+Configuration read from the `[embedding]` section
+of `worker.toml`. Determines which `EmbeddingProvider`
+implementation to instantiate at daemon startup.
+
+```ts
+interface EmbeddingConfig {
+   provider: 'none' | 'onnx' | 'api';
+   model?: string;
+   apiEndpoint?: string;
+   apiKey?: string;
+   dimensions?: number;
+}
+```
+
+| Property      | Description                              |
+|---------------|------------------------------------------|
+| `provider`    | Selects the backend: `'none'` →          |
+|               | `NullEmbeddingProvider`; `'onnx'` →      |
+|               | `OnnxEmbeddingProvider`; `'api'` →       |
+|               | `ApiEmbeddingProvider`                   |
+| `model`       | Path to the local ONNX model file;       |
+|               | required when `provider` is `'onnx'`     |
+| `apiEndpoint` | Remote HTTP API URL; required when       |
+|               | `provider` is `'api'`                    |
+| `apiKey`      | Bearer token for the remote API;         |
+|               | required when `provider` is `'api'`      |
+| `dimensions`  | Vector length; inferred from the model   |
+|               | when possible; required for API          |
+|               | providers that do not report dimensions  |
+
+### `VectorSearchResult`
+
+**File:** `vector-search-result.typing.ts`
+
+A single result from a vector similarity search.
+Used as input to the RRF fusion step alongside
+lexical search results.
+
+```ts
+interface VectorSearchResult {
+   kbid: string;
+   similarity: number;
+}
+```
+
+| Property     | Description                             |
+|--------------|-----------------------------------------|
+| `kbid`       | Knowledge-base identifier of the        |
+|              | matched section                         |
+| `similarity` | Cosine similarity score in [-1, 1]      |
+
+### `VectorIndexMeta`
+
+**File:** `vector-index-meta.typing.ts`
+
+Metadata stored in `vectors-meta.toml` alongside
+`vectors.idx`. Used to detect stale indexes: when
+`model` differs from the current provider's
+`modelName()`, vectors must be rebuilt.
+
+```ts
+interface VectorIndexMeta {
+   model: string;
+   dimensions: number;
+   recordCount: number;
+   version: number;
+}
+```
+
+| Property      | Description                             |
+|---------------|-----------------------------------------|
+| `model`       | `modelName()` value at index build time |
+| `dimensions`  | Vector length used when building the    |
+|               | index                                   |
+| `recordCount` | Number of vectors stored in the index   |
+| `version`     | Format version; currently `1`           |
+
+No spec -- typings are compile-time only.

package/docs/overview.md

@@ -1,188 +1,192 @@
 # @dikolab/kbdb -- Knowledge Base Database
 
-*A persistent, file-based knowledge store and MCP server purpose-built for AI agents -- your second brain, on disk.*
+*A file-based knowledge store and MCP server for
+AI agents -- your second brain, on disk.*
 
-## What Is @dikolab/kbdb?
+## What Is kbdb?
 
-AI agents operate in ephemeral sessions. They lose context between conversations, forget prior research, and have no built-in way to persist or retrieve knowledge across runs. When an agent needs to recall documentation, search a codebase's API surface, or query a corpus of notes, there is no lightweight, local solution optimized for how agents actually retrieve information -- keyword-driven, section-level, ranked.
+AI agents forget everything between sessions. kbdb
+gives them a searchable, persistent memory.
 
-@dikolab/kbdb is a CLI tool and MCP server that imports documents and code documentation, extracts keywords, builds indexes, and enables ranked search -- all backed by a directory-based file database. No database server, no cloud dependency. A folder on disk is your entire knowledge base.
+Import your documents, and kbdb indexes them
+automatically. Search returns ranked results
+instantly. No database server to install, no cloud
+account to create -- your knowledge base is just a
+folder on disk.
 
-The computationally intensive work -- keyword extraction, indexing, ranking -- is offloaded to a Rust engine compiled to WebAssembly, delivering near-native performance inside a Deno or Node.js runtime. The TypeScript CLI provides the developer-facing interface and MCP server; Rust/WASM does the heavy lifting underneath.
+kbdb works as both a command-line tool and an
+[MCP](https://modelcontextprotocol.io/) server.
+Humans use it from the terminal; AI agents connect
+to it as a tool.
 
 ## Who Is This For?
 
-- **AI agent developers** -- Anyone building agentic workflows (with Claude, LangChain, or custom harnesses) who needs persistent, searchable knowledge across sessions.
-- **MCP tool authors** -- Developers creating Model Context Protocol servers who need a local knowledge backend their tools can query.
-- **Knowledge workers using AI assistants** -- Power users who want their AI to have a second brain -- a searchable, structured store of documentation, research, and reference material that survives session boundaries.
-- **CLI-first developers** -- Those who prefer composable command-line tools over GUI applications for managing knowledge.
-
-## Core Features
-
-### CLI
-
-1. **Markdown import and indexing** -- Import `*.md` files, extract keywords and sections, and store them in the file system database.
-2. **Codebase documentation scanning** -- Scan a codebase and import code documentation (TypeDoc, JSDoc, PHPDoc), index it, and store it in the file system database.
-3. **Ranked search** -- Search and display a ranked list of results by section or by whole document, with output in `.md` or `.json` format.
-
-### MCP
-
-1. **MCP service endpoints** -- Expose the knowledge base as MCP service endpoints for Claude and other AI agents to query, search, and retrieve knowledge programmatically.
-
-### Distribution
-
-This tool is available via **NPM** for Node.js or **JSR** for Deno.
-
-- **Node.js**: `npm install -g @dikolab/kbdb`
-- **Deno**: `deno add @dikolab/kbdb` (via [jsr.io](https://jsr.io))
-
-## High-Level Architecture
-
-The system is organized in five layers. Data flows
-downward from user commands or agent requests through
-the worker client and daemon into the WASM engine,
-which performs heavy processing and manages the file
-database.
-
-![System Architecture](goals/architecture.svg)
-
-**The kbdb Command** -- A single executable with two
-run modes. CLI mode (`kbdb learn`, `kbdb search`,
-`kbdb unlearn`) handles human and script invocation --
-each command is a short-lived process. MCP mode
-(`kbdb mcp`) starts a long-lived MCP server that AI
-agents connect to over stdio.
-
-**Worker Client** -- Shared TypeScript module imported
-by both run modes. Discovers or spawns the worker
-daemon and provides a typed async API over IPC.
-
-**Worker Daemon** -- A detached background process
-that loads WASM modules, manages an in-memory cache,
-and serves JSON-RPC 2.0 requests over local IPC (Unix
-sockets or Windows named pipes). The daemon has no
-knowledge of its callers.
-
-**Rust/WASM Engine** -- Three WASM modules:
-`query-parser.wasm` (tokenization, query parsing,
-ranking), `fs-database.wasm` (file database I/O), and
-`kb-worker.wasm` (imports both, orchestrates). Rust
-provides memory safety and performance; WASM provides
-portability across runtimes.
-
-**File Database** -- A structured directory on disk.
-No external database process required. Documents,
-indexes, keywords, and metadata are stored in a
-defined directory layout. The database is portable,
-version-controllable, and human-inspectable.
-
-## Technology Stack
-
-| Layer | Technology | Role |
-|-------|-----------|------|
-| Runtime | Deno (primary), Node.js (compatible) | Executes the CLI, MCP server, and TypeScript orchestration |
-| Language | TypeScript | CLI, command parsing, MCP protocol, I/O coordination |
-| Engine | Rust compiled to WebAssembly | Keyword processing, indexing, ranking, file DB operations |
-| Protocol | MCP (Model Context Protocol) | AI agent integration interface |
-| Distribution | NPM (Node.js), JSR (Deno) | Package distribution for both ecosystems |
-| Storage | Directory-based file database | Persistent, portable, server-less data storage |
+- **AI agent developers** -- persistent, searchable
+  knowledge across agent sessions
+- **MCP tool authors** -- a local knowledge backend
+  for Model Context Protocol servers
+- **Knowledge workers** -- a second brain your AI
+  assistant can search across conversations
+- **Vibe coders** -- a searchable doc store for
+  your AI-assisted projects
+- **CLI-first developers** -- composable commands
+  for managing knowledge from the terminal
+
+## What Can It Do?
+
+- **Import** Markdown files with automatic keyword
+  extraction and indexing
+- **Search** three ways: keyword (default),
+  AI similarity, or hybrid (both combined) via
+  `--algo`
+- **Auto-fallback** -- loosens your query when
+  exact matches return nothing
+- **Recall** sections with progressive context
+  (depth 0--3)
+- **Smart updates** -- re-learning a file replaces
+  the old version, not duplicates it
+- **Near-duplicate detection** -- warns when
+  imported content closely matches existing data
+- **Skills** -- reusable prompt templates with
+  fill-in-the-blank arguments
+- **Agents** -- AI profiles combining a persona
+  with skills
+- **Auto-capture** -- MCP server proactively
+  suggests knowledge to store from conversations
+- **Export and sync** -- snapshot your knowledge
+  base for backup or sharing across machines
+- **Verify** database integrity and clean up stale
+  data
+- **MCP server** with 20 tools, prompts, resources,
+  and completions
+
+### Install
+
+```sh
+# Node.js
+npm install -g @dikolab/kbdb
+
+# Deno
+deno add @dikolab/kbdb
+```
+
+## Architecture
+
+Five layers, top to bottom:
+
+```
+User / AI Agent
+  -> kbdb command (CLI or MCP mode)
+    -> Worker Client (TypeScript IPC)
+      -> Worker Daemon (background process)
+        -> Rust/WASM Engine
+          -> File Database (folder on disk)
+```
+
+**kbdb command** -- One executable, two modes. CLI
+mode runs short-lived commands (`learn`, `search`,
+`recall`). MCP mode starts a long-lived server for
+AI agents over stdio.
+
+**Worker Client** -- Connects to the background
+daemon and gives your code a typed API.
+
+**Worker Daemon** -- Background process that loads
+the search engine, caches data in memory, and
+handles requests.
+
+**Search Engine** -- Compiled from Rust to
+WebAssembly for speed. Three modules handle
+tokenization, indexing, ranking, and file I/O.
+
+**File Database** -- Structured folder on disk.
+No server process. Portable, version-controllable,
+and human-readable.
+
+### Technology Stack
+
+| Layer | Technology | What it does |
+|-------|-----------|--------------|
+| Runtime | Node.js, Deno | Runs the tools |
+| Language | TypeScript | CLI, MCP server, I/O |
+| Engine | Rust -> WebAssembly | Fast indexing and search |
+| Protocol | MCP | AI agent integration |
+| Distribution | NPM, JSR | Install from either |
+| Storage | File database | Your data, in a folder |
 
 ## Design Principles
 
-1. **Offline-first, server-less** -- No database server, no cloud dependency. A directory on disk is the entire database. Works anywhere a filesystem exists.
-2. **AI-agent-native** -- Designed from the ground up to be invoked by AI agents, not just humans. Commands, output formats, and the MCP interface prioritize machine consumption.
-3. **MCP-first integration** -- The MCP server is a primary interface, not an afterthought. AI agents connect to the knowledge base as naturally as they connect to any other MCP service.
-4. **Performance through Rust/WASM** -- Computationally intensive work runs in compiled Rust via WebAssembly, keeping the TypeScript layer thin and the heavy processing fast.
-5. **Portability** -- Runs on Deno or Node.js. The file database is a plain directory. No platform lock-in at any layer.
-6. **Section-level precision** -- Documents are decomposed into sections. Search returns the specific passage that matters, not an entire document.
-7. **Incremental design** -- The project ships working slices. Storage format and engine capabilities will evolve; the architecture accommodates change without requiring rewrites.
-
-## Details
-
-The following documents describe the intended behavior and
-design goals for each major subsystem. They serve as
-implementation specifications, not descriptions of current
-functionality.
-
-- **[Database](goals/database.md)** -- Filesystem storage
-  engine (`fs-database.wasm`). The `.kbdb` directory
-  structure, section and document CRUD operations, index
-  management, and storage design.
-- **[Query Parser](goals/query-parser.md)** -- Query
-  processing and tokenization (`query-parser.wasm`).
-  Standalone WASM module for tokenization, query
-  parsing, keyword extraction, and ranking.
-- **[Document](goals/document.md)** -- Text content
-  format specification. Markdown body text with
-  embedding markers and cross-reference link markers.
+1. **No server needed** -- A folder is your whole
+   database. Works anywhere.
+2. **Built for AI agents** -- Commands and output
+   are optimized for machine use.
+3. **MCP-first** -- The MCP server is a primary
+   interface, not an add-on.
+4. **Fast** -- Heavy work runs in compiled Rust
+   via WebAssembly.
+5. **Portable** -- Runs on Node.js or Deno. The
+   database is a plain directory.
+6. **Section-level search** -- Returns the specific
+   passage that matters, not a whole document.
+
+## Design Documents
+
+Deeper docs for each part of the system. You don't
+need to read these to use kbdb -- they're here for
+contributors and anyone curious about internals.
+
+- **[Database](goals/database.md)** -- File storage
+  engine, directory structure, section and document
+  operations
+- **[Query Parser](goals/query-parser.md)** --
+  Tokenization, query parsing, keyword extraction,
+  ranking
+- **[Hybrid Search](goals/hybrid-search.md)** --
+  Lexical + vector retrieval with score fusion
+- **[Document](goals/document.md)** -- Content format
+  with markers and cross-references
 - **[Content Parser](goals/content-parser.md)** --
-  Marker extraction engine (`kbdb-shared` rlib).
-  Scans section content for `$(type: value)` markers
-  and returns structured reference lists.
+  Marker extraction from section content
 - **[Content Composer](goals/content-composer.md)** --
-  Content composition engine (`kbdb-shared` rlib).
-  Resolves sections and markers, produces rendered
-  Markdown documents from kbid and docid identifiers.
-- **[CLI](goals/cli.md)** -- The `kbdb` command and its
-  two run modes. CLI mode commands (`learn`, `unlearn`,
-  `search`), MCP mode dispatch, output formats, and the
-  pass-through execution model.
-- **[MCP](goals/mcp.md)** -- MCP server mode
-  (`kbdb mcp`). Invocation, database path
-  configuration, stdio transport, MCP tools and
-  resources, agent wiring examples, and relationship
-  to CLI mode.
-- **[Worker Daemon](goals/worker-daemon.md)** -- Cache
-  daemon (`kb-worker.wasm` + `src/worker.ts`).
-  The detached TypeScript daemon, WASM module loading,
-  caching, IPC protocol, lifecycle, and configuration.
-- **[Worker Client](goals/worker-client.md)** -- Shared
-  TypeScript client module (`src/shared/worker-client/`).
-  Daemon discovery, spawning, IPC connection, typed
-  API, and the sole entry point used by `src/cli.ts`.
+  Rendered Markdown composition from identifiers
+- **[CLI](goals/cli.md)** -- Commands, run modes,
+  output formats
+- **[MCP](goals/mcp.md)** -- MCP server, tools,
+  resources, agent wiring
+- **[Auto-Capture](goals/auto-capture.md)** --
+  Proactive knowledge capture via MCP sampling
+- **[Sync](goals/sync.md)** -- Cross-machine export
+  and import
+- **[Worker Daemon](goals/worker-daemon.md)** --
+  Background daemon, WASM loading, caching, IPC
+- **[Worker Client](goals/worker-client.md)** --
+  Client module, daemon discovery, typed API
 
 ## Project Status
 
-@dikolab/kbdb is in active development. The full
-five-layer stack (CLI/MCP -> Worker Client -> Worker
-Daemon -> WASM Engine -> File Database) is wired and
-working end-to-end for all core operations: learn,
-search, recall, content retrieval, skills, agents,
-and maintenance commands.
-
-E2E smoke tests run on both Node.js and Deno via
-Docker, covering 380+ assertions across CLI commands,
-18 MCP tools, MCP protocol features, and worker IPC
-methods.
-
-### Completed
-
-- CLI with 18 commands across 6 groups (tested)
-- MCP server with 18 tools, prompts, resources,
-  and completions (tested)
-- Worker client with typed async IPC API (tested)
-- Worker daemon with WASM module loading (tested)
-- Rust/WASM engine: search, recall, indexing,
-  reference graph, KWIC snippets, pagination
-- Paginated search with rich metadata (heading,
-  type, snippet, matched_terms, docids)
+All layers are wired and working end-to-end. Every
+core operation -- learn, search, recall, content
+retrieval, skills, agents, export, and maintenance
+-- runs through the full stack.
+
+**548 e2e assertions** pass on both Node.js and Deno
+with zero failures and zero skips.
+
+- 18 CLI commands across 6 groups
+- 20 MCP tools with prompts, resources, completions
+- Three search algorithms: lexical, vector, hybrid
+- Sequential MCP request processing
+- Worker daemon with fully implemented WASM host I/O
+- Paginated search with rich metadata
 - Progressive recall at depths 0--3
-- Bidirectional reference graph (references.idx)
-- Skills (template-based prompts) and agents
-  (persona + skills compositions)
-- esbuild bundling pipeline (ESM + CJS + .d.ts)
-- Docker build containers (Deno + Node + Rust +
-  wasm-pack)
-- npm and JSR packaging with documentation
-
-### Remaining
-
-- Complete WASM host I/O imports (some operations
-  still hit stubs: document removal, agent document
-  persistence)
-- Database format versioning and migration
-  infrastructure
+- Bidirectional reference graph
+- Skills and agent document management
+- Auto-capture hooks via MCP sampling
+- Export/import sync primitives
+- esbuild bundling (ESM + CJS + .d.ts)
+- Docker build containers
+- npm and JSR packages with documentation
 
 ## Links
 
-- **Repository**: https://gitlab.com/tech-slaves-internal/knowledge-base-db
+- **Repository**: https://gitlab.com/tech-slaves/knowledge-base-db