@dikolab/kbdb 0.3.2 → 0.4.1

package/docs/goals/hybrid-search.md

@@ -34,8 +34,9 @@ both modes.
 ## Architecture
 
 Two retrieval paths run in parallel. The vector path
-runs entirely in TypeScript. No WASM code is added or
-modified.
+runs in TypeScript, backed by a lightweight WASM crate
+(`kbdb-default-embedding`) for the built-in TF-IDF
+provider.
 
 ```
 Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
@@ -43,12 +44,19 @@ Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
         └─ Vector path (TS)   ─→ Cosine ranked list ──┘
 ```
 
-The WASM boundary is untouched. `query-parser.wasm`
-remains a pure-computation lexical module. `kb-worker.wasm`
-continues to orchestrate lexical retrieval via
-`fs-database.wasm` exactly as today. No new WASM crate,
-no new WASM export, and no new host import is needed for
-the vector path.
+The existing WASM modules are untouched.
+`query-parser.wasm` remains a pure-computation lexical
+module. `kb-worker.wasm` continues to orchestrate
+lexical retrieval via `fs-database.wasm` exactly as
+today. No new exports or host imports are added to
+any existing WASM module.
+
+A new standalone WASM crate, `kbdb-default-embedding`,
+provides the built-in TF-IDF embedding computation.
+It depends only on `kbdb-shared` and requires no host
+imports -- pure computation. It is loaded as an
+independent WASM instance by the worker daemon
+alongside the existing three modules.
 
 The TypeScript daemon layer adds the vector path as an
 independent retrieval step that runs after the lexical
@@ -93,15 +101,29 @@ interface EmbeddingProvider {
 
 ### Implementations
 
-Three concrete implementations are specified.
+Four concrete implementations are specified.
+
+**`TfidfEmbeddingProvider`** -- lightweight built-in
+provider backed by the `kbdb-default-embedding` WASM
+crate. Computes feature-hashed sublinear TF vectors
+(256 dimensions, L2-normalized) using kbdb-shared's
+tokenization pipeline (NFC normalization, Unicode
+segmentation, camelCase splitting, stop-word removal,
+synonym expansion, Snowball stemming). This ensures
+token consistency between the lexical index and the
+TF-IDF vectors -- hybrid search fuses two views of
+the same vocabulary. The default when no `[embedding]`
+section is present in `worker.toml`. Enables hybrid
+search out of the box with zero configuration and
+zero external dependencies. Uses FNV-1a feature
+hashing with the sign trick (Weinberger et al., 2009).
+Model name: `tfidf-v1`.
 
 **`NullEmbeddingProvider`** -- no-op, returns empty
-arrays. The default when no `[embedding]` section is
-present in `worker.toml`. Requesting `vector` or
-`hybrid` search mode with this provider returns a
-structured error rather than silently returning empty
-results. Satisfies POLA: the absence of configuration
-does not silently degrade results.
+arrays. Available for explicit use but no longer the
+default. Returns 0 dimensions, which causes
+`assertProvider` to reject hybrid/vector search
+requests with a structured error.
 
 **`OnnxEmbeddingProvider`** -- local offline inference
 using ONNX Runtime. Default model: `all-MiniLM-L6-v2`
@@ -305,9 +327,12 @@ The default is `lexical`. Existing integrations that do
 not supply `search_mode` are unchanged. This is a
 strict backward-compatibility guarantee.
 
-Requesting `vector` or `hybrid` when no embedding
-provider is configured (i.e. `NullEmbeddingProvider` is
-active) returns a structured error:
+When `[embedding]` is absent from `worker.toml`, the
+`TfidfEmbeddingProvider` is active as the default.
+Hybrid and vector search work out of the box using
+feature-hashed TF vectors. If the user explicitly sets
+`provider = "none"`, `NullEmbeddingProvider` is used
+and hybrid/vector search returns a structured error:
 
 ```json
 {
@@ -320,21 +345,21 @@ active) returns a structured error:
 
 ### CLI
 
-The `search` subcommand gains an optional `--mode` flag:
+The `search` subcommand gains an optional `--algo` flag:
 
 ```
 kbdb search "database rationale"
-kbdb search --mode hybrid "database rationale"
-kbdb search --mode vector "database rationale"
+kbdb search --algo hybrid "database rationale"
+kbdb search --algo vector "database rationale"
 ```
 
-Omitting `--mode` defaults to `lexical`. The flag is
+Omitting `--algo` defaults to `lexical`. The flag is
 not shown in the short help unless an embedding provider
 is configured.
 
 ### MCP
 
-The `search` MCP tool gains an optional `mode` parameter
+The `search` MCP tool gains an optional `algo` parameter
 (string enum: `"lexical" | "vector" | "hybrid"`). The
 parameter is documented as optional in the tool schema
 with a default of `"lexical"`.
@@ -378,8 +403,10 @@ api_key_env = "KBDB_API_KEY"
 | `api_key_env` | `"KBDB_API_KEY"`     | Environment variable holding the API key |
 
 When `[embedding]` is absent from `worker.toml`,
-`provider` defaults to `"none"` and `NullEmbeddingProvider`
-is used.
+`provider` defaults to `"none"` and the built-in
+`TfidfEmbeddingProvider` is used (256-dimensional
+feature-hashed TF vectors). To fully disable
+embeddings, set `provider = "none"` explicitly.
 
 ## Fallback Behaviour
 
@@ -388,11 +415,12 @@ in the search response with a specific warning field
 rather than silently degrading.
 
 **No embedding provider configured.**
-`search_mode` defaults to `lexical`. Requesting `hybrid`
-or `vector` returns a structured error with code
-`EMBEDDING_PROVIDER_NOT_CONFIGURED`. No silent fallback
-occurs -- the caller receives an actionable error
-message.
+The built-in `TfidfEmbeddingProvider` is active by
+default, so hybrid and vector search work without
+configuration. If the user explicitly sets
+`provider = "none"`, requesting `hybrid` or `vector`
+returns a structured error with code
+`EMBEDDING_PROVIDER_NOT_CONFIGURED`.
 
 **Stale vector index (model mismatch).**
 When `vectors-meta.toml` model name does not match the
@@ -413,24 +441,25 @@ results until a rebuild is run.
 
 ## Provider Tradeoffs
 
-| Aspect        | ONNX (local)              | API (cloud)                  |
-|---------------|---------------------------|------------------------------|
-| Latency       | ~10 ms/embed              | ~100 ms/embed                |
-| Offline       | Yes                       | No                           |
-| Cost          | Free                      | Per-token                    |
-| Quality       | Good (384 dimensions)     | Excellent (1536+ dimensions) |
-| Setup         | Model download on first use | API key required            |
-| Dual runtime  | Deno + Node.js >= 20      | Deno + Node.js >= 20         |
-
-Both providers implement the same `EmbeddingProvider`
+| Aspect        | TF-IDF (built-in)         | ONNX (local)              | API (cloud)                  |
+|---------------|---------------------------|---------------------------|------------------------------|
+| Latency       | < 1 ms/embed              | ~10 ms/embed              | ~100 ms/embed                |
+| Offline       | Yes                       | Yes                       | No                           |
+| Cost          | Free                      | Free                      | Per-token                    |
+| Quality       | Adequate (256 dimensions) | Good (384 dimensions)     | Excellent (1536+ dimensions) |
+| Setup         | None (default)            | Model download on first use | API key required            |
+| Dependencies  | None                      | onnxruntime-node          | None                         |
+| Dual runtime  | Deno + Node.js >= 20      | Deno + Node.js >= 20      | Deno + Node.js >= 20         |
+
+All providers implement the same `EmbeddingProvider`
 interface. The daemon interacts only with the interface.
 
 ## Design Constraints
 
 | Constraint            | Details                                        |
 |-----------------------|------------------------------------------------|
-| WASM purity           | `query-parser.wasm` remains pure lexical computation with no I/O. No new WASM crate or export is added. |
+| WASM purity           | `query-parser.wasm` remains pure lexical computation with no I/O. A new standalone crate, `kbdb-default-embedding`, is added for TF-IDF embedding but has no host imports. Existing WASM modules are unchanged. |
 | Backward compatibility | Default `search_mode` is `lexical`. Callers that do not supply the parameter receive identical behaviour to today. |
 | File-based storage    | `vectors.idx` and `vectors-meta.toml` follow the same derived-and-rebuildable pattern as all other index files. |
 | Dual runtime          | `EmbeddingProvider` and all embedding logic must work on Deno and Node.js >= 20. No Node-specific or Deno-specific APIs in the implementation. |
-| No new WASM deps      | All new dependencies are TypeScript-side only. The WASM modules (`fs-database.wasm`, `query-parser.wasm`, `kb-worker.wasm`) are not recompiled for this feature. |
+| No new deps on existing WASM | The existing WASM modules (`fs-database.wasm`, `query-parser.wasm`, `kb-worker.wasm`) are not modified or recompiled. A new standalone module (`default-embedding.wasm`) is added, bringing the total to four WASM modules. |

package/docs/goals/mcp.md

@@ -27,25 +27,30 @@ identically.
 ## Invocation
 
 ```
-kbdb mcp [--db <path>]
+kbdb mcp [--db <path>] [--timeout-ms <ms>]
+   [--retry <n>] [--non-interactive]
 ```
 
-The `--db` flag specifies the absolute path to the
-`.kbdb` database directory. If omitted, the server
-reads the `KBDB_DIR` environment variable. One of the
-two must be provided -- the MCP server does not fall
-back to the current working directory.
+All flags are optional. `--db` specifies the path
+to the `.kbdb` database directory. If omitted, the
+server reads the `KBDB_DB_DIR` environment variable,
+then falls back to the current working directory.
 
 Resolution order:
 
 1. `--db <path>` command-line flag.
-2. `KBDB_DIR` environment variable.
-3. Error: no database directory specified.
+2. `KBDB_DB_DIR` environment variable.
+3. Current working directory.
 
-The MCP server does not fall back to the current
-working directory because the server's working
-directory is determined by the host application, not
-the user.
+### Environment Variables
+
+| Variable              | Description             |
+|-----------------------|-------------------------|
+| `KBDB_DB_DIR`         | Fallback database directory when `--db` is not specified |
+| `KBDB_DAEMON_TIMEOUT` | Request timeout fallback (ms); default 30 000 |
+| `KBDB_RETRY`          | Retry count fallback; default 5 |
+| `KBDB_LOG_LEVEL`      | Log verbosity: `debug`, `info`, `warning`, `error` |
+| `KBDB_NON_INTERACTIVE`| Suppress prompts (`1` or `true`) |
 
 See [CLI -- Run Modes](cli.md#run-modes) for how
 the `kbdb` executable dispatches to MCP mode.
@@ -127,7 +132,7 @@ Add to `.claude/settings.json` or project settings:
 
 ### Environment Variable
 
-The `--db` flag can be replaced by `KBDB_DIR`:
+The `--db` flag can be replaced by `KBDB_DB_DIR`:
 
 ```json
 {
@@ -136,7 +141,7 @@ The `--db` flag can be replaced by `KBDB_DIR`:
          "command": "npx",
          "args": ["@dikolab/kbdb", "mcp"],
          "env": {
-            "KBDB_DIR": "/path/to/.kbdb"
+            "KBDB_DB_DIR": "/path/to/.kbdb"
          }
       }
    }
@@ -1125,7 +1130,7 @@ interface McpToolAnnotations {
 }
 ```
 
-Annotations for all 19 kbdb tools:
+Annotations for all 20 kbdb tools:
 
 | Tool           | title             | readOnly | destructive | idempotent | openWorld |
 |----------------|-------------------|----------|-------------|------------|-----------|
@@ -1148,6 +1153,7 @@ Annotations for all 19 kbdb tools:
 | `agent-list`   | List Agents       | true     | false       | true       | false     |
 | `agent-get`    | Get Agent         | true     | false       | true       | false     |
 | `agent-delete` | Delete Agent      | false    | true        | true       | false     |
+| `auto-capture-review` | Review Captures | false | false    | false      | false     |
 
 ## Completions
 
@@ -1361,7 +1367,7 @@ The difference is the interface:
 | Interface | Command-line arguments | MCP protocol (stdio)   |
 | Consumer  | Human / shell script   | AI agent               |
 | Output    | stdout (JSON/text/MCP) | MCP tool result format |
-| DB path   | CWD or `--db` flag     | `--db` flag or `KBDB_DIR` env |
+| DB path   | CWD or `--db` flag     | `--db` flag or `KBDB_DB_DIR` env |
 
 See [CLI](cli.md) for CLI mode details,
 [Worker Client](worker-client.md) for the shared
@@ -1513,9 +1519,8 @@ returns `-32601` if they are called.
 - The server works with both Deno and Node.js
   runtimes, consistent with the project's
   dual-runtime requirement.
-- The database path must be explicitly configured
-  via `--db` or `KBDB_DIR`. No working directory
-  fallback.
+- The database path is resolved from `--db`,
+  `KBDB_DB_DIR`, or the current working directory.
 - The MCP server does not perform keyword extraction,
   indexing, ranking, or database I/O itself. All
   processing is delegated to the worker daemon via

package/docs/goals/overview.md

@@ -114,7 +114,7 @@ for both the Rust engine and the TypeScript layer.
   lands in the TypeScript source tree, and gets
   bundled into `dist/`. Start here.
 - **[Rust/WASM Modules](../modules/rust/overview.md)**
-  -- Cargo workspace, 4 crates, module namespaces,
+  -- Cargo workspace, 5 crates, module namespaces,
   WASM exports/imports, size optimization, and
   testing.
 - **[TypeScript Modules](../modules/typescript/overview.md)**

package/docs/goals/query-parser.md

@@ -193,6 +193,14 @@ hardcoded. Future versions may allow user-defined
 synonym maps via a `synonyms.toml` configuration file
 in the `.kbdb` directory.
 
+**Implementation status:** Synonym expansion is
+implemented in the Rust pipeline
+(`shared/src/pipeline/synonyms.rs`). The
+built-in map covers common technical synonyms.
+The `query-guidance` default skill complements this
+by instructing AI agents to rephrase queries with
+alternative terms before searching.
+
 ## Query Parsing
 
 Search queries are parsed into a structured query plan

package/docs/goals/semantic-dedup.md

@@ -58,9 +58,13 @@ with no further processing.
 
 ### Pass 2: Cosine (when embeddings available)
 
-Pass 2 runs only when an `EmbeddingProvider` other than
-`NullEmbeddingProvider` is active and
-`vectors.idx` is present and not stale.
+Pass 2 runs when the active `EmbeddingProvider` has
+non-zero dimensions and `vectors.idx` is present and
+not stale. With the built-in `TfidfEmbeddingProvider`
+as the default (256 dimensions), Pass 2 runs
+automatically without configuration. It is skipped
+only when embeddings are explicitly disabled
+(`provider = "none"` in `worker.toml`).
 
 For each gray-zone candidate from Pass 1, retrieve the
 candidate's embedding from `vectors.idx`. If the

package/docs/goals/worker-client.md

@@ -270,7 +270,9 @@ The spawning sequence:
    daemon started.
 4. **On PID file found (success):** worker client
    closes the stderr pipe, detaches the child, unrefs
-   it, and connects to the IPC socket.
+   it, and connects to the IPC socket. Because the
+   daemon writes the PID file only after the socket
+   is listening, the client can connect immediately.
 5. **On timeout or child exit (failure):** worker
    client reads all buffered stderr from the daemon,
    includes it in the thrown error (see Startup
@@ -337,6 +339,65 @@ error reason even though the daemon runs as a separate
 detached process. The daemon's stderr is the single
 source of truth for startup failure diagnostics.
 
+## Connection Retry
+
+When the worker client calls `connect()`, transient
+errors (`ENOENT` -- socket not yet created, or
+`ECONNREFUSED` -- server not yet accepting) are
+retried with exponential backoff:
+
+- **Max retries**: 5
+- **Initial delay**: 50 ms
+- **Backoff**: doubles each attempt (50, 100, 200,
+  400, 800 ms = ~1.55 s worst case)
+- **Non-transient errors** (`EACCES`, `EINVAL`, etc.)
+  fail immediately without retry.
+
+This covers the narrow window where the PID file
+exists but the socket has not yet finished binding
+(e.g. under heavy load or slow I/O).
+
+## Post-Connect Failure Handling
+
+After a successful `connect()`, if the daemon crashes
+or the IPC socket dies:
+
+1. All pending RPC calls (in `this.pending`) are
+   rejected with a descriptive error.
+2. The client marks itself as disconnected
+   (`this.socket = null`).
+3. The socket is destroyed.
+
+The `isConnected` getter exposes the connection
+state so callers (and the cached client factory) can
+check before reusing a client.
+
+## Cached Client Factory
+
+`getOrCreateClient(options?)` returns a cached
+`WorkerClient` for the given context path, or creates
+and caches a new one if the cached client is
+disconnected or absent.
+
+```ts
+import { getOrCreateClient } from '@dikolab/kbdb';
+
+const client = await getOrCreateClient({
+   dbPath: '/path/to/project',
+});
+```
+
+Intended for library consumers and the MCP server.
+CLI one-shots should use `createWorkerClient()`
+directly (each is a separate process anyway).
+
+## Socket Keepalive
+
+Both the client and server enable `SO_KEEPALIVE`
+with a 30-second interval. This ensures both ends
+detect a dead peer within ~30 seconds, even if no
+application-level traffic is flowing.
+
 ## `createWorkerClient(options?)`
 
 Factory function. Returns a connected `WorkerClient`

package/docs/goals/worker-daemon.md

@@ -14,8 +14,9 @@ retrieval and caching management. It is accessible
 
 The daemon:
 
-1. Loads and compiles the `kb-worker` WASM module once
-   at startup.
+1. Loads and compiles the WASM modules once at startup
+   (`kb-worker` for database operations, `embedding`
+   for default TF-IDF vector computation).
 2. Holds indexes and content objects in an in-memory
    cache managed by the WASM module.
 3. Listens for IPC requests from worker client
@@ -163,18 +164,21 @@ daemon process performs the following startup sequence:
    to startup. The health status (`integrity_ok`,
    `integrity_errors`) is included in subsequent
    `status` responses.
-7. Writes its PID to `kbdb-{ctx}.pid` in the OS
-   temporary directory.
+7. Removes the stale IPC socket if present.
 8. Creates the IPC socket (`kbdb-{ctx}.sock` on Unix,
-   named pipe on Windows).
-9. Begins listening for connections.
+   named pipe on Windows) and begins listening
+   (async, with backlog 256).
+9. Writes its PID to `kbdb-{ctx}.pid` in the OS
+   temporary directory.
 
-The daemon only writes its PID file and creates the
-IPC socket after all preconditions pass and the WASM
-module is loaded. If the daemon exits before step 7,
-the worker client's PID-file poll times out, and the
-worker client reads the crash log (or the daemon's
-stderr) to report the failure reason.
+The daemon writes its PID file only after the IPC
+socket is bound and listening. This guarantees the
+worker client's PID-file poll succeeds only when
+the socket is ready to accept connections. If the
+daemon exits before step 9, the worker client's
+PID-file poll times out, and the worker client
+reads the crash log (or the daemon's stderr) to
+report the failure reason.
 
 ### Startup Preconditions
 
@@ -273,6 +277,14 @@ a client connects, disconnects, or sends a request.
 The daemon stays alive as long as at least one client
 is connected, regardless of the idle timer.
 
+`SO_KEEPALIVE` (30-second interval) provides a
+secondary mechanism for detecting dead connections
+beyond the counter-based idle timer. If a client
+process crashes without closing the socket, the OS
+will detect the dead peer within ~30 seconds and
+close the connection, allowing the daemon to
+decrement its counter and eventually idle-shutdown.
+
 ## Graceful Shutdown
 
 On shutdown (idle timeout or OS signal SIGTERM/SIGINT),
@@ -288,24 +300,28 @@ the daemon:
    exists.
 7. Calls `process.exit(0)`.
 
-## WASM Module
+## WASM Modules
 
-The daemon loads a single WASM module: `kb-worker.wasm`
-(file: `kbdb_worker_bg.wasm`). This module statically
-links the `fs-database` and `query-parser` crates, so
-only one module needs to be compiled and instantiated.
+The daemon loads four WASM modules at startup. Three
+share a single `WebAssembly.Memory` and are wired
+together via imports; the fourth is a standalone
+pure-computation module.
 
 ```
-kbdb_worker_bg.wasm
-  (statically links fs-database + query-parser)
+query-parser.wasm  (standalone)
+fs-database.wasm   (standalone)
+default-embedding.wasm (standalone, no host imports)
+       ↓
+kb-worker.wasm
+  (imports query-parser + fs-database)
        ↕
   host I/O imports (TypeScript)
 ```
 
 ### WASM Module: kb-worker.wasm
 
-`kb-worker.wasm` is the single module loaded by the
-daemon. Its responsibilities:
+`kb-worker.wasm` is the orchestration module. Its
+responsibilities:
 
 - **Query orchestration** -- Parses queries, performs
   index lookups, scores and ranks results.
@@ -447,6 +463,26 @@ Configuration is set once when the daemon spawns and
 does not change until the daemon shuts down and a new
 one is spawned.
 
+### Client vs Daemon Timeouts
+
+The `KBDB_DAEMON_TIMEOUT` environment variable (or
+`--timeout-ms` CLI flag) controls the **client-side
+request timeout** -- how long a CLI or MCP client
+waits for a response from the daemon before giving
+up and optionally retrying.
+
+The `timeout_ms` setting in `worker.toml` controls
+the **daemon-side idle timeout** -- how long the
+daemon stays alive after all client connections have
+closed.
+
+These are independent. A long-running query may
+exceed the client request timeout while the daemon
+continues processing. Conversely, a short idle
+timeout shuts the daemon down quickly between
+invocations but has no effect on in-flight
+requests.
+
 ## IPC Transport
 
 The daemon listens on a local IPC channel:
@@ -471,6 +507,17 @@ No third-party IPC library is required. The daemon
 creates the IPC socket at startup and removes it on
 shutdown.
 
+### Socket Options
+
+- **Backlog**: `256`. Handles connection bursts from
+  concurrent CLI invocations and benchmarks without
+  dropping connections.
+- **SO_KEEPALIVE**: Enabled on every accepted
+  connection with a 30-second interval. Detects dead
+  clients without relying solely on the idle timer.
+  Both the server and client enable keepalive (see
+  [Worker Client](worker-client.md)).
+
 ## IPC Protocol
 
 Communication uses **JSON-RPC 2.0** over the IPC
@@ -1481,14 +1528,20 @@ The worker daemon (`src/worker.ts` and its compiled
 `kbdb-worker.js`) ships as part of the npm/JSR package.
 No per-platform native binaries are required.
 
-The package includes one WASM module:
-
-- `kb-worker/kbdb_worker_bg.wasm` -- single module
-  that statically links the query-parser and
-  fs-database crates
-
-It is compiled from Rust via `wasm-pack` and works on
-all platforms. The daemon loads and compiles it at
-startup using the runtime's `WebAssembly` API. The
+The package includes four WASM modules:
+
+- `kb-worker/kbdb_worker_bg.wasm` -- orchestration
+  module that imports query-parser and fs-database
+- `query-parser/query_parser_bg.wasm` -- tokenization
+  and query parsing
+- `fs-database/fs_database_bg.wasm` -- filesystem
+  storage engine
+- `default-embedding/kbdb_default_embedding_bg.wasm`
+  -- built-in TF-IDF embedding (standalone, no host
+  imports)
+
+All are compiled from Rust via `wasm-pack` and work
+on all platforms. The daemon loads and compiles them
+at startup using the runtime's `WebAssembly` API. The
 daemon runs on the same runtime (Node.js or Deno) as
 the worker client that spawns it.