@dikolab/kbdb 0.3.0 → 0.3.1

package/docs/goals/auto-capture.md

@@ -0,0 +1,378 @@
+# Auto-Capture -- Proactive Knowledge Capture Hooks
+
+*Server-side enforcement of knowledge capture that
+closes the gap between agent intent and reliable
+storage -- without requiring the agent to remember
+to act.*
+
+## Goal
+
+The `memory-protocol` skill (see [Skills](skills.md))
+teaches an agent what to store, but enforcement
+depends entirely on the agent choosing to act.
+Agents are unreliable at remembering to call `learn`
+after every significant exchange. Auto-capture closes
+this gap by having the MCP server proactively issue
+a sampling request to the host LLM, asking it to
+review recent context and identify storable knowledge.
+
+This is gap #2 from the
+[Opus 4.8 second brain review](../issues/2026-06-23-opus-48-second-brain-review.md):
+
+> A memory-protocol *skill* is intent, not
+> enforcement -- agents are unreliable at remembering
+> to write. Real second brains lean on auto-capture
+> hooks.
+
+Auto-capture is opt-in and default-disabled. It never
+writes to the database without the host LLM's
+explicit response identifying items to store.
+
+## MCP Sampling Approach
+
+Auto-capture uses the MCP `sampling/createMessage`
+capability. When a trigger fires, the kbdb MCP server
+sends a `sampling/createMessage` request to the host
+asking it to identify storable knowledge from the
+current conversation context.
+
+If the host's `initialize` response does not include
+a `sampling` capability entry, auto-capture is
+silently disabled for that session regardless of
+configuration.
+
+### Sampling Prompt Template
+
+```
+Review the recent conversation context. Identify any
+durable facts, decisions, corrections, architecture
+choices, or configuration details that should be
+stored in the knowledge base for future sessions.
+
+For each item, provide:
+- title: short heading
+- content: the fact or decision
+- tags: relevant topic tags
+
+Respond with JSON: {"items": [...]} or {"items": []}
+if nothing is worth storing. Apply the same criteria
+as the memory-protocol skill: store durable facts,
+not transient context.
+```
+
+The server sets `maxTokens` to a fixed ceiling (e.g.
+2048) to bound cost. The sampling request carries no
+`systemPrompt` -- context comes from the host's own
+conversation window.
+
+### Sampling Response Schema
+
+The host LLM must respond with a JSON object at the
+top level. The server parses the text content of the
+sampling response and extracts:
+
+```json
+{
+   "items": [
+      {
+         "title": "Postgres chosen for JSONB support",
+         "content": "We chose Postgres because JSONB
+            support simplifies document storage.",
+         "tags": ["architecture", "database"]
+      }
+   ]
+}
+```
+
+When `items` is empty or the response cannot be parsed
+as valid JSON, the server takes no action and logs a
+`debug`-level notification.
+
+Each item in `items` is stored via `addSection` with
+`sectionType: 'text/markdown'` and the
+`auto-capture` tag appended to `tags`.
+
+## Trigger Points
+
+Three independent trigger mechanisms fire a capture
+review. All three can be active simultaneously.
+
+### 1. Tool-Call Count Trigger
+
+The MCP server maintains a per-session counter of
+tool invocations. When the count reaches a multiple
+of `trigger_interval`, a capture review fires.
+
+- The counter increments on every `tools/call`
+  dispatch, including read-only tools.
+- The counter does not increment during the capture
+  review itself (no recursion).
+- The review fires after the triggering tool call
+  returns its result to the agent, not before.
+
+Default: every 10 tool calls.
+
+### 2. Explicit Tool Trigger
+
+The `auto-capture-review` tool is always registered,
+regardless of whether auto-capture is enabled in
+`worker.toml`. Agents or users can call it at any
+time to request an immediate capture review.
+
+When `dry_run: true` is passed, the review runs but
+no sections are written. See [MCP Surface](#mcp-surface).
+
+### 3. Session-End Trigger
+
+When the MCP connection closes gracefully (stdin EOF
+or explicit disconnect), one final capture review
+runs before the server exits.
+
+The session-end trigger fires only when
+`auto_capture.enabled` is `true`. It does not fire
+when the connection is lost abruptly (SIGKILL, socket
+drop).
+
+## Configuration
+
+Auto-capture is configured in `worker.toml` under the
+`[auto_capture]` table:
+
+```toml
+[auto_capture]
+enabled = false          # opt-in, default off
+trigger_interval = 10    # tool calls between reviews
+max_captures_per_session = 20  # rate limit
+dry_run = false          # log without writing
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `enabled` | bool | `false` | Enable auto-capture for all MCP sessions |
+| `trigger_interval` | integer | `10` | Tool calls between automatic reviews |
+| `max_captures_per_session` | integer | `20` | Maximum sections stored per session |
+| `dry_run` | bool | `false` | Log what would be stored; write nothing |
+
+When `enabled` is `false`, only the `auto-capture-review`
+tool remains active. The tool-call count and
+session-end triggers are suppressed.
+
+## Guard Rails
+
+### Rate Limiting
+
+The server tracks the number of sections stored
+during the current MCP session. When
+`max_captures_per_session` is reached, further
+automatic reviews are skipped. The
+`auto-capture-review` tool still runs but returns a
+rate-limit message instead of executing the review.
+
+The counter resets when the MCP session ends. It is
+not persisted across sessions.
+
+### Dry-Run Mode
+
+When `dry_run` is `true` (either from `worker.toml`
+or the `auto-capture-review` tool parameter), the
+server executes the full review -- sends the sampling
+request, parses the response, and evaluates items --
+but does not call `addSection`. Instead, it emits a
+`notifications/message` at `info` level describing
+what would have been stored.
+
+Dry-run mode is useful for tuning `trigger_interval`
+and verifying that the sampling prompt produces
+reasonable output before enabling writes.
+
+### Deduplication
+
+Before storing each item from the sampling response,
+the server searches the database for existing sections
+with similar content. If any existing section has a
+Jaccard similarity >= 0.85 to the candidate item, the
+item is skipped and logged at `debug` level.
+
+This uses the same near-duplicate detection that the
+`learn` tool uses, reflected in the `near_duplicates`
+field of `AddSectionResult`.
+
+### No Recursion
+
+The tool-call counter does not increment during an
+active capture review. The `addSection` calls made
+by auto-capture do not trigger a nested review.
+
+### Agent Override
+
+Auto-capture never overrides an agent's explicit
+decision not to store something. If an agent
+calls `unlearn` to remove a section, auto-capture
+will not re-add it in the same session (the content's
+kbid is added to a per-session exclusion set after
+removal).
+
+## Interaction with the memory-protocol Skill
+
+| Aspect | memory-protocol | Auto-capture |
+|--------|-----------------|--------------|
+| Audience | Agent (reads skill text) | MCP server (server-side) |
+| Mechanism | Teaches filtering criteria | Sends sampling request |
+| Enforcement | Agent must choose to act | Server acts automatically |
+| Criteria | Defined in skill body | Sampling prompt mirrors skill |
+| Override | Agent decides | Agent's `unlearn` is respected |
+
+Both mechanisms apply the same filtering criteria --
+store durable facts, skip transient context. They are
+complementary: `memory-protocol` is agent-facing
+intent; auto-capture is server-side enforcement.
+
+## CLI Surface
+
+The `--auto-capture` flag on `kbdb mcp` enables
+auto-capture for the current session, overriding the
+`enabled` value in `worker.toml`:
+
+```
+kbdb mcp --db <path> [--auto-capture]
+```
+
+The flag does not accept a value -- its presence sets
+`enabled: true`. It does not override other fields
+(`trigger_interval`, `max_captures_per_session`,
+`dry_run`), which continue to come from `worker.toml`
+or their defaults.
+
+## MCP Surface
+
+### auto-capture-review Tool
+
+Triggers an immediate capture review regardless of
+whether auto-capture is enabled in configuration.
+
+**Parameters:**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `dry_run` | boolean | no | When `true`, evaluate but do not write (default: `false`) |
+
+**Returns:**
+
+```json
+{
+   "captured": 2,
+   "items": [
+      {
+         "title": "Postgres chosen for JSONB support",
+         "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3"
+      }
+   ],
+   "skipped": 1,
+   "dry_run": false
+}
+```
+
+| Field | Description |
+|-------|-------------|
+| `captured` | Number of sections written (0 when `dry_run: true`) |
+| `items` | Array of `{ title, kbid }` for each section stored |
+| `skipped` | Items returned by LLM but not stored (duplicates, parse errors, rate limit) |
+| `dry_run` | Reflects the effective dry-run mode used |
+
+When sampling is not supported by the host, the tool
+returns `isError: true` with message:
+`"Host does not support sampling; auto-capture-review
+unavailable"`.
+
+**Tool annotation:**
+
+| Attribute | Value |
+|-----------|-------|
+| `readOnlyHint` | `false` |
+| `destructiveHint` | `false` |
+| `idempotentHint` | `false` |
+| `openWorldHint` | `false` |
+
+### Capability Advertisement
+
+When `sampling` is present in the host's `initialize`
+capabilities, the server advertises the `sampling`
+capability in its own `initialize` response. When
+`sampling` is absent, the capability is omitted from
+the response and auto-capture is silently disabled.
+
+## Response Flow
+
+```
+Agent uses kbdb tools
+        |
+        v
+Tool call count reaches N
+        |
+        v
+Server sends sampling/createMessage
+        |
+        v
+Host LLM reviews conversation context
+        |
+        v
+Returns {"items": [...]} or {"items": []}
+        |
+        v
+Server deduplicates against existing sections
+        |
+        v
+Server stores each new item via addSection
+(with "auto-capture" tag appended)
+        |
+        v
+notifications/message logged (info level)
+```
+
+When the host does not support sampling, or when
+`enabled` is `false` and the trigger is automatic
+(not the explicit tool), the flow terminates after
+the trigger fires with a `debug`-level log and no
+sampling request is sent.
+
+## Tagging
+
+Every section stored by auto-capture receives the
+`auto-capture` tag appended to any tags the sampling
+response supplied. This makes auto-captured sections
+filterable and identifiable in search results and
+integrity checks.
+
+Example stored section tags when the sampling
+response supplies `["architecture", "database"]`:
+
+```
+["architecture", "database", "auto-capture"]
+```
+
+## Design Constraints
+
+- **Opt-in by default.** `enabled` is `false` in
+  `worker.toml` until explicitly set. No sections are
+  written without explicit configuration or the
+  `--auto-capture` flag.
+- **Sampling required.** Auto-capture cannot function
+  without MCP sampling support. It is silently
+  disabled when the host omits the `sampling`
+  capability, with no error and no user-visible
+  change to other tools.
+- **No recursion.** Auto-capture calls do not
+  increment the tool-call counter and do not trigger
+  nested reviews.
+- **Agent override respected.** Sections removed by
+  an explicit `unlearn` call are not re-added by
+  auto-capture within the same session.
+- **Rate-limited.** `max_captures_per_session`
+  prevents runaway storage from a misconfigured
+  trigger interval or a verbose LLM response.
+- **Traceability.** Auto-captured sections always
+  carry the `auto-capture` tag so they can be
+  distinguished from manually learned content.
+- **Dry-run available.** `dry_run` mode allows
+  operators to verify sampling output and tune
+  configuration before enabling writes.

package/docs/goals/benchmarking.md

@@ -0,0 +1,296 @@
+# Scale Benchmarking -- Methodology and Performance Targets
+
+*Quantify search latency, rebuild time, and memory
+usage across realistic corpus sizes before trusting
+kbdb as a long-lived knowledge base.*
+
+## Goal
+
+kbdb is intended as a long-lived knowledge base for AI
+agents. Before trusting it at scale, search latency,
+rebuild time, and memory usage must be quantified across
+realistic corpus sizes.
+
+This document specifies the benchmark methodology, corpus
+generation, metrics, and regression targets. It addresses
+gap #4 from the
+[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md):
+
+> Scale of the lexical index at 50k-100k+ sections --
+> search latency and rebuild time are worth benchmarking
+> before trusting it as a long-lived brain.
+
+## Corpus Generation
+
+A synthetic corpus generator lives in `bench/` and
+creates realistic sections for repeatable benchmarks.
+
+### Generator Properties
+
+- **Term distribution** -- Word frequencies follow a
+  Zipfian distribution. A small vocabulary of high-
+  frequency terms dominates each section, with a long
+  tail of rare terms. This matches real-world prose.
+- **Title length** -- 3 to 8 words drawn from the
+  vocabulary.
+- **Body length** -- 100 to 500 words, configurable via
+  `avg body length` parameter.
+- **Vocabulary** -- Terms drawn from a 50,000-word
+  English dictionary subset. The same dictionary is used
+  for all corpus sizes so cross-size comparisons are
+  valid.
+- **Cross-references** -- Sections reference each other
+  via `$(link:)` markers at a configurable density
+  (default: 5% of sections link to one other section).
+- **Tags** -- Each section is assigned one to three tags
+  from a pool of 20 topic tags.
+- **Determinism** -- The generator accepts a seed
+  parameter. Fixed seed produces identical corpus across
+  runs. Results are reproducible across machines.
+
+### Generator Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `--count` | 1000 | Number of sections to generate |
+| `--avg-body-length` | 250 | Average body word count |
+| `--ref-density` | 0.05 | Fraction of sections with a cross-reference |
+| `--seed` | 42 | RNG seed for reproducibility |
+| `--out` | `./bench/corpus/` | Output directory |
+
+## Corpus Sizes
+
+| Label | Sections | Estimated disk size |
+|-------|----------|---------------------|
+| Small | 1,000 | ~5 MB |
+| Medium | 10,000 | ~50 MB |
+| Large | 50,000 | ~250 MB |
+| XLarge | 100,000 | ~500 MB |
+
+All corpus sizes use the same generator with the same
+seed and vocabulary. The only difference is the
+`--count` parameter.
+
+## Operations to Benchmark
+
+| Operation | Description | Variants |
+|-----------|-------------|----------|
+| `search` | Single-query search | cold cache, warm cache |
+| `learn` | Add a single section | with/without embedding |
+| `batch-learn` | Add 100 sections | sequential |
+| `rebuild` | Full index rebuild | lexical only, hybrid |
+| `gc` | Garbage collection | after superseding 10% of sections |
+| `check` | Integrity check | full scan |
+
+### Variant Definitions
+
+**Cold cache** -- the worker daemon is started fresh
+immediately before the timed operation. The WASM
+linear memory page table and any in-process caches
+are empty.
+
+**Warm cache** -- the timed operation is preceded by
+10 identical operations that are not counted. The
+daemon's in-process caches are populated.
+
+**With embedding** -- an embedding provider is
+configured. The `learn` operation includes embedding
+computation time.
+
+**After superseding 10%** -- before timing `gc`, 10%
+of all sections are superseded (replaced) using the
+`replace` flag, leaving their originals as unreferenced
+candidates for collection.
+
+## Metrics
+
+For each operation at each corpus size, the harness
+collects:
+
+- **p50, p95, p99 latency** (ms) -- percentiles across
+  N runs (default N = 10).
+- **Throughput** (ops/sec) -- for batch operations
+  (`batch-learn`), calculated as
+  `(sections_per_batch * N) / total_elapsed_s`.
+- **WASM linear memory high watermark** -- the
+  `WebAssembly.Memory` buffer byte size sampled
+  immediately after the operation completes.
+- **Index file sizes** -- sizes in bytes of
+  `terms.idx`, `positions.idx`, `corpus.dat`, and
+  `vectors.idx` (if present) at each corpus size.
+- **Resident memory of the worker daemon** -- OS-
+  reported RSS of the `kbdb-worker` process, sampled
+  after warm-up.
+
+## Performance Targets
+
+The targets below define the maximum acceptable latency
+for each operation at each corpus size. Results
+exceeding a target mark that combination as a
+regression.
+
+| Operation | Corpus | Target |
+|-----------|--------|--------|
+| `search` (warm) | 1k | < 5 ms |
+| `search` (warm) | 10k | < 15 ms |
+| `search` (warm) | 50k | < 30 ms |
+| `search` (warm) | 100k | < 50 ms |
+| `search` (cold) | 100k | < 200 ms |
+| `learn` (single) | any | < 100 ms |
+| `rebuild` | 1k | < 2 s |
+| `rebuild` | 10k | < 10 s |
+| `rebuild` | 100k | < 30 s |
+| `gc` | 100k | < 10 s |
+| `check` | 100k | < 15 s |
+
+Targets are per-operation p99 latency unless noted.
+`learn (single)` and `rebuild` targets apply to p99
+of the N-run sample.
+
+## Benchmark Harness
+
+All benchmark tooling lives in `bench/`:
+
+```
+bench/
+   generate-corpus.ts    corpus generator script
+   run-benchmark.ts      harness: run operations,
+                          collect timing, emit JSON
+   report.ts             format JSON results as
+                          Markdown table
+   corpus/               generated corpus files
+                          (gitignored)
+   results/              timestamped JSON result files
+   baselines/            per-platform baseline files
+                          for regression comparison
+```
+
+### Harness Procedure
+
+1. Generate the corpus at target size using
+   `bench/generate-corpus.ts` with the configured
+   seed.
+2. Run `db init` to create a fresh `.kbdb` directory.
+3. Run `batch-learn` to populate the database with the
+   full generated corpus.
+4. For each operation variant, run the operation N
+   times (default N = 10), collecting wall-clock
+   elapsed time per run.
+5. Compute p50, p95, p99 from the N samples.
+6. Collect WASM memory and daemon RSS after each
+   operation.
+7. Write a JSON result file to `bench/results/` with
+   a timestamp suffix.
+8. Optionally compare against the baseline file in
+   `bench/baselines/` and emit a pass/fail summary.
+
+The harness uses the public CLI and library API
+exclusively. It does not import internal modules or
+call WASM exports directly.
+
+### Result File Format
+
+Each run produces one JSON file:
+
+```json
+{
+  "timestamp": "2026-06-23T10:00:00Z",
+  "platform": "linux-x86_64",
+  "runtime": "node-22",
+  "corpus_size": "medium",
+  "results": [
+    {
+      "operation": "search",
+      "variant": "warm",
+      "n": 10,
+      "p50_ms": 8,
+      "p95_ms": 11,
+      "p99_ms": 14,
+      "wasm_memory_bytes": 4194304,
+      "daemon_rss_bytes": 52428800
+    }
+  ],
+  "index_sizes": {
+    "terms_idx_bytes": 1048576,
+    "positions_idx_bytes": 524288,
+    "corpus_dat_bytes": 65536,
+    "vectors_idx_bytes": null
+  }
+}
+```
+
+`vectors_idx_bytes` is `null` when no embedding
+provider is configured.
+
+## Hybrid Search Benchmarks
+
+When an embedding provider is configured, the harness
+adds measurements for the vector path. These are
+collected as a separate benchmark suite and not mixed
+with the lexical-only results.
+
+| Additional measurement | Description |
+|------------------------|-------------|
+| Embedding time per section | Elapsed ms for embedding computation during `learn` |
+| Vector search latency | p50/p95/p99 of vector-only retrieval |
+| Lexical search latency | p50/p95/p99 of lexical-only retrieval for the same queries |
+| RRF fusion overhead | p50/p95/p99 of the fusion step alone (subtract lexical + vector from total) |
+| Vector index rebuild time | p50/p95/p99 of `rebuild` with embedding enabled |
+| `vectors.idx` size | File size in bytes at each corpus size |
+
+Hybrid benchmarks require a running embedding provider.
+The harness accepts `--embedding-provider` as a CLI
+flag. When the flag is absent, hybrid measurements are
+skipped and the harness prints a notice.
+
+## Regression Testing
+
+Benchmark results from the current release are stored
+in `bench/baselines/`. Baseline files are named by
+platform and runtime:
+
+```
+bench/baselines/
+   linux-x86_64-node-22.json
+   linux-x86_64-deno-2.json
+```
+
+The harness compares fresh results against the
+baseline and flags any p99 value that exceeds the
+baseline by more than 20%. A flagged regression prints
+to stderr and exits with a non-zero code.
+
+Baselines are updated manually when a performance
+improvement is intentional. The update command:
+
+```sh
+node bench/run-benchmark.ts --update-baseline
+```
+
+CI runs the full benchmark suite on every release
+branch push. Regression failures block the release.
+
+## Design Constraints
+
+- Benchmarks run on a single machine. No distributed
+  or multi-node testing is in scope.
+- Corpus generation is deterministic. A fixed seed
+  always produces the same sections in the same order.
+  Results are reproducible across machines with the
+  same platform and runtime.
+- The harness uses the public CLI and library API, not
+  internal modules. This ensures benchmarks measure the
+  end-to-end path that production callers experience.
+- WASM memory measurement uses the
+  `WebAssembly.Memory` buffer byte size sampled via
+  the daemon's status response, which exposes the
+  current WASM linear memory page count. This is an
+  approximation of the high watermark, not a
+  continuous trace.
+- Index file sizes are measured after the harness
+  operation completes by reading file sizes from the
+  `.kbdb/indexes/` directory. They are informational
+  and not subject to regression thresholds.
+- Baseline files are per-platform and per-runtime.
+  A Linux x86_64 Node.js baseline is not compared
+  against a Deno run or a different architecture.