@dikolab/kbdb 0.3.0 → 0.3.2

package/docs/goals/semantic-dedup.md

@@ -0,0 +1,229 @@
+# Semantic Near-Duplicate Detection
+
+*Cosine similarity on embeddings as a complementary
+detection layer alongside Jaccard -- closes the
+vocabulary-gap ceiling identified in gap #6.*
+
+## Goal
+
+The existing near-duplicate detection computes a Jaccard
+coefficient on stemmed term sets of the incoming section
+versus all indexed sections. Sections that exceed a
+similarity threshold of 0.85 are reported as
+near-duplicates in `AddSectionResult.near_duplicates`.
+
+This mechanism catches lexically similar content
+reliably. It does not catch semantically equivalent
+sections that use different vocabulary. Example: a
+section stored as "We selected PostgreSQL for its JSONB
+support" and a later section written as "Chose Postgres
+due to native JSON column type" express the same
+decision. Their stemmed term sets share few tokens --
+Jaccard produces a low score, no duplicate is flagged,
+and the knowledge base silently accumulates redundant
+content.
+
+This gap is identified as the semantic dedup ceiling in
+[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md)
+(gap #6). Semantic dedup closes it by adding a cosine
+similarity pass on embedding vectors as a complementary
+detection method when an embedding provider is
+configured. The feature is conditional on
+[Hybrid Search](hybrid-search.md) -- if no
+`EmbeddingProvider` is configured, only the Jaccard pass
+runs and behaviour is identical to today.
+
+## Two-Pass Detection Algorithm
+
+Detection proceeds in two sequential passes on every
+`add_section` call.
+
+### Pass 1: Jaccard (always runs)
+
+Compute the Jaccard coefficient between the incoming
+section's stemmed term set and each existing section's
+term set. The term sets are derived from the inverted
+index -- no section files are read.
+
+Sections whose Jaccard score meets or exceeds 0.85 are
+flagged as lexical near-duplicates and appended to
+`near_duplicates` with `method: "jaccard"`.
+
+Sections whose Jaccard score falls in the range
+[0.40, 0.85) -- the gray zone -- are retained as
+candidates for Pass 2.
+
+Sections with a Jaccard score below 0.40 are discarded
+with no further processing.
+
+### Pass 2: Cosine (when embeddings available)
+
+Pass 2 runs only when an `EmbeddingProvider` other than
+`NullEmbeddingProvider` is active and
+`vectors.idx` is present and not stale.
+
+For each gray-zone candidate from Pass 1, retrieve the
+candidate's embedding from `vectors.idx`. If the
+candidate's embedding is missing (e.g. the embedding
+provider failed when that section was learned), skip
+that candidate silently. Compute cosine similarity
+between the incoming section's embedding and the
+candidate's embedding.
+
+The incoming section's embedding is computed by the
+daemon as part of the same learn operation that triggers
+the add (see the embedding lifecycle in
+[Hybrid Search](hybrid-search.md)). If the embedding
+computation for the incoming section itself failed, Pass
+2 is skipped entirely for that call.
+
+Candidates whose cosine similarity meets or exceeds 0.92
+are flagged as semantic near-duplicates and appended to
+`near_duplicates` with `method: "cosine"`.
+
+### Efficiency Rationale
+
+Jaccard operates on integer set intersections over the
+inverted index -- it is cheap. Cosine operates on
+floating-point vector dot products and requires reading
+from `vectors.idx` -- it is more expensive. Restricting
+cosine to the gray zone (Jaccard >= 0.40) means cosine
+is invoked only for sections that already share moderate
+lexical overlap with the candidate. Sections with very
+low Jaccard scores are unlikely to be semantic
+duplicates, so skipping them avoids unnecessary work
+without meaningful loss in detection coverage.
+
+## Detection Thresholds
+
+| Method           | Threshold | Meaning                      |
+|------------------|-----------|------------------------------|
+| Jaccard          | >= 0.85   | Lexical near-duplicate       |
+| Jaccard pre-filter | >= 0.40 | Candidate for cosine pass    |
+| Cosine           | >= 0.92   | Semantic near-duplicate      |
+
+Thresholds are not configurable in v1. They are
+hardcoded constants. If operational experience shows
+that the defaults produce excessive false positives or
+false negatives, they can be exposed as `worker.toml`
+configuration keys in a later version.
+
+## Response Format
+
+The `near_duplicates` array in `AddSectionResult` gains
+a `method` field on every entry. The field identifies
+which detection pass produced the entry.
+
+```json
+{
+   "kbid": "<26-char hash>",
+   "near_duplicates": [
+      {
+         "kbid": "abc123...",
+         "similarity": 0.89,
+         "method": "jaccard"
+      },
+      {
+         "kbid": "def456...",
+         "similarity": 0.95,
+         "method": "cosine"
+      }
+   ]
+}
+```
+
+The `method` field is always present on every
+near-duplicate entry, including in responses from
+deployments where only Jaccard runs. Callers can rely
+on the field being present unconditionally.
+
+Entries produced by Pass 1 always carry `method:
+"jaccard"`. Entries produced by Pass 2 always carry
+`method: "cosine"`. No entry can carry both methods --
+a section that crosses the Jaccard threshold of 0.85
+is already flagged in Pass 1 and is not re-evaluated
+in Pass 2.
+
+## Dependency on Hybrid Search
+
+Semantic dedup reuses the infrastructure introduced by
+[Hybrid Search](hybrid-search.md) without adding any
+new infrastructure of its own.
+
+- The `EmbeddingProvider` interface defined in Hybrid
+  Search provides the embedding computation at learn
+  time.
+- `vectors.idx` provides the embedding lookup at
+  duplicate-detection time. No second embedding call is
+  made -- the stored vectors are read directly.
+- When `NullEmbeddingProvider` is active (no `[embedding]`
+  section in `worker.toml`), Pass 2 never runs. The
+  `near_duplicates` array may still contain Jaccard
+  entries; it will never contain cosine entries.
+- Stale vector index conditions (model name mismatch
+  between `vectors-meta.toml` and the configured
+  provider) are treated the same as a missing
+  `vectors.idx`: Pass 2 is skipped for that call.
+
+## Contradiction Detection Enhancement
+
+The existing `divergent_content` integrity check
+identifies sections that share the same `source_key`
+but have different content -- contradictory knowledge
+from the same origin that supersession should have
+resolved.
+
+With embeddings available, integrity check can extend
+this check to surface potential semantic contradictions
+that lack a shared source key: sections with high cosine
+similarity (>= 0.92) and conflicting content signals.
+This is a secondary, advisory output in the integrity
+check response -- it does not block any operation.
+
+The enhancement runs only when the integrity check is
+invoked and `vectors.idx` is present and not stale. It
+adds a new `semantic_contradictions` array to the
+integrity check response, alongside the existing
+`divergent_content` error entries:
+
+```json
+{
+   "semantic_contradictions": [
+      {
+         "kbid_a": "abc123...",
+         "kbid_b": "def456...",
+         "cosine_similarity": 0.94
+      }
+   ]
+}
+```
+
+When no embedding provider is configured or
+`vectors.idx` is absent, `semantic_contradictions` is
+an empty array.
+
+## Design Constraints
+
+- Semantic dedup is additive. It never replaces the
+  Jaccard pass. Both passes coexist, and the Jaccard
+  pass always runs regardless of embedding configuration.
+- The `method` field is always present on
+  `near_duplicates` entries, even in responses from
+  deployments where only Jaccard runs.
+- Cosine similarity computation reuses existing
+  embedding vectors stored in `vectors.idx`. No
+  additional embedding API call is made during
+  duplicate detection.
+- If Pass 2 is skipped (no provider, stale index, or
+  embedding failure for the incoming section), the
+  response omits cosine entries silently. No warning
+  field is added for a skipped cosine pass -- the
+  absence of cosine entries is not an error condition
+  and is expected in lexical-only deployments.
+- Thresholds are hardcoded in v1. Exposure as
+  configuration keys is deferred to a later version
+  after operational validation.
+- The `semantic_contradictions` enhancement to integrity
+  check is advisory. It does not change `ok` to `false`
+  on its own -- it provides supplementary information
+  for human review.

package/docs/goals/sync.md

@@ -0,0 +1,217 @@
+# Sync -- Cross-Machine Synchronization
+
+*Export the database as a git-friendly directory, move
+it with any transport, import on the other side.*
+
+## Goal
+
+kbdb is currently single-machine -- "persists across
+sessions" means across sessions on the same box. For
+teams, multi-device use, or backup, the database needs
+a sync story.
+
+The design uses export/import as the primitive, with
+git as the recommended transport. Sync is manual and
+explicit. There is no background replication.
+
+This goal addresses gap #5 from
+[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md):
+"No sync/remote story unless you build one on top,
+which limits 'persists across sessions' to 'across
+sessions on this box.'"
+
+## Sync Primitives
+
+Two operations form the sync foundation.
+
+### Export
+
+Serialize the database into a portable, git-friendly
+directory:
+
+```
+kbdb export --format git --db .kbdb --out ./kbdb-sync/
+```
+
+The exported directory mirrors `.kbdb/` minus indexes:
+
+- `catalog.toml` -- database table of contents
+- `sections/**/*.section` -- all section files
+- `documents/**/*.toml` -- all document manifests
+
+Excluded:
+
+- `indexes/` -- derived data, rebuildable
+- Vector indexes (`vectors.idx`, `vectors-meta.toml`)
+- PID files and lock files
+
+The exported directory is a valid git subtree. It
+contains no binary blobs except section content
+(image sections, which are stored as base64 text).
+Every file is human-readable and git-diffable.
+
+### Import
+
+Restore a database from an exported directory:
+
+```
+kbdb import --from ./kbdb-sync/ --db .kbdb
+```
+
+Import reads the exported directory and writes sections
+and documents into the target database. After import,
+run `rebuild` to regenerate all indexes (lexical and
+vector, if configured).
+
+Import is **additive** -- existing sections with
+different kbids are preserved. Only the conflict
+resolution rules below can cause existing sections to
+be superseded.
+
+## Conflict Resolution
+
+When importing sections that conflict with existing
+ones, the following rules apply in priority order:
+
+| Condition | Action |
+|-----------|--------|
+| Same kbid | Skip -- identical content already exists |
+| Same `source_key`, imported `created_at` is newer | Supersede -- imported section replaces local one |
+| Same `source_key`, local `created_at` is newer or equal | Keep -- local version is retained |
+| No `source_key` match | Add as new section |
+
+This leverages the existing supersession mechanism
+documented in [Database](database.md). The content
+hash (kbid) remains the primary identity. The source
+key is the logical identity used for supersession
+matching across machines.
+
+## Git-Based Workflow
+
+Recommended workflow for teams:
+
+```
+Machine A                    Git repo                    Machine B
+─────────                    ────────                    ─────────
+kbdb export --format git ──→ git push ──→ git pull ──→ kbdb import
+                                                         kbdb rebuild
+```
+
+Steps:
+
+1. Machine A exports to a directory within a git repo.
+2. Machine A commits and pushes.
+3. Machine B pulls and imports.
+4. Machine B rebuilds indexes.
+
+This is a manual, explicit sync -- not automatic
+real-time replication. Any transport that moves the
+exported directory works: git, rsync, scp, a shared
+drive, an archive file. Git is recommended because the
+export format is designed to be git-diffable.
+
+## Vector Index Handling
+
+Vector indexes (`vectors.idx`, `vectors-meta.toml`)
+are excluded from export -- they are derived data, the
+same as lexical indexes.
+
+After import:
+
+- If an embedding provider is configured, `rebuild`
+  regenerates vectors using the local model.
+- If the importing machine uses a different embedding
+  model than the exporting machine, vectors are
+  computed fresh with the local model. No action is
+  required to handle this case -- vectors are always
+  rebuilt from content, never transferred.
+
+## Export Formats
+
+| Format | Description | Use Case |
+|--------|-------------|----------|
+| `git` | Flat directory, text-friendly | Git-based sync |
+| `archive` | Single tarball (`.tar.gz`) | Backup, transfer |
+
+The default format is `git` (the directory format).
+
+## CLI Surface
+
+```
+kbdb export [--format git|archive] [--out <path>]
+kbdb import --from <path> [--dry-run]
+```
+
+- `kbdb export` -- exports the database to the
+  specified output path. Defaults to `--format git`.
+  The `--out` path is created if it does not exist.
+- `kbdb import` -- imports from an exported directory
+  or archive. Runs conflict resolution for each
+  incoming section and document.
+- `--dry-run` on import -- shows what would change
+  (sections added, superseded, skipped) without
+  writing anything.
+
+## Import Report
+
+Import returns a structured report:
+
+```json
+{
+  "sections_added": 42,
+  "sections_skipped": 3,
+  "sections_superseded": 5,
+  "documents_added": 2,
+  "documents_skipped": 1,
+  "rebuild_required": true
+}
+```
+
+`rebuild_required` is always `true` after a successful
+import that adds or supersedes any sections, because
+the lexical index is out of date. The caller is
+responsible for running `rebuild` after import.
+
+## Why Not CRDTs
+
+CRDTs (Conflict-free Replicated Data Types) would
+enable automatic, real-time sync without conflicts.
+The design chooses export/import instead for three
+reasons:
+
+1. **Identity conflict** -- The kbid is a content
+   hash. Content changes create new identities rather
+   than updating existing ones. This conflicts with
+   CRDT merge semantics, which assume stable object
+   identity across mutations.
+
+2. **Complexity** -- CRDT support at the current user
+   scale is speculative complexity (YAGNI). The
+   export/import model covers the primary use cases
+   (backup, team sharing, multi-device) with familiar
+   tooling and no new dependencies.
+
+3. **Existing supersession** -- The source key
+   mechanism already provides a logical identity layer
+   above content addressing. Conflict resolution at
+   import time uses the same supersession logic that
+   `kbdb learn --replace` uses. No new primitives are
+   required.
+
+CRDT support may be revisited in a future version if
+demand warrants it.
+
+## Design Constraints
+
+- Sync is manual and explicit -- no background
+  replication.
+- Export excludes all derived data (indexes, vectors).
+- Import respects the existing supersession model
+  from [Database](database.md).
+- No new dependencies -- uses existing file I/O.
+- Git is recommended but not required -- any transport
+  that moves the exported directory works.
+- The single-machine write lock documented in
+  [Concurrent Write Safety](concurrency.md) is not
+  relevant to sync. Import is a local write operation
+  and acquires the lock normally.

package/docs/modules/rust/fs-database/indexes/vectors.md

@@ -0,0 +1,110 @@
+# indexes::vectors -- Vector Embedding Index
+
+*Maps kb-ids to embedding vectors for similarity search. Stored in
+`indexes/vectors.idx` with metadata in `indexes/vectors-meta.toml`.*
+
+## Source
+
+`engine/crates/fs-database/src/index/vectors.rs`
+
+## Implementation Boundary
+
+Unlike other index files which are written by the Rust/WASM engine,
+the vector index is written and read by TypeScript. The Rust module
+defines the binary format specification and provides read/write
+utilities for potential future use, but the primary producer is the
+TypeScript embedding module. This separation exists because embedding
+computation requires network I/O to call external API providers, and
+the WASM target (`wasm32-unknown-unknown`) has no network access.
+ONNX models are also too large to fit in WASM linear memory.
+
+## Binary Format: vectors.idx
+
+Fixed-width binary file. A 32-byte header is followed by N records,
+one per section. Record size is fixed for a given file because all
+embeddings share the same `dimensions` value from the header.
+
+**Header (32 bytes):**
+
+| Offset | Size | Type      | Field                          |
+|--------|------|-----------|--------------------------------|
+| 0      | 4    | u32 LE    | magic (`0x56454358` = "VECX")  |
+| 4      | 4    | u32 LE    | version (1)                    |
+| 8      | 4    | u32 LE    | dimensions                     |
+| 12     | 4    | u32 LE    | record_count                   |
+| 16     | 16   | [u8; 16]  | reserved (zero-filled)         |
+
+**Record (variable, but fixed per file):**
+
+| Offset | Size          | Type           | Field                     |
+|--------|---------------|----------------|---------------------------|
+| 0      | 64            | [u8; 64]       | kbid (UTF-8, null-padded) |
+| 64     | dims × 4      | [f32 LE; dims] | embedding vector          |
+
+Record size = 64 + (dimensions × 4). For 384-dimensional embeddings,
+each record is 1,600 bytes.
+
+## Metadata: vectors-meta.toml
+
+```toml
+model = "all-MiniLM-L6-v2"
+dimensions = 384
+record_count = 1042
+version = 1
+```
+
+The metadata file is read before opening `vectors.idx`. If the
+`model` field differs from the current embedding provider's model
+name, the vector index is considered stale and must be rebuilt before
+it can be used for search.
+
+## Type Definitions
+
+```rust
+pub struct VectorIndex {
+    dimensions: u32,
+    entries: Vec<VectorEntry>,
+}
+
+pub struct VectorEntry {
+    pub kbid: KbId,
+    pub vector: Vec<f32>,
+}
+```
+
+`VectorIndex` owns all loaded entries in memory. `dimensions` is
+stored separately from the per-entry vector length so that the header
+value can be validated against each record during a read. Both structs
+are not exposed at the WASM boundary -- the index is consumed
+internally by `search` and `lookup`.
+
+## Methods
+
+```rust
+impl VectorIndex {
+    pub fn read_from(path: &str) -> Result<Self, FsError>;
+    pub fn write_to(&self, path: &str) -> Result<(), FsError>;
+    pub fn lookup(&self, kbid: &KbId) -> Option<&[f32]>;
+    pub fn search(
+        &self,
+        query_vector: &[f32],
+        limit: usize,
+    ) -> Vec<(KbId, f32)>;
+}
+```
+
+`read_from` parses the binary file header, validates the magic bytes
+and version, then reads all records into memory. `write_to` serializes
+the header followed by each entry in record order. `lookup` returns
+the stored embedding slice for a single kb-id, or `None` if the kb-id
+is not present. `search` computes cosine similarity between
+`query_vector` and every stored entry, then returns the top `limit`
+results as `(KbId, score)` pairs sorted by descending similarity.
+
+## Rebuild Semantics
+
+`vectors.idx` is derived data and can be regenerated from section
+content at any time. During rebuild, the TypeScript layer iterates all
+sections, computes embeddings via the configured `EmbeddingProvider`,
+and writes the binary file. If no embedding provider is configured,
+`vectors.idx` is not generated and similarity search is unavailable.

package/docs/modules/typescript/shared/auto-capture/module.md

@@ -0,0 +1,73 @@
+# auto-capture -- MCP Sampling-Based Automatic Knowledge Capture
+
+**Source:** `src/shared/auto-capture/`
+
+## Responsibility
+
+Automatic knowledge capture from AI agent conversations. Uses MCP
+`sampling/createMessage` to ask the host LLM to identify knowledge
+worth storing, then feeds extracted knowledge into kbdb via `learn`.
+Operates as an MCP server-side feature -- the server proactively
+requests captures based on configurable triggers.
+
+## Directory Layout
+
+```
+src/shared/auto-capture/
+   index.ts
+   auto-capture.controller.ts
+   trigger.ts
+   sampling-prompt.ts
+   rate-limiter.ts
+```
+
+## Components
+
+- **AutoCaptureController** (`auto-capture.controller.ts`) --
+  Orchestrates the capture flow. Tracks tool call count, checks
+  trigger conditions, invokes sampling, and feeds results to
+  `learn`. Stateful per MCP session.
+- **Trigger logic** (`trigger.ts`) -- Evaluates whether a capture
+  check should fire. Conditions: (a) after N tool calls (default:
+  10), (b) configurable time interval, (c) explicit
+  `auto-capture-review` tool invocation. Multiple triggers can be
+  active simultaneously.
+- **Sampling prompts** (`sampling-prompt.ts`) -- Prompt templates
+  sent via `sampling/createMessage`. The prompt asks the host LLM
+  to extract knowledge from the conversation that matches the
+  `memory-protocol` skill's storage criteria: facts, decisions,
+  rationale, procedures.
+- **Rate limiter** (`rate-limiter.ts`) -- Prevents excessive
+  capture attempts. Enforces minimum interval between captures
+  (default: 60 s) and maximum captures per session (default: 20).
+
+## Configuration
+
+Configured via the `[auto_capture]` section of `worker.toml`:
+
+```toml
+[auto_capture]
+enabled = false
+trigger_after_tool_calls = 10
+min_interval_seconds = 60
+max_captures_per_session = 20
+dry_run = false
+```
+
+When `dry_run = true`, the controller logs what would be captured
+without actually writing to the database. Useful for tuning
+triggers.
+
+## Dependency Direction
+
+```
+shared/auto-capture/
+  └── (no shared module deps -- uses MCP sampling API)
+
+shared/mcp/
+  └── shared/auto-capture/   (controller lifecycle)
+```
+
+The auto-capture module is instantiated by the MCP server module.
+It uses the MCP protocol's sampling capability -- not direct imports
+of other shared modules.