@dikolab/kbdb 0.2.2 → 0.3.1

package/docs/goals/sync.md

@@ -0,0 +1,217 @@
+# Sync -- Cross-Machine Synchronization
+
+*Export the database as a git-friendly directory, move
+it with any transport, import on the other side.*
+
+## Goal
+
+kbdb is currently single-machine -- "persists across
+sessions" means across sessions on the same box. For
+teams, multi-device use, or backup, the database needs
+a sync story.
+
+The design uses export/import as the primitive, with
+git as the recommended transport. Sync is manual and
+explicit. There is no background replication.
+
+This goal addresses gap #5 from
+[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md):
+"No sync/remote story unless you build one on top,
+which limits 'persists across sessions' to 'across
+sessions on this box.'"
+
+## Sync Primitives
+
+Two operations form the sync foundation.
+
+### Export
+
+Serialize the database into a portable, git-friendly
+directory:
+
+```
+kbdb export --format git --db .kbdb --out ./kbdb-sync/
+```
+
+The exported directory mirrors `.kbdb/` minus indexes:
+
+- `catalog.toml` -- database table of contents
+- `sections/**/*.section` -- all section files
+- `documents/**/*.toml` -- all document manifests
+
+Excluded:
+
+- `indexes/` -- derived data, rebuildable
+- Vector indexes (`vectors.idx`, `vectors-meta.toml`)
+- PID files and lock files
+
+The exported directory is a valid git subtree. It
+contains no binary blobs except section content
+(image sections, which are stored as base64 text).
+Every file is human-readable and git-diffable.
+
+### Import
+
+Restore a database from an exported directory:
+
+```
+kbdb import --from ./kbdb-sync/ --db .kbdb
+```
+
+Import reads the exported directory and writes sections
+and documents into the target database. After import,
+run `rebuild` to regenerate all indexes (lexical and
+vector, if configured).
+
+Import is **additive** -- existing sections with
+different kbids are preserved. Only the conflict
+resolution rules below can cause existing sections to
+be superseded.
+
+## Conflict Resolution
+
+When importing sections that conflict with existing
+ones, the following rules apply in priority order:
+
+| Condition | Action |
+|-----------|--------|
+| Same kbid | Skip -- identical content already exists |
+| Same `source_key`, imported `created_at` is newer | Supersede -- imported section replaces local one |
+| Same `source_key`, local `created_at` is newer or equal | Keep -- local version is retained |
+| No `source_key` match | Add as new section |
+
+This leverages the existing supersession mechanism
+documented in [Database](database.md). The content
+hash (kbid) remains the primary identity. The source
+key is the logical identity used for supersession
+matching across machines.
+
+## Git-Based Workflow
+
+Recommended workflow for teams:
+
+```
+Machine A                    Git repo                    Machine B
+─────────                    ────────                    ─────────
+kbdb export --format git ──→ git push ──→ git pull ──→ kbdb import
+                                                         kbdb rebuild
+```
+
+Steps:
+
+1. Machine A exports to a directory within a git repo.
+2. Machine A commits and pushes.
+3. Machine B pulls and imports.
+4. Machine B rebuilds indexes.
+
+This is a manual, explicit sync -- not automatic
+real-time replication. Any transport that moves the
+exported directory works: git, rsync, scp, a shared
+drive, an archive file. Git is recommended because the
+export format is designed to be git-diffable.
+
+## Vector Index Handling
+
+Vector indexes (`vectors.idx`, `vectors-meta.toml`)
+are excluded from export -- they are derived data, the
+same as lexical indexes.
+
+After import:
+
+- If an embedding provider is configured, `rebuild`
+  regenerates vectors using the local model.
+- If the importing machine uses a different embedding
+  model than the exporting machine, vectors are
+  computed fresh with the local model. No action is
+  required to handle this case -- vectors are always
+  rebuilt from content, never transferred.
+
+## Export Formats
+
+| Format | Description | Use Case |
+|--------|-------------|----------|
+| `git` | Flat directory, text-friendly | Git-based sync |
+| `archive` | Single tarball (`.tar.gz`) | Backup, transfer |
+
+The default format is `git` (the directory format).
+
+## CLI Surface
+
+```
+kbdb export [--format git|archive] [--out <path>]
+kbdb import --from <path> [--dry-run]
+```
+
+- `kbdb export` -- exports the database to the
+  specified output path. Defaults to `--format git`.
+  The `--out` path is created if it does not exist.
+- `kbdb import` -- imports from an exported directory
+  or archive. Runs conflict resolution for each
+  incoming section and document.
+- `--dry-run` on import -- shows what would change
+  (sections added, superseded, skipped) without
+  writing anything.
+
+## Import Report
+
+Import returns a structured report:
+
+```json
+{
+  "sections_added": 42,
+  "sections_skipped": 3,
+  "sections_superseded": 5,
+  "documents_added": 2,
+  "documents_skipped": 1,
+  "rebuild_required": true
+}
+```
+
+`rebuild_required` is always `true` after a successful
+import that adds or supersedes any sections, because
+the lexical index is out of date. The caller is
+responsible for running `rebuild` after import.
+
+## Why Not CRDTs
+
+CRDTs (Conflict-free Replicated Data Types) would
+enable automatic, real-time sync without conflicts.
+The design chooses export/import instead for three
+reasons:
+
+1. **Identity conflict** -- The kbid is a content
+   hash. Content changes create new identities rather
+   than updating existing ones. This conflicts with
+   CRDT merge semantics, which assume stable object
+   identity across mutations.
+
+2. **Complexity** -- CRDT support at the current user
+   scale is speculative complexity (YAGNI). The
+   export/import model covers the primary use cases
+   (backup, team sharing, multi-device) with familiar
+   tooling and no new dependencies.
+
+3. **Existing supersession** -- The source key
+   mechanism already provides a logical identity layer
+   above content addressing. Conflict resolution at
+   import time uses the same supersession logic that
+   `kbdb learn --replace` uses. No new primitives are
+   required.
+
+CRDT support may be revisited in a future version if
+demand warrants it.
+
+## Design Constraints
+
+- Sync is manual and explicit -- no background
+  replication.
+- Export excludes all derived data (indexes, vectors).
+- Import respects the existing supersession model
+  from [Database](database.md).
+- No new dependencies -- uses existing file I/O.
+- Git is recommended but not required -- any transport
+  that moves the exported directory works.
+- The single-machine write lock documented in
+  [Concurrent Write Safety](concurrency.md) is not
+  relevant to sync. Import is a local write operation
+  and acquires the lock normally.

package/docs/goals/worker-daemon.md

@@ -107,6 +107,34 @@ Each database directory is an independent context with
 its own daemon, its own cache, and its own WASM module
 instance. There is no shared state across contexts.
 
+## Write Serialization
+
+The daemon serializes write operations using an
+exclusive file lock on `kbdb-{ctx}.lock` in the OS
+temporary directory. The lock file uses the same
+context ID as the PID file and IPC socket.
+
+- **Read operations** proceed without acquiring the
+  lock. Concurrent reads are fully parallel.
+- **Write operations** acquire the exclusive lock
+  before calling the WASM write export. If another
+  write is already in progress, the new write queues
+  behind the lock and waits.
+- **Lock duration** -- the lock is held only for the
+  duration of the single write operation and released
+  immediately after it completes. It is not held
+  across multiple operations or across client
+  connections.
+
+This serialization model is equivalent to SQLite's
+default locking: readers never block, and writers
+take turns one at a time.
+
+Write methods subject to this lock are:
+`addSection`, `updateSection`, `removeSection`,
+`groupSections`, `removeGrouping`, `initDirectory`,
+`gc`, and `rebuildIndexes`.
+
 ## Daemon Startup
 
 When the worker client spawns a new daemon (see
@@ -127,15 +155,23 @@ daemon process performs the following startup sequence:
    module (see WASM Module below).
 5. Calls `worker_init` on the WASM module to open the
    database context.
-6. Writes its PID to `kbdb-{ctx}.pid` in the OS
+6. Calls `db_check_integrity` on the WASM module to
+   run a startup health check. If `ok` is false, the
+   daemon logs a structured warning to stderr listing
+   the error count and error types. The daemon
+   continues operating -- health issues are not fatal
+   to startup. The health status (`integrity_ok`,
+   `integrity_errors`) is included in subsequent
+   `status` responses.
+7. Writes its PID to `kbdb-{ctx}.pid` in the OS
    temporary directory.
-7. Creates the IPC socket (`kbdb-{ctx}.sock` on Unix,
+8. Creates the IPC socket (`kbdb-{ctx}.sock` on Unix,
    named pipe on Windows).
-8. Begins listening for connections.
+9. Begins listening for connections.
 
 The daemon only writes its PID file and creates the
 IPC socket after all preconditions pass and the WASM
-module is loaded. If the daemon exits before step 6,
+module is loaded. If the daemon exits before step 7,
 the worker client's PID-file poll times out, and the
 worker client reads the crash log (or the daemon's
 stderr) to report the failure reason.
@@ -494,6 +530,7 @@ for each WASM operation:
 | `listByType`, `list_by_type`            | `worker_list_by_type`    |
 | `listDocumentsByType`                   | `worker_list_docs_by_type` |
 | `recall`                                | `worker_recall`          |
+| `export`                                | `db_export`              |
 
 `migrateDatabase` is listed in the map but throws
 `IpcError(-32601, 'migrateDatabase is not yet
@@ -520,12 +557,22 @@ Returns the daemon's current state.
 // Response
 {
    "jsonrpc": "2.0", "id": 1,
-   "result": "busy"
+   "result": {
+      "state": "busy",
+      "integrity_ok": true,
+      "integrity_errors": []
+   }
 }
 ```
 
-Result: `"busy"` if the connection count is greater
-than zero, `"idle"` otherwise.
+Result fields:
+
+| Field              | Type     | Description                               |
+|--------------------|----------|-------------------------------------------|
+| `state`            | string   | `"busy"` if connections > 0, else `"idle"` |
+| `integrity_ok`     | boolean  | `true` if startup health check passed     |
+| `integrity_errors` | string[] | Error types from `db_check_integrity`;    |
+|                    |          | empty when `integrity_ok` is `true`       |
 
 ### `idle_at`
 

package/docs/modules/rust/fs-database/functions/integrity.md

@@ -44,12 +44,23 @@ pub fn rebuild_indexes()
     manifest must exist.
 - Detect circular references in `$(text:)` and
   `$(document:)` marker chains.
+- Detect **divergent content**: sections sharing the
+  same `source_key` that have different content.
+  Indicates contradictory versions that should be
+  resolved via supersession. Reported as
+  `divergent_content` entries in `IntegrityReport`.
 
 ### `garbage_collect`
 
 Remove section files not referenced by any document
 manifest. Returns the number of removed files.
 
+Also reports `stale_sources` -- an advisory count of
+sections whose `source_path` points to a file that no
+longer contains their heading text. Stale sections
+are not automatically removed (they may still be
+referenced by documents); the count is informational.
+
 ### `rebuild_indexes`
 
 Reconstruct all index files (`terms.idx`,

package/docs/modules/rust/fs-database/functions/section.md

@@ -30,17 +30,34 @@ pub fn remove_section(kbid: &KbId)
 ### `add_section`
 
 1. Compute kb-id from content via `compute_kbid`.
-2. Create shard directory if needed (`host_mkdir`).
-3. Serialize TOML header + `---` separator + body.
+2. **Supersession check**: if `source_key` is
+   provided and a section with the same `source_key`
+   already exists, set `supersedes` to the old
+   section's kbid, set `updated_at` to the current
+   timestamp, remove the old kbid from document
+   manifests, and mark the old section for GC. The
+   new section replaces the old one as the current
+   version.
+3. Create shard directory if needed (`host_mkdir`).
+4. Serialize TOML header + `---` separator + body.
    TOML header includes: kbid, docids, section_type,
    title, description, size, created_at, checksum,
    token_count, heading_token_count,
-   body_token_count, code_token_count.
-4. Write atomically via `atomic_write`.
-5. Update catalog via `increment_sections(1)`.
+   body_token_count, code_token_count, source_key,
+   source_path, source_mtime, updated_at, tags,
+   supersedes.
+5. Write atomically via `atomic_write`.
+6. **Near-duplicate detection**: compute Jaccard
+   coefficient on stemmed term sets against existing
+   sections. Sections with similarity >= 0.85 are
+   reported as `near_duplicates` in the result
+   (advisory, not a block).
+7. Update catalog via `increment_sections(1)`.
 
 Validation: image sections (`image/*` type) require
-a `title` in `AddSectionParams`.
+a `title` in `AddSectionParams`. Text sections
+without a `title` emit a `heading_missing` warning
+in the result.
 
 ### `update_section`
 

package/docs/modules/rust/fs-database/indexes/vectors.md

@@ -0,0 +1,110 @@
+# indexes::vectors -- Vector Embedding Index
+
+*Maps kb-ids to embedding vectors for similarity search. Stored in
+`indexes/vectors.idx` with metadata in `indexes/vectors-meta.toml`.*
+
+## Source
+
+`engine/crates/fs-database/src/index/vectors.rs`
+
+## Implementation Boundary
+
+Unlike other index files which are written by the Rust/WASM engine,
+the vector index is written and read by TypeScript. The Rust module
+defines the binary format specification and provides read/write
+utilities for potential future use, but the primary producer is the
+TypeScript embedding module. This separation exists because embedding
+computation requires network I/O to call external API providers, and
+the WASM target (`wasm32-unknown-unknown`) has no network access.
+ONNX models are also too large to fit in WASM linear memory.
+
+## Binary Format: vectors.idx
+
+Fixed-width binary file. A 32-byte header is followed by N records,
+one per section. Record size is fixed for a given file because all
+embeddings share the same `dimensions` value from the header.
+
+**Header (32 bytes):**
+
+| Offset | Size | Type      | Field                          |
+|--------|------|-----------|--------------------------------|
+| 0      | 4    | u32 LE    | magic (`0x56454358` = "VECX")  |
+| 4      | 4    | u32 LE    | version (1)                    |
+| 8      | 4    | u32 LE    | dimensions                     |
+| 12     | 4    | u32 LE    | record_count                   |
+| 16     | 16   | [u8; 16]  | reserved (zero-filled)         |
+
+**Record (variable, but fixed per file):**
+
+| Offset | Size          | Type           | Field                     |
+|--------|---------------|----------------|---------------------------|
+| 0      | 64            | [u8; 64]       | kbid (UTF-8, null-padded) |
+| 64     | dims × 4      | [f32 LE; dims] | embedding vector          |
+
+Record size = 64 + (dimensions × 4). For 384-dimensional embeddings,
+each record is 1,600 bytes.
+
+## Metadata: vectors-meta.toml
+
+```toml
+model = "all-MiniLM-L6-v2"
+dimensions = 384
+record_count = 1042
+version = 1
+```
+
+The metadata file is read before opening `vectors.idx`. If the
+`model` field differs from the current embedding provider's model
+name, the vector index is considered stale and must be rebuilt before
+it can be used for search.
+
+## Type Definitions
+
+```rust
+pub struct VectorIndex {
+    dimensions: u32,
+    entries: Vec<VectorEntry>,
+}
+
+pub struct VectorEntry {
+    pub kbid: KbId,
+    pub vector: Vec<f32>,
+}
+```
+
+`VectorIndex` owns all loaded entries in memory. `dimensions` is
+stored separately from the per-entry vector length so that the header
+value can be validated against each record during a read. Both structs
+are not exposed at the WASM boundary -- the index is consumed
+internally by `search` and `lookup`.
+
+## Methods
+
+```rust
+impl VectorIndex {
+    pub fn read_from(path: &str) -> Result<Self, FsError>;
+    pub fn write_to(&self, path: &str) -> Result<(), FsError>;
+    pub fn lookup(&self, kbid: &KbId) -> Option<&[f32]>;
+    pub fn search(
+        &self,
+        query_vector: &[f32],
+        limit: usize,
+    ) -> Vec<(KbId, f32)>;
+}
+```
+
+`read_from` parses the binary file header, validates the magic bytes
+and version, then reads all records into memory. `write_to` serializes
+the header followed by each entry in record order. `lookup` returns
+the stored embedding slice for a single kb-id, or `None` if the kb-id
+is not present. `search` computes cosine similarity between
+`query_vector` and every stored entry, then returns the top `limit`
+results as `(KbId, score)` pairs sorted by descending similarity.
+
+## Rebuild Semantics
+
+`vectors.idx` is derived data and can be regenerated from section
+content at any time. During rebuild, the TypeScript layer iterates all
+sections, computes embeddings via the configured `EmbeddingProvider`,
+and writes the binary file. If no embedding provider is configured,
+`vectors.idx` is not generated and similarity search is unavailable.

package/docs/modules/rust/fs-database/module.md

@@ -120,6 +120,7 @@ engine/crates/fs-database/
 | `db_status`            | Return database metadata   |
 | `db_check_integrity`   | Run integrity checks       |
 | `db_garbage_collect`   | Remove orphaned sections   |
+| `db_export`            | Export database snapshot    |
 | `alloc` / `free`       | Memory management (from `kbdb-shared`) |
 
 ## Dependencies

package/docs/modules/rust/fs-database/types/section.md

@@ -26,6 +26,12 @@ pub struct SectionHeader {
    pub heading_token_count: u32,
    pub body_token_count: u32,
    pub code_token_count: u32,
+   pub source_key: Option<String>,
+   pub source_path: Option<String>,
+   pub source_mtime: Option<String>,
+   pub updated_at: Option<String>,
+   pub tags: Vec<String>,
+   pub supersedes: Option<KbId>,
 }
 ```
 
@@ -43,6 +49,21 @@ and stored in the header for query-time retrieval.
 See [FieldTokenCounts](../../shared/query/types.md)
 for the shared struct.
 
+`source_key` is a stable logical identity derived
+from `sha256("${source_path}\0${heading}")`,
+truncated to 16 hex chars. Separate from `kbid`
+(content hash) -- identifies *what the section is
+about* rather than *what it currently says*. `null`
+for stdin input. `source_path` is the original file
+path that was learned. `source_mtime` is the ISO 8601
+modified time of the source file at learn time.
+`updated_at` is the ISO 8601 timestamp of the last
+supersession (`null` on first create). `tags` are
+user-supplied strings for scoping and filtering.
+`supersedes` is the kbid of the section this one
+replaced (`null` on first create). See
+[Database -- Logical Identity](../../../goals/database.md).
+
 ### `IntegrityReport`
 
 ```rust
@@ -53,6 +74,7 @@ pub struct IntegrityReport {
    pub broken_refs: Vec<(KbId, String, KbId)>,
    pub type_mismatches: Vec<(KbId, String, KbId)>,
    pub circular_refs: Vec<(KbId, Vec<KbId>)>,
+   pub divergent_content: Vec<(String, Vec<KbId>)>,
 }
 ```
 
@@ -64,7 +86,11 @@ covering all six marker types (`text`, `code`,
 target_kbid)` where the target section type does
 not match the marker type. `circular_refs` are
 `(start_kbid, cycle_path)` for reference chains
-that form loops.
+that form loops. `divergent_content` entries are
+`(source_key, conflicting_kbids)` -- sections
+sharing the same `source_key` that have different
+content, indicating contradictory versions that
+should be resolved via supersession.
 
 ## Functions
 

package/docs/modules/rust/overview.md

@@ -266,7 +266,7 @@ engine/crates/query-parser/
     ├── lib.rs               ← WASM exports
     ├── mem.rs               ← shared-memory helpers
     │
-    ├── pipeline/            ← 8-stage tokenizer
+    ├── pipeline/            ← 9-stage tokenizer
     │   ├── mod.rs
     │   ├── markers.rs
     │   ├── unicode.rs
@@ -274,6 +274,7 @@ engine/crates/query-parser/
     │   ├── identifiers.rs
     │   ├── case.rs
     │   ├── stopwords.rs
+    │   ├── synonyms.rs
     │   ├── stem.rs
     │   └── ngram.rs
     │

package/docs/modules/rust/query-parser/functions/module.md

@@ -5,9 +5,9 @@ extraction, and ranking.*
 
 ## Modules
 
-- **[pipeline](pipeline.md)** -- 8-stage tokenization
+- **[pipeline](pipeline.md)** -- 9-stage tokenization
   pipeline. Converts raw text into normalized, stemmed
-  tokens with n-grams.
+  tokens with synonym expansion and n-grams.
 - **[parse](parse.md)** -- Query string parsing into
   structured `QueryPlan`.
 - **[extract](extract.md)** -- RAKE keyphrase

package/docs/modules/rust/query-parser/functions/pipeline.md

@@ -1,8 +1,8 @@
 # functions::pipeline -- Tokenization Pipeline
 
-*8-stage pipeline that converts raw text into
-normalized, stemmed tokens. Used for both indexing
-and query processing.*
+*9-stage pipeline that converts raw text into
+normalized, stemmed tokens with synonym expansion.
+Used for both indexing and query processing.*
 
 ## Source
 
@@ -14,7 +14,7 @@ and query processing.*
 pub fn tokenize(input: &str) -> TokenStream;
 ```
 
-Runs all eight stages in sequence. Each stage is a
+Runs all nine stages in sequence. Each stage is a
 pure function.
 
 ## Stages
@@ -83,7 +83,32 @@ pub fn remove_stop_words(
 Filters common function words. Single-character
 tokens also removed.
 
-### 7. Stemming (`stem.rs`)
+### 7. Synonym Expansion (`synonyms.rs`)
+
+```rust
+pub fn expand_synonyms(
+   word: &str,
+   map: &SynonymMap,
+) -> Vec<String>;
+```
+
+Looks up the lowercased word in the built-in synonym
+map. Returns a vector containing the original word
+plus any synonyms. When no synonyms exist, returns a
+single-element vector containing only the original.
+
+The synonym map is a bidirectional
+`HashMap<String, Vec<String>>` built at compile time.
+Expansion is applied after stop-word removal and
+before stemming, so both original and synonym tokens
+are stemmed consistently.
+
+The map is small (~30-50 entries) and focused on
+common IT/development vocabulary mismatches:
+login/sign-in, auth/authentication, remove/delete,
+config/configuration, credential/password, etc.
+
+### 8. Stemming (`stem.rs`)
 
 ```rust
 pub fn stem_word(word: &str) -> StemResult;
@@ -92,7 +117,7 @@ pub fn stem_word(word: &str) -> StemResult;
 Snowball / Porter2. Preserves both original and
 stemmed forms.
 
-### 8. N-gram Extraction (`ngram.rs`)
+### 9. N-gram Extraction (`ngram.rs`)
 
 ```rust
 pub fn extract_ngrams(
@@ -101,3 +126,16 @@ pub fn extract_ngrams(
 ```
 
 Bigrams and trigrams for phrase indexing.
+
+## Relaxed Query Mode
+
+The tokenization pipeline is used for both the
+primary AND-mode query plan and the relaxed OR-mode
+fallback plan. `qp_parse_query` produces a
+`QueryPlan` with a `match_mode` field (`All` | `Any`).
+When the AND plan returns zero matches,
+`kb-worker.wasm` retries with the OR plan using the
+same tokenized terms. No additional pipeline stage or
+WASM export is needed. See
+[Query Parser -- Relaxed Query
+Mode](../../../goals/query-parser.md).

package/docs/modules/rust/query-parser/module.md

@@ -49,8 +49,9 @@ engine/crates/query-parser/
     │   ├── identifiers.rs       # stage 4: camelCase split
     │   ├── case.rs              # stage 5: case folding
     │   ├── stopwords.rs         # stage 6: stop words
-    │   ├── stem.rs              # stage 7: Snowball stemmer
-    │   └── ngram.rs             # stage 8: n-gram extract
+    │   ├── synonyms.rs          # stage 7: synonym expansion
+    │   ├── stem.rs              # stage 8: Snowball stemmer
+    │   └── ngram.rs             # stage 9: n-gram extract
     ├── parse.rs                 # query string → QueryPlan
     ├── extract/
     │   ├── mod.rs               # extraction facade
@@ -78,10 +79,10 @@ engine/crates/query-parser/
 
 ### [functions/](functions/module.md) -- Operations
 
-- **[pipeline](functions/pipeline.md)** -- 8-stage
+- **[pipeline](functions/pipeline.md)** -- 9-stage
   tokenization pipeline. Marker stripping, NFC,
   segmentation, identifier splitting, case folding,
-  stop words, stemming, n-grams.
+  stop words, synonym expansion, stemming, n-grams.
 - **[parse](functions/parse.md)** -- Query string
   parsing into `QueryPlan`.
 - **[extract](functions/extract.md)** -- RAKE