@dikolab/kbdb 0.2.2 → 0.3.1

package/docs/goals/cli.md

@@ -53,12 +53,14 @@ kbdb --version | -v
 kbdb learn [options] <path | ->...
 kbdb unlearn [options] <kbid>...
 kbdb search [options] <query>...
+kbdb recall [options] <kbid>...
 kbdb content [options] <id>...
 kbdb mcp [options]
 kbdb check [options]
 kbdb gc [options]
 kbdb rebuild [options]
 kbdb status [options]
+kbdb export [options] [<path>]
 kbdb db init [options]
 kbdb db migrate [options]
 kbdb skill learn [options] --name <n> --body <file|->
@@ -196,6 +198,17 @@ Import and index documents into the knowledge base.
   to this document ID. If the document does not
   exist, it is created. Applies to all sections
   ingested in this invocation.
+- `--tags <t,...>` -- Comma-separated tags for scoping
+  and filtering. Applies to all sections ingested in
+  this invocation. Example: `--tags auth,api,v2`.
+- `--replace` -- Enable supersession. When set, the
+  system matches each incoming section by source key
+  (source path + heading). If an existing section has
+  the same source key, it is superseded: the old
+  section's document references are transferred to the
+  new one and the old section becomes eligible for GC.
+  Without this flag, re-learning edited content creates
+  a sibling rather than replacing the original.
 
 **Supported file extensions:**
 
@@ -239,7 +252,10 @@ sections in the same invocation.
         "kbid": "abcdef1234567890abcdef1234",
         "path": "/absolute/path/to/file.md",
         "size": 1234,
-        "skipped": false
+        "skipped": false,
+        "superseded": null,
+        "warnings": [],
+        "near_duplicates": []
      }
   ]
   ```
@@ -247,7 +263,12 @@ sections in the same invocation.
   is the UTF-8 character count of the content. The
   `skipped` field is always `false` for results that
   appear in the array -- files that are skipped produce
-  no entry.
+  no entry. `superseded` is the kbid of the section
+  that was replaced via `--replace` (null when no
+  supersession occurred). `warnings` is an array of
+  warning strings (e.g. `"heading_missing"` when no
+  title is provided). `near_duplicates` lists sections
+  with similarity >= 0.85 as `{kbid, similarity}`.
 - **stderr**: errors only.
 
 **Exit code:** `0` on success, `1` on error.
@@ -425,15 +446,19 @@ array (see [SearchDisplay type](#searchdisplay-type)).
          "type": "text/markdown",
          "docids": ["doc-intro-ml"],
          "score": 8.7432,
+         "confidence": 1.0,
          "snippet": "Machine learning is a subfield...",
          "matched_terms": ["machine", "learning"],
+         "matched_fields": ["heading", "body"],
+         "created_at": "2026-06-15T10:00:00Z",
          "preview": ""
       }
    ],
    "total": 42,
    "offset": 0,
    "limit": 20,
-   "has_more": true
+   "has_more": true,
+   "relaxed": false
 }
 ```
 
@@ -448,15 +473,19 @@ section text:
          "type": "text/markdown",
          "docids": ["doc-intro-ml"],
          "score": 8.7432,
+         "confidence": 1.0,
          "snippet": "Machine learning is a subfield...",
          "matched_terms": ["machine", "learning"],
+         "matched_fields": ["heading", "body"],
+         "created_at": "2026-06-15T10:00:00Z",
          "preview": "Machine learning is a subfield..."
       }
    ],
    "total": 42,
    "offset": 0,
    "limit": 20,
-   "has_more": true
+   "has_more": true,
+   "relaxed": false
 }
 ```
 
@@ -557,6 +586,131 @@ src/cli.ts  parseArgs()
 See [Content Composer](content-composer.md) for the
 full composition specification.
 
+### `recall` -- progressive context retrieval
+
+```
+kbdb recall [options] <kbid>...
+```
+
+Retrieve full content and progressive context for one
+or more sections by kbid. Use after `search` to drill
+into relevant results. Returns section text, metadata,
+references, and back-references at configurable
+expansion depths.
+
+**Arguments:**
+- `<kbid>...` -- one or more 26-character kb-id
+  strings.
+
+**Options:**
+
+| Flag            | Alias | Description                    |
+|-----------------|-------|--------------------------------|
+| `--depth <n>`   | `-d`  | Expansion depth (0--3,         |
+|                 |       | default: 0, clamped to 3)      |
+| `--db <path>`   |       | Database directory path        |
+
+**Expansion levels:**
+
+| Depth | Returns                               |
+|-------|---------------------------------------|
+| 0     | Full section content with heading,    |
+|       | type, and docids.                     |
+| 1     | Depth-0 content, plus parent document |
+|       | manifests (title, all section kbids   |
+|       | with headings and types) for each     |
+|       | docid, plus back-references (sections |
+|       | that reference this section, from the |
+|       | reference graph index, each with      |
+|       | marker_type and context_snippet).     |
+| 2     | Depth-1 content, plus full text of    |
+|       | sibling sections (other sections in   |
+|       | the same documents), plus forward     |
+|       | references from the section's own     |
+|       | content markers.                      |
+| 3     | Depth-2 content, plus full text of    |
+|       | referenced sections (one level deep), |
+|       | plus their back-references.           |
+
+Depth values above 3 are clamped to 3.
+
+**Output:**
+- **stdout**: JSON `RecallResult` object (single kbid)
+  or `RecallResult[]` array (multiple kbids).
+
+Single result shape (depth 0):
+
+```json
+{
+   "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+   "heading": "Architecture Overview",
+   "type": "text/markdown",
+   "content": "Full section text...",
+   "docids": ["doc-5678"]
+}
+```
+
+Single result shape (depth 1+):
+
+```json
+{
+   "kbid": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+   "heading": "Architecture Overview",
+   "type": "text/markdown",
+   "content": "Full section text...",
+   "docids": ["doc-5678"],
+   "documents": [
+      {
+         "docid": "doc-5678",
+         "title": "System Architecture",
+         "sections": [
+            {
+               "kbid": "...",
+               "heading": "...",
+               "type": "..."
+            }
+         ]
+      }
+   ],
+   "back_references": [
+      {
+         "kbid": "ref-section-kbid",
+         "heading": "Related Section",
+         "type": "text/markdown",
+         "marker_type": "link",
+         "context_snippet": "For details, see ...",
+         "direction": "back"
+      }
+   ],
+   "siblings": [ ... ],
+   "references": [ ... ]
+}
+```
+
+Fields beyond `content` and `docids` are present only
+at the corresponding depth level. At depth 0,
+`documents`, `back_references`, `siblings`, and
+`references` are absent. At depth 1, `siblings` and
+`references` are absent. And so on.
+
+- **stderr**: errors only.
+
+**Exit code:** `0` on success, `1` on error.
+
+**Delegation chain:**
+```
+src/cli.ts  parseArgs()
+  -> findDb() / promptUser() / runDbInit()
+  -> createWorkerClient({ contextPath })
+  -> versionGate(client, dbDir)
+  -> dispatchCommand(client, options)
+       -> runRecall(client, options)
+            -> client.recall({ kbids, depth })
+                 -> IPC -> worker daemon
+                      -> kb-worker.wasm
+                           -> fs-database.wasm (fetch)
+```
+
 ### `mcp` -- start MCP server
 
 ```
@@ -604,7 +758,8 @@ The MCP server exposes `retrieve` (section/document
 lookup by ID) and `status` (database metadata) tools
 that have no CLI equivalents. These are programmatic
 operations suited to MCP clients (AI agents). CLI
-users use `kbdb search` to find content and
+users use `kbdb search` to find content,
+`kbdb recall` to expand context progressively, and
 `kbdb content` to compose documents by ID.
 
 ### `check` -- integrity check
@@ -614,8 +769,9 @@ kbdb check [options]
 ```
 
 Verify the integrity of the database. Checks section
-file checksums, document manifest references, and
-type consistency.
+file checksums, document manifest references, type
+consistency, and divergent content (sections sharing
+the same source key with different content).
 
 **Output:**
 - **stdout**: JSON integrity report:
@@ -672,11 +828,15 @@ manifest references it (empty `docids` array).
   ```json
   {
      "removed_sections": 3,
-     "removed_bytes": 15360
+     "removed_bytes": 15360,
+     "stale_sources": 1
   }
   ```
   When no sections are eligible, `removed_sections`
   is `0` and `removed_bytes` is `0`.
+  `stale_sources` reports sections whose
+  `source_path` file no longer contains their heading
+  text (advisory, not automatically removed).
 - **stderr**: errors only.
 
 **Exit code:** `0` on success, `1` on error.
@@ -734,6 +894,55 @@ src/cli.ts  parseArgs()
   -> stdout
 ```
 
+### `export` -- snapshot database
+
+```
+kbdb export [options] [<path>]
+```
+
+Export the database as a self-contained directory
+snapshot for backup, transfer, or version control.
+The export includes sections, documents, and the
+catalog. Indexes are excluded -- they are derived
+data and can be rebuilt with `kbdb rebuild` after
+importing.
+
+**Arguments:**
+- `<path>` -- Output directory path. Defaults to
+  `./kbdb-export/` in the current working directory.
+  If the directory exists, it is overwritten.
+
+**Options:**
+- `--db <path>` -- Source database directory.
+
+**Output:**
+- **stdout**: JSON report:
+  ```json
+  {
+     "path": "/absolute/path/kbdb-export",
+     "sections_exported": 42,
+     "documents_exported": 5
+  }
+  ```
+- **stderr**: errors only.
+
+**Exit code:** `0` on success, `1` on error.
+
+**Delegation chain:**
+```
+src/cli.ts  parseArgs()
+  -> findDb() / promptUser() / runDbInit()
+  -> createWorkerClient({ contextPath })
+  -> versionGate(client, dbDir)
+  -> dispatchCommand(client, options)
+       -> runExport(client, options)
+            -> client.export({ path })
+                 -> IPC -> worker daemon
+                      -> kb-worker.wasm
+                           -> fs-database.wasm
+  -> stdout
+```
+
 ### `kbdb db` -- database management
 
 The `db` command group manages the `.kbdb` database
@@ -1520,10 +1729,13 @@ processes the raw argument vector
 5. All other flags (`--db`, `--format`, `--mode`,
    `--limit`, `--offset`, `--title`, `--description`,
    `--docid`, `--name`, `--persona`, `--body`,
-   `--skills`) consume the next token as their value.
-6. `--content` / `-c` is a boolean flag; it sets
-   `options.content = true` without consuming the next
-   token.
+   `--skills`, `--depth` / `-d`, `--tags`) consume
+   the next token as their value. `--depth` is parsed
+   as a number and clamped to the range 0--3. `--tags`
+   splits its value on commas into a `string[]`.
+6. `--content` / `-c` and `--replace` are boolean
+   flags; they set their respective option to `true`
+   without consuming the next token.
 7. The first non-flag, non-group token becomes the
    **command**.
 8. Subsequent non-flag tokens become **positional
@@ -1584,6 +1796,12 @@ interface CliOptions {
    /** Comma-separated skill kbids
     *  (agent create). Parsed to string[]. */
    skills?: string[];
+   /** Recall expansion depth (0-3). Defaults to 0. */
+   depth?: number;
+   /** Comma-separated tags for scoping (learn). */
+   tags?: string[];
+   /** Enable supersession by source key (learn). */
+   replace?: boolean;
    /** Positional arguments following the sub-command. */
    args: string[];
 }
@@ -1638,6 +1856,7 @@ Knowledge Base:
    learn               Import content from files/stdin
    unlearn             Remove sections by kbid
    search              Query the knowledge base
+   recall              Retrieve section context by kbid
    content             Compose rendered content from ids
 
 Database:
@@ -1650,6 +1869,7 @@ Maintenance:
                        sections
    rebuild             Reconstruct all index files
    status              Show database status
+   export              Snapshot database for backup
 
 Skills:
    skill learn         Create a new skill definition
@@ -1734,12 +1954,20 @@ interface SearchDisplay {
    docids: string[];
    /** Relevance score from the ranking engine. */
    score: number;
+   /** Normalized score (0.0-1.0) relative to the top
+    *  result in this result set. */
+   confidence: number;
    /** KWIC excerpt (~150 chars) centered on the
     *  highest-scoring matched term. Empty string
     *  for image sections. */
    snippet: string;
    /** Query terms that matched, as a flat string[]. */
    matched_terms: string[];
+   /** Fields that contained matches: subset of
+    *  ["heading", "body", "code"]. */
+   matched_fields: string[];
+   /** ISO 8601 creation timestamp. */
+   created_at: string;
    /** Full section text. Empty string when --content
     *  is not used. */
    preview: string;
@@ -1764,12 +1992,23 @@ Produced by `runLearn()` for each file ingested.
 interface LearnResult {
    /** Knowledge-base identifier assigned to the section. */
    kbid: string;
-   /** Absolute path to the source file (or "-" for stdin). */
+   /** Absolute path to the source file (or "-" for
+    *  stdin). */
    path: string;
    /** UTF-8 character count of the ingested content. */
    size: number;
-   /** true when the file was skipped without being ingested. */
+   /** true when the file was skipped. */
    skipped: boolean;
+   /** Kbid of the section this one superseded via
+    *  --replace, or null. */
+   superseded: string | null;
+   /** Warning strings (e.g. "heading_missing"). */
+   warnings: string[];
+   /** Sections with similarity >= 0.85. */
+   near_duplicates: Array<{
+      kbid: string;
+      similarity: number;
+   }>;
 }
 ```
 
@@ -1860,10 +2099,12 @@ Routing table:
 | `learn`          | `runLearn`         | yes             |
 | `unlearn`        | `runUnlearn`       | yes             |
 | `search`         | `runSearch`        | yes             |
+| `recall`         | `runRecall`        | yes             |
 | `content`        | `runContent`       | yes             |
 | `check`          | `runCheck`         | yes             |
 | `gc`             | `runGc`            | yes             |
 | `rebuild`        | `runRebuild`       | yes             |
+| `export`         | `runExport`        | yes             |
 | `db-init`        | `runDbInit`        | no              |
 | `db-migrate`     | `runDbMigrate`     | yes             |
 | `skill-learn`    | `runSkillLearn`    | yes             |

package/docs/goals/concurrency.md

@@ -0,0 +1,259 @@
+# Concurrent Write Safety
+
+*One writer at a time, unlimited concurrent readers --
+formalizing the locking model, multi-daemon prevention,
+crash recovery, and stress-test methodology.*
+
+## Goal
+
+The worker daemon serializes all write operations
+through a single-writer lock. Multiple concurrent
+readers proceed without any lock. This document
+formalizes the locking model, multi-daemon prevention,
+atomic write guarantees, crash recovery, and
+stress-test methodology.
+
+This goal addresses gap #3 from the issue
+[2026-06-23 Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md):
+the single-writer lock in the worker daemon must be
+validated under concurrent load before concurrent
+multi-agent use can be trusted.
+
+## Single-Writer Lock
+
+The daemon acquires an exclusive file lock on
+`kbdb-{ctx}.lock` in the OS temporary directory during
+write operations. The lock file uses the same context
+ID as the PID file and IPC socket (see
+[Worker Daemon](worker-daemon.md) for context ID
+derivation).
+
+- **Read operations** -- search, recall, retrieve,
+  content, dbStatus, and listByType -- proceed without
+  acquiring the lock. Concurrent read requests from
+  multiple worker client connections are fully parallel.
+- **Write operations** -- addSection, updateSection,
+  removeSection, groupSections, removeGrouping,
+  initDirectory, gc, rebuildIndexes, and export --
+  acquire the exclusive lock before calling the
+  corresponding WASM export. If another write is
+  already in progress, the new write queues behind the
+  lock and waits.
+- The lock is held only for the duration of a single
+  write operation. It is released immediately after
+  the operation completes. It is never held across
+  multiple operations or across client connections.
+- The locking model is implemented in TypeScript inside
+  the worker daemon (`src/shared/worker-daemon/`), not
+  inside any WASM module.
+
+This model is equivalent to SQLite's default locking:
+readers never block, and writers take turns one at a
+time.
+
+### Write Operations Subject to the Lock
+
+| IPC Method               | WASM Export                |
+|--------------------------|----------------------------|
+| `addSection`             | `worker_add_section`       |
+| `updateSection`          | `worker_update_section`    |
+| `removeSection`          | `worker_remove_section`    |
+| `groupSections`          | `worker_group_sections`    |
+| `removeGrouping`         | `worker_remove_grouping`   |
+| `initDirectory`          | `worker_init_directory`    |
+| `gc`                     | `worker_garbage_collect`   |
+| `rebuildIndexes`         | `worker_rebuild_indexes`   |
+| `export`                 | `db_export`                |
+
+## Multi-Daemon Prevention
+
+Each database context gets exactly one daemon process
+at a time, enforced by a PID file keyed to the context
+ID.
+
+- The PID file is named `kbdb-{ctx}.pid` and lives in
+  the OS temporary directory. Its presence means a
+  daemon for that context is either running or died
+  without cleaning up.
+- Before spawning a new daemon, the worker client reads
+  the PID file (if present) and checks whether the
+  recorded process is alive.
+  - **PID file absent** -- no running daemon; spawn a
+    new one.
+  - **PID file present, process alive** -- connect to
+    the existing daemon; do not spawn.
+  - **PID file present, process dead** -- stale PID
+    file from a prior crash; remove the file and spawn
+    a new daemon.
+- The daemon writes its PID file only after all startup
+  preconditions pass and the WASM module is loaded
+  successfully. See [Worker Daemon](worker-daemon.md)
+  for the full startup sequence.
+- On graceful shutdown, the daemon removes its PID
+  file before exiting.
+
+This protocol ensures that the same database directory
+is never served by two concurrent daemon processes.
+
+## Atomic Write Guarantees
+
+All file writes in the database use the atomic
+write pattern: content is written to a temporary file
+first, then the temporary file is renamed to the final
+path. Rename is an atomic operation on all supported
+filesystems (POSIX and Windows NTFS).
+
+- **Section files** (`sections/{prefix}/{kbid}.sec`)
+  are written to `sections/{prefix}/{kbid}.sec.tmp`
+  then renamed.
+- **Document manifests** (`documents/{docid}.toml`)
+  are written to `documents/{docid}.toml.tmp` then
+  renamed.
+- **Index files** (`indexes/terms.idx`, etc.) are
+  rebuilt to temporary paths then renamed into place.
+- **Catalog** (`catalog.toml`) is written last,
+  after all data files are in place. A crash
+  before the catalog rename leaves the catalog
+  consistent with the previous complete state.
+
+If the daemon crashes mid-write, the temporary file
+is orphaned but the original file is intact. On next
+daemon startup, orphaned temporary files (matching
+`*.tmp`) in the database directory are cleaned up.
+
+The atomic write pattern is specified in
+[Database](database.md) under Storage Design.
+
+## Crash Recovery
+
+The table below describes every identified failure
+scenario, the resulting disk state, and the recovery
+action.
+
+| Scenario | State After Crash | Recovery |
+|---|---|---|
+| Crash during section write | `*.sec.tmp` orphaned, original `.sec` intact | Startup cleans `*.tmp` files |
+| Crash during index rebuild | Partial `*.idx.tmp`, originals intact | Startup cleans `*.tmp`; next `rebuildIndexes` overwrites |
+| Crash during GC | Some unreferenced sections deleted, others not | Re-run `gc`; idempotent |
+| Crash during catalog write | `catalog.toml.tmp` orphaned, original intact | Startup cleans `*.tmp` |
+| PID file not removed | Stale `kbdb-{ctx}.pid` on disk | Worker client detects dead process, removes PID, spawns new daemon |
+| Lock file not released | `kbdb-{ctx}.lock` orphaned | Daemon startup acquires the exclusive lock fresh; OS releases orphaned file locks when the process that held them exits |
+| WASM crash inside write | Partial in-memory state, temp file may exist | Daemon exits; startup cleans `*.tmp`; next operation starts clean |
+
+All recovery actions are automatic. No manual
+intervention is required for any scenario in this
+table.
+
+## Stress Test Specification
+
+The following test harness validates concurrent write
+safety. It must pass on both Deno and Node.js runtimes
+before a release can claim concurrent safety.
+
+### Setup
+
+1. Create a fresh, empty database directory.
+2. Prepare N unique documents (N in the range 10--50).
+   Each document must have distinct content to avoid
+   content-address collisions that would make section
+   count verification meaningless.
+
+### Execution
+
+3. Launch N concurrent `kbdb learn` invocations, all
+   targeting the same database directory. Each
+   invocation learns one of the N unique documents.
+4. Use a process pool or parallel shell to launch all
+   N invocations simultaneously -- not sequentially.
+5. Wait for all N invocations to complete. Record any
+   non-zero exit codes as failures.
+
+### Verification
+
+6. Run `kbdb check` against the database.
+   The response must have `ok: true`. Any errors in the
+   response are test failures.
+7. Query the section count. The count must equal the
+   expected number of sections derived from the N
+   documents (no duplicates from race conditions, no
+   losses from interleaved writes).
+8. Verify that no section has a checksum mismatch.
+   A mismatch indicates a torn write -- the atomic
+   rename pattern failed or was bypassed.
+
+### Pass Criteria
+
+| Check | Expected Result |
+|---|---|
+| All `kbdb learn` exit codes | `0` |
+| `kbdb check` response | `ok: true`, zero errors |
+| Section count | Exactly the expected count |
+| Checksum mismatches | Zero |
+
+### Runtime Matrix
+
+Run the full harness on both runtimes:
+
+| Runtime | Command prefix |
+|---|---|
+| Node.js >= 20 | `node dist/kbdb.mjs` |
+| Deno | `deno run --allow-all src/cli.ts` |
+
+Both must pass. A failure on one runtime but not the
+other indicates a runtime-specific locking bug.
+
+## Edge Cases
+
+### Two CLI Invocations Through the Same Daemon
+
+When two `kbdb` CLI processes connect to the same
+daemon (same database directory), their write requests
+share the single-writer lock automatically. No user
+action is required. The second write waits for the
+first to complete.
+
+### MCP Server Connections
+
+An MCP server connection routes all requests through
+a worker client to the daemon, the same as a CLI
+invocation. Multiple MCP client connections routed to
+the same daemon share the write lock:
+
+- Concurrent read requests from multiple MCP clients
+  are served in parallel.
+- Concurrent write requests from multiple MCP clients
+  are serialized through the lock one at a time.
+
+There is no per-connection or per-client lock; the
+lock is global to the daemon process.
+
+### Direct File Access
+
+Accessing or modifying `.kbdb` files directly -- via
+text editor, shell commands, or any process other than
+the daemon -- is not supported. Direct writes bypass
+the single-writer lock and the atomic write pattern,
+and may corrupt the database. All mutations must go
+through the daemon's IPC API.
+
+## Design Constraints
+
+- **Per-daemon locking, not per-file** -- The lock
+  covers all writes to a database context, not writes
+  to individual files. This means writes to unrelated
+  sections still serialize behind the same lock. The
+  simplicity of a single lock outweighs the throughput
+  cost at current scale. If per-section parallelism
+  becomes necessary, this constraint is the place to
+  revisit.
+- **Single-machine only** -- The locking model
+  relies on OS-level file locks and process identity
+  on a single machine. It provides no guarantees across
+  network filesystems or multiple machines. Cross-machine
+  scenarios require an external sync mechanism (see
+  [Sync](sync.md) when available).
+- **Lock implemented in TypeScript** -- The write lock
+  is managed by the TypeScript worker daemon layer, not
+  by any WASM module. WASM modules have no knowledge of
+  concurrent callers; they process one call at a time
+  through the host.