@dikolab/kbdb 0.2.2 → 0.3.1

package/docs/goals/database.md

@@ -278,6 +278,40 @@ for storage. They are pre-computed once at insertion
 time and reused on every search -- trading a small
 write-time cost for significant query-time speed.
 
+Additional metadata fields support truth maintenance
+and provenance tracking:
+
+- **`source_key`** -- Logical identity hash (16
+  lowercase hex characters). Derived from the source
+  file path and heading text:
+  `sha256("${source_path}\0${heading}")[..8]`. Used
+  for supersession -- matching existing sections when
+  content is re-learned. `null` for stdin input and
+  sections without a source path. See **Logical
+  Identity (Source Key)** below.
+- **`source_path`** -- Absolute path of the file that
+  was learned. `null` for stdin input and
+  programmatic API calls. Stored for provenance
+  tracking and source-aware garbage collection.
+- **`source_mtime`** -- ISO 8601 timestamp of the
+  source file's last modification time at learn time.
+  `null` when no source file exists. Allows staleness
+  detection: if the source file's mtime has changed
+  since this timestamp, the section may be outdated.
+- **`updated_at`** -- ISO 8601 timestamp of the last
+  supersession. `null` on first creation. Set when a
+  section supersedes an existing one with the same
+  source key.
+- **`tags`** -- Array of user-supplied string tags for
+  scoping and filtering. Empty array when no tags are
+  provided. Tags are not indexed for search -- they
+  are metadata for query-time filtering.
+- **`supersedes`** -- The kbid of the section this one
+  replaced via supersession. `null` on first creation
+  and for non-superseding additions. Creates a
+  supersession chain that can be traversed to find
+  prior versions.
+
 The `add_section` operation rejects requests with
 `image/*` type that do not include a `title`. Text
 and code sections may omit both fields.
@@ -322,6 +356,12 @@ body_token_count = 187
 code_token_count = 0
 created_at = "2026-06-15T10:00:00Z"
 checksum = "sha256:abcdef..."
+source_key = "a1b2c3d4e5f67890"
+source_path = "/home/user/docs/architecture.md"
+source_mtime = "2026-06-15T09:45:00Z"
+updated_at = ""
+tags = ["architecture", "overview"]
+supersedes = ""
 ---
 Section body text here.
 
@@ -347,6 +387,12 @@ body_token_count = 0
 code_token_count = 22
 created_at = "2026-06-15T10:15:00Z"
 checksum = "sha256:1a2b3c..."
+source_key = "f8e7d6c5b4a39281"
+source_path = "/home/user/src/greet.ts"
+source_mtime = "2026-06-15T10:10:00Z"
+updated_at = ""
+tags = ["api", "greeting"]
+supersedes = ""
 ---
 export function greet(name: string): string {
    return `Hello, ${name}!`;
@@ -368,6 +414,12 @@ body_token_count = 3
 code_token_count = 0
 created_at = "2026-06-15T10:30:00Z"
 checksum = "sha256:fedcba..."
+source_key = "1029384756abcdef"
+source_path = "/home/user/docs/arch-diagram.png"
+source_mtime = "2026-06-15T10:25:00Z"
+updated_at = ""
+tags = ["diagram", "architecture"]
+supersedes = ""
 ---
 iVBORw0KGgoAAAANSUhEUgAA...base64 data...
 ```
@@ -463,6 +515,48 @@ Updating a section's content creates a new file with
 a new kbid; the old file is eligible for garbage
 collection.
 
+#### Logical Identity (Source Key)
+
+Content-addressing identifies *what a section
+currently says*. The **source key** identifies *what
+a section is about* -- it is a stable logical
+identity that persists across content edits.
+
+The source key is derived from the section's origin:
+
+```
+source_key = sha256("${source_path}\0${heading}")[..8]
+           → 16 lowercase hex characters
+```
+
+- `source_path` -- the absolute path of the file that
+  was learned (normalized: resolved symlinks, no
+  trailing separator).
+- `heading` -- the section's title text (or empty
+  string when no title is provided).
+- `\0` -- a null byte separator.
+
+The SHA-256 digest is truncated to 8 bytes (64 bits)
+and hex-encoded to produce a 16-character identifier.
+
+Source keys enable **supersession**: when content is
+re-learned from the same source with the same
+heading, the system can identify the existing section
+and replace it rather than creating a sibling. This
+prevents the brain from silently accumulating
+contradictions when facts are corrected.
+
+Source key is `null` for:
+- Stdin input (`-`) -- no source path exists.
+- Sections created without a source path (e.g.
+  programmatic API calls that omit it).
+
+When `source_key` is `null`, supersession is not
+available and the section behaves as a pure
+content-addressed entry (current behavior).
+
+See **Add section** below for supersession semantics.
+
 ### Documents (`documents/`)
 
 One TOML file per document. A document groups sections
@@ -667,14 +761,15 @@ directly.
    `fs-database.wasm`, which writes the TOML header
    (kbid, docids, type, title, description, size,
    token_count, heading_token_count, body_token_count,
-   code_token_count, timestamp, checksum), the
-   separator, and the content body (preprocessed text
-   for text and code sections, base64 for image
-   sections). Updates `catalog.toml` counts. Updates
-   the inverted index with per-field term frequencies,
-   the positional index with term positions, and
-   corpus statistics (including per-field average
-   lengths).
+   code_token_count, created_at, checksum, source_key,
+   source_path, source_mtime, updated_at, tags,
+   supersedes), the separator, and the content body
+   (preprocessed text for text and code sections,
+   base64 for image sections). Updates `catalog.toml`
+   counts. Updates the inverted index with per-field
+   term frequencies, the positional index with term
+   positions, and corpus statistics (including
+   per-field average lengths).
 
    Image sections (`image/*` type) have their binary
    content stored but not indexed. However, their
@@ -682,9 +777,50 @@ directly.
    attributes are tokenized and indexed, making image
    sections searchable via their metadata.
 
+   **Supersession:** When the `replace` flag is set
+   and the new section has a non-null `source_key`,
+   `fs-database.wasm` searches for an existing section
+   with the same `source_key`. If found:
+
+   - The new section's `supersedes` field is set to
+     the old section's kbid.
+   - The new section's `updated_at` is set to the
+     current timestamp.
+   - The old kbid is replaced with the new kbid in all
+     document manifests that reference it.
+   - The old section's `docids` array is emptied,
+     making it eligible for garbage collection.
+   - Index entries for the old section are removed and
+     replaced with entries for the new section.
+
+   When `replace` is not set, or `source_key` is
+   `null`, the add operation behaves as before (pure
+   content-addressed addition with deduplication).
+
+   **Near-duplicate detection:** After the kbid is
+   computed and deduplication is checked, the add
+   operation computes a Jaccard similarity coefficient
+   between the new section's stemmed term set and
+   existing sections' term sets. Sections exceeding a
+   similarity threshold of 0.85 are reported as
+   near-duplicates in the response
+   (`near_duplicates: [{kbid, similarity}]`). This is
+   advisory -- the section is still stored. The check
+   uses the inverted index for efficient comparison
+   without reading every section's content.
+
+   **Heading-less import warning:** Text and code
+   sections added without a `title` emit a warning in
+   the response: `heading_missing`. The heading field
+   carries 2.0x ranking weight; omitting it means the
+   section loses its strongest searchability signal.
+   The section is still stored -- this is a warning,
+   not an error.
+
    **Validation:** requests with `image/*` type that
    omit `title` are rejected with an error. Text and
-   code sections may omit `title`.
+   code sections may omit `title` (with the
+   heading-less warning above).
 
 3. **Update section** -- Given a kb-id and new content,
    replace the section's content. Since kb-ids are
@@ -795,12 +931,26 @@ directly.
     sections (not referenced by any document),
     orphaned documents (referencing missing sections),
     broken embedding references -- markers whose
-    target does not exist or has the wrong type -- and
+    target does not exist or has the wrong type,
     circular reference chains in `$(text:)` and
-    `$(document:)` markers.
+    `$(document:)` markers, and **divergent content**
+    -- multiple sections sharing the same `source_key`
+    (same origin) but with different content,
+    indicating contradictory knowledge that should
+    have been resolved by supersession.
 13. **Garbage collection** -- Remove section files not
-    referenced by any document manifest.
-14. **Index rebuild** -- Reconstruct all index files
+    referenced by any document manifest. Additionally
+    report `stale_sources` -- sections whose
+    `source_path` points to a file that no longer
+    contains their heading text. Stale sources are
+    advisory (not automatically removed) and surfaced
+    in the GC response for manual review.
+14. **Export** -- Copy sections, documents, and
+    catalog to a self-contained output directory (no
+    indexes -- they are rebuildable). Provides a
+    portable, git-friendly snapshot of the knowledge
+    base for backup and versioning.
+15. **Index rebuild** -- Reconstruct all index files
     from sections and documents. Re-tokenizes all text
     and code sections per field, recomputes per-field
     term frequencies and token counts (updating
@@ -975,8 +1125,11 @@ Response per mode:
         "type": "text/markdown",
         "heading": "Section Heading",
         "score": 8.74,
+        "confidence": 1.0,
         "snippet": "...matching context...",
-        "matched_terms": ["term1", "term2"]
+        "matched_terms": ["term1", "term2"],
+        "matched_fields": ["heading", "body"],
+        "created_at": "2026-06-15T10:00:00Z"
       }
     ],
     "total": 42,
@@ -1058,7 +1211,10 @@ Response per mode:
   "type": "text/markdown",
   "title": "Architecture Overview",
   "description": "High-level overview",
-  "docid": "<doc id>"
+  "docid": "<doc id>",
+  "source_path": "/home/user/docs/architecture.md",
+  "tags": ["architecture", "overview"],
+  "replace": false
 }
 
 // Response
@@ -1069,7 +1225,10 @@ Response per mode:
   "description": "High-level overview",
   "indexed_terms_count": 42,
   "elapsed_ms": 15,
-  "attached": false
+  "attached": false,
+  "superseded": null,
+  "warnings": [],
+  "near_duplicates": []
 }
 ```
 
@@ -1302,6 +1461,10 @@ The integrity check verifies:
 - **circular_reference** -- a `$(text:)` or
   `$(document:)` marker chain loops back to an entity
   already in the expansion path.
+- **divergent_content** -- two or more sections share
+  the same `source_key` but have different content
+  (different kbids). Indicates contradictory knowledge
+  that should have been resolved by supersession.
 
 When `ok` is `true`, the `errors` array is empty.
 
@@ -1315,6 +1478,7 @@ When `ok` is `true`, the `errors` array is empty.
 {
   "removed_sections": 3,
   "removed_bytes": 15360,
+  "stale_sources": 1,
   "elapsed_ms": 25
 }
 ```
@@ -1327,6 +1491,12 @@ remove documents, indexes, or the catalog. After
 removal, all caches are invalidated to prevent stale
 reads.
 
+`stale_sources` reports the number of sections whose
+`source_path` points to a file that no longer
+contains their heading text. These sections may be
+outdated but are not automatically removed -- they
+are reported for manual review.
+
 **Index rebuild:**
 
 ```json
@@ -1357,6 +1527,30 @@ entries). Equivalent to deleting `indexes/` and
 re-adding every section. After rebuild, all caches
 are invalidated.
 
+**Export:**
+
+```json
+// Request
+{
+  "path": "/home/user/backup/kbdb-export"
+}
+
+// Response
+{
+  "path": "/home/user/backup/kbdb-export",
+  "sections_exported": 42,
+  "documents_exported": 5,
+  "elapsed_ms": 120
+}
+```
+
+Export copies sections, documents, and `catalog.toml`
+to the specified output directory. Indexes are not
+included -- they are derived data and can be rebuilt
+with `rebuild` after importing. The export directory
+is a self-contained, git-friendly snapshot suitable
+for backup, transfer, or version control.
+
 ### Error Responses
 
 Errors are returned as JSON-RPC 2.0 error objects:
@@ -1411,6 +1605,34 @@ See [Query Parser](query-parser.md) for full details.
 | SHA-256 + Truncation + Base32 | SHA-256 of the typed header, truncated to 128 bits (16 bytes), base32 lowercase (no padding) -- 26 alphanumeric characters |
 | Content Preprocessing       | Text and code: BOM removal, `\r\n`/`\r` to `\n`, Unicode NFC, whitespace trimming. Images: base64 string used as-is |
 
+## Concurrent Write Behavior
+
+The daemon serializes all write operations through a
+**single-writer lock**. The lock is an exclusive file
+lock on `kbdb-{ctx}.lock` in the OS temporary
+directory, keyed to the same context ID used for the
+PID file and IPC socket.
+
+- **Reads** proceed without acquiring the lock. The
+  daemon serves concurrent read requests from
+  multiple worker client connections.
+- **Writes** (add section, update section, remove
+  section, group/remove document, garbage collection,
+  index rebuild, export) acquire the lock before
+  modifying any file. If another write is in progress,
+  the request queues behind the lock.
+- The lock is released after each write operation
+  completes, not held across multiple operations.
+- CLI invocations sharing the same daemon are
+  serialized automatically.
+- Direct file access to `.kbdb` outside the daemon is
+  not supported and may cause corruption. All
+  mutations must go through the daemon's IPC API.
+
+This model is equivalent to SQLite's default locking:
+one writer at a time, concurrent readers, no external
+coordination required.
+
 ## Design Constraints
 
 - `fs-database.wasm` is the sole owner of `.kbdb` I/O.
@@ -1423,6 +1645,10 @@ See [Query Parser](query-parser.md) for full details.
   their content hash. `fs-database.wasm` never modifies
   a section file in place -- updates produce new files
   with new kb-ids.
+- Source keys are advisory logical identity. The
+  content hash (kbid) remains the primary identifier.
+  Supersession is opt-in via the `replace` flag --
+  default learn behavior is unchanged.
 - Indexes are derived data. They can be rebuilt entirely
   from sections and document manifests. Loss of index
   files is recoverable.
@@ -1437,7 +1663,8 @@ See [Query Parser](query-parser.md) for full details.
 - No external database process or server dependency.
 - The database must support concurrent read access from
   multiple CLI invocations via the shared daemon without
-  corruption.
+  corruption. Write operations are serialized through
+  the daemon's single-writer lock.
 - Garbage collection of unreferenced sections is
   `fs-database.wasm`'s responsibility. Sections not
   listed in any document manifest may be removed.

package/docs/goals/document.md

@@ -289,7 +289,7 @@ The tokenization pipeline removes them in Stage 1
 search index.
 
 See [Query Parser](query-parser.md) for the full
-eight-stage tokenization pipeline and
+nine-stage tokenization pipeline and
 [Content Parser](content-parser.md) for the marker
 extraction functions used during stripping.