@dikolab/kbdb 0.2.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (120) hide show
  1. package/README.md +59 -24
  2. package/dist/cli.cjs +462 -73
  3. package/dist/cli.cjs.map +4 -4
  4. package/dist/cli.mjs +454 -65
  5. package/dist/cli.mjs.map +4 -4
  6. package/dist/kbdb-worker.cjs +395 -60
  7. package/dist/kbdb-worker.cjs.map +4 -4
  8. package/dist/mod.cjs +18 -1
  9. package/dist/mod.cjs.map +2 -2
  10. package/dist/mod.mjs +18 -1
  11. package/dist/mod.mjs.map +2 -2
  12. package/dist/src/shared/cli/functions/dispatch-command.function.d.ts +3 -1
  13. package/dist/src/shared/cli/functions/install-default-skills.function.d.ts +12 -0
  14. package/dist/src/shared/cli/functions/run-check.function.d.ts +47 -5
  15. package/dist/src/shared/cli/functions/run-export.function.d.ts +34 -0
  16. package/dist/src/shared/cli/functions/run-gc.function.d.ts +6 -0
  17. package/dist/src/shared/cli/functions/run-learn.function.d.ts +35 -6
  18. package/dist/src/shared/cli/functions/run-recall.function.d.ts +45 -0
  19. package/dist/src/shared/cli/functions/run-search.function.d.ts +52 -15
  20. package/dist/src/shared/cli/index.d.ts +4 -0
  21. package/dist/src/shared/cli/typings/cli-options.interface.d.ts +13 -0
  22. package/dist/src/shared/cli/typings/learn-result.model.d.ts +19 -0
  23. package/dist/src/shared/cli/typings/search-display.model.d.ts +18 -0
  24. package/dist/src/shared/mcp/classes/mcp-server.class.d.ts +8 -0
  25. package/dist/src/shared/mcp/typings/mcp-client.interface.d.ts +43 -0
  26. package/dist/src/shared/version/constants/version.constant.d.ts +1 -1
  27. package/dist/src/shared/wasm-codec/functions/decode-add-section-result.function.d.ts +48 -0
  28. package/dist/src/shared/wasm-codec/functions/decode-scored-results.function.d.ts +58 -0
  29. package/dist/src/shared/wasm-codec/functions/decode-section-records.function.d.ts +33 -0
  30. package/dist/src/shared/wasm-codec/functions/encode-add-section-params.function.d.ts +52 -0
  31. package/dist/src/shared/wasm-codec/index.d.ts +2 -0
  32. package/dist/src/shared/worker-client/classes/worker-client.class.d.ts +16 -0
  33. package/dist/src/shared/worker-client/index.d.ts +1 -1
  34. package/dist/src/shared/worker-client/typings/add-section-result.model.d.ts +29 -15
  35. package/dist/src/shared/worker-client/typings/export-result.model.d.ts +14 -0
  36. package/dist/src/shared/worker-client/typings/gc-result.model.d.ts +6 -0
  37. package/dist/src/shared/worker-client/typings/search-result.model.d.ts +12 -0
  38. package/dist/src/shared/worker-client/typings/section.model.d.ts +16 -0
  39. package/dist/src/shared/worker-daemon/classes/worker-daemon.class.d.ts +35 -0
  40. package/dist/wasm/fs-database/kbdb_fs_database.d.ts +22 -1
  41. package/dist/wasm/fs-database/kbdb_fs_database.js +28 -1
  42. package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
  43. package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm.d.ts +1 -0
  44. package/dist/wasm/kb-worker/kbdb_worker.d.ts +48 -1
  45. package/dist/wasm/kb-worker/kbdb_worker.js +60 -1
  46. package/dist/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
  47. package/dist/wasm/kb-worker/kbdb_worker_bg.wasm.d.ts +2 -0
  48. package/dist/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
  49. package/dist/worker.mjs +396 -60
  50. package/dist/worker.mjs.map +4 -4
  51. package/docs/details/cli.md +102 -15
  52. package/docs/details/library-api.md +55 -6
  53. package/docs/details/mcp-server.md +8 -4
  54. package/docs/details/search-and-ranking.md +63 -5
  55. package/docs/details/storage.md +2 -0
  56. package/docs/goals/cli.md +255 -14
  57. package/docs/goals/database.md +244 -17
  58. package/docs/goals/document.md +1 -1
  59. package/docs/goals/mcp.md +83 -14
  60. package/docs/goals/query-parser.md +98 -4
  61. package/docs/goals/query-result.md +177 -10
  62. package/docs/goals/skills.md +72 -0
  63. package/docs/goals/worker-daemon.md +54 -7
  64. package/docs/modules/rust/fs-database/functions/integrity.md +11 -0
  65. package/docs/modules/rust/fs-database/functions/section.md +23 -6
  66. package/docs/modules/rust/fs-database/module.md +1 -0
  67. package/docs/modules/rust/fs-database/types/section.md +27 -1
  68. package/docs/modules/rust/overview.md +2 -1
  69. package/docs/modules/rust/query-parser/functions/module.md +2 -2
  70. package/docs/modules/rust/query-parser/functions/pipeline.md +44 -6
  71. package/docs/modules/rust/query-parser/module.md +5 -4
  72. package/docs/modules/rust/query-parser/types/keyphrase.md +11 -0
  73. package/docs/modules/rust/query-parser/types/module.md +3 -3
  74. package/docs/modules/typescript/shared/cli/functions.md +68 -3
  75. package/package.json +1 -1
  76. package/src/shared/cli/functions/dispatch-command.function.ts +53 -1
  77. package/src/shared/cli/functions/format-output.function.ts +36 -12
  78. package/src/shared/cli/functions/generate-help.function.ts +56 -1
  79. package/src/shared/cli/functions/install-default-skills.function.ts +296 -0
  80. package/src/shared/cli/functions/parse-args.function.ts +13 -0
  81. package/src/shared/cli/functions/run-check.function.ts +87 -6
  82. package/src/shared/cli/functions/run-export.function.ts +40 -0
  83. package/src/shared/cli/functions/run-gc.function.ts +6 -0
  84. package/src/shared/cli/functions/run-init.function.ts +2 -0
  85. package/src/shared/cli/functions/run-learn.function.ts +68 -16
  86. package/src/shared/cli/functions/run-recall.function.ts +66 -0
  87. package/src/shared/cli/functions/run-search.function.ts +58 -15
  88. package/src/shared/cli/index.ts +4 -0
  89. package/src/shared/cli/typings/cli-options.interface.ts +13 -0
  90. package/src/shared/cli/typings/learn-result.model.ts +16 -0
  91. package/src/shared/cli/typings/search-display.model.ts +18 -0
  92. package/src/shared/mcp/classes/mcp-server.class.ts +30 -1
  93. package/src/shared/mcp/functions/create-tool-definitions.function.ts +46 -5
  94. package/src/shared/mcp/functions/handle-tool-call.function.ts +70 -2
  95. package/src/shared/mcp/typings/mcp-client.interface.ts +45 -0
  96. package/src/shared/version/constants/version.constant.ts +1 -1
  97. package/src/shared/wasm-codec/functions/decode-add-section-result.function.ts +112 -0
  98. package/src/shared/wasm-codec/functions/decode-scored-results.function.ts +86 -0
  99. package/src/shared/wasm-codec/functions/decode-section-records.function.ts +87 -0
  100. package/src/shared/wasm-codec/functions/encode-add-section-params.function.ts +68 -0
  101. package/src/shared/wasm-codec/index.ts +5 -0
  102. package/src/shared/worker-client/classes/worker-client.class.ts +19 -0
  103. package/src/shared/worker-client/index.ts +4 -1
  104. package/src/shared/worker-client/typings/add-section-result.model.ts +30 -15
  105. package/src/shared/worker-client/typings/export-result.model.ts +14 -0
  106. package/src/shared/worker-client/typings/gc-result.model.ts +6 -0
  107. package/src/shared/worker-client/typings/search-result.model.ts +12 -0
  108. package/src/shared/worker-client/typings/section.model.ts +16 -0
  109. package/src/shared/worker-daemon/classes/worker-daemon.class.ts +315 -46
  110. package/src/shared/worker-daemon/functions/create-host-imports.function.ts +19 -4
  111. package/src/shared/worker-daemon/functions/create-ipc-server.function.ts +5 -1
  112. package/src/wasm/fs-database/kbdb_fs_database.d.ts +22 -1
  113. package/src/wasm/fs-database/kbdb_fs_database.js +28 -1
  114. package/src/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
  115. package/src/wasm/fs-database/kbdb_fs_database_bg.wasm.d.ts +1 -0
  116. package/src/wasm/kb-worker/kbdb_worker.d.ts +48 -1
  117. package/src/wasm/kb-worker/kbdb_worker.js +60 -1
  118. package/src/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
  119. package/src/wasm/kb-worker/kbdb_worker_bg.wasm.d.ts +2 -0
  120. package/src/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
@@ -278,6 +278,40 @@ for storage. They are pre-computed once at insertion
278
278
  time and reused on every search -- trading a small
279
279
  write-time cost for significant query-time speed.
280
280
 
281
+ Additional metadata fields support truth maintenance
282
+ and provenance tracking:
283
+
284
+ - **`source_key`** -- Logical identity hash (16
285
+ lowercase hex characters). Derived from the source
286
+ file path and heading text:
287
+ `sha256("${source_path}\0${heading}")[..8]`. Used
288
+ for supersession -- matching existing sections when
289
+ content is re-learned. `null` for stdin input and
290
+ sections without a source path. See **Logical
291
+ Identity (Source Key)** below.
292
+ - **`source_path`** -- Absolute path of the file that
293
+ was learned. `null` for stdin input and
294
+ programmatic API calls. Stored for provenance
295
+ tracking and source-aware garbage collection.
296
+ - **`source_mtime`** -- ISO 8601 timestamp of the
297
+ source file's last modification time at learn time.
298
+ `null` when no source file exists. Allows staleness
299
+ detection: if the source file's mtime has changed
300
+ since this timestamp, the section may be outdated.
301
+ - **`updated_at`** -- ISO 8601 timestamp of the last
302
+ supersession. `null` on first creation. Set when a
303
+ section supersedes an existing one with the same
304
+ source key.
305
+ - **`tags`** -- Array of user-supplied string tags for
306
+ scoping and filtering. Empty array when no tags are
307
+ provided. Tags are not indexed for search -- they
308
+ are metadata for query-time filtering.
309
+ - **`supersedes`** -- The kbid of the section this one
310
+ replaced via supersession. `null` on first creation
311
+ and for non-superseding additions. Creates a
312
+ supersession chain that can be traversed to find
313
+ prior versions.
314
+
281
315
  The `add_section` operation rejects requests with
282
316
  `image/*` type that do not include a `title`. Text
283
317
  and code sections may omit both fields.
@@ -322,6 +356,12 @@ body_token_count = 187
322
356
  code_token_count = 0
323
357
  created_at = "2026-06-15T10:00:00Z"
324
358
  checksum = "sha256:abcdef..."
359
+ source_key = "a1b2c3d4e5f67890"
360
+ source_path = "/home/user/docs/architecture.md"
361
+ source_mtime = "2026-06-15T09:45:00Z"
362
+ updated_at = ""
363
+ tags = ["architecture", "overview"]
364
+ supersedes = ""
325
365
  ---
326
366
  Section body text here.
327
367
 
@@ -347,6 +387,12 @@ body_token_count = 0
347
387
  code_token_count = 22
348
388
  created_at = "2026-06-15T10:15:00Z"
349
389
  checksum = "sha256:1a2b3c..."
390
+ source_key = "f8e7d6c5b4a39281"
391
+ source_path = "/home/user/src/greet.ts"
392
+ source_mtime = "2026-06-15T10:10:00Z"
393
+ updated_at = ""
394
+ tags = ["api", "greeting"]
395
+ supersedes = ""
350
396
  ---
351
397
  export function greet(name: string): string {
352
398
  return `Hello, ${name}!`;
@@ -368,6 +414,12 @@ body_token_count = 3
368
414
  code_token_count = 0
369
415
  created_at = "2026-06-15T10:30:00Z"
370
416
  checksum = "sha256:fedcba..."
417
+ source_key = "1029384756abcdef"
418
+ source_path = "/home/user/docs/arch-diagram.png"
419
+ source_mtime = "2026-06-15T10:25:00Z"
420
+ updated_at = ""
421
+ tags = ["diagram", "architecture"]
422
+ supersedes = ""
371
423
  ---
372
424
  iVBORw0KGgoAAAANSUhEUgAA...base64 data...
373
425
  ```
@@ -463,6 +515,48 @@ Updating a section's content creates a new file with
463
515
  a new kbid; the old file is eligible for garbage
464
516
  collection.
465
517
 
518
+ #### Logical Identity (Source Key)
519
+
520
+ Content-addressing identifies *what a section
521
+ currently says*. The **source key** identifies *what
522
+ a section is about* -- it is a stable logical
523
+ identity that persists across content edits.
524
+
525
+ The source key is derived from the section's origin:
526
+
527
+ ```
528
+ source_key = sha256("${source_path}\0${heading}")[..8]
529
+ → 16 lowercase hex characters
530
+ ```
531
+
532
+ - `source_path` -- the absolute path of the file that
533
+ was learned (normalized: resolved symlinks, no
534
+ trailing separator).
535
+ - `heading` -- the section's title text (or empty
536
+ string when no title is provided).
537
+ - `\0` -- a null byte separator.
538
+
539
+ The SHA-256 digest is truncated to 8 bytes (64 bits)
540
+ and hex-encoded to produce a 16-character identifier.
541
+
542
+ Source keys enable **supersession**: when content is
543
+ re-learned from the same source with the same
544
+ heading, the system can identify the existing section
545
+ and replace it rather than creating a sibling. This
546
+ prevents the brain from silently accumulating
547
+ contradictions when facts are corrected.
548
+
549
+ Source key is `null` for:
550
+ - Stdin input (`-`) -- no source path exists.
551
+ - Sections created without a source path (e.g.
552
+ programmatic API calls that omit it).
553
+
554
+ When `source_key` is `null`, supersession is not
555
+ available and the section behaves as a pure
556
+ content-addressed entry (current behavior).
557
+
558
+ See **Add section** below for supersession semantics.
559
+
466
560
  ### Documents (`documents/`)
467
561
 
468
562
  One TOML file per document. A document groups sections
@@ -667,14 +761,15 @@ directly.
667
761
  `fs-database.wasm`, which writes the TOML header
668
762
  (kbid, docids, type, title, description, size,
669
763
  token_count, heading_token_count, body_token_count,
670
- code_token_count, timestamp, checksum), the
671
- separator, and the content body (preprocessed text
672
- for text and code sections, base64 for image
673
- sections). Updates `catalog.toml` counts. Updates
674
- the inverted index with per-field term frequencies,
675
- the positional index with term positions, and
676
- corpus statistics (including per-field average
677
- lengths).
764
+ code_token_count, created_at, checksum, source_key,
765
+ source_path, source_mtime, updated_at, tags,
766
+ supersedes), the separator, and the content body
767
+ (preprocessed text for text and code sections,
768
+ base64 for image sections). Updates `catalog.toml`
769
+ counts. Updates the inverted index with per-field
770
+ term frequencies, the positional index with term
771
+ positions, and corpus statistics (including
772
+ per-field average lengths).
678
773
 
679
774
  Image sections (`image/*` type) have their binary
680
775
  content stored but not indexed. However, their
@@ -682,9 +777,50 @@ directly.
682
777
  attributes are tokenized and indexed, making image
683
778
  sections searchable via their metadata.
684
779
 
780
+ **Supersession:** When the `replace` flag is set
781
+ and the new section has a non-null `source_key`,
782
+ `fs-database.wasm` searches for an existing section
783
+ with the same `source_key`. If found:
784
+
785
+ - The new section's `supersedes` field is set to
786
+ the old section's kbid.
787
+ - The new section's `updated_at` is set to the
788
+ current timestamp.
789
+ - The old kbid is replaced with the new kbid in all
790
+ document manifests that reference it.
791
+ - The old section's `docids` array is emptied,
792
+ making it eligible for garbage collection.
793
+ - Index entries for the old section are removed and
794
+ replaced with entries for the new section.
795
+
796
+ When `replace` is not set, or `source_key` is
797
+ `null`, the add operation behaves as before (pure
798
+ content-addressed addition with deduplication).
799
+
800
+ **Near-duplicate detection:** After the kbid is
801
+ computed and deduplication is checked, the add
802
+ operation computes a Jaccard similarity coefficient
803
+ between the new section's stemmed term set and
804
+ existing sections' term sets. Sections exceeding a
805
+ similarity threshold of 0.85 are reported as
806
+ near-duplicates in the response
807
+ (`near_duplicates: [{kbid, similarity}]`). This is
808
+ advisory -- the section is still stored. The check
809
+ uses the inverted index for efficient comparison
810
+ without reading every section's content.
811
+
812
+ **Heading-less import warning:** Text and code
813
+ sections added without a `title` emit a warning in
814
+ the response: `heading_missing`. The heading field
815
+ carries 2.0x ranking weight; omitting it means the
816
+ section loses its strongest searchability signal.
817
+ The section is still stored -- this is a warning,
818
+ not an error.
819
+
685
820
  **Validation:** requests with `image/*` type that
686
821
  omit `title` are rejected with an error. Text and
687
- code sections may omit `title`.
822
+ code sections may omit `title` (with the
823
+ heading-less warning above).
688
824
 
689
825
  3. **Update section** -- Given a kb-id and new content,
690
826
  replace the section's content. Since kb-ids are
@@ -795,12 +931,26 @@ directly.
795
931
  sections (not referenced by any document),
796
932
  orphaned documents (referencing missing sections),
797
933
  broken embedding references -- markers whose
798
- target does not exist or has the wrong type -- and
934
+ target does not exist or has the wrong type,
799
935
  circular reference chains in `$(text:)` and
800
- `$(document:)` markers.
936
+ `$(document:)` markers, and **divergent content**
937
+ -- multiple sections sharing the same `source_key`
938
+ (same origin) but with different content,
939
+ indicating contradictory knowledge that should
940
+ have been resolved by supersession.
801
941
  13. **Garbage collection** -- Remove section files not
802
- referenced by any document manifest.
803
- 14. **Index rebuild** -- Reconstruct all index files
942
+ referenced by any document manifest. Additionally
943
+ report `stale_sources` -- sections whose
944
+ `source_path` points to a file that no longer
945
+ contains their heading text. Stale sources are
946
+ advisory (not automatically removed) and surfaced
947
+ in the GC response for manual review.
948
+ 14. **Export** -- Copy sections, documents, and
949
+ catalog to a self-contained output directory (no
950
+ indexes -- they are rebuildable). Provides a
951
+ portable, git-friendly snapshot of the knowledge
952
+ base for backup and versioning.
953
+ 15. **Index rebuild** -- Reconstruct all index files
804
954
  from sections and documents. Re-tokenizes all text
805
955
  and code sections per field, recomputes per-field
806
956
  term frequencies and token counts (updating
@@ -975,8 +1125,11 @@ Response per mode:
975
1125
  "type": "text/markdown",
976
1126
  "heading": "Section Heading",
977
1127
  "score": 8.74,
1128
+ "confidence": 1.0,
978
1129
  "snippet": "...matching context...",
979
- "matched_terms": ["term1", "term2"]
1130
+ "matched_terms": ["term1", "term2"],
1131
+ "matched_fields": ["heading", "body"],
1132
+ "created_at": "2026-06-15T10:00:00Z"
980
1133
  }
981
1134
  ],
982
1135
  "total": 42,
@@ -1058,7 +1211,10 @@ Response per mode:
1058
1211
  "type": "text/markdown",
1059
1212
  "title": "Architecture Overview",
1060
1213
  "description": "High-level overview",
1061
- "docid": "<doc id>"
1214
+ "docid": "<doc id>",
1215
+ "source_path": "/home/user/docs/architecture.md",
1216
+ "tags": ["architecture", "overview"],
1217
+ "replace": false
1062
1218
  }
1063
1219
 
1064
1220
  // Response
@@ -1069,7 +1225,10 @@ Response per mode:
1069
1225
  "description": "High-level overview",
1070
1226
  "indexed_terms_count": 42,
1071
1227
  "elapsed_ms": 15,
1072
- "attached": false
1228
+ "attached": false,
1229
+ "superseded": null,
1230
+ "warnings": [],
1231
+ "near_duplicates": []
1073
1232
  }
1074
1233
  ```
1075
1234
 
@@ -1302,6 +1461,10 @@ The integrity check verifies:
1302
1461
  - **circular_reference** -- a `$(text:)` or
1303
1462
  `$(document:)` marker chain loops back to an entity
1304
1463
  already in the expansion path.
1464
+ - **divergent_content** -- two or more sections share
1465
+ the same `source_key` but have different content
1466
+ (different kbids). Indicates contradictory knowledge
1467
+ that should have been resolved by supersession.
1305
1468
 
1306
1469
  When `ok` is `true`, the `errors` array is empty.
1307
1470
 
@@ -1315,6 +1478,7 @@ When `ok` is `true`, the `errors` array is empty.
1315
1478
  {
1316
1479
  "removed_sections": 3,
1317
1480
  "removed_bytes": 15360,
1481
+ "stale_sources": 1,
1318
1482
  "elapsed_ms": 25
1319
1483
  }
1320
1484
  ```
@@ -1327,6 +1491,12 @@ remove documents, indexes, or the catalog. After
1327
1491
  removal, all caches are invalidated to prevent stale
1328
1492
  reads.
1329
1493
 
1494
+ `stale_sources` reports the number of sections whose
1495
+ `source_path` points to a file that no longer
1496
+ contains their heading text. These sections may be
1497
+ outdated but are not automatically removed -- they
1498
+ are reported for manual review.
1499
+
1330
1500
  **Index rebuild:**
1331
1501
 
1332
1502
  ```json
@@ -1357,6 +1527,30 @@ entries). Equivalent to deleting `indexes/` and
1357
1527
  re-adding every section. After rebuild, all caches
1358
1528
  are invalidated.
1359
1529
 
1530
+ **Export:**
1531
+
1532
+ ```json
1533
+ // Request
1534
+ {
1535
+ "path": "/home/user/backup/kbdb-export"
1536
+ }
1537
+
1538
+ // Response
1539
+ {
1540
+ "path": "/home/user/backup/kbdb-export",
1541
+ "sections_exported": 42,
1542
+ "documents_exported": 5,
1543
+ "elapsed_ms": 120
1544
+ }
1545
+ ```
1546
+
1547
+ Export copies sections, documents, and `catalog.toml`
1548
+ to the specified output directory. Indexes are not
1549
+ included -- they are derived data and can be rebuilt
1550
+ with `rebuild` after importing. The export directory
1551
+ is a self-contained, git-friendly snapshot suitable
1552
+ for backup, transfer, or version control.
1553
+
1360
1554
  ### Error Responses
1361
1555
 
1362
1556
  Errors are returned as JSON-RPC 2.0 error objects:
@@ -1411,6 +1605,34 @@ See [Query Parser](query-parser.md) for full details.
1411
1605
  | SHA-256 + Truncation + Base32 | SHA-256 of the typed header, truncated to 128 bits (16 bytes), base32 lowercase (no padding) -- 26 alphanumeric characters |
1412
1606
  | Content Preprocessing | Text and code: BOM removal, `\r\n`/`\r` to `\n`, Unicode NFC, whitespace trimming. Images: base64 string used as-is |
1413
1607
 
1608
+ ## Concurrent Write Behavior
1609
+
1610
+ The daemon serializes all write operations through a
1611
+ **single-writer lock**. The lock is an exclusive file
1612
+ lock on `kbdb-{ctx}.lock` in the OS temporary
1613
+ directory, keyed to the same context ID used for the
1614
+ PID file and IPC socket.
1615
+
1616
+ - **Reads** proceed without acquiring the lock. The
1617
+ daemon serves concurrent read requests from
1618
+ multiple worker client connections.
1619
+ - **Writes** (add section, update section, remove
1620
+ section, group/remove document, garbage collection,
1621
+ index rebuild, export) acquire the lock before
1622
+ modifying any file. If another write is in progress,
1623
+ the request queues behind the lock.
1624
+ - The lock is released after each write operation
1625
+ completes, not held across multiple operations.
1626
+ - CLI invocations sharing the same daemon are
1627
+ serialized automatically.
1628
+ - Direct file access to `.kbdb` outside the daemon is
1629
+ not supported and may cause corruption. All
1630
+ mutations must go through the daemon's IPC API.
1631
+
1632
+ This model is equivalent to SQLite's default locking:
1633
+ one writer at a time, concurrent readers, no external
1634
+ coordination required.
1635
+
1414
1636
  ## Design Constraints
1415
1637
 
1416
1638
  - `fs-database.wasm` is the sole owner of `.kbdb` I/O.
@@ -1423,6 +1645,10 @@ See [Query Parser](query-parser.md) for full details.
1423
1645
  their content hash. `fs-database.wasm` never modifies
1424
1646
  a section file in place -- updates produce new files
1425
1647
  with new kb-ids.
1648
+ - Source keys are advisory logical identity. The
1649
+ content hash (kbid) remains the primary identifier.
1650
+ Supersession is opt-in via the `replace` flag --
1651
+ default learn behavior is unchanged.
1426
1652
  - Indexes are derived data. They can be rebuilt entirely
1427
1653
  from sections and document manifests. Loss of index
1428
1654
  files is recoverable.
@@ -1437,7 +1663,8 @@ See [Query Parser](query-parser.md) for full details.
1437
1663
  - No external database process or server dependency.
1438
1664
  - The database must support concurrent read access from
1439
1665
  multiple CLI invocations via the shared daemon without
1440
- corruption.
1666
+ corruption. Write operations are serialized through
1667
+ the daemon's single-writer lock.
1441
1668
  - Garbage collection of unreferenced sections is
1442
1669
  `fs-database.wasm`'s responsibility. Sections not
1443
1670
  listed in any document manifest may be removed.
@@ -289,7 +289,7 @@ The tokenization pipeline removes them in Stage 1
289
289
  search index.
290
290
 
291
291
  See [Query Parser](query-parser.md) for the full
292
- eight-stage tokenization pipeline and
292
+ nine-stage tokenization pipeline and
293
293
  [Content Parser](content-parser.md) for the marker
294
294
  extraction functions used during stripping.
295
295
 
package/docs/goals/mcp.md CHANGED
@@ -50,6 +50,17 @@ the user.
50
50
  See [CLI -- Run Modes](cli.md#run-modes) for how
51
51
  the `kbdb` executable dispatches to MCP mode.
52
52
 
53
+ ## Startup Health Check
54
+
55
+ When the MCP server starts, it runs
56
+ `client.integrityCheck()` automatically after the
57
+ worker daemon connection is established. If the
58
+ check returns `ok: false`, the server emits a
59
+ `notifications/message` log with level `warning`
60
+ listing the error summary. The server continues to
61
+ operate -- health issues are not fatal. This ensures
62
+ corruption surfaces before it produces wrong answers.
63
+
53
64
  ## Installation and Wiring
54
65
 
55
66
  Installation is the same as the CLI:
@@ -332,6 +343,7 @@ daemon via the worker client.
332
343
 
333
344
  Search the knowledge base. Returns ranked results
334
345
  with headings and snippets for relevance evaluation.
346
+ Use multiple short keyword queries for best coverage.
335
347
  Use `recall` to get full content of specific results.
336
348
 
337
349
  **Parameters:**
@@ -358,13 +370,16 @@ additional calls:
358
370
  "type": "text/markdown",
359
371
  "docids": ["doc-5678"],
360
372
  "snippet": "The system is organized...",
361
- "matched_terms": ["architecture", "system"]
373
+ "matched_terms": ["architecture", "system"],
374
+ "matched_fields": ["heading", "body"],
375
+ "created_at": "2026-06-20T14:30:00Z"
362
376
  }
363
377
  ],
364
378
  "total": 42,
365
379
  "offset": 0,
366
380
  "limit": 20,
367
- "has_more": true
381
+ "has_more": true,
382
+ "relaxed": false
368
383
  }
369
384
  ```
370
385
 
@@ -379,6 +394,15 @@ additional calls:
379
394
  matched term. Empty string for image sections.
380
395
  - `matched_terms` -- query terms that matched, as a
381
396
  flat `string[]`. No frequency counts.
397
+ - `matched_fields` -- subset of `["heading", "body",
398
+ "code"]` indicating which fields contained matches.
399
+ - `created_at` -- ISO 8601 timestamp of when the
400
+ section was first indexed.
401
+ - `relaxed` -- `true` when the result came from an
402
+ automatic AND-to-OR fallback (see
403
+ [Query Result -- Relaxed-Match
404
+ Fallback](query-result.md)). `false` for normal
405
+ results.
382
406
 
383
407
  When `content` is `true`, each item also includes
384
408
  a `content` field with the full section text.
@@ -393,20 +417,24 @@ Maps to `client.search()`.
393
417
 
394
418
  ### learn
395
419
 
396
- Import content into the knowledge base. The daemon
397
- decomposes the content into sections, extracts
398
- keywords, builds indexes, and assigns ranking
399
- weights.
420
+ Import content into the knowledge base. Use for
421
+ durable facts, decisions, and corrections. Use
422
+ `--replace` when updating existing knowledge. Attach
423
+ tags for scoping. The daemon decomposes the content
424
+ into sections, extracts keywords, builds indexes,
425
+ and assigns ranking weights.
400
426
 
401
427
  **Parameters:**
402
428
 
403
429
  | Name | Type | Required | Description |
404
430
  |---------------|--------|----------|------------------|
405
- | `content` | string | yes | Text content or base64-encoded image data |
406
- | `type` | string | no | `"text/markdown"` (default), `text/*` MIME type, `"application/json"`, or `image/*` MIME type |
407
- | `title` | string | no* | Section title. ***Required** for `image/*` types.* Optional for text and code types. |
408
- | `description` | string | no | Optional section description |
409
- | `docid` | string | no | Group under this document ID |
431
+ | `content` | string | yes | Text content or base64-encoded image data |
432
+ | `type` | string | no | `"text/markdown"` (default), `text/*` MIME type, `"application/json"`, or `image/*` MIME type |
433
+ | `title` | string | no* | Section title. ***Required** for `image/*` types.* Optional for text and code types. |
434
+ | `description` | string | no | Optional section description |
435
+ | `docid` | string | no | Group under this document ID |
436
+ | `tags` | string[] | no | Tags for scoping and filtering |
437
+ | `replace` | boolean | no | Enable supersession by source key. When `true` and a section with the same source key exists, the old section is replaced. |
410
438
 
411
439
  **Content type rules:**
412
440
 
@@ -441,6 +469,16 @@ Text and code sections return a positive
441
469
  `indexed_terms_count`. Image sections return
442
470
  `indexed_terms_count: 0`.
443
471
 
472
+ Additional result fields when applicable:
473
+ - `superseded` -- kbid of the section this one
474
+ replaced via `replace: true`, or `null`.
475
+ - `warnings` -- string array of advisory warnings
476
+ (e.g. `"heading_missing"` when a text section has
477
+ no title).
478
+ - `near_duplicates` -- array of
479
+ `{ kbid, similarity }` objects for existing
480
+ sections with Jaccard similarity >= 0.85.
481
+
444
482
  If the content already exists in the database (same
445
483
  kb-id) and `docid` is provided, the section is not
446
484
  duplicated. Instead, the docid is added to the
@@ -662,8 +700,12 @@ checks pass, `ok` is `true` and `errors` is empty.
662
700
 
663
701
  Error types: `checksum_mismatch`, `orphan_section`,
664
702
  `orphan_document`, `broken_reference`,
665
- `type_mismatch`, `circular_reference`. See
666
- [Database -- Integrity and
703
+ `type_mismatch`, `circular_reference`,
704
+ `divergent_content`. The `divergent_content` error
705
+ flags sections sharing the same `source_key` that
706
+ have different content -- indicates contradictory
707
+ versions that should be resolved via supersession.
708
+ See [Database -- Integrity and
667
709
  Maintenance](database.md).
668
710
 
669
711
  Maps to `client.integrityCheck()`.
@@ -696,6 +738,32 @@ this to recover from index corruption.
696
738
 
697
739
  Maps to `client.rebuildIndexes()`.
698
740
 
741
+ ### export
742
+
743
+ Export the knowledge base as a self-contained
744
+ directory snapshot for backup or versioning. Copies
745
+ sections, documents, and catalog (no indexes -- they
746
+ are rebuildable via `rebuild`).
747
+
748
+ **Parameters:**
749
+
750
+ | Name | Type | Required | Description |
751
+ |--------|--------|----------|---------------------------|
752
+ | `path` | string | no | Export destination path (default: `./kbdb-export/`) |
753
+
754
+ **Returns:** Compact JSON with export summary:
755
+
756
+ ```json
757
+ {
758
+ "path": "/home/user/kbdb-export",
759
+ "sections_exported": 142,
760
+ "documents_exported": 23,
761
+ "elapsed_ms": 350
762
+ }
763
+ ```
764
+
765
+ Maps to `client.export()`.
766
+
699
767
  ### skill-learn
700
768
 
701
769
  Create a new skill definition and store it in the
@@ -1057,7 +1125,7 @@ interface McpToolAnnotations {
1057
1125
  }
1058
1126
  ```
1059
1127
 
1060
- Annotations for all 18 kbdb tools:
1128
+ Annotations for all 19 kbdb tools:
1061
1129
 
1062
1130
  | Tool | title | readOnly | destructive | idempotent | openWorld |
1063
1131
  |----------------|-------------------|----------|-------------|------------|-----------|
@@ -1071,6 +1139,7 @@ Annotations for all 18 kbdb tools:
1071
1139
  | `check` | Integrity Check | true | false | true | false |
1072
1140
  | `gc` | Garbage Collect | false | true | false | false |
1073
1141
  | `rebuild` | Rebuild Indexes | false | false | false | false |
1142
+ | `export` | Export Snapshot | false | false | true | false |
1074
1143
  | `skill-learn` | Create Skill | false | false | true | false |
1075
1144
  | `skill-list` | List Skills | true | false | true | false |
1076
1145
  | `skill-get` | Get Skill | true | false | true | false |
@@ -115,9 +115,11 @@ Stages:
115
115
  decomposition).
116
116
  5. **Case folding** (lowercase).
117
117
  6. **Stop word removal** (configurable list).
118
- 7. **Stemming** (Snowball / Porter2 -- stores both
118
+ 7. **Synonym expansion** (bidirectional synonym map --
119
+ inserts synonym tokens alongside originals).
120
+ 8. **Stemming** (Snowball / Porter2 -- stores both
119
121
  stemmed and original forms).
120
- 8. **N-gram extraction** (bigrams, trigrams for phrase
122
+ 9. **N-gram extraction** (bigrams, trigrams for phrase
121
123
  indexing and multi-word term matching).
122
124
 
123
125
  `kb-worker.wasm` calls the tokenization pipeline for
@@ -126,6 +128,71 @@ and search (when a query is submitted). It feeds the
126
128
  tokenized output to `fs-database.wasm` for index
127
129
  storage or index lookups as needed.
128
130
 
131
+ ### Synonym Expansion (Stage 7)
132
+
133
+ Bridges the lexical gap between user vocabulary and
134
+ indexed content. Without synonym expansion, a search
135
+ for `"login credentials"` returns zero results when
136
+ the indexed document uses "signs in" and "identity"
137
+ instead -- different root words with the same meaning.
138
+
139
+ **Placement rationale:** Applied after stop-word
140
+ removal (stage 6) and before stemming (stage 8). This
141
+ ensures synonyms are expanded on lowercased,
142
+ stop-word-filtered tokens, and both the original and
143
+ synonym tokens are then stemmed consistently.
144
+
145
+ **Built-in synonym map:** A small, curated
146
+ `HashMap<String, Vec<String>>` of bidirectional
147
+ synonym pairs. Bidirectional means if `login` maps to
148
+ `sign-in`, then `sign-in` also maps to `login`.
149
+
150
+ **Example map entries:**
151
+
152
+ | Term | Synonyms |
153
+ |--------------|------------------------------------|
154
+ | login | sign-in, sign_in, signin |
155
+ | auth | authentication, authenticate |
156
+ | remove | delete, erase |
157
+ | create | add, insert |
158
+ | update | modify, change, edit |
159
+ | error | exception, fault, failure |
160
+ | config | configuration, settings |
161
+ | credential | password, identity |
162
+ | user | account, member |
163
+ | deploy | release, publish, ship |
164
+ | search | query, find, lookup |
165
+ | doc | document, documentation |
166
+ | api | endpoint, interface |
167
+ | db | database |
168
+ | env | environment |
169
+ | repo | repository |
170
+ | param | parameter, argument |
171
+ | dir | directory, folder |
172
+ | msg | message |
173
+ | info | information |
174
+
175
+ **Expansion behaviour:** For each token in the stream,
176
+ look up the synonym map. If found, insert the synonym
177
+ tokens at the same position as the original. Both the
178
+ original and its synonyms proceed to the stemming
179
+ stage. This preserves the original token's position
180
+ for proximity/zone scoring while adding synonym
181
+ coverage.
182
+
183
+ **Applied symmetrically:** The same synonym expansion
184
+ applies during both `learn` (indexing) and `search`
185
+ (querying). If a document uses "login" and the user
186
+ searches "sign-in", the expansion during indexing
187
+ produced "sign-in" tokens from "login", and the
188
+ expansion during querying produced "login" tokens
189
+ from "sign-in".
190
+
191
+ **Not user-configurable in v1.** The built-in map is
192
+ hardcoded. Future versions may allow user-defined
193
+ synonym maps via a `synonyms.toml` configuration file
194
+ in the `.kbdb` directory.
195
+
129
196
  ## Query Parsing
130
197
 
131
198
  Search queries are parsed into a structured query plan
@@ -148,8 +215,35 @@ Parsing stages:
148
215
  The query plan is the query parser's output --
149
216
  `kb-worker.wasm` receives it and passes it to
150
217
  `fs-database.wasm` for index lookups, then calls the
151
- query parser's ranking functions to score and order the raw
152
- results.
218
+ query parser's ranking functions to score and order
219
+ the raw results.
220
+
221
+ ### Relaxed Query Mode
222
+
223
+ When a multi-term AND query returns zero results,
224
+ `kb-worker.wasm` retries the query in **relaxed
225
+ mode** -- OR semantics where any term matches, not
226
+ all required.
227
+
228
+ The query parser generates both the primary (AND)
229
+ plan and a relaxed (OR) plan from the same parsed
230
+ query. `kb-worker.wasm` executes the AND plan first.
231
+ If the AND plan produces zero matches, it falls back
232
+ to the OR plan. No new WASM export is needed --
233
+ `qp_parse_query` returns a `QueryPlan` that supports
234
+ mode switching via a `match_mode` field
235
+ (`All` | `Any`).
236
+
237
+ Relaxed results are flagged with `relaxed: true` in
238
+ the `PagedSearchResult` envelope (see
239
+ [Query Result -- Relaxed-Match
240
+ Fallback](query-result.md)). The fallback is
241
+ transparent to the caller.
242
+
243
+ Single-term queries are unaffected (AND and OR are
244
+ equivalent for one term). Phrase queries and
245
+ exclusions are preserved in both modes -- only the
246
+ inter-term combination changes.
153
247
 
154
248
  ## Query Cache
155
249