@dikolab/kbdb 0.2.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -15
- package/dist/cli.cjs +461 -72
- package/dist/cli.cjs.map +4 -4
- package/dist/cli.mjs +453 -64
- package/dist/cli.mjs.map +4 -4
- package/dist/kbdb-worker.cjs +395 -60
- package/dist/kbdb-worker.cjs.map +4 -4
- package/dist/mod.cjs +17 -0
- package/dist/mod.cjs.map +2 -2
- package/dist/mod.mjs +17 -0
- package/dist/mod.mjs.map +2 -2
- package/dist/src/shared/cli/functions/dispatch-command.function.d.ts +3 -1
- package/dist/src/shared/cli/functions/install-default-skills.function.d.ts +12 -0
- package/dist/src/shared/cli/functions/run-check.function.d.ts +47 -5
- package/dist/src/shared/cli/functions/run-export.function.d.ts +34 -0
- package/dist/src/shared/cli/functions/run-gc.function.d.ts +6 -0
- package/dist/src/shared/cli/functions/run-learn.function.d.ts +35 -6
- package/dist/src/shared/cli/functions/run-recall.function.d.ts +45 -0
- package/dist/src/shared/cli/functions/run-search.function.d.ts +52 -15
- package/dist/src/shared/cli/index.d.ts +4 -0
- package/dist/src/shared/cli/typings/cli-options.interface.d.ts +13 -0
- package/dist/src/shared/cli/typings/learn-result.model.d.ts +19 -0
- package/dist/src/shared/cli/typings/search-display.model.d.ts +18 -0
- package/dist/src/shared/mcp/classes/mcp-server.class.d.ts +8 -0
- package/dist/src/shared/mcp/typings/mcp-client.interface.d.ts +43 -0
- package/dist/src/shared/wasm-codec/functions/decode-add-section-result.function.d.ts +48 -0
- package/dist/src/shared/wasm-codec/functions/decode-scored-results.function.d.ts +58 -0
- package/dist/src/shared/wasm-codec/functions/decode-section-records.function.d.ts +33 -0
- package/dist/src/shared/wasm-codec/functions/encode-add-section-params.function.d.ts +52 -0
- package/dist/src/shared/wasm-codec/index.d.ts +2 -0
- package/dist/src/shared/worker-client/classes/worker-client.class.d.ts +16 -0
- package/dist/src/shared/worker-client/index.d.ts +1 -1
- package/dist/src/shared/worker-client/typings/add-section-result.model.d.ts +29 -15
- package/dist/src/shared/worker-client/typings/export-result.model.d.ts +14 -0
- package/dist/src/shared/worker-client/typings/gc-result.model.d.ts +6 -0
- package/dist/src/shared/worker-client/typings/search-result.model.d.ts +12 -0
- package/dist/src/shared/worker-client/typings/section.model.d.ts +16 -0
- package/dist/src/shared/worker-daemon/classes/worker-daemon.class.d.ts +35 -0
- package/dist/wasm/fs-database/kbdb_fs_database.d.ts +22 -1
- package/dist/wasm/fs-database/kbdb_fs_database.js +28 -1
- package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
- package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm.d.ts +1 -0
- package/dist/wasm/kb-worker/kbdb_worker.d.ts +48 -1
- package/dist/wasm/kb-worker/kbdb_worker.js +60 -1
- package/dist/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
- package/dist/wasm/kb-worker/kbdb_worker_bg.wasm.d.ts +2 -0
- package/dist/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
- package/dist/worker.mjs +396 -60
- package/dist/worker.mjs.map +4 -4
- package/docs/details/cli.md +102 -15
- package/docs/details/library-api.md +55 -6
- package/docs/details/mcp-server.md +8 -4
- package/docs/details/search-and-ranking.md +63 -5
- package/docs/details/storage.md +2 -0
- package/docs/goals/cli.md +255 -14
- package/docs/goals/database.md +244 -17
- package/docs/goals/document.md +1 -1
- package/docs/goals/mcp.md +83 -14
- package/docs/goals/query-parser.md +98 -4
- package/docs/goals/query-result.md +177 -10
- package/docs/goals/skills.md +72 -0
- package/docs/goals/worker-daemon.md +54 -7
- package/docs/modules/rust/fs-database/functions/integrity.md +11 -0
- package/docs/modules/rust/fs-database/functions/section.md +23 -6
- package/docs/modules/rust/fs-database/module.md +1 -0
- package/docs/modules/rust/fs-database/types/section.md +27 -1
- package/docs/modules/rust/overview.md +2 -1
- package/docs/modules/rust/query-parser/functions/module.md +2 -2
- package/docs/modules/rust/query-parser/functions/pipeline.md +44 -6
- package/docs/modules/rust/query-parser/module.md +5 -4
- package/docs/modules/rust/query-parser/types/keyphrase.md +11 -0
- package/docs/modules/rust/query-parser/types/module.md +3 -3
- package/docs/modules/typescript/shared/cli/functions.md +68 -3
- package/package.json +1 -1
- package/src/shared/cli/functions/dispatch-command.function.ts +53 -1
- package/src/shared/cli/functions/format-output.function.ts +36 -12
- package/src/shared/cli/functions/generate-help.function.ts +56 -1
- package/src/shared/cli/functions/install-default-skills.function.ts +296 -0
- package/src/shared/cli/functions/parse-args.function.ts +13 -0
- package/src/shared/cli/functions/run-check.function.ts +87 -6
- package/src/shared/cli/functions/run-export.function.ts +40 -0
- package/src/shared/cli/functions/run-gc.function.ts +6 -0
- package/src/shared/cli/functions/run-init.function.ts +2 -0
- package/src/shared/cli/functions/run-learn.function.ts +68 -16
- package/src/shared/cli/functions/run-recall.function.ts +66 -0
- package/src/shared/cli/functions/run-search.function.ts +58 -15
- package/src/shared/cli/index.ts +4 -0
- package/src/shared/cli/typings/cli-options.interface.ts +13 -0
- package/src/shared/cli/typings/learn-result.model.ts +16 -0
- package/src/shared/cli/typings/search-display.model.ts +18 -0
- package/src/shared/mcp/classes/mcp-server.class.ts +30 -1
- package/src/shared/mcp/functions/create-tool-definitions.function.ts +46 -5
- package/src/shared/mcp/functions/handle-tool-call.function.ts +70 -2
- package/src/shared/mcp/typings/mcp-client.interface.ts +45 -0
- package/src/shared/wasm-codec/functions/decode-add-section-result.function.ts +112 -0
- package/src/shared/wasm-codec/functions/decode-scored-results.function.ts +86 -0
- package/src/shared/wasm-codec/functions/decode-section-records.function.ts +87 -0
- package/src/shared/wasm-codec/functions/encode-add-section-params.function.ts +68 -0
- package/src/shared/wasm-codec/index.ts +5 -0
- package/src/shared/worker-client/classes/worker-client.class.ts +19 -0
- package/src/shared/worker-client/index.ts +4 -1
- package/src/shared/worker-client/typings/add-section-result.model.ts +30 -15
- package/src/shared/worker-client/typings/export-result.model.ts +14 -0
- package/src/shared/worker-client/typings/gc-result.model.ts +6 -0
- package/src/shared/worker-client/typings/search-result.model.ts +12 -0
- package/src/shared/worker-client/typings/section.model.ts +16 -0
- package/src/shared/worker-daemon/classes/worker-daemon.class.ts +315 -46
- package/src/shared/worker-daemon/functions/create-host-imports.function.ts +19 -4
- package/src/shared/worker-daemon/functions/create-ipc-server.function.ts +5 -1
- package/src/wasm/fs-database/kbdb_fs_database.d.ts +22 -1
- package/src/wasm/fs-database/kbdb_fs_database.js +28 -1
- package/src/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
- package/src/wasm/fs-database/kbdb_fs_database_bg.wasm.d.ts +1 -0
- package/src/wasm/kb-worker/kbdb_worker.d.ts +48 -1
- package/src/wasm/kb-worker/kbdb_worker.js +60 -1
- package/src/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
- package/src/wasm/kb-worker/kbdb_worker_bg.wasm.d.ts +2 -0
- package/src/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
package/docs/goals/database.md
CHANGED
|
@@ -278,6 +278,40 @@ for storage. They are pre-computed once at insertion
|
|
|
278
278
|
time and reused on every search -- trading a small
|
|
279
279
|
write-time cost for significant query-time speed.
|
|
280
280
|
|
|
281
|
+
Additional metadata fields support truth maintenance
|
|
282
|
+
and provenance tracking:
|
|
283
|
+
|
|
284
|
+
- **`source_key`** -- Logical identity hash (16
|
|
285
|
+
lowercase hex characters). Derived from the source
|
|
286
|
+
file path and heading text:
|
|
287
|
+
`sha256("${source_path}\0${heading}")[..8]`. Used
|
|
288
|
+
for supersession -- matching existing sections when
|
|
289
|
+
content is re-learned. `null` for stdin input and
|
|
290
|
+
sections without a source path. See **Logical
|
|
291
|
+
Identity (Source Key)** below.
|
|
292
|
+
- **`source_path`** -- Absolute path of the file that
|
|
293
|
+
was learned. `null` for stdin input and
|
|
294
|
+
programmatic API calls. Stored for provenance
|
|
295
|
+
tracking and source-aware garbage collection.
|
|
296
|
+
- **`source_mtime`** -- ISO 8601 timestamp of the
|
|
297
|
+
source file's last modification time at learn time.
|
|
298
|
+
`null` when no source file exists. Allows staleness
|
|
299
|
+
detection: if the source file's mtime has changed
|
|
300
|
+
since this timestamp, the section may be outdated.
|
|
301
|
+
- **`updated_at`** -- ISO 8601 timestamp of the last
|
|
302
|
+
supersession. `null` on first creation. Set when a
|
|
303
|
+
section supersedes an existing one with the same
|
|
304
|
+
source key.
|
|
305
|
+
- **`tags`** -- Array of user-supplied string tags for
|
|
306
|
+
scoping and filtering. Empty array when no tags are
|
|
307
|
+
provided. Tags are not indexed for search -- they
|
|
308
|
+
are metadata for query-time filtering.
|
|
309
|
+
- **`supersedes`** -- The kbid of the section this one
|
|
310
|
+
replaced via supersession. `null` on first creation
|
|
311
|
+
and for non-superseding additions. Creates a
|
|
312
|
+
supersession chain that can be traversed to find
|
|
313
|
+
prior versions.
|
|
314
|
+
|
|
281
315
|
The `add_section` operation rejects requests with
|
|
282
316
|
`image/*` type that do not include a `title`. Text
|
|
283
317
|
and code sections may omit both fields.
|
|
@@ -322,6 +356,12 @@ body_token_count = 187
|
|
|
322
356
|
code_token_count = 0
|
|
323
357
|
created_at = "2026-06-15T10:00:00Z"
|
|
324
358
|
checksum = "sha256:abcdef..."
|
|
359
|
+
source_key = "a1b2c3d4e5f67890"
|
|
360
|
+
source_path = "/home/user/docs/architecture.md"
|
|
361
|
+
source_mtime = "2026-06-15T09:45:00Z"
|
|
362
|
+
updated_at = ""
|
|
363
|
+
tags = ["architecture", "overview"]
|
|
364
|
+
supersedes = ""
|
|
325
365
|
---
|
|
326
366
|
Section body text here.
|
|
327
367
|
|
|
@@ -347,6 +387,12 @@ body_token_count = 0
|
|
|
347
387
|
code_token_count = 22
|
|
348
388
|
created_at = "2026-06-15T10:15:00Z"
|
|
349
389
|
checksum = "sha256:1a2b3c..."
|
|
390
|
+
source_key = "f8e7d6c5b4a39281"
|
|
391
|
+
source_path = "/home/user/src/greet.ts"
|
|
392
|
+
source_mtime = "2026-06-15T10:10:00Z"
|
|
393
|
+
updated_at = ""
|
|
394
|
+
tags = ["api", "greeting"]
|
|
395
|
+
supersedes = ""
|
|
350
396
|
---
|
|
351
397
|
export function greet(name: string): string {
|
|
352
398
|
return `Hello, ${name}!`;
|
|
@@ -368,6 +414,12 @@ body_token_count = 3
|
|
|
368
414
|
code_token_count = 0
|
|
369
415
|
created_at = "2026-06-15T10:30:00Z"
|
|
370
416
|
checksum = "sha256:fedcba..."
|
|
417
|
+
source_key = "1029384756abcdef"
|
|
418
|
+
source_path = "/home/user/docs/arch-diagram.png"
|
|
419
|
+
source_mtime = "2026-06-15T10:25:00Z"
|
|
420
|
+
updated_at = ""
|
|
421
|
+
tags = ["diagram", "architecture"]
|
|
422
|
+
supersedes = ""
|
|
371
423
|
---
|
|
372
424
|
iVBORw0KGgoAAAANSUhEUgAA...base64 data...
|
|
373
425
|
```
|
|
@@ -463,6 +515,48 @@ Updating a section's content creates a new file with
|
|
|
463
515
|
a new kbid; the old file is eligible for garbage
|
|
464
516
|
collection.
|
|
465
517
|
|
|
518
|
+
#### Logical Identity (Source Key)
|
|
519
|
+
|
|
520
|
+
Content-addressing identifies *what a section
|
|
521
|
+
currently says*. The **source key** identifies *what
|
|
522
|
+
a section is about* -- it is a stable logical
|
|
523
|
+
identity that persists across content edits.
|
|
524
|
+
|
|
525
|
+
The source key is derived from the section's origin:
|
|
526
|
+
|
|
527
|
+
```
|
|
528
|
+
source_key = sha256("${source_path}\0${heading}")[..8]
|
|
529
|
+
→ 16 lowercase hex characters
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
- `source_path` -- the absolute path of the file that
|
|
533
|
+
was learned (normalized: resolved symlinks, no
|
|
534
|
+
trailing separator).
|
|
535
|
+
- `heading` -- the section's title text (or empty
|
|
536
|
+
string when no title is provided).
|
|
537
|
+
- `\0` -- a null byte separator.
|
|
538
|
+
|
|
539
|
+
The SHA-256 digest is truncated to 8 bytes (64 bits)
|
|
540
|
+
and hex-encoded to produce a 16-character identifier.
|
|
541
|
+
|
|
542
|
+
Source keys enable **supersession**: when content is
|
|
543
|
+
re-learned from the same source with the same
|
|
544
|
+
heading, the system can identify the existing section
|
|
545
|
+
and replace it rather than creating a sibling. This
|
|
546
|
+
prevents the brain from silently accumulating
|
|
547
|
+
contradictions when facts are corrected.
|
|
548
|
+
|
|
549
|
+
Source key is `null` for:
|
|
550
|
+
- Stdin input (`-`) -- no source path exists.
|
|
551
|
+
- Sections created without a source path (e.g.
|
|
552
|
+
programmatic API calls that omit it).
|
|
553
|
+
|
|
554
|
+
When `source_key` is `null`, supersession is not
|
|
555
|
+
available and the section behaves as a pure
|
|
556
|
+
content-addressed entry (current behavior).
|
|
557
|
+
|
|
558
|
+
See **Add section** below for supersession semantics.
|
|
559
|
+
|
|
466
560
|
### Documents (`documents/`)
|
|
467
561
|
|
|
468
562
|
One TOML file per document. A document groups sections
|
|
@@ -667,14 +761,15 @@ directly.
|
|
|
667
761
|
`fs-database.wasm`, which writes the TOML header
|
|
668
762
|
(kbid, docids, type, title, description, size,
|
|
669
763
|
token_count, heading_token_count, body_token_count,
|
|
670
|
-
code_token_count,
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
|
|
675
|
-
the
|
|
676
|
-
|
|
677
|
-
|
|
764
|
+
code_token_count, created_at, checksum, source_key,
|
|
765
|
+
source_path, source_mtime, updated_at, tags,
|
|
766
|
+
supersedes), the separator, and the content body
|
|
767
|
+
(preprocessed text for text and code sections,
|
|
768
|
+
base64 for image sections). Updates `catalog.toml`
|
|
769
|
+
counts. Updates the inverted index with per-field
|
|
770
|
+
term frequencies, the positional index with term
|
|
771
|
+
positions, and corpus statistics (including
|
|
772
|
+
per-field average lengths).
|
|
678
773
|
|
|
679
774
|
Image sections (`image/*` type) have their binary
|
|
680
775
|
content stored but not indexed. However, their
|
|
@@ -682,9 +777,50 @@ directly.
|
|
|
682
777
|
attributes are tokenized and indexed, making image
|
|
683
778
|
sections searchable via their metadata.
|
|
684
779
|
|
|
780
|
+
**Supersession:** When the `replace` flag is set
|
|
781
|
+
and the new section has a non-null `source_key`,
|
|
782
|
+
`fs-database.wasm` searches for an existing section
|
|
783
|
+
with the same `source_key`. If found:
|
|
784
|
+
|
|
785
|
+
- The new section's `supersedes` field is set to
|
|
786
|
+
the old section's kbid.
|
|
787
|
+
- The new section's `updated_at` is set to the
|
|
788
|
+
current timestamp.
|
|
789
|
+
- The old kbid is replaced with the new kbid in all
|
|
790
|
+
document manifests that reference it.
|
|
791
|
+
- The old section's `docids` array is emptied,
|
|
792
|
+
making it eligible for garbage collection.
|
|
793
|
+
- Index entries for the old section are removed and
|
|
794
|
+
replaced with entries for the new section.
|
|
795
|
+
|
|
796
|
+
When `replace` is not set, or `source_key` is
|
|
797
|
+
`null`, the add operation behaves as before (pure
|
|
798
|
+
content-addressed addition with deduplication).
|
|
799
|
+
|
|
800
|
+
**Near-duplicate detection:** After the kbid is
|
|
801
|
+
computed and deduplication is checked, the add
|
|
802
|
+
operation computes a Jaccard similarity coefficient
|
|
803
|
+
between the new section's stemmed term set and
|
|
804
|
+
existing sections' term sets. Sections exceeding a
|
|
805
|
+
similarity threshold of 0.85 are reported as
|
|
806
|
+
near-duplicates in the response
|
|
807
|
+
(`near_duplicates: [{kbid, similarity}]`). This is
|
|
808
|
+
advisory -- the section is still stored. The check
|
|
809
|
+
uses the inverted index for efficient comparison
|
|
810
|
+
without reading every section's content.
|
|
811
|
+
|
|
812
|
+
**Heading-less import warning:** Text and code
|
|
813
|
+
sections added without a `title` emit a warning in
|
|
814
|
+
the response: `heading_missing`. The heading field
|
|
815
|
+
carries 2.0x ranking weight; omitting it means the
|
|
816
|
+
section loses its strongest searchability signal.
|
|
817
|
+
The section is still stored -- this is a warning,
|
|
818
|
+
not an error.
|
|
819
|
+
|
|
685
820
|
**Validation:** requests with `image/*` type that
|
|
686
821
|
omit `title` are rejected with an error. Text and
|
|
687
|
-
code sections may omit `title
|
|
822
|
+
code sections may omit `title` (with the
|
|
823
|
+
heading-less warning above).
|
|
688
824
|
|
|
689
825
|
3. **Update section** -- Given a kb-id and new content,
|
|
690
826
|
replace the section's content. Since kb-ids are
|
|
@@ -795,12 +931,26 @@ directly.
|
|
|
795
931
|
sections (not referenced by any document),
|
|
796
932
|
orphaned documents (referencing missing sections),
|
|
797
933
|
broken embedding references -- markers whose
|
|
798
|
-
target does not exist or has the wrong type
|
|
934
|
+
target does not exist or has the wrong type,
|
|
799
935
|
circular reference chains in `$(text:)` and
|
|
800
|
-
`$(document:)` markers
|
|
936
|
+
`$(document:)` markers, and **divergent content**
|
|
937
|
+
-- multiple sections sharing the same `source_key`
|
|
938
|
+
(same origin) but with different content,
|
|
939
|
+
indicating contradictory knowledge that should
|
|
940
|
+
have been resolved by supersession.
|
|
801
941
|
13. **Garbage collection** -- Remove section files not
|
|
802
|
-
referenced by any document manifest.
|
|
803
|
-
|
|
942
|
+
referenced by any document manifest. Additionally
|
|
943
|
+
report `stale_sources` -- sections whose
|
|
944
|
+
`source_path` points to a file that no longer
|
|
945
|
+
contains their heading text. Stale sources are
|
|
946
|
+
advisory (not automatically removed) and surfaced
|
|
947
|
+
in the GC response for manual review.
|
|
948
|
+
14. **Export** -- Copy sections, documents, and
|
|
949
|
+
catalog to a self-contained output directory (no
|
|
950
|
+
indexes -- they are rebuildable). Provides a
|
|
951
|
+
portable, git-friendly snapshot of the knowledge
|
|
952
|
+
base for backup and versioning.
|
|
953
|
+
15. **Index rebuild** -- Reconstruct all index files
|
|
804
954
|
from sections and documents. Re-tokenizes all text
|
|
805
955
|
and code sections per field, recomputes per-field
|
|
806
956
|
term frequencies and token counts (updating
|
|
@@ -975,8 +1125,11 @@ Response per mode:
|
|
|
975
1125
|
"type": "text/markdown",
|
|
976
1126
|
"heading": "Section Heading",
|
|
977
1127
|
"score": 8.74,
|
|
1128
|
+
"confidence": 1.0,
|
|
978
1129
|
"snippet": "...matching context...",
|
|
979
|
-
"matched_terms": ["term1", "term2"]
|
|
1130
|
+
"matched_terms": ["term1", "term2"],
|
|
1131
|
+
"matched_fields": ["heading", "body"],
|
|
1132
|
+
"created_at": "2026-06-15T10:00:00Z"
|
|
980
1133
|
}
|
|
981
1134
|
],
|
|
982
1135
|
"total": 42,
|
|
@@ -1058,7 +1211,10 @@ Response per mode:
|
|
|
1058
1211
|
"type": "text/markdown",
|
|
1059
1212
|
"title": "Architecture Overview",
|
|
1060
1213
|
"description": "High-level overview",
|
|
1061
|
-
"docid": "<doc id>"
|
|
1214
|
+
"docid": "<doc id>",
|
|
1215
|
+
"source_path": "/home/user/docs/architecture.md",
|
|
1216
|
+
"tags": ["architecture", "overview"],
|
|
1217
|
+
"replace": false
|
|
1062
1218
|
}
|
|
1063
1219
|
|
|
1064
1220
|
// Response
|
|
@@ -1069,7 +1225,10 @@ Response per mode:
|
|
|
1069
1225
|
"description": "High-level overview",
|
|
1070
1226
|
"indexed_terms_count": 42,
|
|
1071
1227
|
"elapsed_ms": 15,
|
|
1072
|
-
"attached": false
|
|
1228
|
+
"attached": false,
|
|
1229
|
+
"superseded": null,
|
|
1230
|
+
"warnings": [],
|
|
1231
|
+
"near_duplicates": []
|
|
1073
1232
|
}
|
|
1074
1233
|
```
|
|
1075
1234
|
|
|
@@ -1302,6 +1461,10 @@ The integrity check verifies:
|
|
|
1302
1461
|
- **circular_reference** -- a `$(text:)` or
|
|
1303
1462
|
`$(document:)` marker chain loops back to an entity
|
|
1304
1463
|
already in the expansion path.
|
|
1464
|
+
- **divergent_content** -- two or more sections share
|
|
1465
|
+
the same `source_key` but have different content
|
|
1466
|
+
(different kbids). Indicates contradictory knowledge
|
|
1467
|
+
that should have been resolved by supersession.
|
|
1305
1468
|
|
|
1306
1469
|
When `ok` is `true`, the `errors` array is empty.
|
|
1307
1470
|
|
|
@@ -1315,6 +1478,7 @@ When `ok` is `true`, the `errors` array is empty.
|
|
|
1315
1478
|
{
|
|
1316
1479
|
"removed_sections": 3,
|
|
1317
1480
|
"removed_bytes": 15360,
|
|
1481
|
+
"stale_sources": 1,
|
|
1318
1482
|
"elapsed_ms": 25
|
|
1319
1483
|
}
|
|
1320
1484
|
```
|
|
@@ -1327,6 +1491,12 @@ remove documents, indexes, or the catalog. After
|
|
|
1327
1491
|
removal, all caches are invalidated to prevent stale
|
|
1328
1492
|
reads.
|
|
1329
1493
|
|
|
1494
|
+
`stale_sources` reports the number of sections whose
|
|
1495
|
+
`source_path` points to a file that no longer
|
|
1496
|
+
contains their heading text. These sections may be
|
|
1497
|
+
outdated but are not automatically removed -- they
|
|
1498
|
+
are reported for manual review.
|
|
1499
|
+
|
|
1330
1500
|
**Index rebuild:**
|
|
1331
1501
|
|
|
1332
1502
|
```json
|
|
@@ -1357,6 +1527,30 @@ entries). Equivalent to deleting `indexes/` and
|
|
|
1357
1527
|
re-adding every section. After rebuild, all caches
|
|
1358
1528
|
are invalidated.
|
|
1359
1529
|
|
|
1530
|
+
**Export:**
|
|
1531
|
+
|
|
1532
|
+
```json
|
|
1533
|
+
// Request
|
|
1534
|
+
{
|
|
1535
|
+
"path": "/home/user/backup/kbdb-export"
|
|
1536
|
+
}
|
|
1537
|
+
|
|
1538
|
+
// Response
|
|
1539
|
+
{
|
|
1540
|
+
"path": "/home/user/backup/kbdb-export",
|
|
1541
|
+
"sections_exported": 42,
|
|
1542
|
+
"documents_exported": 5,
|
|
1543
|
+
"elapsed_ms": 120
|
|
1544
|
+
}
|
|
1545
|
+
```
|
|
1546
|
+
|
|
1547
|
+
Export copies sections, documents, and `catalog.toml`
|
|
1548
|
+
to the specified output directory. Indexes are not
|
|
1549
|
+
included -- they are derived data and can be rebuilt
|
|
1550
|
+
with `rebuild` after importing. The export directory
|
|
1551
|
+
is a self-contained, git-friendly snapshot suitable
|
|
1552
|
+
for backup, transfer, or version control.
|
|
1553
|
+
|
|
1360
1554
|
### Error Responses
|
|
1361
1555
|
|
|
1362
1556
|
Errors are returned as JSON-RPC 2.0 error objects:
|
|
@@ -1411,6 +1605,34 @@ See [Query Parser](query-parser.md) for full details.
|
|
|
1411
1605
|
| SHA-256 + Truncation + Base32 | SHA-256 of the typed header, truncated to 128 bits (16 bytes), base32 lowercase (no padding) -- 26 alphanumeric characters |
|
|
1412
1606
|
| Content Preprocessing | Text and code: BOM removal, `\r\n`/`\r` to `\n`, Unicode NFC, whitespace trimming. Images: base64 string used as-is |
|
|
1413
1607
|
|
|
1608
|
+
## Concurrent Write Behavior
|
|
1609
|
+
|
|
1610
|
+
The daemon serializes all write operations through a
|
|
1611
|
+
**single-writer lock**. The lock is an exclusive file
|
|
1612
|
+
lock on `kbdb-{ctx}.lock` in the OS temporary
|
|
1613
|
+
directory, keyed to the same context ID used for the
|
|
1614
|
+
PID file and IPC socket.
|
|
1615
|
+
|
|
1616
|
+
- **Reads** proceed without acquiring the lock. The
|
|
1617
|
+
daemon serves concurrent read requests from
|
|
1618
|
+
multiple worker client connections.
|
|
1619
|
+
- **Writes** (add section, update section, remove
|
|
1620
|
+
section, group/remove document, garbage collection,
|
|
1621
|
+
index rebuild, export) acquire the lock before
|
|
1622
|
+
modifying any file. If another write is in progress,
|
|
1623
|
+
the request queues behind the lock.
|
|
1624
|
+
- The lock is released after each write operation
|
|
1625
|
+
completes, not held across multiple operations.
|
|
1626
|
+
- CLI invocations sharing the same daemon are
|
|
1627
|
+
serialized automatically.
|
|
1628
|
+
- Direct file access to `.kbdb` outside the daemon is
|
|
1629
|
+
not supported and may cause corruption. All
|
|
1630
|
+
mutations must go through the daemon's IPC API.
|
|
1631
|
+
|
|
1632
|
+
This model is equivalent to SQLite's default locking:
|
|
1633
|
+
one writer at a time, concurrent readers, no external
|
|
1634
|
+
coordination required.
|
|
1635
|
+
|
|
1414
1636
|
## Design Constraints
|
|
1415
1637
|
|
|
1416
1638
|
- `fs-database.wasm` is the sole owner of `.kbdb` I/O.
|
|
@@ -1423,6 +1645,10 @@ See [Query Parser](query-parser.md) for full details.
|
|
|
1423
1645
|
their content hash. `fs-database.wasm` never modifies
|
|
1424
1646
|
a section file in place -- updates produce new files
|
|
1425
1647
|
with new kb-ids.
|
|
1648
|
+
- Source keys are advisory logical identity. The
|
|
1649
|
+
content hash (kbid) remains the primary identifier.
|
|
1650
|
+
Supersession is opt-in via the `replace` flag --
|
|
1651
|
+
default learn behavior is unchanged.
|
|
1426
1652
|
- Indexes are derived data. They can be rebuilt entirely
|
|
1427
1653
|
from sections and document manifests. Loss of index
|
|
1428
1654
|
files is recoverable.
|
|
@@ -1437,7 +1663,8 @@ See [Query Parser](query-parser.md) for full details.
|
|
|
1437
1663
|
- No external database process or server dependency.
|
|
1438
1664
|
- The database must support concurrent read access from
|
|
1439
1665
|
multiple CLI invocations via the shared daemon without
|
|
1440
|
-
corruption.
|
|
1666
|
+
corruption. Write operations are serialized through
|
|
1667
|
+
the daemon's single-writer lock.
|
|
1441
1668
|
- Garbage collection of unreferenced sections is
|
|
1442
1669
|
`fs-database.wasm`'s responsibility. Sections not
|
|
1443
1670
|
listed in any document manifest may be removed.
|
package/docs/goals/document.md
CHANGED
|
@@ -289,7 +289,7 @@ The tokenization pipeline removes them in Stage 1
|
|
|
289
289
|
search index.
|
|
290
290
|
|
|
291
291
|
See [Query Parser](query-parser.md) for the full
|
|
292
|
-
|
|
292
|
+
nine-stage tokenization pipeline and
|
|
293
293
|
[Content Parser](content-parser.md) for the marker
|
|
294
294
|
extraction functions used during stripping.
|
|
295
295
|
|
package/docs/goals/mcp.md
CHANGED
|
@@ -50,6 +50,17 @@ the user.
|
|
|
50
50
|
See [CLI -- Run Modes](cli.md#run-modes) for how
|
|
51
51
|
the `kbdb` executable dispatches to MCP mode.
|
|
52
52
|
|
|
53
|
+
## Startup Health Check
|
|
54
|
+
|
|
55
|
+
When the MCP server starts, it runs
|
|
56
|
+
`client.integrityCheck()` automatically after the
|
|
57
|
+
worker daemon connection is established. If the
|
|
58
|
+
check returns `ok: false`, the server emits a
|
|
59
|
+
`notifications/message` log with level `warning`
|
|
60
|
+
listing the error summary. The server continues to
|
|
61
|
+
operate -- health issues are not fatal. This ensures
|
|
62
|
+
corruption surfaces before it produces wrong answers.
|
|
63
|
+
|
|
53
64
|
## Installation and Wiring
|
|
54
65
|
|
|
55
66
|
Installation is the same as the CLI:
|
|
@@ -332,6 +343,7 @@ daemon via the worker client.
|
|
|
332
343
|
|
|
333
344
|
Search the knowledge base. Returns ranked results
|
|
334
345
|
with headings and snippets for relevance evaluation.
|
|
346
|
+
Use multiple short keyword queries for best coverage.
|
|
335
347
|
Use `recall` to get full content of specific results.
|
|
336
348
|
|
|
337
349
|
**Parameters:**
|
|
@@ -358,13 +370,16 @@ additional calls:
|
|
|
358
370
|
"type": "text/markdown",
|
|
359
371
|
"docids": ["doc-5678"],
|
|
360
372
|
"snippet": "The system is organized...",
|
|
361
|
-
"matched_terms": ["architecture", "system"]
|
|
373
|
+
"matched_terms": ["architecture", "system"],
|
|
374
|
+
"matched_fields": ["heading", "body"],
|
|
375
|
+
"created_at": "2026-06-20T14:30:00Z"
|
|
362
376
|
}
|
|
363
377
|
],
|
|
364
378
|
"total": 42,
|
|
365
379
|
"offset": 0,
|
|
366
380
|
"limit": 20,
|
|
367
|
-
"has_more": true
|
|
381
|
+
"has_more": true,
|
|
382
|
+
"relaxed": false
|
|
368
383
|
}
|
|
369
384
|
```
|
|
370
385
|
|
|
@@ -379,6 +394,15 @@ additional calls:
|
|
|
379
394
|
matched term. Empty string for image sections.
|
|
380
395
|
- `matched_terms` -- query terms that matched, as a
|
|
381
396
|
flat `string[]`. No frequency counts.
|
|
397
|
+
- `matched_fields` -- subset of `["heading", "body",
|
|
398
|
+
"code"]` indicating which fields contained matches.
|
|
399
|
+
- `created_at` -- ISO 8601 timestamp of when the
|
|
400
|
+
section was first indexed.
|
|
401
|
+
- `relaxed` -- `true` when the result came from an
|
|
402
|
+
automatic AND-to-OR fallback (see
|
|
403
|
+
[Query Result -- Relaxed-Match
|
|
404
|
+
Fallback](query-result.md)). `false` for normal
|
|
405
|
+
results.
|
|
382
406
|
|
|
383
407
|
When `content` is `true`, each item also includes
|
|
384
408
|
a `content` field with the full section text.
|
|
@@ -393,20 +417,24 @@ Maps to `client.search()`.
|
|
|
393
417
|
|
|
394
418
|
### learn
|
|
395
419
|
|
|
396
|
-
Import content into the knowledge base.
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
420
|
+
Import content into the knowledge base. Use for
|
|
421
|
+
durable facts, decisions, and corrections. Use
|
|
422
|
+
`--replace` when updating existing knowledge. Attach
|
|
423
|
+
tags for scoping. The daemon decomposes the content
|
|
424
|
+
into sections, extracts keywords, builds indexes,
|
|
425
|
+
and assigns ranking weights.
|
|
400
426
|
|
|
401
427
|
**Parameters:**
|
|
402
428
|
|
|
403
429
|
| Name | Type | Required | Description |
|
|
404
430
|
|---------------|--------|----------|------------------|
|
|
405
|
-
| `content` | string
|
|
406
|
-
| `type` | string
|
|
407
|
-
| `title` | string
|
|
408
|
-
| `description` | string
|
|
409
|
-
| `docid` | string
|
|
431
|
+
| `content` | string | yes | Text content or base64-encoded image data |
|
|
432
|
+
| `type` | string | no | `"text/markdown"` (default), `text/*` MIME type, `"application/json"`, or `image/*` MIME type |
|
|
433
|
+
| `title` | string | no* | Section title. ***Required** for `image/*` types.* Optional for text and code types. |
|
|
434
|
+
| `description` | string | no | Optional section description |
|
|
435
|
+
| `docid` | string | no | Group under this document ID |
|
|
436
|
+
| `tags` | string[] | no | Tags for scoping and filtering |
|
|
437
|
+
| `replace` | boolean | no | Enable supersession by source key. When `true` and a section with the same source key exists, the old section is replaced. |
|
|
410
438
|
|
|
411
439
|
**Content type rules:**
|
|
412
440
|
|
|
@@ -441,6 +469,16 @@ Text and code sections return a positive
|
|
|
441
469
|
`indexed_terms_count`. Image sections return
|
|
442
470
|
`indexed_terms_count: 0`.
|
|
443
471
|
|
|
472
|
+
Additional result fields when applicable:
|
|
473
|
+
- `superseded` -- kbid of the section this one
|
|
474
|
+
replaced via `replace: true`, or `null`.
|
|
475
|
+
- `warnings` -- string array of advisory warnings
|
|
476
|
+
(e.g. `"heading_missing"` when a text section has
|
|
477
|
+
no title).
|
|
478
|
+
- `near_duplicates` -- array of
|
|
479
|
+
`{ kbid, similarity }` objects for existing
|
|
480
|
+
sections with Jaccard similarity >= 0.85.
|
|
481
|
+
|
|
444
482
|
If the content already exists in the database (same
|
|
445
483
|
kb-id) and `docid` is provided, the section is not
|
|
446
484
|
duplicated. Instead, the docid is added to the
|
|
@@ -662,8 +700,12 @@ checks pass, `ok` is `true` and `errors` is empty.
|
|
|
662
700
|
|
|
663
701
|
Error types: `checksum_mismatch`, `orphan_section`,
|
|
664
702
|
`orphan_document`, `broken_reference`,
|
|
665
|
-
`type_mismatch`, `circular_reference
|
|
666
|
-
|
|
703
|
+
`type_mismatch`, `circular_reference`,
|
|
704
|
+
`divergent_content`. The `divergent_content` error
|
|
705
|
+
flags sections sharing the same `source_key` that
|
|
706
|
+
have different content -- indicates contradictory
|
|
707
|
+
versions that should be resolved via supersession.
|
|
708
|
+
See [Database -- Integrity and
|
|
667
709
|
Maintenance](database.md).
|
|
668
710
|
|
|
669
711
|
Maps to `client.integrityCheck()`.
|
|
@@ -696,6 +738,32 @@ this to recover from index corruption.
|
|
|
696
738
|
|
|
697
739
|
Maps to `client.rebuildIndexes()`.
|
|
698
740
|
|
|
741
|
+
### export
|
|
742
|
+
|
|
743
|
+
Export the knowledge base as a self-contained
|
|
744
|
+
directory snapshot for backup or versioning. Copies
|
|
745
|
+
sections, documents, and catalog (no indexes -- they
|
|
746
|
+
are rebuildable via `rebuild`).
|
|
747
|
+
|
|
748
|
+
**Parameters:**
|
|
749
|
+
|
|
750
|
+
| Name | Type | Required | Description |
|
|
751
|
+
|--------|--------|----------|---------------------------|
|
|
752
|
+
| `path` | string | no | Export destination path (default: `./kbdb-export/`) |
|
|
753
|
+
|
|
754
|
+
**Returns:** Compact JSON with export summary:
|
|
755
|
+
|
|
756
|
+
```json
|
|
757
|
+
{
|
|
758
|
+
"path": "/home/user/kbdb-export",
|
|
759
|
+
"sections_exported": 142,
|
|
760
|
+
"documents_exported": 23,
|
|
761
|
+
"elapsed_ms": 350
|
|
762
|
+
}
|
|
763
|
+
```
|
|
764
|
+
|
|
765
|
+
Maps to `client.export()`.
|
|
766
|
+
|
|
699
767
|
### skill-learn
|
|
700
768
|
|
|
701
769
|
Create a new skill definition and store it in the
|
|
@@ -1057,7 +1125,7 @@ interface McpToolAnnotations {
|
|
|
1057
1125
|
}
|
|
1058
1126
|
```
|
|
1059
1127
|
|
|
1060
|
-
Annotations for all
|
|
1128
|
+
Annotations for all 19 kbdb tools:
|
|
1061
1129
|
|
|
1062
1130
|
| Tool | title | readOnly | destructive | idempotent | openWorld |
|
|
1063
1131
|
|----------------|-------------------|----------|-------------|------------|-----------|
|
|
@@ -1071,6 +1139,7 @@ Annotations for all 18 kbdb tools:
|
|
|
1071
1139
|
| `check` | Integrity Check | true | false | true | false |
|
|
1072
1140
|
| `gc` | Garbage Collect | false | true | false | false |
|
|
1073
1141
|
| `rebuild` | Rebuild Indexes | false | false | false | false |
|
|
1142
|
+
| `export` | Export Snapshot | false | false | true | false |
|
|
1074
1143
|
| `skill-learn` | Create Skill | false | false | true | false |
|
|
1075
1144
|
| `skill-list` | List Skills | true | false | true | false |
|
|
1076
1145
|
| `skill-get` | Get Skill | true | false | true | false |
|
|
@@ -115,9 +115,11 @@ Stages:
|
|
|
115
115
|
decomposition).
|
|
116
116
|
5. **Case folding** (lowercase).
|
|
117
117
|
6. **Stop word removal** (configurable list).
|
|
118
|
-
7. **
|
|
118
|
+
7. **Synonym expansion** (bidirectional synonym map --
|
|
119
|
+
inserts synonym tokens alongside originals).
|
|
120
|
+
8. **Stemming** (Snowball / Porter2 -- stores both
|
|
119
121
|
stemmed and original forms).
|
|
120
|
-
|
|
122
|
+
9. **N-gram extraction** (bigrams, trigrams for phrase
|
|
121
123
|
indexing and multi-word term matching).
|
|
122
124
|
|
|
123
125
|
`kb-worker.wasm` calls the tokenization pipeline for
|
|
@@ -126,6 +128,71 @@ and search (when a query is submitted). It feeds the
|
|
|
126
128
|
tokenized output to `fs-database.wasm` for index
|
|
127
129
|
storage or index lookups as needed.
|
|
128
130
|
|
|
131
|
+
### Synonym Expansion (Stage 7)
|
|
132
|
+
|
|
133
|
+
Bridges the lexical gap between user vocabulary and
|
|
134
|
+
indexed content. Without synonym expansion, a search
|
|
135
|
+
for `"login credentials"` returns zero results when
|
|
136
|
+
the indexed document uses "signs in" and "identity"
|
|
137
|
+
instead -- different root words with the same meaning.
|
|
138
|
+
|
|
139
|
+
**Placement rationale:** Applied after stop-word
|
|
140
|
+
removal (stage 6) and before stemming (stage 8). This
|
|
141
|
+
ensures synonyms are expanded on lowercased,
|
|
142
|
+
stop-word-filtered tokens, and both the original and
|
|
143
|
+
synonym tokens are then stemmed consistently.
|
|
144
|
+
|
|
145
|
+
**Built-in synonym map:** A small, curated
|
|
146
|
+
`HashMap<String, Vec<String>>` of bidirectional
|
|
147
|
+
synonym pairs. Bidirectional means if `login` maps to
|
|
148
|
+
`sign-in`, then `sign-in` also maps to `login`.
|
|
149
|
+
|
|
150
|
+
**Example map entries:**
|
|
151
|
+
|
|
152
|
+
| Term | Synonyms |
|
|
153
|
+
|--------------|------------------------------------|
|
|
154
|
+
| login | sign-in, sign_in, signin |
|
|
155
|
+
| auth | authentication, authenticate |
|
|
156
|
+
| remove | delete, erase |
|
|
157
|
+
| create | add, insert |
|
|
158
|
+
| update | modify, change, edit |
|
|
159
|
+
| error | exception, fault, failure |
|
|
160
|
+
| config | configuration, settings |
|
|
161
|
+
| credential | password, identity |
|
|
162
|
+
| user | account, member |
|
|
163
|
+
| deploy | release, publish, ship |
|
|
164
|
+
| search | query, find, lookup |
|
|
165
|
+
| doc | document, documentation |
|
|
166
|
+
| api | endpoint, interface |
|
|
167
|
+
| db | database |
|
|
168
|
+
| env | environment |
|
|
169
|
+
| repo | repository |
|
|
170
|
+
| param | parameter, argument |
|
|
171
|
+
| dir | directory, folder |
|
|
172
|
+
| msg | message |
|
|
173
|
+
| info | information |
|
|
174
|
+
|
|
175
|
+
**Expansion behaviour:** For each token in the stream,
|
|
176
|
+
look up the synonym map. If found, insert the synonym
|
|
177
|
+
tokens at the same position as the original. Both the
|
|
178
|
+
original and its synonyms proceed to the stemming
|
|
179
|
+
stage. This preserves the original token's position
|
|
180
|
+
for proximity/zone scoring while adding synonym
|
|
181
|
+
coverage.
|
|
182
|
+
|
|
183
|
+
**Applied symmetrically:** The same synonym expansion
|
|
184
|
+
applies during both `learn` (indexing) and `search`
|
|
185
|
+
(querying). If a document uses "login" and the user
|
|
186
|
+
searches "sign-in", the expansion during indexing
|
|
187
|
+
produced "sign-in" tokens from "login", and the
|
|
188
|
+
expansion during querying produced "login" tokens
|
|
189
|
+
from "sign-in".
|
|
190
|
+
|
|
191
|
+
**Not user-configurable in v1.** The built-in map is
|
|
192
|
+
hardcoded. Future versions may allow user-defined
|
|
193
|
+
synonym maps via a `synonyms.toml` configuration file
|
|
194
|
+
in the `.kbdb` directory.
|
|
195
|
+
|
|
129
196
|
## Query Parsing
|
|
130
197
|
|
|
131
198
|
Search queries are parsed into a structured query plan
|
|
@@ -148,8 +215,35 @@ Parsing stages:
|
|
|
148
215
|
The query plan is the query parser's output --
|
|
149
216
|
`kb-worker.wasm` receives it and passes it to
|
|
150
217
|
`fs-database.wasm` for index lookups, then calls the
|
|
151
|
-
query parser's ranking functions to score and order
|
|
152
|
-
results.
|
|
218
|
+
query parser's ranking functions to score and order
|
|
219
|
+
the raw results.
|
|
220
|
+
|
|
221
|
+
### Relaxed Query Mode
|
|
222
|
+
|
|
223
|
+
When a multi-term AND query returns zero results,
|
|
224
|
+
`kb-worker.wasm` retries the query in **relaxed
|
|
225
|
+
mode** -- OR semantics where any term matches, not
|
|
226
|
+
all required.
|
|
227
|
+
|
|
228
|
+
The query parser generates both the primary (AND)
|
|
229
|
+
plan and a relaxed (OR) plan from the same parsed
|
|
230
|
+
query. `kb-worker.wasm` executes the AND plan first.
|
|
231
|
+
If the AND plan produces zero matches, it falls back
|
|
232
|
+
to the OR plan. No new WASM export is needed --
|
|
233
|
+
`qp_parse_query` returns a `QueryPlan` that supports
|
|
234
|
+
mode switching via a `match_mode` field
|
|
235
|
+
(`All` | `Any`).
|
|
236
|
+
|
|
237
|
+
Relaxed results are flagged with `relaxed: true` in
|
|
238
|
+
the `PagedSearchResult` envelope (see
|
|
239
|
+
[Query Result -- Relaxed-Match
|
|
240
|
+
Fallback](query-result.md)). The fallback is
|
|
241
|
+
transparent to the caller.
|
|
242
|
+
|
|
243
|
+
Single-term queries are unaffected (AND and OR are
|
|
244
|
+
equivalent for one term). Phrase queries and
|
|
245
|
+
exclusions are preserved in both modes -- only the
|
|
246
|
+
inter-term combination changes.
|
|
153
247
|
|
|
154
248
|
## Query Cache
|
|
155
249
|
|