@dikolab/kbdb 0.2.2 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -31
- package/dist/bench/constants/corpus-defaults.constant.d.ts +3 -0
- package/dist/bench/functions/format-report.function.d.ts +11 -0
- package/dist/bench/functions/generate-corpus.function.d.ts +12 -0
- package/dist/bench/functions/measure-operation.function.d.ts +14 -0
- package/dist/bench/functions/seeded-rng.function.d.ts +11 -0
- package/dist/bench/functions/zipf-word.function.d.ts +13 -0
- package/dist/bench/index.d.ts +10 -0
- package/dist/bench/typings/bench-result.interface.d.ts +11 -0
- package/dist/bench/typings/bench-suite-result.interface.d.ts +10 -0
- package/dist/bench/typings/corpus-config.interface.d.ts +11 -0
- package/dist/bench/typings/corpus-section.interface.d.ts +9 -0
- package/dist/cli.cjs +523 -80
- package/dist/cli.cjs.map +4 -4
- package/dist/cli.mjs +515 -72
- package/dist/cli.mjs.map +4 -4
- package/dist/kbdb-worker.cjs +395 -60
- package/dist/kbdb-worker.cjs.map +4 -4
- package/dist/mod.cjs +17 -0
- package/dist/mod.cjs.map +2 -2
- package/dist/mod.mjs +17 -0
- package/dist/mod.mjs.map +2 -2
- package/dist/src/shared/auto-capture/classes/auto-capture-controller.class.d.ts +33 -0
- package/dist/src/shared/auto-capture/constants/auto-capture-defaults.constant.d.ts +8 -0
- package/dist/src/shared/auto-capture/functions/build-capture-prompt.function.d.ts +11 -0
- package/dist/src/shared/auto-capture/functions/evaluate-trigger.function.d.ts +12 -0
- package/dist/src/shared/auto-capture/index.d.ts +7 -0
- package/dist/src/shared/auto-capture/typings/auto-capture-config.interface.d.ts +13 -0
- package/dist/src/shared/auto-capture/typings/auto-capture-state.interface.d.ts +13 -0
- package/dist/src/shared/auto-capture/typings/auto-capture-trigger.interface.d.ts +7 -0
- package/dist/src/shared/cli/constants/global-help.constant.d.ts +8 -0
- package/dist/src/shared/cli/constants/leaf-help.constant.d.ts +7 -0
- package/dist/src/shared/cli/constants/memory-protocol-body.constant.d.ts +7 -0
- package/dist/src/shared/cli/constants/query-guidance-body.constant.d.ts +7 -0
- package/dist/src/shared/cli/constants/subcommand-help.constant.d.ts +8 -0
- package/dist/src/shared/cli/functions/collect-files.function.d.ts +13 -0
- package/dist/src/shared/cli/functions/compute-section-kbid.function.d.ts +15 -0
- package/dist/src/shared/cli/functions/dispatch-command.function.d.ts +3 -1
- package/dist/src/shared/cli/functions/format-mcp-output.function.d.ts +9 -0
- package/dist/src/shared/cli/functions/format-text-output.function.d.ts +9 -0
- package/dist/src/shared/cli/functions/generate-group-help.function.d.ts +11 -0
- package/dist/src/shared/cli/functions/handle-data-command.function.d.ts +16 -0
- package/dist/src/shared/cli/functions/handle-mcp.function.d.ts +15 -0
- package/dist/src/shared/cli/functions/handle-migrate.function.d.ts +15 -0
- package/dist/src/shared/cli/functions/hydrate-search-previews.function.d.ts +14 -0
- package/dist/src/shared/cli/functions/install-default-skills.function.d.ts +12 -0
- package/dist/src/shared/cli/functions/resolve-skill-names.function.d.ts +16 -0
- package/dist/src/shared/cli/functions/run-check.function.d.ts +47 -5
- package/dist/src/shared/cli/functions/run-export.function.d.ts +34 -0
- package/dist/src/shared/cli/functions/run-gc.function.d.ts +6 -0
- package/dist/src/shared/cli/functions/run-learn.function.d.ts +35 -6
- package/dist/src/shared/cli/functions/run-recall.function.d.ts +45 -0
- package/dist/src/shared/cli/functions/run-search.function.d.ts +53 -15
- package/dist/src/shared/cli/functions/version-gate.function.d.ts +15 -0
- package/dist/src/shared/cli/functions/wrap-command.function.d.ts +15 -0
- package/dist/src/shared/cli/index.d.ts +4 -0
- package/dist/src/shared/cli/typings/agent-create-client.interface.d.ts +42 -0
- package/dist/src/shared/cli/typings/agent-get-client.interface.d.ts +36 -0
- package/dist/src/shared/cli/typings/check-client.interface.d.ts +58 -0
- package/dist/src/shared/cli/typings/cli-deps.interface.d.ts +48 -0
- package/dist/src/shared/cli/typings/cli-options.interface.d.ts +15 -0
- package/dist/src/shared/cli/typings/disconnectable-client.interface.d.ts +26 -0
- package/dist/src/shared/cli/typings/dispatch-client.type.d.ts +27 -0
- package/dist/src/shared/cli/typings/learn-client.interface.d.ts +52 -0
- package/dist/src/shared/cli/typings/learn-io.interface.d.ts +5 -0
- package/dist/src/shared/cli/typings/learn-result.model.d.ts +19 -0
- package/dist/src/shared/cli/typings/search-client.interface.d.ts +71 -0
- package/dist/src/shared/cli/typings/search-display.model.d.ts +18 -0
- package/dist/src/shared/embedding/classes/null-embedding-provider.class.d.ts +20 -0
- package/dist/src/shared/embedding/constants/near-duplicate-thresholds.constant.d.ts +15 -0
- package/dist/src/shared/embedding/functions/build-vector-index.function.d.ts +14 -0
- package/dist/src/shared/embedding/functions/cosine-similarity.function.d.ts +10 -0
- package/dist/src/shared/embedding/functions/detect-near-duplicates.function.d.ts +18 -0
- package/dist/src/shared/embedding/functions/jaccard-coefficient.function.d.ts +9 -0
- package/dist/src/shared/embedding/functions/read-vector-index-meta.function.d.ts +10 -0
- package/dist/src/shared/embedding/functions/rrf-fuse.function.d.ts +14 -0
- package/dist/src/shared/embedding/functions/search-vectors.function.d.ts +14 -0
- package/dist/src/shared/embedding/index.d.ts +16 -0
- package/dist/src/shared/embedding/typings/embedding-config.interface.d.ts +20 -0
- package/dist/src/shared/embedding/typings/embedding-provider.interface.d.ts +23 -0
- package/dist/src/shared/embedding/typings/near-duplicate-candidate.model.d.ts +23 -0
- package/dist/src/shared/embedding/typings/near-duplicate-result.model.d.ts +24 -0
- package/dist/src/shared/embedding/typings/vector-index-entry.model.d.ts +12 -0
- package/dist/src/shared/embedding/typings/vector-index-meta.model.d.ts +11 -0
- package/dist/src/shared/embedding/typings/vector-search-result.model.d.ts +7 -0
- package/dist/src/shared/mcp/classes/mcp-server.class.d.ts +8 -0
- package/dist/src/shared/mcp/constants/tool-agent-create.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-agent-delete.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-agent-get.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-agent-list.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-auto-capture-review.constant.d.ts +8 -0
- package/dist/src/shared/mcp/constants/tool-check.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-content.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-export.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-gc.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-learn.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-rebuild.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-recall.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-retrieve.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-search.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-skill-delete.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-skill-get.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-skill-learn.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-skill-list.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-status.constant.d.ts +3 -0
- package/dist/src/shared/mcp/constants/tool-unlearn.constant.d.ts +3 -0
- package/dist/src/shared/mcp/functions/as-boolean-or-undefined.function.d.ts +7 -0
- package/dist/src/shared/mcp/functions/as-string-array-or-undefined.function.d.ts +7 -0
- package/dist/src/shared/mcp/functions/as-string-array.function.d.ts +7 -0
- package/dist/src/shared/mcp/functions/as-string.function.d.ts +8 -0
- package/dist/src/shared/mcp/functions/build-skills-text.function.d.ts +18 -0
- package/dist/src/shared/mcp/functions/handle-agent-prompt.function.d.ts +14 -0
- package/dist/src/shared/mcp/functions/handle-mcp-request.function.d.ts +14 -0
- package/dist/src/shared/mcp/functions/handle-prompt-get.function.d.ts +4 -3
- package/dist/src/shared/mcp/functions/handle-skill-prompt.function.d.ts +16 -0
- package/dist/src/shared/mcp/functions/handle-tool-auto-capture-review.function.d.ts +12 -0
- package/dist/src/shared/mcp/functions/handle-tool-check.function.d.ts +11 -0
- package/dist/src/shared/mcp/functions/handle-tool-content.function.d.ts +12 -0
- package/dist/src/shared/mcp/functions/handle-tool-export.function.d.ts +12 -0
- package/dist/src/shared/mcp/functions/handle-tool-gc.function.d.ts +11 -0
- package/dist/src/shared/mcp/functions/handle-tool-learn.function.d.ts +12 -0
- package/dist/src/shared/mcp/functions/handle-tool-rebuild.function.d.ts +11 -0
- package/dist/src/shared/mcp/functions/handle-tool-retrieve.function.d.ts +12 -0
- package/dist/src/shared/mcp/functions/handle-tool-search.function.d.ts +13 -0
- package/dist/src/shared/mcp/functions/handle-tool-skill-delete.function.d.ts +12 -0
- package/dist/src/shared/mcp/functions/handle-tool-skill-get.function.d.ts +13 -0
- package/dist/src/shared/mcp/functions/handle-tool-skill-learn.function.d.ts +13 -0
- package/dist/src/shared/mcp/functions/handle-tool-skill-list.function.d.ts +11 -0
- package/dist/src/shared/mcp/functions/handle-tool-status.function.d.ts +11 -0
- package/dist/src/shared/mcp/functions/handle-tool-unlearn.function.d.ts +12 -0
- package/dist/src/shared/mcp/functions/parse-agent-meta.function.d.ts +14 -0
- package/dist/src/shared/mcp/functions/parse-skill-arguments.function.d.ts +11 -0
- package/dist/src/shared/mcp/functions/read-agent-meta.function.d.ts +17 -0
- package/dist/src/shared/mcp/functions/resolve-skill-refs.function.d.ts +23 -0
- package/dist/src/shared/mcp/functions/run-startup-health-check.function.d.ts +14 -0
- package/dist/src/shared/mcp/typings/mcp-capabilities.interface.d.ts +16 -0
- package/dist/src/shared/mcp/typings/mcp-client.interface.d.ts +45 -0
- package/dist/src/shared/mcp/typings/mcp-completion.interface.d.ts +43 -0
- package/dist/src/shared/mcp/typings/mcp-content-result.model.d.ts +9 -0
- package/dist/src/shared/mcp/typings/mcp-export-result.model.d.ts +11 -0
- package/dist/src/shared/mcp/typings/mcp-gc-result.model.d.ts +15 -0
- package/dist/src/shared/mcp/typings/mcp-integrity-check-result.model.d.ts +16 -0
- package/dist/src/shared/mcp/typings/mcp-integrity-error.model.d.ts +9 -0
- package/dist/src/shared/mcp/typings/mcp-notification.interface.d.ts +9 -0
- package/dist/src/shared/mcp/typings/mcp-rebuild-result.model.d.ts +9 -0
- package/dist/src/shared/mcp/typings/mcp-request.interface.d.ts +11 -0
- package/dist/src/shared/mcp/typings/mcp-resource-result.interface.d.ts +12 -0
- package/dist/src/shared/mcp/typings/mcp-resource-template.interface.d.ts +11 -0
- package/dist/src/shared/mcp/typings/mcp-response.interface.d.ts +18 -0
- package/dist/src/shared/mcp/typings/mcp-server-info.interface.d.ts +7 -0
- package/dist/src/shared/mcp/typings/mcp-server-state.interface.d.ts +16 -0
- package/dist/src/shared/mcp/typings/mcp-tool-annotations.interface.d.ts +32 -0
- package/dist/src/shared/mcp/typings/mcp-tool-definition.interface.d.ts +15 -0
- package/dist/src/shared/mcp/typings/mcp-tool-result.interface.d.ts +19 -0
- package/dist/src/shared/mcp/typings/mcp-worker-client.interface.d.ts +220 -0
- package/dist/src/shared/mcp/typings/tool-call-result.model.d.ts +17 -0
- package/dist/src/shared/sync/functions/build-import-report.function.d.ts +18 -0
- package/dist/src/shared/sync/functions/parse-export-entry.function.d.ts +11 -0
- package/dist/src/shared/sync/functions/resolve-conflict.function.d.ts +13 -0
- package/dist/src/shared/sync/functions/serialize-export-entry.function.d.ts +8 -0
- package/dist/src/shared/sync/index.d.ts +9 -0
- package/dist/src/shared/sync/typings/conflict-resolution.type.d.ts +8 -0
- package/dist/src/shared/sync/typings/export-entry.interface.d.ts +13 -0
- package/dist/src/shared/sync/typings/export-options.interface.d.ts +9 -0
- package/dist/src/shared/sync/typings/import-options.interface.d.ts +7 -0
- package/dist/src/shared/sync/typings/import-report.interface.d.ts +15 -0
- package/dist/src/shared/wasm-codec/functions/decode-add-section-result.function.d.ts +48 -0
- package/dist/src/shared/wasm-codec/functions/decode-one-recall-result.function.d.ts +14 -0
- package/dist/src/shared/wasm-codec/functions/decode-option-string.function.d.ts +12 -0
- package/dist/src/shared/wasm-codec/functions/decode-recall-document.function.d.ts +14 -0
- package/dist/src/shared/wasm-codec/functions/decode-recall-reference.function.d.ts +14 -0
- package/dist/src/shared/wasm-codec/functions/decode-recall-sibling.function.d.ts +13 -0
- package/dist/src/shared/wasm-codec/functions/decode-scored-results.function.d.ts +58 -0
- package/dist/src/shared/wasm-codec/functions/decode-section-match.function.d.ts +15 -0
- package/dist/src/shared/wasm-codec/functions/decode-section-record-trailing.function.d.ts +14 -0
- package/dist/src/shared/wasm-codec/functions/decode-section-record.function.d.ts +14 -0
- package/dist/src/shared/wasm-codec/functions/decode-section-records.function.d.ts +33 -0
- package/dist/src/shared/wasm-codec/functions/decode-section-type.function.d.ts +13 -0
- package/dist/src/shared/wasm-codec/functions/decode-wasm-list.function.d.ts +17 -0
- package/dist/src/shared/wasm-codec/functions/encode-add-section-params.function.d.ts +52 -0
- package/dist/src/shared/wasm-codec/index.d.ts +2 -0
- package/dist/src/shared/wasm-codec/typings/add-section-input.model.d.ts +45 -0
- package/dist/src/shared/wasm-codec/typings/add-section-result.model.d.ts +27 -0
- package/dist/src/shared/wasm-codec/typings/document-manifest.model.d.ts +19 -0
- package/dist/src/shared/wasm-codec/typings/document-section.model.d.ts +12 -0
- package/dist/src/shared/wasm-codec/typings/item-decoder.type.d.ts +6 -0
- package/dist/src/shared/wasm-codec/typings/near-duplicate.model.d.ts +16 -0
- package/dist/src/shared/wasm-codec/typings/paged-search-result.model.d.ts +21 -0
- package/dist/src/shared/wasm-codec/typings/section-match.model.d.ts +31 -0
- package/dist/src/shared/wasm-codec/typings/section-record-trailing.model.d.ts +38 -0
- package/dist/src/shared/wasm-codec/typings/section-record.model.d.ts +66 -0
- package/dist/src/shared/wasm-codec/typings/section-type-result.model.d.ts +12 -0
- package/dist/src/shared/worker-client/classes/worker-client.class.d.ts +16 -0
- package/dist/src/shared/worker-client/functions/handle-ipc-data.function.d.ts +16 -0
- package/dist/src/shared/worker-client/functions/send-ipc-request.function.d.ts +19 -0
- package/dist/src/shared/worker-client/index.d.ts +1 -1
- package/dist/src/shared/worker-client/typings/add-section-result.model.d.ts +31 -15
- package/dist/src/shared/worker-client/typings/export-result.model.d.ts +14 -0
- package/dist/src/shared/worker-client/typings/gc-result.model.d.ts +6 -0
- package/dist/src/shared/worker-client/typings/json-rpc-response.interface.d.ts +16 -0
- package/dist/src/shared/worker-client/typings/pending-call.interface.d.ts +7 -0
- package/dist/src/shared/worker-client/typings/recall-document.model.d.ts +16 -0
- package/dist/src/shared/worker-client/typings/recall-params.model.d.ts +15 -0
- package/dist/src/shared/worker-client/typings/recall-reference.model.d.ts +32 -0
- package/dist/src/shared/worker-client/typings/recall-sibling.model.d.ts +11 -0
- package/dist/src/shared/worker-client/typings/recall-single-result.model.d.ts +40 -0
- package/dist/src/shared/worker-client/typings/search-result.model.d.ts +12 -0
- package/dist/src/shared/worker-client/typings/section.model.d.ts +16 -0
- package/dist/src/shared/worker-daemon/classes/ipc-error.class.d.ts +17 -0
- package/dist/src/shared/worker-daemon/classes/worker-daemon.class.d.ts +35 -0
- package/dist/src/shared/worker-daemon/constants/wasm-method-map.constant.d.ts +16 -0
- package/dist/src/shared/worker-daemon/functions/build-ipc-error-response.function.d.ts +9 -0
- package/dist/src/shared/worker-daemon/functions/check-catalog-version.function.d.ts +9 -0
- package/dist/src/shared/worker-daemon/functions/check-preconditions.function.d.ts +10 -0
- package/dist/src/shared/worker-daemon/functions/detect-stale-pid.function.d.ts +15 -0
- package/dist/src/shared/worker-daemon/functions/dispatch-request.function.d.ts +35 -0
- package/dist/src/shared/worker-daemon/functions/handle-check-version.function.d.ts +23 -0
- package/dist/src/shared/worker-daemon/functions/handle-config-method.function.d.ts +11 -0
- package/dist/src/shared/worker-daemon/functions/handle-read-ops.function.d.ts +52 -0
- package/dist/src/shared/worker-daemon/functions/handle-write-ops.function.d.ts +44 -0
- package/dist/src/shared/worker-daemon/functions/parse-auto-capture-section.function.d.ts +14 -0
- package/dist/src/shared/worker-daemon/functions/parse-catalog-version.function.d.ts +10 -0
- package/dist/src/shared/worker-daemon/functions/parse-embedding-section.function.d.ts +13 -0
- package/dist/src/shared/worker-daemon/functions/parse-worker-toml.function.d.ts +11 -0
- package/dist/src/shared/worker-daemon/functions/run-startup-integrity-check.function.d.ts +18 -0
- package/dist/src/shared/worker-daemon/functions/scan-near-duplicates.function.d.ts +21 -0
- package/dist/src/shared/worker-daemon/functions/wasm-bridge.function.d.ts +51 -0
- package/dist/src/shared/worker-daemon/functions/wasm-memory-helpers.function.d.ts +22 -0
- package/dist/src/shared/worker-daemon/functions/write-lock.function.d.ts +32 -0
- package/dist/src/shared/worker-daemon/typings/allocator-ref.interface.d.ts +16 -0
- package/dist/src/shared/worker-daemon/typings/daemon-config.interface.d.ts +6 -0
- package/dist/src/shared/worker-daemon/typings/fs-adapter.interface.d.ts +22 -0
- package/dist/src/shared/worker-daemon/typings/handler-context.interface.d.ts +18 -0
- package/dist/src/shared/worker-daemon/typings/memory-ref.interface.d.ts +13 -0
- package/dist/src/shared/worker-daemon/typings/wasm-fn.type.d.ts +2 -0
- package/dist/src/shared/worker-daemon/typings/write-lock.interface.d.ts +15 -0
- package/dist/wasm/fs-database/kbdb_fs_database.d.ts +22 -1
- package/dist/wasm/fs-database/kbdb_fs_database.js +28 -1
- package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
- package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm.d.ts +1 -0
- package/dist/wasm/kb-worker/kbdb_worker.d.ts +48 -1
- package/dist/wasm/kb-worker/kbdb_worker.js +60 -1
- package/dist/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
- package/dist/wasm/kb-worker/kbdb_worker_bg.wasm.d.ts +2 -0
- package/dist/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
- package/dist/worker.mjs +396 -60
- package/dist/worker.mjs.map +4 -4
- package/docs/details/cli.md +102 -15
- package/docs/details/library-api.md +55 -6
- package/docs/details/mcp-server.md +10 -4
- package/docs/details/search-and-ranking.md +63 -5
- package/docs/details/storage.md +2 -0
- package/docs/goals/auto-capture.md +378 -0
- package/docs/goals/benchmarking.md +296 -0
- package/docs/goals/cli.md +255 -14
- package/docs/goals/concurrency.md +259 -0
- package/docs/goals/database.md +244 -17
- package/docs/goals/document.md +1 -1
- package/docs/goals/hybrid-search.md +436 -0
- package/docs/goals/mcp.md +83 -14
- package/docs/goals/query-parser.md +98 -4
- package/docs/goals/query-result.md +177 -10
- package/docs/goals/semantic-dedup.md +229 -0
- package/docs/goals/skills.md +72 -0
- package/docs/goals/sync.md +217 -0
- package/docs/goals/worker-daemon.md +54 -7
- package/docs/modules/rust/fs-database/functions/integrity.md +11 -0
- package/docs/modules/rust/fs-database/functions/section.md +23 -6
- package/docs/modules/rust/fs-database/indexes/vectors.md +110 -0
- package/docs/modules/rust/fs-database/module.md +1 -0
- package/docs/modules/rust/fs-database/types/section.md +27 -1
- package/docs/modules/rust/overview.md +2 -1
- package/docs/modules/rust/query-parser/functions/module.md +2 -2
- package/docs/modules/rust/query-parser/functions/pipeline.md +44 -6
- package/docs/modules/rust/query-parser/module.md +5 -4
- package/docs/modules/rust/query-parser/types/keyphrase.md +11 -0
- package/docs/modules/rust/query-parser/types/module.md +3 -3
- package/docs/modules/typescript/shared/auto-capture/module.md +73 -0
- package/docs/modules/typescript/shared/cli/functions.md +68 -3
- package/docs/modules/typescript/shared/embedding/functions.md +94 -0
- package/docs/modules/typescript/shared/embedding/module.md +71 -0
- package/docs/modules/typescript/shared/embedding/typings.md +122 -0
- package/docs/overview.md +148 -166
- package/docs/release-notes/0.2.0.md +3 -3
- package/docs/release-notes/0.3.0.md +89 -0
- package/docs/release-notes/0.3.1.md +111 -0
- package/docs/release-notes/README.md +2 -0
- package/package.json +1 -1
- package/src/shared/auto-capture/classes/auto-capture-controller.class.ts +62 -0
- package/src/shared/auto-capture/constants/auto-capture-defaults.constant.ts +15 -0
- package/src/shared/auto-capture/functions/build-capture-prompt.function.ts +27 -0
- package/src/shared/auto-capture/functions/evaluate-trigger.function.ts +50 -0
- package/src/shared/auto-capture/index.ts +7 -0
- package/src/shared/auto-capture/typings/auto-capture-config.interface.ts +13 -0
- package/src/shared/auto-capture/typings/auto-capture-state.interface.ts +13 -0
- package/src/shared/auto-capture/typings/auto-capture-trigger.interface.ts +7 -0
- package/src/shared/cli/constants/global-help.constant.ts +73 -0
- package/src/shared/cli/constants/leaf-help.constant.ts +219 -0
- package/src/shared/cli/constants/memory-protocol-body.constant.ts +39 -0
- package/src/shared/cli/constants/query-guidance-body.constant.ts +37 -0
- package/src/shared/cli/constants/subcommand-help.constant.ts +195 -0
- package/src/shared/cli/functions/collect-files.function.ts +55 -0
- package/src/shared/cli/functions/compute-section-kbid.function.ts +49 -0
- package/src/shared/cli/functions/dispatch-command.function.ts +53 -1
- package/src/shared/cli/functions/format-mcp-output.function.ts +41 -0
- package/src/shared/cli/functions/format-output.function.ts +36 -12
- package/src/shared/cli/functions/format-text-output.function.ts +42 -0
- package/src/shared/cli/functions/generate-group-help.function.ts +41 -0
- package/src/shared/cli/functions/generate-help.function.ts +56 -1
- package/src/shared/cli/functions/handle-data-command.function.ts +67 -0
- package/src/shared/cli/functions/handle-mcp.function.ts +56 -0
- package/src/shared/cli/functions/handle-migrate.function.ts +64 -0
- package/src/shared/cli/functions/hydrate-search-previews.function.ts +29 -0
- package/src/shared/cli/functions/install-default-skills.function.ts +296 -0
- package/src/shared/cli/functions/parse-args.function.ts +16 -0
- package/src/shared/cli/functions/resolve-skill-names.function.ts +43 -0
- package/src/shared/cli/functions/run-check.function.ts +87 -6
- package/src/shared/cli/functions/run-export.function.ts +40 -0
- package/src/shared/cli/functions/run-gc.function.ts +6 -0
- package/src/shared/cli/functions/run-init.function.ts +2 -0
- package/src/shared/cli/functions/run-learn.function.ts +68 -16
- package/src/shared/cli/functions/run-recall.function.ts +66 -0
- package/src/shared/cli/functions/run-search.function.ts +60 -15
- package/src/shared/cli/functions/version-gate.function.ts +44 -0
- package/src/shared/cli/functions/wrap-command.function.ts +34 -0
- package/src/shared/cli/index.ts +4 -0
- package/src/shared/cli/typings/agent-create-client.interface.ts +47 -0
- package/src/shared/cli/typings/agent-get-client.interface.ts +39 -0
- package/src/shared/cli/typings/check-client.interface.ts +66 -0
- package/src/shared/cli/typings/cli-deps.interface.ts +54 -0
- package/src/shared/cli/typings/cli-options.interface.ts +15 -0
- package/src/shared/cli/typings/disconnectable-client.interface.ts +29 -0
- package/src/shared/cli/typings/dispatch-client.type.ts +46 -0
- package/src/shared/cli/typings/learn-client.interface.ts +53 -0
- package/src/shared/cli/typings/learn-io.interface.ts +5 -0
- package/src/shared/cli/typings/learn-result.model.ts +16 -0
- package/src/shared/cli/typings/search-client.interface.ts +73 -0
- package/src/shared/cli/typings/search-display.model.ts +18 -0
- package/src/shared/embedding/classes/null-embedding-provider.class.ts +29 -0
- package/src/shared/embedding/constants/near-duplicate-thresholds.constant.ts +17 -0
- package/src/shared/embedding/functions/build-vector-index.function.ts +79 -0
- package/src/shared/embedding/functions/cosine-similarity.function.ts +31 -0
- package/src/shared/embedding/functions/detect-near-duplicates.function.ts +69 -0
- package/src/shared/embedding/functions/jaccard-coefficient.function.ts +26 -0
- package/src/shared/embedding/functions/read-vector-index-meta.function.ts +44 -0
- package/src/shared/embedding/functions/rrf-fuse.function.ts +33 -0
- package/src/shared/embedding/functions/search-vectors.function.ts +59 -0
- package/src/shared/embedding/index.ts +20 -0
- package/src/shared/embedding/typings/embedding-config.interface.ts +23 -0
- package/src/shared/embedding/typings/embedding-provider.interface.ts +25 -0
- package/src/shared/embedding/typings/near-duplicate-candidate.model.ts +25 -0
- package/src/shared/embedding/typings/near-duplicate-result.model.ts +26 -0
- package/src/shared/embedding/typings/vector-index-entry.model.ts +13 -0
- package/src/shared/embedding/typings/vector-index-meta.model.ts +14 -0
- package/src/shared/embedding/typings/vector-search-result.model.ts +8 -0
- package/src/shared/mcp/classes/mcp-server.class.ts +36 -4
- package/src/shared/mcp/constants/tool-agent-create.constant.ts +39 -0
- package/src/shared/mcp/constants/tool-agent-delete.constant.ts +26 -0
- package/src/shared/mcp/constants/tool-agent-get.constant.ts +24 -0
- package/src/shared/mcp/constants/tool-agent-list.constant.ts +18 -0
- package/src/shared/mcp/constants/tool-auto-capture-review.constant.ts +32 -0
- package/src/shared/mcp/constants/tool-check.constant.ts +21 -0
- package/src/shared/mcp/constants/tool-content.constant.ts +29 -0
- package/src/shared/mcp/constants/tool-export.constant.ts +27 -0
- package/src/shared/mcp/constants/tool-gc.constant.ts +21 -0
- package/src/shared/mcp/constants/tool-learn.constant.ts +58 -0
- package/src/shared/mcp/constants/tool-rebuild.constant.ts +21 -0
- package/src/shared/mcp/constants/tool-recall.constant.ts +36 -0
- package/src/shared/mcp/constants/tool-retrieve.constant.ts +31 -0
- package/src/shared/mcp/constants/tool-search.constant.ts +51 -0
- package/src/shared/mcp/constants/tool-skill-delete.constant.ts +24 -0
- package/src/shared/mcp/constants/tool-skill-get.constant.ts +27 -0
- package/src/shared/mcp/constants/tool-skill-learn.constant.ts +37 -0
- package/src/shared/mcp/constants/tool-skill-list.constant.ts +18 -0
- package/src/shared/mcp/constants/tool-status.constant.ts +20 -0
- package/src/shared/mcp/constants/tool-unlearn.constant.ts +25 -0
- package/src/shared/mcp/functions/as-boolean-or-undefined.function.ts +11 -0
- package/src/shared/mcp/functions/as-string-array-or-undefined.function.ts +17 -0
- package/src/shared/mcp/functions/as-string-array.function.ts +15 -0
- package/src/shared/mcp/functions/as-string.function.ts +10 -0
- package/src/shared/mcp/functions/build-skills-text.function.ts +43 -0
- package/src/shared/mcp/functions/create-tool-definitions.function.ts +74 -5
- package/src/shared/mcp/functions/handle-agent-create.function.ts +1 -1
- package/src/shared/mcp/functions/handle-agent-prompt.function.ts +68 -0
- package/src/shared/mcp/functions/handle-mcp-request.function.ts +202 -0
- package/src/shared/mcp/functions/handle-prompt-get.function.ts +18 -12
- package/src/shared/mcp/functions/handle-skill-prompt.function.ts +44 -0
- package/src/shared/mcp/functions/handle-tool-auto-capture-review.function.ts +24 -0
- package/src/shared/mcp/functions/handle-tool-call.function.ts +88 -2
- package/src/shared/mcp/functions/handle-tool-check.function.ts +21 -0
- package/src/shared/mcp/functions/handle-tool-content.function.ts +23 -0
- package/src/shared/mcp/functions/handle-tool-export.function.ts +25 -0
- package/src/shared/mcp/functions/handle-tool-gc.function.ts +21 -0
- package/src/shared/mcp/functions/handle-tool-learn.function.ts +69 -0
- package/src/shared/mcp/functions/handle-tool-rebuild.function.ts +21 -0
- package/src/shared/mcp/functions/handle-tool-retrieve.function.ts +40 -0
- package/src/shared/mcp/functions/handle-tool-search.function.ts +47 -0
- package/src/shared/mcp/functions/handle-tool-skill-delete.function.ts +34 -0
- package/src/shared/mcp/functions/handle-tool-skill-get.function.ts +59 -0
- package/src/shared/mcp/functions/handle-tool-skill-learn.function.ts +61 -0
- package/src/shared/mcp/functions/handle-tool-skill-list.function.ts +39 -0
- package/src/shared/mcp/functions/handle-tool-status.function.ts +21 -0
- package/src/shared/mcp/functions/handle-tool-unlearn.function.ts +24 -0
- package/src/shared/mcp/functions/parse-agent-meta.function.ts +27 -0
- package/src/shared/mcp/functions/parse-skill-arguments.function.ts +36 -0
- package/src/shared/mcp/functions/read-agent-meta.function.ts +41 -0
- package/src/shared/mcp/functions/resolve-skill-refs.function.ts +50 -0
- package/src/shared/mcp/functions/run-startup-health-check.function.ts +34 -0
- package/src/shared/mcp/typings/mcp-capabilities.interface.ts +16 -0
- package/src/shared/mcp/typings/mcp-client.interface.ts +47 -0
- package/src/shared/mcp/typings/mcp-completion.interface.ts +45 -0
- package/src/shared/mcp/typings/mcp-content-result.model.ts +9 -0
- package/src/shared/mcp/typings/mcp-export-result.model.ts +11 -0
- package/src/shared/mcp/typings/mcp-gc-result.model.ts +15 -0
- package/src/shared/mcp/typings/mcp-integrity-check-result.model.ts +17 -0
- package/src/shared/mcp/typings/mcp-integrity-error.model.ts +15 -0
- package/src/shared/mcp/typings/mcp-notification.interface.ts +9 -0
- package/src/shared/mcp/typings/mcp-rebuild-result.model.ts +9 -0
- package/src/shared/mcp/typings/mcp-request.interface.ts +11 -0
- package/src/shared/mcp/typings/mcp-resource-result.interface.ts +12 -0
- package/src/shared/mcp/typings/mcp-resource-template.interface.ts +11 -0
- package/src/shared/mcp/typings/mcp-response.interface.ts +18 -0
- package/src/shared/mcp/typings/mcp-server-info.interface.ts +7 -0
- package/src/shared/mcp/typings/mcp-server-state.interface.ts +17 -0
- package/src/shared/mcp/typings/mcp-tool-annotations.interface.ts +32 -0
- package/src/shared/mcp/typings/mcp-tool-definition.interface.ts +16 -0
- package/src/shared/mcp/typings/mcp-tool-result.interface.ts +19 -0
- package/src/shared/mcp/typings/mcp-worker-client.interface.ts +243 -0
- package/src/shared/mcp/typings/tool-call-result.model.ts +18 -0
- package/src/shared/sync/functions/build-import-report.function.ts +40 -0
- package/src/shared/sync/functions/parse-export-entry.function.ts +45 -0
- package/src/shared/sync/functions/resolve-conflict.function.ts +31 -0
- package/src/shared/sync/functions/serialize-export-entry.function.ts +18 -0
- package/src/shared/sync/index.ts +9 -0
- package/src/shared/sync/typings/conflict-resolution.type.ts +8 -0
- package/src/shared/sync/typings/export-entry.interface.ts +13 -0
- package/src/shared/sync/typings/export-options.interface.ts +9 -0
- package/src/shared/sync/typings/import-options.interface.ts +7 -0
- package/src/shared/sync/typings/import-report.interface.ts +15 -0
- package/src/shared/wasm-codec/functions/decode-add-section-result.function.ts +112 -0
- package/src/shared/wasm-codec/functions/decode-one-recall-result.function.ts +86 -0
- package/src/shared/wasm-codec/functions/decode-option-string.function.ts +24 -0
- package/src/shared/wasm-codec/functions/decode-recall-document.function.ts +59 -0
- package/src/shared/wasm-codec/functions/decode-recall-reference.function.ts +52 -0
- package/src/shared/wasm-codec/functions/decode-recall-sibling.function.ts +43 -0
- package/src/shared/wasm-codec/functions/decode-scored-results.function.ts +86 -0
- package/src/shared/wasm-codec/functions/decode-section-match.function.ts +80 -0
- package/src/shared/wasm-codec/functions/decode-section-record-trailing.function.ts +75 -0
- package/src/shared/wasm-codec/functions/decode-section-record.function.ts +85 -0
- package/src/shared/wasm-codec/functions/decode-section-records.function.ts +87 -0
- package/src/shared/wasm-codec/functions/decode-section-type.function.ts +30 -0
- package/src/shared/wasm-codec/functions/decode-wasm-list.function.ts +31 -0
- package/src/shared/wasm-codec/functions/encode-add-section-params.function.ts +68 -0
- package/src/shared/wasm-codec/index.ts +5 -0
- package/src/shared/wasm-codec/typings/add-section-input.model.ts +45 -0
- package/src/shared/wasm-codec/typings/add-section-result.model.ts +28 -0
- package/src/shared/wasm-codec/typings/document-manifest.model.ts +20 -0
- package/src/shared/wasm-codec/typings/document-section.model.ts +12 -0
- package/src/shared/wasm-codec/typings/item-decoder.type.ts +10 -0
- package/src/shared/wasm-codec/typings/near-duplicate.model.ts +16 -0
- package/src/shared/wasm-codec/typings/paged-search-result.model.ts +22 -0
- package/src/shared/wasm-codec/typings/section-match.model.ts +31 -0
- package/src/shared/wasm-codec/typings/section-record-trailing.model.ts +38 -0
- package/src/shared/wasm-codec/typings/section-record.model.ts +66 -0
- package/src/shared/wasm-codec/typings/section-type-result.model.ts +12 -0
- package/src/shared/worker-client/classes/worker-client.class.ts +19 -0
- package/src/shared/worker-client/functions/handle-ipc-data.function.ts +49 -0
- package/src/shared/worker-client/functions/send-ipc-request.function.ts +39 -0
- package/src/shared/worker-client/index.ts +4 -1
- package/src/shared/worker-client/typings/add-section-result.model.ts +32 -15
- package/src/shared/worker-client/typings/export-result.model.ts +14 -0
- package/src/shared/worker-client/typings/gc-result.model.ts +6 -0
- package/src/shared/worker-client/typings/json-rpc-response.interface.ts +17 -0
- package/src/shared/worker-client/typings/pending-call.interface.ts +7 -0
- package/src/shared/worker-client/typings/recall-document.model.ts +16 -0
- package/src/shared/worker-client/typings/recall-params.model.ts +15 -0
- package/src/shared/worker-client/typings/recall-reference.model.ts +32 -0
- package/src/shared/worker-client/typings/recall-sibling.model.ts +11 -0
- package/src/shared/worker-client/typings/recall-single-result.model.ts +41 -0
- package/src/shared/worker-client/typings/search-result.model.ts +12 -0
- package/src/shared/worker-client/typings/section.model.ts +16 -0
- package/src/shared/worker-daemon/classes/ipc-error.class.ts +22 -0
- package/src/shared/worker-daemon/classes/worker-daemon.class.ts +315 -46
- package/src/shared/worker-daemon/constants/wasm-method-map.constant.ts +69 -0
- package/src/shared/worker-daemon/functions/build-ipc-error-response.function.ts +16 -0
- package/src/shared/worker-daemon/functions/check-catalog-version.function.ts +54 -0
- package/src/shared/worker-daemon/functions/check-preconditions.function.ts +28 -0
- package/src/shared/worker-daemon/functions/create-host-imports.function.ts +19 -4
- package/src/shared/worker-daemon/functions/create-ipc-server.function.ts +5 -1
- package/src/shared/worker-daemon/functions/detect-stale-pid.function.ts +33 -0
- package/src/shared/worker-daemon/functions/dispatch-request.function.ts +190 -0
- package/src/shared/worker-daemon/functions/handle-check-version.function.ts +74 -0
- package/src/shared/worker-daemon/functions/handle-config-method.function.ts +27 -0
- package/src/shared/worker-daemon/functions/handle-read-ops.function.ts +236 -0
- package/src/shared/worker-daemon/functions/handle-write-ops.function.ts +257 -0
- package/src/shared/worker-daemon/functions/parse-auto-capture-section.function.ts +51 -0
- package/src/shared/worker-daemon/functions/parse-catalog-version.function.ts +20 -0
- package/src/shared/worker-daemon/functions/parse-embedding-section.function.ts +38 -0
- package/src/shared/worker-daemon/functions/parse-worker-toml.function.ts +104 -0
- package/src/shared/worker-daemon/functions/run-startup-integrity-check.function.ts +55 -0
- package/src/shared/worker-daemon/functions/scan-near-duplicates.function.ts +92 -0
- package/src/shared/worker-daemon/functions/wasm-bridge.function.ts +144 -0
- package/src/shared/worker-daemon/functions/wasm-memory-helpers.function.ts +41 -0
- package/src/shared/worker-daemon/functions/write-lock.function.ts +49 -0
- package/src/shared/worker-daemon/typings/allocator-ref.interface.ts +16 -0
- package/src/shared/worker-daemon/typings/daemon-config.interface.ts +7 -0
- package/src/shared/worker-daemon/typings/fs-adapter.interface.ts +22 -0
- package/src/shared/worker-daemon/typings/handler-context.interface.ts +19 -0
- package/src/shared/worker-daemon/typings/memory-ref.interface.ts +13 -0
- package/src/shared/worker-daemon/typings/wasm-fn.type.ts +2 -0
- package/src/shared/worker-daemon/typings/write-lock.interface.ts +13 -0
- package/src/wasm/fs-database/kbdb_fs_database.d.ts +22 -1
- package/src/wasm/fs-database/kbdb_fs_database.js +28 -1
- package/src/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
- package/src/wasm/fs-database/kbdb_fs_database_bg.wasm.d.ts +1 -0
- package/src/wasm/kb-worker/kbdb_worker.d.ts +48 -1
- package/src/wasm/kb-worker/kbdb_worker.js +60 -1
- package/src/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
- package/src/wasm/kb-worker/kbdb_worker_bg.wasm.d.ts +2 -0
- package/src/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
|
@@ -115,9 +115,11 @@ Stages:
|
|
|
115
115
|
decomposition).
|
|
116
116
|
5. **Case folding** (lowercase).
|
|
117
117
|
6. **Stop word removal** (configurable list).
|
|
118
|
-
7. **
|
|
118
|
+
7. **Synonym expansion** (bidirectional synonym map --
|
|
119
|
+
inserts synonym tokens alongside originals).
|
|
120
|
+
8. **Stemming** (Snowball / Porter2 -- stores both
|
|
119
121
|
stemmed and original forms).
|
|
120
|
-
|
|
122
|
+
9. **N-gram extraction** (bigrams, trigrams for phrase
|
|
121
123
|
indexing and multi-word term matching).
|
|
122
124
|
|
|
123
125
|
`kb-worker.wasm` calls the tokenization pipeline for
|
|
@@ -126,6 +128,71 @@ and search (when a query is submitted). It feeds the
|
|
|
126
128
|
tokenized output to `fs-database.wasm` for index
|
|
127
129
|
storage or index lookups as needed.
|
|
128
130
|
|
|
131
|
+
### Synonym Expansion (Stage 7)
|
|
132
|
+
|
|
133
|
+
Bridges the lexical gap between user vocabulary and
|
|
134
|
+
indexed content. Without synonym expansion, a search
|
|
135
|
+
for `"login credentials"` returns zero results when
|
|
136
|
+
the indexed document uses "signs in" and "identity"
|
|
137
|
+
instead -- different root words with the same meaning.
|
|
138
|
+
|
|
139
|
+
**Placement rationale:** Applied after stop-word
|
|
140
|
+
removal (stage 6) and before stemming (stage 8). This
|
|
141
|
+
ensures synonyms are expanded on lowercased,
|
|
142
|
+
stop-word-filtered tokens, and both the original and
|
|
143
|
+
synonym tokens are then stemmed consistently.
|
|
144
|
+
|
|
145
|
+
**Built-in synonym map:** A small, curated
|
|
146
|
+
`HashMap<String, Vec<String>>` of bidirectional
|
|
147
|
+
synonym pairs. Bidirectional means if `login` maps to
|
|
148
|
+
`sign-in`, then `sign-in` also maps to `login`.
|
|
149
|
+
|
|
150
|
+
**Example map entries:**
|
|
151
|
+
|
|
152
|
+
| Term | Synonyms |
|
|
153
|
+
|--------------|------------------------------------|
|
|
154
|
+
| login | sign-in, sign_in, signin |
|
|
155
|
+
| auth | authentication, authenticate |
|
|
156
|
+
| remove | delete, erase |
|
|
157
|
+
| create | add, insert |
|
|
158
|
+
| update | modify, change, edit |
|
|
159
|
+
| error | exception, fault, failure |
|
|
160
|
+
| config | configuration, settings |
|
|
161
|
+
| credential | password, identity |
|
|
162
|
+
| user | account, member |
|
|
163
|
+
| deploy | release, publish, ship |
|
|
164
|
+
| search | query, find, lookup |
|
|
165
|
+
| doc | document, documentation |
|
|
166
|
+
| api | endpoint, interface |
|
|
167
|
+
| db | database |
|
|
168
|
+
| env | environment |
|
|
169
|
+
| repo | repository |
|
|
170
|
+
| param | parameter, argument |
|
|
171
|
+
| dir | directory, folder |
|
|
172
|
+
| msg | message |
|
|
173
|
+
| info | information |
|
|
174
|
+
|
|
175
|
+
**Expansion behaviour:** For each token in the stream,
|
|
176
|
+
look up the synonym map. If found, insert the synonym
|
|
177
|
+
tokens at the same position as the original. Both the
|
|
178
|
+
original and its synonyms proceed to the stemming
|
|
179
|
+
stage. This preserves the original token's position
|
|
180
|
+
for proximity/zone scoring while adding synonym
|
|
181
|
+
coverage.
|
|
182
|
+
|
|
183
|
+
**Applied symmetrically:** The same synonym expansion
|
|
184
|
+
applies during both `learn` (indexing) and `search`
|
|
185
|
+
(querying). If a document uses "login" and the user
|
|
186
|
+
searches "sign-in", the expansion during indexing
|
|
187
|
+
produced "sign-in" tokens from "login", and the
|
|
188
|
+
expansion during querying produced "login" tokens
|
|
189
|
+
from "sign-in".
|
|
190
|
+
|
|
191
|
+
**Not user-configurable in v1.** The built-in map is
|
|
192
|
+
hardcoded. Future versions may allow user-defined
|
|
193
|
+
synonym maps via a `synonyms.toml` configuration file
|
|
194
|
+
in the `.kbdb` directory.
|
|
195
|
+
|
|
129
196
|
## Query Parsing
|
|
130
197
|
|
|
131
198
|
Search queries are parsed into a structured query plan
|
|
@@ -148,8 +215,35 @@ Parsing stages:
|
|
|
148
215
|
The query plan is the query parser's output --
|
|
149
216
|
`kb-worker.wasm` receives it and passes it to
|
|
150
217
|
`fs-database.wasm` for index lookups, then calls the
|
|
151
|
-
query parser's ranking functions to score and order
|
|
152
|
-
results.
|
|
218
|
+
query parser's ranking functions to score and order
|
|
219
|
+
the raw results.
|
|
220
|
+
|
|
221
|
+
### Relaxed Query Mode
|
|
222
|
+
|
|
223
|
+
When a multi-term AND query returns zero results,
|
|
224
|
+
`kb-worker.wasm` retries the query in **relaxed
|
|
225
|
+
mode** -- OR semantics where any term matches, not
|
|
226
|
+
all required.
|
|
227
|
+
|
|
228
|
+
The query parser generates both the primary (AND)
|
|
229
|
+
plan and a relaxed (OR) plan from the same parsed
|
|
230
|
+
query. `kb-worker.wasm` executes the AND plan first.
|
|
231
|
+
If the AND plan produces zero matches, it falls back
|
|
232
|
+
to the OR plan. No new WASM export is needed --
|
|
233
|
+
`qp_parse_query` returns a `QueryPlan` that supports
|
|
234
|
+
mode switching via a `match_mode` field
|
|
235
|
+
(`All` | `Any`).
|
|
236
|
+
|
|
237
|
+
Relaxed results are flagged with `relaxed: true` in
|
|
238
|
+
the `PagedSearchResult` envelope (see
|
|
239
|
+
[Query Result -- Relaxed-Match
|
|
240
|
+
Fallback](query-result.md)). The fallback is
|
|
241
|
+
transparent to the caller.
|
|
242
|
+
|
|
243
|
+
Single-term queries are unaffected (AND and OR are
|
|
244
|
+
equivalent for one term). Phrase queries and
|
|
245
|
+
exclusions are preserved in both modes -- only the
|
|
246
|
+
inter-term combination changes.
|
|
153
247
|
|
|
154
248
|
## Query Cache
|
|
155
249
|
|
|
@@ -434,6 +434,34 @@ order** (highest relevance first). Ties are broken
|
|
|
434
434
|
by kbid lexicographic order for deterministic
|
|
435
435
|
ordering.
|
|
436
436
|
|
|
437
|
+
## Score Normalization
|
|
438
|
+
|
|
439
|
+
Raw BM25F scores vary in magnitude across queries and
|
|
440
|
+
corpora, making them difficult to interpret in
|
|
441
|
+
isolation. The ranking pipeline produces a
|
|
442
|
+
**normalized confidence** score alongside the raw
|
|
443
|
+
score:
|
|
444
|
+
|
|
445
|
+
```
|
|
446
|
+
confidence = final_score / max_score_in_result_set
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
- The top-ranked result always has `confidence = 1.0`.
|
|
450
|
+
- A result with half the raw score of the top result
|
|
451
|
+
has `confidence = 0.5`.
|
|
452
|
+
- When the result set is empty, no normalization is
|
|
453
|
+
needed.
|
|
454
|
+
|
|
455
|
+
`confidence` is a relative measure within a single
|
|
456
|
+
query's result set -- it does not compare across
|
|
457
|
+
different queries. Its purpose is to let the caller
|
|
458
|
+
distinguish strong matches from weak tail results
|
|
459
|
+
without needing to understand BM25F score magnitudes.
|
|
460
|
+
|
|
461
|
+
The CLI exposes `confidence` in `SearchDisplay` for
|
|
462
|
+
human inspection. The MCP projection omits it --
|
|
463
|
+
result ordering is the ranking signal for AI agents.
|
|
464
|
+
|
|
437
465
|
## Result Truncation and Pagination
|
|
438
466
|
|
|
439
467
|
After sorting, results are windowed by `offset` and
|
|
@@ -452,6 +480,7 @@ interface PagedSearchResult {
|
|
|
452
480
|
offset: number;
|
|
453
481
|
limit: number;
|
|
454
482
|
has_more: boolean;
|
|
483
|
+
relaxed: boolean;
|
|
455
484
|
}
|
|
456
485
|
```
|
|
457
486
|
|
|
@@ -462,6 +491,7 @@ interface PagedSearchResult {
|
|
|
462
491
|
| `offset` | Echo of requested offset (or 0) |
|
|
463
492
|
| `limit` | Echo of effective limit (or 20) |
|
|
464
493
|
| `has_more` | `true` when `offset + items.length < total` |
|
|
494
|
+
| `relaxed` | `true` when results came from relaxed-match fallback (see below) |
|
|
465
495
|
|
|
466
496
|
`has_more` is a convenience boolean so the caller
|
|
467
497
|
does not need to compute whether more pages exist.
|
|
@@ -489,14 +519,26 @@ interface SectionMatch {
|
|
|
489
519
|
type: string;
|
|
490
520
|
heading: string;
|
|
491
521
|
score: number;
|
|
522
|
+
confidence: number;
|
|
492
523
|
snippet: string;
|
|
493
524
|
matched_terms: string[];
|
|
525
|
+
matched_fields: string[];
|
|
526
|
+
created_at: string;
|
|
494
527
|
}
|
|
495
528
|
```
|
|
496
529
|
|
|
497
530
|
`heading` is `string | null` -- `null` when the
|
|
498
|
-
section has no title. `
|
|
499
|
-
|
|
531
|
+
section has no title. `confidence` is the normalized
|
|
532
|
+
score (0.0--1.0) relative to the top result (see
|
|
533
|
+
Score Normalization above). `matched_fields` is a
|
|
534
|
+
subset of `["heading", "body", "code"]` indicating
|
|
535
|
+
which fields contained term matches -- lets the
|
|
536
|
+
caller see whether the match was in a title vs body
|
|
537
|
+
text. `created_at` is the section's ISO 8601
|
|
538
|
+
creation timestamp.
|
|
539
|
+
|
|
540
|
+
`snippet` is always populated for text and code
|
|
541
|
+
sections using KWIC
|
|
500
542
|
(keyword-in-context): ~150 characters centered on
|
|
501
543
|
the highest-scoring matched term position, with
|
|
502
544
|
`...` ellipsis at boundaries. If the best match is
|
|
@@ -576,17 +618,25 @@ interface McpSearchResult {
|
|
|
576
618
|
docids: string[];
|
|
577
619
|
snippet: string;
|
|
578
620
|
matched_terms: string[];
|
|
621
|
+
matched_fields: string[];
|
|
622
|
+
created_at: string;
|
|
579
623
|
content?: string;
|
|
580
624
|
}
|
|
581
625
|
```
|
|
582
626
|
|
|
583
|
-
The `score`
|
|
584
|
-
intentionally omitted. Results
|
|
585
|
-
relevance descending; the position
|
|
586
|
-
is the ranking signal. Exposing
|
|
587
|
-
scores provides no actionable
|
|
588
|
-
agent and risks confusing score
|
|
589
|
-
different queries.
|
|
627
|
+
The `score` and `confidence` fields from
|
|
628
|
+
`SectionMatch` are intentionally omitted. Results
|
|
629
|
+
are ordered by relevance descending; the position
|
|
630
|
+
in the array is the ranking signal. Exposing
|
|
631
|
+
uncalibrated BM25F scores provides no actionable
|
|
632
|
+
information to the agent and risks confusing score
|
|
633
|
+
magnitudes across different queries.
|
|
634
|
+
|
|
635
|
+
`matched_fields` is included because it tells the
|
|
636
|
+
agent whether the match was in a heading (strong
|
|
637
|
+
topical signal) vs body text (weaker). `created_at`
|
|
638
|
+
is included to let agents prefer recent results
|
|
639
|
+
when multiple sections cover the same topic.
|
|
590
640
|
|
|
591
641
|
`matched_terms` is a flat `string[]` of the query
|
|
592
642
|
terms that matched -- no per-term frequency counts.
|
|
@@ -609,7 +659,40 @@ inspection and debugging.
|
|
|
609
659
|
|
|
610
660
|
The MCP result uses the same `PagedSearchResult`
|
|
611
661
|
envelope with `items`, `total`, `offset`, `limit`,
|
|
612
|
-
and `
|
|
662
|
+
`has_more`, and `relaxed`.
|
|
663
|
+
|
|
664
|
+
## Relaxed-Match Fallback
|
|
665
|
+
|
|
666
|
+
When the primary query (implicit AND -- all terms
|
|
667
|
+
required) returns **zero results**, the ranking
|
|
668
|
+
engine automatically retries with relaxed matching:
|
|
669
|
+
|
|
670
|
+
1. **OR mode** -- any query term may match, not all
|
|
671
|
+
required. The query plan is re-executed with OR
|
|
672
|
+
semantics using the same tokenized terms.
|
|
673
|
+
|
|
674
|
+
2. Results from the relaxed retry are flagged:
|
|
675
|
+
`relaxed: true` in the `PagedSearchResult`
|
|
676
|
+
envelope. Normal (AND) results have
|
|
677
|
+
`relaxed: false`.
|
|
678
|
+
|
|
679
|
+
Relaxed mode is transparent to the caller -- no
|
|
680
|
+
special syntax or parameters are needed. The engine
|
|
681
|
+
tries AND first and falls back to OR only when AND
|
|
682
|
+
produces zero matches. This converts a silent miss
|
|
683
|
+
("nothing found") into a soft hit ("weaker matches
|
|
684
|
+
found") that the caller can reason about.
|
|
685
|
+
|
|
686
|
+
The AI agent (or human user) can check the `relaxed`
|
|
687
|
+
flag to know that the results are partial-term
|
|
688
|
+
matches rather than full-term matches. Relaxed
|
|
689
|
+
results typically have lower relevance -- the
|
|
690
|
+
ordering and confidence scores still reflect relative
|
|
691
|
+
quality within the relaxed set.
|
|
692
|
+
|
|
693
|
+
When the primary AND query returns at least one
|
|
694
|
+
result, no fallback is attempted. The `relaxed` flag
|
|
695
|
+
is always `false` for non-empty AND results.
|
|
613
696
|
|
|
614
697
|
## References Resolution
|
|
615
698
|
|
|
@@ -691,3 +774,87 @@ of the ranking coin:
|
|
|
691
774
|
|
|
692
775
|
See [Query Parser](query-parser.md) for the full
|
|
693
776
|
tokenization pipeline and extraction algorithms.
|
|
777
|
+
|
|
778
|
+
## Content Authoring Best Practices
|
|
779
|
+
|
|
780
|
+
### How Field Weights Affect Searchability
|
|
781
|
+
|
|
782
|
+
kbdb uses three weighted fields for ranking (see
|
|
783
|
+
the [Field Weight Mapping](#field-weight-mapping)
|
|
784
|
+
table above). Content authors can influence how
|
|
785
|
+
well their content is found by understanding these
|
|
786
|
+
weights:
|
|
787
|
+
|
|
788
|
+
- **Title / heading text** carries **2.0x** the
|
|
789
|
+
ranking weight of body text. A match in a heading
|
|
790
|
+
has twice the influence on ranking as the same
|
|
791
|
+
match in the body.
|
|
792
|
+
- **Code blocks** carry **1.5x** weight. Code
|
|
793
|
+
identifiers (function names, variable names, class
|
|
794
|
+
names) are split on camelCase/snake_case boundaries
|
|
795
|
+
and indexed as separate terms.
|
|
796
|
+
- **Body text and descriptions** are the baseline at
|
|
797
|
+
**1.0x**.
|
|
798
|
+
- **Binary content** (images, attachments) is not
|
|
799
|
+
indexed at all -- searchability depends entirely on
|
|
800
|
+
the `title` (required) and `description` (optional)
|
|
801
|
+
attributes.
|
|
802
|
+
|
|
803
|
+
### Writing Index-Friendly Headings
|
|
804
|
+
|
|
805
|
+
The heading is the single highest-impact field for
|
|
806
|
+
searchability. Follow these guidelines:
|
|
807
|
+
|
|
808
|
+
- Use **descriptive, keyword-rich headings**. Prefer
|
|
809
|
+
"Authentication and Login Flow" over just "Auth
|
|
810
|
+
Flow" -- the expanded heading gives the search
|
|
811
|
+
engine more term surface.
|
|
812
|
+
- Include **synonyms and alternative phrasing** in
|
|
813
|
+
headings when natural. If a section covers logging
|
|
814
|
+
in, mention both "login" and "sign-in" if
|
|
815
|
+
applicable.
|
|
816
|
+
- Avoid generic headings like "Overview",
|
|
817
|
+
"Introduction", or "Notes" that carry no
|
|
818
|
+
discriminative value for search.
|
|
819
|
+
- Stemming is automatic ("connections" matches
|
|
820
|
+
"connect"), but different root words are NOT
|
|
821
|
+
equivalent without synonym expansion --
|
|
822
|
+
"login" does not match "sign-in" by default.
|
|
823
|
+
|
|
824
|
+
### Structuring Content for Recall
|
|
825
|
+
|
|
826
|
+
Content structure affects the progressive context
|
|
827
|
+
expansion provided by the `recall` command:
|
|
828
|
+
|
|
829
|
+
- Place the most important terms in the **first 20%**
|
|
830
|
+
of a section. The zone scoring algorithm gives
|
|
831
|
+
extra weight to terms in this heading zone.
|
|
832
|
+
- Use explicit cross-references (`$(link:kbid)`
|
|
833
|
+
markers) between related sections. The reference
|
|
834
|
+
graph enables progressive context expansion via
|
|
835
|
+
`recall` at depth 1+.
|
|
836
|
+
- Group related sections into documents. The `recall`
|
|
837
|
+
command at depth 2+ returns sibling sections,
|
|
838
|
+
giving AI agents full surrounding context.
|
|
839
|
+
- For code sections: use meaningful identifiers.
|
|
840
|
+
camelCase and snake_case names are automatically
|
|
841
|
+
split and indexed. `getUserById` produces four
|
|
842
|
+
searchable terms: "get", "user", "by", "id".
|
|
843
|
+
|
|
844
|
+
### Leveraging the Query Syntax
|
|
845
|
+
|
|
846
|
+
Guidance for AI agents and human users on writing
|
|
847
|
+
effective queries:
|
|
848
|
+
|
|
849
|
+
- Use **multiple short keyword queries** rather than
|
|
850
|
+
one long natural-language question. Lexical search
|
|
851
|
+
matches on token overlap, not semantic meaning.
|
|
852
|
+
- Use **exact phrases** (`"error handling"`) when term
|
|
853
|
+
order matters.
|
|
854
|
+
- Use **boolean operators** (`auth AND token`,
|
|
855
|
+
`cache OR redis`) to broaden or narrow results.
|
|
856
|
+
- Use **term exclusion** (`auth -oauth`) to filter out
|
|
857
|
+
irrelevant matches.
|
|
858
|
+
- After finding results with `search`, use `recall`
|
|
859
|
+
with increasing depth to progressively expand
|
|
860
|
+
context.
|
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
# Semantic Near-Duplicate Detection
|
|
2
|
+
|
|
3
|
+
*Cosine similarity on embeddings as a complementary
|
|
4
|
+
detection layer alongside Jaccard -- closes the
|
|
5
|
+
vocabulary-gap ceiling identified in gap #6.*
|
|
6
|
+
|
|
7
|
+
## Goal
|
|
8
|
+
|
|
9
|
+
The existing near-duplicate detection computes a Jaccard
|
|
10
|
+
coefficient on stemmed term sets of the incoming section
|
|
11
|
+
versus all indexed sections. Sections that exceed a
|
|
12
|
+
similarity threshold of 0.85 are reported as
|
|
13
|
+
near-duplicates in `AddSectionResult.near_duplicates`.
|
|
14
|
+
|
|
15
|
+
This mechanism catches lexically similar content
|
|
16
|
+
reliably. It does not catch semantically equivalent
|
|
17
|
+
sections that use different vocabulary. Example: a
|
|
18
|
+
section stored as "We selected PostgreSQL for its JSONB
|
|
19
|
+
support" and a later section written as "Chose Postgres
|
|
20
|
+
due to native JSON column type" express the same
|
|
21
|
+
decision. Their stemmed term sets share few tokens --
|
|
22
|
+
Jaccard produces a low score, no duplicate is flagged,
|
|
23
|
+
and the knowledge base silently accumulates redundant
|
|
24
|
+
content.
|
|
25
|
+
|
|
26
|
+
This gap is identified as the semantic dedup ceiling in
|
|
27
|
+
[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md)
|
|
28
|
+
(gap #6). Semantic dedup closes it by adding a cosine
|
|
29
|
+
similarity pass on embedding vectors as a complementary
|
|
30
|
+
detection method when an embedding provider is
|
|
31
|
+
configured. The feature is conditional on
|
|
32
|
+
[Hybrid Search](hybrid-search.md) -- if no
|
|
33
|
+
`EmbeddingProvider` is configured, only the Jaccard pass
|
|
34
|
+
runs and behaviour is identical to today.
|
|
35
|
+
|
|
36
|
+
## Two-Pass Detection Algorithm
|
|
37
|
+
|
|
38
|
+
Detection proceeds in two sequential passes on every
|
|
39
|
+
`add_section` call.
|
|
40
|
+
|
|
41
|
+
### Pass 1: Jaccard (always runs)
|
|
42
|
+
|
|
43
|
+
Compute the Jaccard coefficient between the incoming
|
|
44
|
+
section's stemmed term set and each existing section's
|
|
45
|
+
term set. The term sets are derived from the inverted
|
|
46
|
+
index -- no section files are read.
|
|
47
|
+
|
|
48
|
+
Sections whose Jaccard score meets or exceeds 0.85 are
|
|
49
|
+
flagged as lexical near-duplicates and appended to
|
|
50
|
+
`near_duplicates` with `method: "jaccard"`.
|
|
51
|
+
|
|
52
|
+
Sections whose Jaccard score falls in the range
|
|
53
|
+
[0.40, 0.85) -- the gray zone -- are retained as
|
|
54
|
+
candidates for Pass 2.
|
|
55
|
+
|
|
56
|
+
Sections with a Jaccard score below 0.40 are discarded
|
|
57
|
+
with no further processing.
|
|
58
|
+
|
|
59
|
+
### Pass 2: Cosine (when embeddings available)
|
|
60
|
+
|
|
61
|
+
Pass 2 runs only when an `EmbeddingProvider` other than
|
|
62
|
+
`NullEmbeddingProvider` is active and
|
|
63
|
+
`vectors.idx` is present and not stale.
|
|
64
|
+
|
|
65
|
+
For each gray-zone candidate from Pass 1, retrieve the
|
|
66
|
+
candidate's embedding from `vectors.idx`. If the
|
|
67
|
+
candidate's embedding is missing (e.g. the embedding
|
|
68
|
+
provider failed when that section was learned), skip
|
|
69
|
+
that candidate silently. Compute cosine similarity
|
|
70
|
+
between the incoming section's embedding and the
|
|
71
|
+
candidate's embedding.
|
|
72
|
+
|
|
73
|
+
The incoming section's embedding is computed by the
|
|
74
|
+
daemon as part of the same learn operation that triggers
|
|
75
|
+
the add (see the embedding lifecycle in
|
|
76
|
+
[Hybrid Search](hybrid-search.md)). If the embedding
|
|
77
|
+
computation for the incoming section itself failed, Pass
|
|
78
|
+
2 is skipped entirely for that call.
|
|
79
|
+
|
|
80
|
+
Candidates whose cosine similarity meets or exceeds 0.92
|
|
81
|
+
are flagged as semantic near-duplicates and appended to
|
|
82
|
+
`near_duplicates` with `method: "cosine"`.
|
|
83
|
+
|
|
84
|
+
### Efficiency Rationale
|
|
85
|
+
|
|
86
|
+
Jaccard operates on integer set intersections over the
|
|
87
|
+
inverted index -- it is cheap. Cosine operates on
|
|
88
|
+
floating-point vector dot products and requires reading
|
|
89
|
+
from `vectors.idx` -- it is more expensive. Restricting
|
|
90
|
+
cosine to the gray zone (Jaccard >= 0.40) means cosine
|
|
91
|
+
is invoked only for sections that already share moderate
|
|
92
|
+
lexical overlap with the candidate. Sections with very
|
|
93
|
+
low Jaccard scores are unlikely to be semantic
|
|
94
|
+
duplicates, so skipping them avoids unnecessary work
|
|
95
|
+
without meaningful loss in detection coverage.
|
|
96
|
+
|
|
97
|
+
## Detection Thresholds
|
|
98
|
+
|
|
99
|
+
| Method | Threshold | Meaning |
|
|
100
|
+
|------------------|-----------|------------------------------|
|
|
101
|
+
| Jaccard | >= 0.85 | Lexical near-duplicate |
|
|
102
|
+
| Jaccard pre-filter | >= 0.40 | Candidate for cosine pass |
|
|
103
|
+
| Cosine | >= 0.92 | Semantic near-duplicate |
|
|
104
|
+
|
|
105
|
+
Thresholds are not configurable in v1. They are
|
|
106
|
+
hardcoded constants. If operational experience shows
|
|
107
|
+
that the defaults produce excessive false positives or
|
|
108
|
+
false negatives, they can be exposed as `worker.toml`
|
|
109
|
+
configuration keys in a later version.
|
|
110
|
+
|
|
111
|
+
## Response Format
|
|
112
|
+
|
|
113
|
+
The `near_duplicates` array in `AddSectionResult` gains
|
|
114
|
+
a `method` field on every entry. The field identifies
|
|
115
|
+
which detection pass produced the entry.
|
|
116
|
+
|
|
117
|
+
```json
|
|
118
|
+
{
|
|
119
|
+
"kbid": "<26-char hash>",
|
|
120
|
+
"near_duplicates": [
|
|
121
|
+
{
|
|
122
|
+
"kbid": "abc123...",
|
|
123
|
+
"similarity": 0.89,
|
|
124
|
+
"method": "jaccard"
|
|
125
|
+
},
|
|
126
|
+
{
|
|
127
|
+
"kbid": "def456...",
|
|
128
|
+
"similarity": 0.95,
|
|
129
|
+
"method": "cosine"
|
|
130
|
+
}
|
|
131
|
+
]
|
|
132
|
+
}
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
The `method` field is always present on every
|
|
136
|
+
near-duplicate entry, including in responses from
|
|
137
|
+
deployments where only Jaccard runs. Callers can rely
|
|
138
|
+
on the field being present unconditionally.
|
|
139
|
+
|
|
140
|
+
Entries produced by Pass 1 always carry `method:
|
|
141
|
+
"jaccard"`. Entries produced by Pass 2 always carry
|
|
142
|
+
`method: "cosine"`. No entry can carry both methods --
|
|
143
|
+
a section that crosses the Jaccard threshold of 0.85
|
|
144
|
+
is already flagged in Pass 1 and is not re-evaluated
|
|
145
|
+
in Pass 2.
|
|
146
|
+
|
|
147
|
+
## Dependency on Hybrid Search
|
|
148
|
+
|
|
149
|
+
Semantic dedup reuses the infrastructure introduced by
|
|
150
|
+
[Hybrid Search](hybrid-search.md) without adding any
|
|
151
|
+
new infrastructure of its own.
|
|
152
|
+
|
|
153
|
+
- The `EmbeddingProvider` interface defined in Hybrid
|
|
154
|
+
Search provides the embedding computation at learn
|
|
155
|
+
time.
|
|
156
|
+
- `vectors.idx` provides the embedding lookup at
|
|
157
|
+
duplicate-detection time. No second embedding call is
|
|
158
|
+
made -- the stored vectors are read directly.
|
|
159
|
+
- When `NullEmbeddingProvider` is active (no `[embedding]`
|
|
160
|
+
section in `worker.toml`), Pass 2 never runs. The
|
|
161
|
+
`near_duplicates` array may still contain Jaccard
|
|
162
|
+
entries; it will never contain cosine entries.
|
|
163
|
+
- Stale vector index conditions (model name mismatch
|
|
164
|
+
between `vectors-meta.toml` and the configured
|
|
165
|
+
provider) are treated the same as a missing
|
|
166
|
+
`vectors.idx`: Pass 2 is skipped for that call.
|
|
167
|
+
|
|
168
|
+
## Contradiction Detection Enhancement
|
|
169
|
+
|
|
170
|
+
The existing `divergent_content` integrity check
|
|
171
|
+
identifies sections that share the same `source_key`
|
|
172
|
+
but have different content -- contradictory knowledge
|
|
173
|
+
from the same origin that supersession should have
|
|
174
|
+
resolved.
|
|
175
|
+
|
|
176
|
+
With embeddings available, integrity check can extend
|
|
177
|
+
this check to surface potential semantic contradictions
|
|
178
|
+
that lack a shared source key: sections with high cosine
|
|
179
|
+
similarity (>= 0.92) and conflicting content signals.
|
|
180
|
+
This is a secondary, advisory output in the integrity
|
|
181
|
+
check response -- it does not block any operation.
|
|
182
|
+
|
|
183
|
+
The enhancement runs only when the integrity check is
|
|
184
|
+
invoked and `vectors.idx` is present and not stale. It
|
|
185
|
+
adds a new `semantic_contradictions` array to the
|
|
186
|
+
integrity check response, alongside the existing
|
|
187
|
+
`divergent_content` error entries:
|
|
188
|
+
|
|
189
|
+
```json
|
|
190
|
+
{
|
|
191
|
+
"semantic_contradictions": [
|
|
192
|
+
{
|
|
193
|
+
"kbid_a": "abc123...",
|
|
194
|
+
"kbid_b": "def456...",
|
|
195
|
+
"cosine_similarity": 0.94
|
|
196
|
+
}
|
|
197
|
+
]
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
When no embedding provider is configured or
|
|
202
|
+
`vectors.idx` is absent, `semantic_contradictions` is
|
|
203
|
+
an empty array.
|
|
204
|
+
|
|
205
|
+
## Design Constraints
|
|
206
|
+
|
|
207
|
+
- Semantic dedup is additive. It never replaces the
|
|
208
|
+
Jaccard pass. Both passes coexist, and the Jaccard
|
|
209
|
+
pass always runs regardless of embedding configuration.
|
|
210
|
+
- The `method` field is always present on
|
|
211
|
+
`near_duplicates` entries, even in responses from
|
|
212
|
+
deployments where only Jaccard runs.
|
|
213
|
+
- Cosine similarity computation reuses existing
|
|
214
|
+
embedding vectors stored in `vectors.idx`. No
|
|
215
|
+
additional embedding API call is made during
|
|
216
|
+
duplicate detection.
|
|
217
|
+
- If Pass 2 is skipped (no provider, stale index, or
|
|
218
|
+
embedding failure for the incoming section), the
|
|
219
|
+
response omits cosine entries silently. No warning
|
|
220
|
+
field is added for a skipped cosine pass -- the
|
|
221
|
+
absence of cosine entries is not an error condition
|
|
222
|
+
and is expected in lexical-only deployments.
|
|
223
|
+
- Thresholds are hardcoded in v1. Exposure as
|
|
224
|
+
configuration keys is deferred to a later version
|
|
225
|
+
after operational validation.
|
|
226
|
+
- The `semantic_contradictions` enhancement to integrity
|
|
227
|
+
check is advisory. It does not change `ok` to `false`
|
|
228
|
+
on its own -- it provides supplementary information
|
|
229
|
+
for human review.
|
package/docs/goals/skills.md
CHANGED
|
@@ -719,12 +719,84 @@ dangling kbid. The agent is not deleted. See
|
|
|
719
719
|
[Agents](agents.md) for how dangling references are
|
|
720
720
|
handled at retrieval time.
|
|
721
721
|
|
|
722
|
+
## Default Skills
|
|
723
|
+
|
|
724
|
+
### Overview
|
|
725
|
+
|
|
726
|
+
When a new database is created via `kbdb db init`, a
|
|
727
|
+
set of default skills are automatically installed.
|
|
728
|
+
These skills provide baseline capabilities that
|
|
729
|
+
improve AI agent effectiveness. Default skills are
|
|
730
|
+
stored as regular skill sections (no special treatment
|
|
731
|
+
in the engine) and can be deleted by users if not
|
|
732
|
+
needed.
|
|
733
|
+
|
|
734
|
+
### `query-guidance` Default Skill
|
|
735
|
+
|
|
736
|
+
A built-in skill that teaches AI agents how to
|
|
737
|
+
effectively search the knowledge base. This addresses
|
|
738
|
+
the limitation that lexical search requires
|
|
739
|
+
disciplined query generation for good recall.
|
|
740
|
+
|
|
741
|
+
**Skill definition:**
|
|
742
|
+
|
|
743
|
+
```json
|
|
744
|
+
{
|
|
745
|
+
"name": "query-guidance",
|
|
746
|
+
"description": "How to effectively search this knowledge base. Read this before issuing search queries.",
|
|
747
|
+
"arguments": [],
|
|
748
|
+
"body": "This knowledge base uses lexical search (stemmed BM25F), not semantic/vector search. Follow these guidelines for effective retrieval:\n\n1. Issue MULTIPLE short keyword queries rather than one long natural-language question. Each query should target 2-3 specific terms.\n\n2. EXPAND SYNONYMS yourself. \"login\" will not match \"sign-in\" unless both terms appear in the content. Try alternative phrasings: search for \"login\", then \"sign-in\", then \"authentication\".\n\n3. USE BOOLEAN AND PHRASE SYNTAX:\n - Exact phrases: \"error handling\"\n - Boolean AND: auth AND token\n - Boolean OR: cache OR redis\n - Exclusion: auth -oauth\n\n4. LEAN ON HEADINGS. Headings carry 2x weight in ranking. Search for terms that are likely in section titles.\n\n5. STEMMING IS AUTOMATIC. \"connections\" matches \"connect\", \"running\" matches \"run\". You do not need to try inflected forms.\n\n6. USE RECALL AFTER SEARCH. After finding relevant sections with search, use recall with increasing depth (0-3) to progressively expand context: depth 0 = full content, depth 1 = parent documents + back-references, depth 2 = siblings + forward references, depth 3 = referenced sections.\n\n7. CODE IDENTIFIERS ARE SPLIT. camelCase and snake_case names are decomposed: getUserById becomes searchable as \"get\", \"user\", \"by\", \"id\".\n\n8. PAGINATION. Default limit is 20 results. Use offset for additional pages if total > limit."
|
|
749
|
+
}
|
|
750
|
+
```
|
|
751
|
+
|
|
752
|
+
### Installation Behaviour
|
|
753
|
+
|
|
754
|
+
Default skills are installed during `kbdb db init`.
|
|
755
|
+
The `runDbInit` function installs each default skill
|
|
756
|
+
via `client.addSection({ sectionType: 'skill', ... })`
|
|
757
|
+
after creating the database scaffold.
|
|
758
|
+
|
|
759
|
+
Default skills receive kbids via the same
|
|
760
|
+
content-addressing scheme as user-created skills.
|
|
761
|
+
Re-running `db init` on a fresh database produces
|
|
762
|
+
identical kbids for the same default skill content.
|
|
763
|
+
|
|
764
|
+
### `memory-protocol` Default Skill
|
|
765
|
+
|
|
766
|
+
A built-in skill that teaches AI agents *when* and
|
|
767
|
+
*what* to store in the knowledge base. This addresses
|
|
768
|
+
the risk of the brain filling with transient junk or
|
|
769
|
+
failing to capture durable facts.
|
|
770
|
+
|
|
771
|
+
**Skill definition:**
|
|
772
|
+
|
|
773
|
+
```json
|
|
774
|
+
{
|
|
775
|
+
"name": "memory-protocol",
|
|
776
|
+
"description": "Guidelines for what to store in this knowledge base and when. Read this before using the learn tool.",
|
|
777
|
+
"arguments": [],
|
|
778
|
+
"body": "Follow these guidelines to keep the knowledge base accurate and useful:\n\n1. STORE DURABLE FACTS: decisions, architecture choices, configuration values, corrections to prior knowledge, API contracts, and resolved incidents. These are the facts that matter weeks later.\n\n2. DO NOT STORE: transient conversation context, debugging output, ephemeral task state, raw log dumps, or information that changes every session. The brain should contain knowledge, not activity logs.\n\n3. SEARCH BEFORE STORING. Always search the knowledge base before adding new content. If a section already covers the topic, use learn with --replace to update it rather than creating a duplicate.\n\n4. USE --replace FOR CORRECTIONS. When a fact changes (e.g., rate limit updated from 100 to 200 req/min), re-learn the corrected content with --replace so the old version is superseded. Without --replace, both the old and new versions coexist with equal ranking.\n\n5. ATTACH TAGS for scoping. Use tags like project name, domain, or topic (e.g., --tags auth,api,v2) so queries can be filtered by scope later.\n\n6. PREFER RECENT RESULTS. When search returns multiple sections covering the same topic, prefer the one with the most recent created_at timestamp -- it is more likely to reflect the current state.\n\n7. USE RECALL AFTER SEARCH. After finding relevant sections, use recall with increasing depth (0-3) to expand context progressively before making decisions.\n\n8. INCLUDE HEADINGS. Always provide a descriptive title when storing content. Headings carry 2x ranking weight -- omitting them significantly reduces searchability."
|
|
779
|
+
}
|
|
780
|
+
```
|
|
781
|
+
|
|
782
|
+
### Future Default Skills
|
|
783
|
+
|
|
784
|
+
Additional default skills may be added in future
|
|
785
|
+
versions (e.g. `content-authoring-tips`,
|
|
786
|
+
`search-syntax-reference`). Each follows the same
|
|
787
|
+
installation pattern and is a regular skill section.
|
|
788
|
+
|
|
722
789
|
## Design Constraints
|
|
723
790
|
|
|
724
791
|
- **Skills are plain sections.** No separate entity
|
|
725
792
|
type, storage schema, or index is introduced. Skills
|
|
726
793
|
use `sectionType = 'skill'` within the existing
|
|
727
794
|
section infrastructure.
|
|
795
|
+
- **Default skills are regular sections.** They use
|
|
796
|
+
the same storage, indexing, and retrieval
|
|
797
|
+
infrastructure. There is no distinction between a
|
|
798
|
+
default skill and a user-created skill beyond the
|
|
799
|
+
installation source.
|
|
728
800
|
- **No skill versioning.** Updating a skill means
|
|
729
801
|
creating a new skill section (new kbid) and
|
|
730
802
|
optionally deleting the old one. The kbid changes
|