@modusensus/dsh-mneme 0.14.0-beta.1 → 0.14.0-beta.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +830 -830
- package/assets/icon.svg +3 -3
- package/bin/cli.mjs +964 -964
- package/bin/dsh-mneme-choro.mjs +262 -262
- package/bin/dsh-mneme-mcp.mjs +510 -510
- package/bin/dsh-mneme-serve.mjs +151 -151
- package/cordis.patch.yml +21 -21
- package/docs/CONFIGURATION.md +407 -404
- package/docs/CONTINUITY.md +139 -139
- package/docs/DAEMON.md +57 -57
- package/docs/DIRECTORIES.md +114 -114
- package/docs/DISTILL.md +397 -371
- package/docs/DSH-GO.md +97 -97
- package/docs/ENTITIES.md +247 -247
- package/docs/FORGET.md +77 -77
- package/docs/KB.md +110 -110
- package/docs/LOCAL_MODEL.md +222 -222
- package/docs/SEMANTIC.md +256 -256
- package/docs/SHARE.md +552 -552
- package/docs/SLEEP.md +202 -202
- package/docs/STORAGE.md +122 -122
- package/docs/WORKFLOWS.md +366 -366
- package/lib/api-standalone.js +1007 -1007
- package/lib/api.js +3600 -3600
- package/lib/attachment-text.js +122 -122
- package/lib/bootstrap-http.js +105 -105
- package/lib/bootstrap.js +215 -215
- package/lib/candidates.js +454 -454
- package/lib/capabilities.js +819 -819
- package/lib/choro-command.js +111 -111
- package/lib/client.js +14420 -14322
- package/lib/commands.js +95 -95
- package/lib/config-keys.js +144 -144
- package/lib/config.js +1361 -1344
- package/lib/content-hash.js +54 -54
- package/lib/continuity.js +258 -258
- package/lib/date-hint.js +32 -0
- package/lib/diary/auto.js +190 -190
- package/lib/diary/compose.js +440 -440
- package/lib/diary/dir.js +59 -59
- package/lib/diary/export.js +97 -97
- package/lib/diary/index.js +26 -26
- package/lib/diary/library.js +189 -189
- package/lib/diary/qq-inbox.js +142 -142
- package/lib/diary/rollover.js +61 -61
- package/lib/diary/session-provider.js +122 -122
- package/lib/document-index.js +94 -94
- package/lib/document.js +322 -322
- package/lib/dream/clustering.js +123 -123
- package/lib/dream/decisions.js +945 -945
- package/lib/dream/narratives.js +44 -44
- package/lib/dream/sleep.js +2027 -2027
- package/lib/dream/triage.js +139 -139
- package/lib/dream.js +1879 -1879
- package/lib/dsh-go/host.js +162 -162
- package/lib/dsh-go/panel.js +2091 -2091
- package/lib/dsh-go/text.js +342 -342
- package/lib/dsh-go/views/common.js +68 -68
- package/lib/dsh-go/views/entity.js +48 -48
- package/lib/dsh-go/views/memory.js +198 -198
- package/lib/dsh-go/views/root.js +30 -30
- package/lib/dsh-go/views/settings.js +398 -398
- package/lib/dsh-go/views/status.js +51 -51
- package/lib/embedding.js +342 -342
- package/lib/entities/extractor.js +285 -285
- package/lib/fence.js +24 -24
- package/lib/graph/anchoring.js +184 -184
- package/lib/guide.js +74 -74
- package/lib/heat.js +144 -144
- package/lib/hot-memory.js +119 -119
- package/lib/image-describe.js +158 -0
- package/lib/index.js +852 -852
- package/lib/inject.js +656 -651
- package/lib/kb/index.js +309 -309
- package/lib/kb/routes.js +219 -219
- package/lib/kb/tools.js +148 -148
- package/lib/lang.js +607 -595
- package/lib/local-embedder.js +358 -358
- package/lib/maintenance.js +162 -162
- package/lib/memory-source.js +9 -9
- package/lib/mirror.js +330 -330
- package/lib/organize.js +456 -456
- package/lib/peak-hours.js +172 -172
- package/lib/pipeline-activities.js +178 -178
- package/lib/pipeline-attribution.js +165 -165
- package/lib/pipeline-command.js +279 -279
- package/lib/pipeline-confirm.js +142 -142
- package/lib/pipeline-exec.js +174 -174
- package/lib/pipeline-nudge-auto.js +129 -129
- package/lib/pipeline-nudge.js +1623 -1623
- package/lib/pipeline-observations.js +181 -181
- package/lib/pipeline-observe.js +133 -133
- package/lib/pipeline-runner.js +361 -361
- package/lib/pipeline-runs.js +442 -442
- package/lib/pipeline.js +716 -716
- package/lib/placement.js +13 -13
- package/lib/pre-inject-gate.js +354 -354
- package/lib/profile-extract.js +164 -164
- package/lib/profile-facts.js +356 -356
- package/lib/provenance-http.js +251 -251
- package/lib/quality-filter.js +162 -162
- package/lib/recall-stats.js +190 -190
- package/lib/reclaim-http.js +150 -150
- package/lib/reranker.js +269 -269
- package/lib/runtime/adopt.js +176 -176
- package/lib/runtime/closure.js +172 -172
- package/lib/runtime/download.js +247 -247
- package/lib/runtime/layout.js +265 -265
- package/lib/runtime/loader.js +252 -252
- package/lib/runtime/model-fetch.js +361 -361
- package/lib/runtime/provision.js +237 -237
- package/lib/runtime/tarball.js +130 -130
- package/lib/runtime/verify.js +223 -223
- package/lib/safe-slice.js +18 -18
- package/lib/scope-state.js +277 -277
- package/lib/scope.js +187 -187
- package/lib/search/adaptive.js +60 -60
- package/lib/search/bm25.js +96 -96
- package/lib/semantic.js +572 -572
- package/lib/sensitive-scan.js +209 -209
- package/lib/serve.js +228 -228
- package/lib/service.js +3268 -3268
- package/lib/session-gate.js +91 -91
- package/lib/session-usage.js +350 -350
- package/lib/settings.js +827 -817
- package/lib/share-http.js +281 -281
- package/lib/share-import.js +234 -234
- package/lib/share.js +335 -335
- package/lib/store.js +5895 -5895
- package/lib/summarize.js +1388 -1343
- package/lib/tools.js +1370 -1356
- package/lib/user-settings.js +230 -230
- package/lib/vector-index.js +180 -180
- package/lib/version-check.js +122 -122
- package/lib/wiki/claims.js +267 -267
- package/lib/wiki/cli-hint.js +90 -90
- package/lib/wiki/inject.js +164 -164
- package/lib/wiki/partition.js +200 -200
- package/lib/wiki/plan.js +214 -214
- package/lib/wiki/queries.js +640 -640
- package/lib/wiki/render.js +261 -261
- package/lib/wiki/scan.js +371 -371
- package/lib/wiki/t0.js +500 -500
- package/lib/wiki/t1-jobs.js +232 -232
- package/lib/wiki/t1.js +609 -609
- package/lib/write-admission.js +404 -404
- package/locale/en.json +6 -6
- package/locale/zh.json +6 -6
- package/package.json +1 -1
- package/runtime-manifest.json +526 -526
- package/scripts/benchmark-embed.js +201 -201
- package/scripts/benchmark-recall.js +249 -249
- package/scripts/benchmark-rerank.js +166 -166
- package/scripts/build-runtime-manifest.mjs +224 -224
- package/scripts/changelog-prep.mjs +91 -91
- package/scripts/check-sync.js +44 -44
- package/scripts/e2e-dsh.js +220 -220
- package/scripts/mneme-runtime.mjs +277 -277
- package/scripts/publish-gate.mjs +197 -197
- package/scripts/release-docs.mjs +210 -210
- package/scripts/release-notes.mjs +164 -164
- package/scripts/stress-dsh.js +255 -255
- package/scripts/sync-lib.js +47 -47
- package/scripts/sync-test-badge.mjs +69 -69
- package/scripts/test-count-sync.mjs +48 -48
- package/src/api-standalone.js +1007 -1007
- package/src/api.js +3600 -3600
- package/src/attachment-text.js +122 -122
- package/src/bootstrap-http.js +105 -105
- package/src/bootstrap.js +215 -215
- package/src/candidates.js +454 -454
- package/src/capabilities.js +819 -819
- package/src/choro-command.js +111 -111
- package/src/commands.js +95 -95
- package/src/config-keys.js +144 -144
- package/src/config.js +1361 -1344
- package/src/content-hash.js +54 -54
- package/src/continuity.js +258 -258
- package/src/date-hint.js +32 -0
- package/src/diary/auto.js +190 -190
- package/src/diary/compose.js +440 -440
- package/src/diary/dir.js +59 -59
- package/src/diary/export.js +97 -97
- package/src/diary/index.js +26 -26
- package/src/diary/library.js +189 -189
- package/src/diary/qq-inbox.js +142 -142
- package/src/diary/rollover.js +61 -61
- package/src/diary/session-provider.js +122 -122
- package/src/document-index.js +94 -94
- package/src/document.js +322 -322
- package/src/dream/clustering.js +123 -123
- package/src/dream/decisions.js +945 -945
- package/src/dream/narratives.js +44 -44
- package/src/dream/sleep.js +2027 -2027
- package/src/dream/triage.js +139 -139
- package/src/dream.js +1879 -1879
- package/src/dsh-go/host.js +162 -162
- package/src/dsh-go/panel.js +2091 -2091
- package/src/dsh-go/text.js +342 -342
- package/src/dsh-go/views/common.js +68 -68
- package/src/dsh-go/views/entity.js +48 -48
- package/src/dsh-go/views/memory.js +198 -198
- package/src/dsh-go/views/root.js +30 -30
- package/src/dsh-go/views/settings.js +398 -398
- package/src/dsh-go/views/status.js +51 -51
- package/src/embedding.js +342 -342
- package/src/entities/extractor.js +285 -285
- package/src/fence.js +24 -24
- package/src/graph/anchoring.js +184 -184
- package/src/guide.js +74 -74
- package/src/heat.js +144 -144
- package/src/hot-memory.js +119 -119
- package/src/image-describe.js +158 -0
- package/src/index.js +852 -852
- package/src/inject.js +656 -651
- package/src/kb/index.js +309 -309
- package/src/kb/routes.js +219 -219
- package/src/kb/tools.js +148 -148
- package/src/lang.js +607 -595
- package/src/local-embedder.js +358 -358
- package/src/maintenance.js +162 -162
- package/src/memory-source.js +9 -9
- package/src/mirror.js +330 -330
- package/src/organize.js +456 -456
- package/src/peak-hours.js +172 -172
- package/src/pipeline-activities.js +178 -178
- package/src/pipeline-attribution.js +165 -165
- package/src/pipeline-command.js +279 -279
- package/src/pipeline-confirm.js +142 -142
- package/src/pipeline-exec.js +174 -174
- package/src/pipeline-nudge-auto.js +129 -129
- package/src/pipeline-nudge.js +1623 -1623
- package/src/pipeline-observations.js +181 -181
- package/src/pipeline-observe.js +133 -133
- package/src/pipeline-runner.js +361 -361
- package/src/pipeline-runs.js +442 -442
- package/src/pipeline.js +716 -716
- package/src/placement.js +13 -13
- package/src/pre-inject-gate.js +354 -354
- package/src/profile-extract.js +164 -164
- package/src/profile-facts.js +356 -356
- package/src/provenance-http.js +251 -251
- package/src/quality-filter.js +162 -162
- package/src/recall-stats.js +190 -190
- package/src/reclaim-http.js +150 -150
- package/src/reranker.js +269 -269
- package/src/runtime/adopt.js +176 -176
- package/src/runtime/closure.js +172 -172
- package/src/runtime/download.js +247 -247
- package/src/runtime/layout.js +265 -265
- package/src/runtime/loader.js +252 -252
- package/src/runtime/model-fetch.js +361 -361
- package/src/runtime/provision.js +237 -237
- package/src/runtime/tarball.js +130 -130
- package/src/runtime/verify.js +223 -223
- package/src/safe-slice.js +18 -18
- package/src/scope-state.js +277 -277
- package/src/scope.js +187 -187
- package/src/search/adaptive.js +60 -60
- package/src/search/bm25.js +96 -96
- package/src/semantic.js +572 -572
- package/src/sensitive-scan.js +209 -209
- package/src/serve.js +228 -228
- package/src/service.js +3268 -3268
- package/src/session-gate.js +91 -91
- package/src/session-usage.js +350 -350
- package/src/settings.js +827 -817
- package/src/share-http.js +281 -281
- package/src/share-import.js +234 -234
- package/src/share.js +335 -335
- package/src/store.js +5895 -5895
- package/src/summarize.js +1388 -1343
- package/src/tools.js +1370 -1356
- package/src/user-settings.js +230 -230
- package/src/vector-index.js +180 -180
- package/src/version-check.js +122 -122
- package/src/wiki/claims.js +267 -267
- package/src/wiki/cli-hint.js +90 -90
- package/src/wiki/inject.js +164 -164
- package/src/wiki/partition.js +200 -200
- package/src/wiki/plan.js +214 -214
- package/src/wiki/queries.js +640 -640
- package/src/wiki/render.js +261 -261
- package/src/wiki/scan.js +371 -371
- package/src/wiki/t0.js +500 -500
- package/src/wiki/t1-jobs.js +232 -232
- package/src/wiki/t1.js +609 -609
- package/src/write-admission.js +404 -404
- package/test/adaptive-inject-budget.test.js +85 -85
- package/test/api-hardening.test.js +173 -173
- package/test/api-nudge-route.test.js +259 -259
- package/test/api-routes-gaps.test.js +1092 -1092
- package/test/api.test.js +3007 -2998
- package/test/attachment-text.test.js +139 -139
- package/test/audit.test.js +448 -448
- package/test/baseline-persist.test.js +248 -248
- package/test/benchmark.test.js +94 -94
- package/test/bm25-corpus-cache.test.js +145 -145
- package/test/bootstrap.test.js +228 -228
- package/test/candidates.test.js +345 -345
- package/test/capabilities.test.js +577 -577
- package/test/capacity-warning.test.js +262 -262
- package/test/changelog-prep.test.js +125 -125
- package/test/choro-command.test.js +218 -218
- package/test/cli-enums.test.js +37 -37
- package/test/cli-flag-value.test.js +86 -86
- package/test/cli-receive.test.js +390 -390
- package/test/cli-share.test.js +437 -437
- package/test/client-share.test.js +225 -225
- package/test/client.test.js +4595 -4425
- package/test/clustering.test.js +100 -100
- package/test/commands.test.js +106 -106
- package/test/config-keys-routes.test.js +233 -233
- package/test/config.test.js +153 -153
- package/test/conflict-freeze.test.js +344 -344
- package/test/conflict-ledger-enforce.test.js +452 -452
- package/test/conflict-ledger.test.js +242 -242
- package/test/conflict-reviewed.test.js +79 -79
- package/test/content-hash.test.js +204 -204
- package/test/continuity.test.js +452 -452
- package/test/contrast.test.js +432 -432
- package/test/date-hint.test.js +156 -0
- package/test/dedupe-candidate-window.test.js +180 -180
- package/test/dependency-overrides.test.js +78 -78
- package/test/diary-library.test.js +187 -187
- package/test/diary.test.js +576 -576
- package/test/differentiate-feedback.test.js +248 -248
- package/test/distill-tx-embed-gap.test.js +110 -110
- package/test/docs-config-keys.test.js +65 -65
- package/test/document-dir.test.js +233 -233
- package/test/document.test.js +703 -703
- package/test/dream-archive-guardrail.test.js +116 -116
- package/test/dream-baseline-clamp.test.js +174 -174
- package/test/dream-candidate-hybrid.test.js +281 -281
- package/test/dream-coverage-degrade.test.js +86 -86
- package/test/dream-decision-evidence.test.js +314 -314
- package/test/dream-failure-backoff.test.js +173 -173
- package/test/dream-fence.test.js +207 -207
- package/test/dream-maxtokens-fallback.test.js +132 -132
- package/test/dream-merge-guardrail.test.js +219 -219
- package/test/dream-peak-hours.test.js +194 -194
- package/test/dream.test.js +1528 -1528
- package/test/dsh-go-host.test.js +134 -134
- package/test/dsh-go-panel-actions.test.js +854 -854
- package/test/dsh-go-panel.test.js +251 -251
- package/test/embed-ready-poll.test.js +94 -94
- package/test/embed-text-reasons.test.js +112 -112
- package/test/embedding-cache-f32.test.js +276 -276
- package/test/embedding-ready.test.js +226 -226
- package/test/entities.test.js +711 -711
- package/test/entity-recall.test.js +179 -179
- package/test/epistemic.test.js +325 -325
- package/test/failure-query.test.js +147 -147
- package/test/failure-retention.test.js +105 -105
- package/test/fnew-0112.test.js +311 -311
- package/test/fnew-03.test.js +422 -422
- package/test/forget-corrections.test.js +70 -70
- package/test/graph-anchoring.test.js +184 -184
- package/test/graph-api.test.js +333 -333
- package/test/graph-boosted-relations.test.js +189 -189
- package/test/graph-inject-budget.test.js +103 -103
- package/test/graph-multipath.test.js +269 -269
- package/test/graph-passive-confirm.test.js +91 -91
- package/test/graph-weight-evolution.test.js +104 -104
- package/test/heat.test.js +175 -175
- package/test/helpers/dream-mock.js +86 -86
- package/test/helpers/peer-worker.mjs +25 -25
- package/test/helpers/runtime-source.js +60 -60
- package/test/helpers/tar-builder.js +80 -80
- package/test/helpers/write-admission-samples.js +312 -312
- package/test/hot-memory.test.js +259 -259
- package/test/image-description.test.js +271 -0
- package/test/image-memory.test.js +170 -170
- package/test/inject-content-cap.test.js +91 -91
- package/test/inject-first-round-bm25.test.js +89 -89
- package/test/inject-heat.test.js +77 -77
- package/test/inject-parent-gate.test.js +134 -109
- package/test/inject-pin-explicit.test.js +300 -300
- package/test/inject-pin-vector-crowd.test.js +225 -225
- package/test/inject-pools.test.js +195 -195
- package/test/inject-recall.test.js +172 -172
- package/test/inject-rotation.test.js +161 -161
- package/test/inject.test.js +553 -553
- package/test/issue-617-step1-query-and-fusion.test.js +361 -361
- package/test/kb-api-wiring.test.js +128 -128
- package/test/kb-collections.test.js +272 -272
- package/test/kb-routes.test.js +379 -379
- package/test/lang.test.js +98 -98
- package/test/lastrunat-persist.test.js +101 -101
- package/test/lib-smoke.test.js +116 -116
- package/test/llm-audit.test.js +815 -815
- package/test/llm-message-source.test.js +207 -207
- package/test/local-embedder.test.js +292 -292
- package/test/maintenance.test.js +501 -501
- package/test/mcp-stdio.test.js +385 -385
- package/test/mint-source.test.js +311 -311
- package/test/mirror-dirty.test.js +424 -424
- package/test/mirror-edit-digest.test.js +187 -187
- package/test/mirror-generation.test.js +499 -499
- package/test/mirror-types.test.js +208 -208
- package/test/mirror.test.js +249 -249
- package/test/narrative-bars.test.js +162 -162
- package/test/organize-tool.test.js +232 -232
- package/test/organize.test.js +418 -418
- package/test/panel-chart-scale.test.js +90 -90
- package/test/panel-tokens.test.js +245 -245
- package/test/peak-hours-observability.test.js +70 -70
- package/test/peer-blockers.test.js +193 -193
- package/test/persona-addressing.test.js +83 -83
- package/test/persona-api.test.js +172 -172
- package/test/persona-by-session.test.js +568 -568
- package/test/persona-flag.test.js +36 -36
- package/test/persona-inject.test.js +145 -145
- package/test/persona-settings.test.js +198 -198
- package/test/pipeline-activities.test.js +155 -155
- package/test/pipeline-attribution.test.js +175 -175
- package/test/pipeline-command.test.js +243 -243
- package/test/pipeline-confirm.test.js +148 -148
- package/test/pipeline-exec.test.js +172 -172
- package/test/pipeline-nudge-auto.test.js +248 -248
- package/test/pipeline-nudge.test.js +1278 -1278
- package/test/pipeline-observations.test.js +157 -157
- package/test/pipeline-observe.test.js +127 -127
- package/test/pipeline-r1r2.test.js +278 -278
- package/test/pipeline-runner.test.js +468 -468
- package/test/pipeline-runs.test.js +767 -767
- package/test/pipeline.test.js +636 -636
- package/test/placement.test.js +36 -36
- package/test/policy-epoch.test.js +259 -259
- package/test/pre-inject-gate.test.js +444 -444
- package/test/profile-extract.test.js +269 -269
- package/test/profile-facts.test.js +492 -492
- package/test/prompt-stance.test.js +128 -128
- package/test/provenance-http.test.js +367 -367
- package/test/publish-gate.test.js +272 -272
- package/test/quality-backfill.test.js +290 -290
- package/test/quality-filter.test.js +295 -295
- package/test/readme-claims.test.js +113 -113
- package/test/reasoning-effort.test.js +752 -752
- package/test/recall-anti-update.test.js +82 -82
- package/test/recall-evals.test.js +253 -253
- package/test/recall-layer.test.js +326 -326
- package/test/recall-retention.test.js +128 -128
- package/test/recall-stats.test.js +274 -274
- package/test/receipt-chain.test.js +451 -451
- package/test/reclaim-http.test.js +209 -209
- package/test/reflection-update-gate.test.js +129 -129
- package/test/reflection.test.js +226 -226
- package/test/reindex-backfill.test.js +475 -475
- package/test/reindex-scheduler.test.js +369 -369
- package/test/release-docs.test.js +156 -156
- package/test/release-notes.test.js +265 -265
- package/test/release-workflow.test.js +93 -93
- package/test/reranker-single-logit.test.js +73 -73
- package/test/reranker.test.js +197 -197
- package/test/resident-status.test.js +83 -83
- package/test/route-policy.test.js +113 -113
- package/test/runtime-adopt.test.js +206 -206
- package/test/runtime-cli.test.js +89 -89
- package/test/runtime-closure.test.js +193 -193
- package/test/runtime-download.test.js +302 -302
- package/test/runtime-layout.test.js +163 -163
- package/test/runtime-loader.test.js +255 -255
- package/test/runtime-manifest.test.js +159 -159
- package/test/runtime-model-fetch.test.js +348 -348
- package/test/runtime-provision.test.js +101 -101
- package/test/runtime-tarball.test.js +81 -81
- package/test/runtime-verify.test.js +319 -319
- package/test/scope-candidates.test.js +491 -491
- package/test/scope-distill-stamp.test.js +134 -134
- package/test/scope-explicit.test.js +452 -452
- package/test/scope-inject-soft.test.js +106 -106
- package/test/scope-optional-service.test.js +110 -110
- package/test/scope-retrieval.test.js +197 -197
- package/test/scope-state.test.js +610 -610
- package/test/scope-storage.test.js +255 -255
- package/test/scope-strict.test.js +367 -367
- package/test/search-fusion.test.js +90 -90
- package/test/security-doc.test.js +188 -188
- package/test/semantic.test.js +124 -124
- package/test/sensitive-scan.test.js +251 -251
- package/test/serve-bin.test.js +121 -121
- package/test/serve.test.js +258 -258
- package/test/service-search.test.js +174 -174
- package/test/service.test.js +410 -410
- package/test/session-gate.test.js +264 -264
- package/test/session-usage.test.js +317 -317
- package/test/settings.test.js +273 -273
- package/test/share-foreign-rows.test.js +144 -144
- package/test/share-http.test.js +144 -144
- package/test/share-import.test.js +577 -577
- package/test/share-imported-guard.test.js +311 -311
- package/test/share-imported-identity.test.js +111 -111
- package/test/share-pack.test.js +511 -511
- package/test/share-receive-http.test.js +756 -756
- package/test/share-routes.test.js +337 -337
- package/test/share-store.test.js +133 -133
- package/test/share-validate.test.js +101 -101
- package/test/skip-invalid-restore.test.js +262 -262
- package/test/sleep-action-set.test.js +441 -441
- package/test/sleep-catchup.test.js +315 -315
- package/test/sleep-failure-backoff.test.js +156 -156
- package/test/sleep-heat.test.js +174 -174
- package/test/sleep-scheduler-rearm.test.js +89 -89
- package/test/sleep.test.js +1455 -1455
- package/test/standalone-api.test.js +1344 -1344
- package/test/store-vector-cache.test.js +98 -98
- package/test/store-write-path-columns.test.js +191 -191
- package/test/store.test.js +744 -744
- package/test/stress.test.js +209 -209
- package/test/summarize-occurred-at.test.js +127 -127
- package/test/summarize-parse-diagnostics.test.js +75 -75
- package/test/summarize-reasoning-effort.test.js +189 -189
- package/test/summarize.test.js +2251 -2251
- package/test/supersede-state-tracking.test.js +394 -394
- package/test/test-count-sync.test.js +71 -71
- package/test/tool-state-ignore.test.js +65 -65
- package/test/tools-dto-schema-matrix.test.js +398 -398
- package/test/tools-memory-get-schema.test.js +65 -65
- package/test/tools.test.js +585 -585
- package/test/updated-at-semantics.test.js +113 -113
- package/test/user-settings-preview.test.js +127 -127
- package/test/user-settings-segments.test.js +527 -527
- package/test/vector-index.test.js +334 -334
- package/test/version-check.test.js +152 -152
- package/test/webServer-optional.test.js +82 -82
- package/test/wiki-api.test.js +734 -734
- package/test/wiki-claims.test.js +194 -194
- package/test/wiki-cli-hint.test.js +118 -118
- package/test/wiki-inject.test.js +265 -265
- package/test/wiki-partition.test.js +112 -112
- package/test/wiki-plan.test.js +80 -80
- package/test/wiki-provenance.test.js +202 -202
- package/test/wiki-queries.test.js +398 -398
- package/test/wiki-scan.test.js +208 -208
- package/test/wiki-store.test.js +190 -190
- package/test/wiki-t0.test.js +456 -456
- package/test/wiki-t1-jobs.test.js +125 -125
- package/test/wiki-t1.test.js +460 -460
- package/test/wiki-tools.test.js +176 -176
- package/test/workflow-guards.test.js +155 -155
- package/test/write-admission.test.js +668 -668
package/docs/SEMANTIC.md
CHANGED
|
@@ -1,256 +1,256 @@
|
|
|
1
|
-
# dsh-mneme 语义增强架构 — 设计文档
|
|
2
|
-
|
|
3
|
-
- **日期**:2026-08-15
|
|
4
|
-
- **状态**:✅ 已实现(本地 Embedding / Rerank / 聚类随 v0.2+ 落地;本文档转为历史设计存档,当前能力以 [README](../README.md) 正文为准)
|
|
5
|
-
- **范围**:为 dsh-mneme 增加"完全离线的语义记忆引擎",覆盖本地 Embedding、Rerank 精排、autoDream 向量聚类三个阶段
|
|
6
|
-
- **相关文档**:[本地模型部署指南](LOCAL_MODEL.md) · [全键配置参考](CONFIGURATION.md)
|
|
7
|
-
|
|
8
|
-
## 1. 背景与目标
|
|
9
|
-
|
|
10
|
-
dsh-mneme v0.1 已具备**可选的向量搜索**:通过 `/api/dsh-mneme/vector-config` 配置 OpenAI 兼容 `/embeddings` 端点,命中"字面不同但语义相近"的记忆;未配置或失败时降级为 LIKE 子串搜索。
|
|
11
|
-
|
|
12
|
-
v0.2 的目标是把这条**可选、依赖外部 API** 的向量链路升级为**默认可用、完全离线**的语义记忆引擎:
|
|
13
|
-
|
|
14
|
-
1. **本地 Embedding**:ONNX(`Xenova/bge-small-zh-v1.5`)或 Ollama,记忆向量不再依赖外网 API
|
|
15
|
-
2. **Rerank 精排**:向量初筛后交叉编码精排,把 Top-K 准确率再抬一档
|
|
16
|
-
3. **autoDream 语义增强**:对记忆向量做聚类,让自动巩固(去重/合并/冲突裁决)从"主题相近"升级为"语义相近"
|
|
17
|
-
|
|
18
|
-
## 2. 总体架构
|
|
19
|
-
|
|
20
|
-
```
|
|
21
|
-
┌──────────────────────────────────────────────────────────────┐
|
|
22
|
-
│ 语义引擎(v0.2,完全离线可选) │
|
|
23
|
-
├──────────────────────────────────────────────────────────────┤
|
|
24
|
-
│ ① Embedder 层(三选一,统一接口,可自动降级) │
|
|
25
|
-
│ LocalEmbedder(ONNX) │ OllamaEmbedder │ OpenAIEmbedder │
|
|
26
|
-
│ init() / embed(texts) / embedSingle() / dimension / hash │
|
|
27
|
-
├──────────────────────────────────────────────────────────────┤
|
|
28
|
-
│ ② VectorIndex 层 │
|
|
29
|
-
│ SQLite memories.embedding 列(JSON 向量) │
|
|
30
|
-
│ + 余弦相似度检索(score 0..1)+ 缺失向量增量补建 reindex │
|
|
31
|
-
├──────────────────────────────────────────────────────────────┤
|
|
32
|
-
│ ③ Rerank 层(可选) │
|
|
33
|
-
│ LocalReranker(Xenova/bge-reranker-base,交叉编码) │
|
|
34
|
-
│ 召回候选 → 精排 → Top-K │
|
|
35
|
-
├──────────────────────────────────────────────────────────────┤
|
|
36
|
-
│ ④ autoDream 聚类(可选) │
|
|
37
|
-
│ clusterMemories(向量 K-Means) / findPotentialConflicts │
|
|
38
|
-
│ 语义分组 + 疑似矛盾检测 → 巩固决策更精准 │
|
|
39
|
-
└──────────────────────────────────────────────────────────────┘
|
|
40
|
-
```
|
|
41
|
-
|
|
42
|
-
- **① Embedder**:`src/local-embedder.js`,三后端统一接口,`createEmbedderByProvider(provider, opts)` 按名创建
|
|
43
|
-
- **② VectorIndex**:复用 v0.1 的 `memories.embedding` 列与 `store.js` 的余弦检索,无新存储层
|
|
44
|
-
- **③ Rerank**:`src/reranker.js`(Phase 2),可选开启
|
|
45
|
-
- **④ 聚类**:`src/dream/clustering.js`(Phase 3),autoDream 调度器集成
|
|
46
|
-
|
|
47
|
-
### 目录结构(新增/变化部分)
|
|
48
|
-
|
|
49
|
-
```
|
|
50
|
-
src/
|
|
51
|
-
├── local-embedder.js # [新增] Local/Ollama/OpenAI 三 Embedder + createEmbedderByProvider
|
|
52
|
-
├── reranker.js # [新增·Phase2] LocalReranker 交叉编码精排
|
|
53
|
-
├── dream/
|
|
54
|
-
│ └── clustering.js # [新增·Phase3] clusterMemories / findPotentialConflicts / kMeans
|
|
55
|
-
└── embedding.js # [保留] v0.1 OpenAI 兼容客户端(作为降级路径之一复用)
|
|
56
|
-
scripts/
|
|
57
|
-
├── benchmark-embed.js # [新增] Embedding 吞吐量基准
|
|
58
|
-
└── benchmark-rerank.js # [新增] Rerank 延迟基准
|
|
59
|
-
docs/
|
|
60
|
-
├── SEMANTIC.md # 本文档
|
|
61
|
-
├── LOCAL_MODEL.md # 本地模型部署指南
|
|
62
|
-
└── CONFIGURATION.md # 全键配置参考
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
## 3. 搜索流水线
|
|
66
|
-
|
|
67
|
-
`memory_search` 在 v0.2 的完整链路(对应 `src/tools.js` 的 `mode: auto | keyword | vector | hybrid`):
|
|
68
|
-
|
|
69
|
-
```
|
|
70
|
-
query
|
|
71
|
-
│
|
|
72
|
-
┌───────────┴───────────┐
|
|
73
|
-
▼ ▼
|
|
74
|
-
keyword 召回 vector 召回
|
|
75
|
-
(LIKE 子串 title/ (Embedder.embed(query)
|
|
76
|
-
content/tags) → 余弦相似度 top-N
|
|
77
|
-
→ 缺失向量自动补建)
|
|
78
|
-
│ │
|
|
79
|
-
└───── 混合融合 ─────────┘
|
|
80
|
-
(auto: 关键词命中优先 + 向量补足;分数归一化合并)
|
|
81
|
-
│
|
|
82
|
-
▼
|
|
83
|
-
[Rerank 开启?]
|
|
84
|
-
├─ 是 → LocalReranker.rerank(query, candidates, topK)
|
|
85
|
-
│ 交叉编码精排,取 topK
|
|
86
|
-
└─ 否 → 按 importance/score 直接截取 topK
|
|
87
|
-
│
|
|
88
|
-
▼
|
|
89
|
-
memory_search 结果
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
- **召回阶段(候选)**:混合召回,宁多勿漏,候选集通常取向量 top-N(默认远超 Top-K,供精排筛选)
|
|
93
|
-
- **精排阶段(Rerank)**:对候选逐条打分(query 与候选的交互编码),取 Top-K
|
|
94
|
-
- **结果阶段**:返回条目 + `score`(召回分/精排分)+ `source` 时间戳,与 v0.1 返回结构保持一致
|
|
95
|
-
|
|
96
|
-
## 4. 模块接口
|
|
97
|
-
|
|
98
|
-
### 4.1 Embedder 统一接口(`src/local-embedder.js`)
|
|
99
|
-
|
|
100
|
-
三个后端共享同一接口,方法失败**抛错**,由上层编排降级链(与 v0.1 `embedding.js` 的"失败返回 null"不同)。
|
|
101
|
-
|
|
102
|
-
| 成员 | 说明 |
|
|
103
|
-
|------|------|
|
|
104
|
-
| `init()` | 加载模型 / 探测服务,失败抛错 |
|
|
105
|
-
| `embed(texts: string[])` | 批量嵌入,返回 `number[][]`(本地模型 mean pooling + L2 归一化) |
|
|
106
|
-
| `embedSingle(text)` | 单条嵌入 → `number[]` |
|
|
107
|
-
| `dimension` | 向量维度(getter) |
|
|
108
|
-
| `modelHash` | 模型指纹(`<model>#<hash>`),用于向量索引一致性校验,模型换名/换维度时提示重建索引 |
|
|
109
|
-
| `dispose()` | 释放模型资源 |
|
|
110
|
-
|
|
111
|
-
**LocalEmbedder(ONNX)**
|
|
112
|
-
|
|
113
|
-
```js
|
|
114
|
-
new LocalEmbedder({
|
|
115
|
-
model: "Xenova/bge-small-zh-v1.5", // 默认;中文优化,全离线
|
|
116
|
-
dimension: 512, // 默认与模型匹配
|
|
117
|
-
device: "cpu", // cpu | wasm | gpu(onnxruntime 后端)
|
|
118
|
-
batchSize: 8, // 分批嵌入,控制峰值内存
|
|
119
|
-
cacheDir: "", // 自定义模型缓存目录,空则用 HF 默认缓存
|
|
120
|
-
useDtype: "q8", // 量化精度 q8/fp32/fp16
|
|
121
|
-
logger: null
|
|
122
|
-
})
|
|
123
|
-
```
|
|
124
|
-
|
|
125
|
-
**OllamaEmbedder**
|
|
126
|
-
|
|
127
|
-
```js
|
|
128
|
-
new OllamaEmbedder({
|
|
129
|
-
baseUrl: "http://localhost:11434", // 默认
|
|
130
|
-
model: "nomic-embed-text", // 默认;维度从首次响应自动推断
|
|
131
|
-
logger: null
|
|
132
|
-
})
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
**OpenAIEmbedder(兼容 v0.1 行为)**
|
|
136
|
-
|
|
137
|
-
```js
|
|
138
|
-
new OpenAIEmbedder({
|
|
139
|
-
baseUrl: "", // 例如 https://api.openai.com/v1,也支持 SiliconFlow/智谱/Ollama 代理
|
|
140
|
-
apiKey: "",
|
|
141
|
-
model: "",
|
|
142
|
-
timeoutMs: 15000
|
|
143
|
-
})
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
**工厂**
|
|
147
|
-
|
|
148
|
-
```js
|
|
149
|
-
import { createEmbedderByProvider } from "../src/local-embedder.js";
|
|
150
|
-
const embedder = createEmbedderByProvider("local", { device: "cpu" }); // local | ollama | openai
|
|
151
|
-
await embedder.init();
|
|
152
|
-
const vecs = await embedder.embed(["你好", "记忆库"]);
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
### 4.2 Reranker(`src/reranker.js`,Phase 2)
|
|
156
|
-
|
|
157
|
-
| 成员 | 说明 |
|
|
158
|
-
|------|------|
|
|
159
|
-
| `constructor({ model, device, batchSize, cacheDir, logger })` | `model` 默认 `Xenova/bge-reranker-base` |
|
|
160
|
-
| `init()` | 加载交叉编码模型 |
|
|
161
|
-
| `rerank(query, candidates, topK?)` | 对候选逐条打分,返回按相关性降序的候选(附 `score`),`topK` 缺省返回全量排序 |
|
|
162
|
-
|
|
163
|
-
示例:
|
|
164
|
-
|
|
165
|
-
```js
|
|
166
|
-
import { LocalReranker } from "../src/reranker.js";
|
|
167
|
-
const reranker = new LocalReranker({ model: "Xenova/bge-reranker-base" });
|
|
168
|
-
await reranker.init();
|
|
169
|
-
const top = await reranker.rerank("怎么部署本地模型", candidates, 5);
|
|
170
|
-
```
|
|
171
|
-
|
|
172
|
-
### 4.3 autoDream 聚类(`src/dream/clustering.js`,Phase 3,已实现)
|
|
173
|
-
|
|
174
|
-
| 成员 | 说明 |
|
|
175
|
-
|------|------|
|
|
176
|
-
| `cosineSimilarity(a, b)` | 余弦相似度,零向量/长度不齐返回 0,数值稳定 |
|
|
177
|
-
| `kMeans(vectors, k, opts)` | K-Means(k-means++ 播种 + 空簇修复),返回每簇索引数组 |
|
|
178
|
-
| `clusterMemories(memories, vectors, k)` | 按向量把记忆分组,返回簇(保留原对象引用) |
|
|
179
|
-
| `findPotentialConflicts(memories, vectors, threshold = 0.85)` | 同类型、高相似(可能矛盾)的记忆对 |
|
|
180
|
-
|
|
181
|
-
autoDream 调度器在触发巩固时:对候选记忆做 `clusterMemories` 分组 → 每组送 LLM 决策(merge/archive/conflict),并用 `findPotentialConflicts` 预先标出疑似矛盾对,减少漏判。
|
|
182
|
-
|
|
183
|
-
## 5. 降级策略
|
|
184
|
-
|
|
185
|
-
```
|
|
186
|
-
首选 回退 1 回退 2 兜底
|
|
187
|
-
LocalEmbedder(ONNX) ──失败──► OllamaEmbedder ──失败──► OpenAIEmbedder ──失败/未配置──► 关键词 LIKE
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
- Embedder 方法**抛错**,由上层 `createEmbedderByProvider` 编排的降级链逐级捕获切换;全链路失败则回退到 v0.1 的 LIKE 子串关键词搜索(本库无 FTS5,关键词召回即 LIKE 子串扫描)
|
|
191
|
-
- **原则**:语义链路任何一环失败都**静默降级、不阻断记忆读写**——写入照常落库,只是当次不补向量
|
|
192
|
-
- **降级是每请求动态的**:同一会话内 ONNX 偶发失败,下一次请求自动重试首选路径,不固化到降级档
|
|
193
|
-
- **索引一致性**:Embedder 的 `modelHash` 变化(换模型/换维度)时,旧向量与新向量不可混算余弦,需触发「重建索引」(复用 v0.1 的 `reindexMissing` 补建缺失向量)
|
|
194
|
-
|
|
195
|
-
## 6. 模型清单
|
|
196
|
-
|
|
197
|
-
| 用途 | 模型 | 后端 | 维度 | 说明 |
|
|
198
|
-
|------|------|------|------|------|
|
|
199
|
-
| Embedding | `Xenova/bge-small-zh-v1.5` | ONNX(默认) | 512 | 中文优化,全离线,体积小 |
|
|
200
|
-
| Embedding | `bge-m3` | Ollama | 1024 | 多语言(中/英/日等),质量高 |
|
|
201
|
-
| Embedding | `nomic-embed-text` | Ollama | 768 | 英文为主,通用 |
|
|
202
|
-
| Embedding | `text-embedding-3-small` | OpenAI 兼容 | 1536 | v0.1 云端默认,升级后仍可用 |
|
|
203
|
-
| Rerank | `Xenova/bge-reranker-base` | ONNX(Phase 2) | — | 交叉编码,首候选精排 |
|
|
204
|
-
| Rerank | `bge-reranker-v2-m3` | 云/本地 | — | 更强精排,多语言 |
|
|
205
|
-
|
|
206
|
-
> 维度随模型而定:换用不同维度的模型后,已存向量与模型指纹不匹配,需重建索引。
|
|
207
|
-
|
|
208
|
-
## 7. 配置新增项
|
|
209
|
-
|
|
210
|
-
见 [CONFIGURATION.md](CONFIGURATION.md) 的检索与运行时键表。核心:`embedProvider`(`local` / `ollama` / `openai`,默认 `openai` 保持 v0.1 行为)、`localEmbedModel`、`localEmbedDimension`、`localEmbedDevice`、`localEmbedBatchSize`、`ollamaBaseUrl`、`ollamaModel`、`embedModelCacheDir`、`rerankEnabled`、`rerankModel`。
|
|
211
|
-
|
|
212
|
-
## 8. 测试与基准
|
|
213
|
-
|
|
214
|
-
- 单元:`src/local-embedder.js` 三后端接口一致性、`clustering.js` 聚类正确性(node:test)
|
|
215
|
-
- 降级链:mock 各后端抛错,断言逐级回退到 LIKE
|
|
216
|
-
- 基准:`scripts/benchmark-embed.js`(嵌入延迟/吞吐)、`scripts/benchmark-rerank.js`(Rerank 延迟 vs 候选规模)
|
|
217
|
-
|
|
218
|
-
## 9. 反思更新(v0.2.1 `update` 决策 + 失败追踪)
|
|
219
|
-
|
|
220
|
-
### 9.1 `update` 决策类型
|
|
221
|
-
|
|
222
|
-
autoDream 整理时,LLM 可输出第 5 种决策 `update`,直接修正单条记忆的过时/错误内容:
|
|
223
|
-
|
|
224
|
-
- **约束**:`ids` 只能含一个 id;必须产生实际字段变化;不能更新 `summary`;新建 < 24h 的记忆不可 update
|
|
225
|
-
- **频率限制**:每次 autoDream 最多 `reflectionUpdateMaxPerRun`(默认 2)个 update,超限整单拒绝
|
|
226
|
-
- **幂等**:字段已与目标一致时跳过(可重放无副作用)
|
|
227
|
-
- **审计**:`dream_runs` 记录 update 的 `_before` 快照,可回溯变更前内容
|
|
228
|
-
- **向量同步**:update 后删除旧向量、按新内容重嵌入,保持索引一致
|
|
229
|
-
|
|
230
|
-
### 9.2 失败追踪(failure_memories)
|
|
231
|
-
|
|
232
|
-
`failure_memories` 表记录记忆纠正/失败事件,为后续反思进化积累数据:
|
|
233
|
-
|
|
234
|
-
```sql
|
|
235
|
-
CREATE TABLE failure_memories (
|
|
236
|
-
id TEXT PRIMARY KEY,
|
|
237
|
-
query TEXT, -- 用户原始查询/意图(可选)
|
|
238
|
-
expected TEXT, -- 期望结果
|
|
239
|
-
actual TEXT, -- 实际结果
|
|
240
|
-
failure_type TEXT, -- outdated / miss / wrong / user_correction
|
|
241
|
-
memory_id TEXT, -- 关联的记忆 id
|
|
242
|
-
created_at TEXT NOT NULL
|
|
243
|
-
);
|
|
244
|
-
```
|
|
245
|
-
|
|
246
|
-
- **触发**:用户调用 `memory_update` 且内容变化时,若 `reflectionFailureTracking` 开启则记录 `user_correction` 行(actual=旧值、expected=新值)
|
|
247
|
-
- **查询**:`store.listFailures`(过滤/分页)、`store.getFailureStats`(按类型计数)
|
|
248
|
-
|
|
249
|
-
### 9.3 配置项
|
|
250
|
-
|
|
251
|
-
```javascript
|
|
252
|
-
reflectionUpdateEnabled: true, // update 决策总开关
|
|
253
|
-
reflectionFailureTracking: true, // 失败追踪总开关
|
|
254
|
-
reflectionUpdateMaxPerRun: 2, // 每次整理最多 update 数
|
|
255
|
-
reflectionUpdateMinAgeHours: 24 // 新建记忆保护期(小时)
|
|
256
|
-
```
|
|
1
|
+
# dsh-mneme 语义增强架构 — 设计文档
|
|
2
|
+
|
|
3
|
+
- **日期**:2026-08-15
|
|
4
|
+
- **状态**:✅ 已实现(本地 Embedding / Rerank / 聚类随 v0.2+ 落地;本文档转为历史设计存档,当前能力以 [README](../README.md) 正文为准)
|
|
5
|
+
- **范围**:为 dsh-mneme 增加"完全离线的语义记忆引擎",覆盖本地 Embedding、Rerank 精排、autoDream 向量聚类三个阶段
|
|
6
|
+
- **相关文档**:[本地模型部署指南](LOCAL_MODEL.md) · [全键配置参考](CONFIGURATION.md)
|
|
7
|
+
|
|
8
|
+
## 1. 背景与目标
|
|
9
|
+
|
|
10
|
+
dsh-mneme v0.1 已具备**可选的向量搜索**:通过 `/api/dsh-mneme/vector-config` 配置 OpenAI 兼容 `/embeddings` 端点,命中"字面不同但语义相近"的记忆;未配置或失败时降级为 LIKE 子串搜索。
|
|
11
|
+
|
|
12
|
+
v0.2 的目标是把这条**可选、依赖外部 API** 的向量链路升级为**默认可用、完全离线**的语义记忆引擎:
|
|
13
|
+
|
|
14
|
+
1. **本地 Embedding**:ONNX(`Xenova/bge-small-zh-v1.5`)或 Ollama,记忆向量不再依赖外网 API
|
|
15
|
+
2. **Rerank 精排**:向量初筛后交叉编码精排,把 Top-K 准确率再抬一档
|
|
16
|
+
3. **autoDream 语义增强**:对记忆向量做聚类,让自动巩固(去重/合并/冲突裁决)从"主题相近"升级为"语义相近"
|
|
17
|
+
|
|
18
|
+
## 2. 总体架构
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
┌──────────────────────────────────────────────────────────────┐
|
|
22
|
+
│ 语义引擎(v0.2,完全离线可选) │
|
|
23
|
+
├──────────────────────────────────────────────────────────────┤
|
|
24
|
+
│ ① Embedder 层(三选一,统一接口,可自动降级) │
|
|
25
|
+
│ LocalEmbedder(ONNX) │ OllamaEmbedder │ OpenAIEmbedder │
|
|
26
|
+
│ init() / embed(texts) / embedSingle() / dimension / hash │
|
|
27
|
+
├──────────────────────────────────────────────────────────────┤
|
|
28
|
+
│ ② VectorIndex 层 │
|
|
29
|
+
│ SQLite memories.embedding 列(JSON 向量) │
|
|
30
|
+
│ + 余弦相似度检索(score 0..1)+ 缺失向量增量补建 reindex │
|
|
31
|
+
├──────────────────────────────────────────────────────────────┤
|
|
32
|
+
│ ③ Rerank 层(可选) │
|
|
33
|
+
│ LocalReranker(Xenova/bge-reranker-base,交叉编码) │
|
|
34
|
+
│ 召回候选 → 精排 → Top-K │
|
|
35
|
+
├──────────────────────────────────────────────────────────────┤
|
|
36
|
+
│ ④ autoDream 聚类(可选) │
|
|
37
|
+
│ clusterMemories(向量 K-Means) / findPotentialConflicts │
|
|
38
|
+
│ 语义分组 + 疑似矛盾检测 → 巩固决策更精准 │
|
|
39
|
+
└──────────────────────────────────────────────────────────────┘
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
- **① Embedder**:`src/local-embedder.js`,三后端统一接口,`createEmbedderByProvider(provider, opts)` 按名创建
|
|
43
|
+
- **② VectorIndex**:复用 v0.1 的 `memories.embedding` 列与 `store.js` 的余弦检索,无新存储层
|
|
44
|
+
- **③ Rerank**:`src/reranker.js`(Phase 2),可选开启
|
|
45
|
+
- **④ 聚类**:`src/dream/clustering.js`(Phase 3),autoDream 调度器集成
|
|
46
|
+
|
|
47
|
+
### 目录结构(新增/变化部分)
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
src/
|
|
51
|
+
├── local-embedder.js # [新增] Local/Ollama/OpenAI 三 Embedder + createEmbedderByProvider
|
|
52
|
+
├── reranker.js # [新增·Phase2] LocalReranker 交叉编码精排
|
|
53
|
+
├── dream/
|
|
54
|
+
│ └── clustering.js # [新增·Phase3] clusterMemories / findPotentialConflicts / kMeans
|
|
55
|
+
└── embedding.js # [保留] v0.1 OpenAI 兼容客户端(作为降级路径之一复用)
|
|
56
|
+
scripts/
|
|
57
|
+
├── benchmark-embed.js # [新增] Embedding 吞吐量基准
|
|
58
|
+
└── benchmark-rerank.js # [新增] Rerank 延迟基准
|
|
59
|
+
docs/
|
|
60
|
+
├── SEMANTIC.md # 本文档
|
|
61
|
+
├── LOCAL_MODEL.md # 本地模型部署指南
|
|
62
|
+
└── CONFIGURATION.md # 全键配置参考
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## 3. 搜索流水线
|
|
66
|
+
|
|
67
|
+
`memory_search` 在 v0.2 的完整链路(对应 `src/tools.js` 的 `mode: auto | keyword | vector | hybrid`):
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
query
|
|
71
|
+
│
|
|
72
|
+
┌───────────┴───────────┐
|
|
73
|
+
▼ ▼
|
|
74
|
+
keyword 召回 vector 召回
|
|
75
|
+
(LIKE 子串 title/ (Embedder.embed(query)
|
|
76
|
+
content/tags) → 余弦相似度 top-N
|
|
77
|
+
→ 缺失向量自动补建)
|
|
78
|
+
│ │
|
|
79
|
+
└───── 混合融合 ─────────┘
|
|
80
|
+
(auto: 关键词命中优先 + 向量补足;分数归一化合并)
|
|
81
|
+
│
|
|
82
|
+
▼
|
|
83
|
+
[Rerank 开启?]
|
|
84
|
+
├─ 是 → LocalReranker.rerank(query, candidates, topK)
|
|
85
|
+
│ 交叉编码精排,取 topK
|
|
86
|
+
└─ 否 → 按 importance/score 直接截取 topK
|
|
87
|
+
│
|
|
88
|
+
▼
|
|
89
|
+
memory_search 结果
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
- **召回阶段(候选)**:混合召回,宁多勿漏,候选集通常取向量 top-N(默认远超 Top-K,供精排筛选)
|
|
93
|
+
- **精排阶段(Rerank)**:对候选逐条打分(query 与候选的交互编码),取 Top-K
|
|
94
|
+
- **结果阶段**:返回条目 + `score`(召回分/精排分)+ `source` 时间戳,与 v0.1 返回结构保持一致
|
|
95
|
+
|
|
96
|
+
## 4. 模块接口
|
|
97
|
+
|
|
98
|
+
### 4.1 Embedder 统一接口(`src/local-embedder.js`)
|
|
99
|
+
|
|
100
|
+
三个后端共享同一接口,方法失败**抛错**,由上层编排降级链(与 v0.1 `embedding.js` 的"失败返回 null"不同)。
|
|
101
|
+
|
|
102
|
+
| 成员 | 说明 |
|
|
103
|
+
|------|------|
|
|
104
|
+
| `init()` | 加载模型 / 探测服务,失败抛错 |
|
|
105
|
+
| `embed(texts: string[])` | 批量嵌入,返回 `number[][]`(本地模型 mean pooling + L2 归一化) |
|
|
106
|
+
| `embedSingle(text)` | 单条嵌入 → `number[]` |
|
|
107
|
+
| `dimension` | 向量维度(getter) |
|
|
108
|
+
| `modelHash` | 模型指纹(`<model>#<hash>`),用于向量索引一致性校验,模型换名/换维度时提示重建索引 |
|
|
109
|
+
| `dispose()` | 释放模型资源 |
|
|
110
|
+
|
|
111
|
+
**LocalEmbedder(ONNX)**
|
|
112
|
+
|
|
113
|
+
```js
|
|
114
|
+
new LocalEmbedder({
|
|
115
|
+
model: "Xenova/bge-small-zh-v1.5", // 默认;中文优化,全离线
|
|
116
|
+
dimension: 512, // 默认与模型匹配
|
|
117
|
+
device: "cpu", // cpu | wasm | gpu(onnxruntime 后端)
|
|
118
|
+
batchSize: 8, // 分批嵌入,控制峰值内存
|
|
119
|
+
cacheDir: "", // 自定义模型缓存目录,空则用 HF 默认缓存
|
|
120
|
+
useDtype: "q8", // 量化精度 q8/fp32/fp16
|
|
121
|
+
logger: null
|
|
122
|
+
})
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
**OllamaEmbedder**
|
|
126
|
+
|
|
127
|
+
```js
|
|
128
|
+
new OllamaEmbedder({
|
|
129
|
+
baseUrl: "http://localhost:11434", // 默认
|
|
130
|
+
model: "nomic-embed-text", // 默认;维度从首次响应自动推断
|
|
131
|
+
logger: null
|
|
132
|
+
})
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
**OpenAIEmbedder(兼容 v0.1 行为)**
|
|
136
|
+
|
|
137
|
+
```js
|
|
138
|
+
new OpenAIEmbedder({
|
|
139
|
+
baseUrl: "", // 例如 https://api.openai.com/v1,也支持 SiliconFlow/智谱/Ollama 代理
|
|
140
|
+
apiKey: "",
|
|
141
|
+
model: "",
|
|
142
|
+
timeoutMs: 15000
|
|
143
|
+
})
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
**工厂**
|
|
147
|
+
|
|
148
|
+
```js
|
|
149
|
+
import { createEmbedderByProvider } from "../src/local-embedder.js";
|
|
150
|
+
const embedder = createEmbedderByProvider("local", { device: "cpu" }); // local | ollama | openai
|
|
151
|
+
await embedder.init();
|
|
152
|
+
const vecs = await embedder.embed(["你好", "记忆库"]);
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
### 4.2 Reranker(`src/reranker.js`,Phase 2)
|
|
156
|
+
|
|
157
|
+
| 成员 | 说明 |
|
|
158
|
+
|------|------|
|
|
159
|
+
| `constructor({ model, device, batchSize, cacheDir, logger })` | `model` 默认 `Xenova/bge-reranker-base` |
|
|
160
|
+
| `init()` | 加载交叉编码模型 |
|
|
161
|
+
| `rerank(query, candidates, topK?)` | 对候选逐条打分,返回按相关性降序的候选(附 `score`),`topK` 缺省返回全量排序 |
|
|
162
|
+
|
|
163
|
+
示例:
|
|
164
|
+
|
|
165
|
+
```js
|
|
166
|
+
import { LocalReranker } from "../src/reranker.js";
|
|
167
|
+
const reranker = new LocalReranker({ model: "Xenova/bge-reranker-base" });
|
|
168
|
+
await reranker.init();
|
|
169
|
+
const top = await reranker.rerank("怎么部署本地模型", candidates, 5);
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
### 4.3 autoDream 聚类(`src/dream/clustering.js`,Phase 3,已实现)
|
|
173
|
+
|
|
174
|
+
| 成员 | 说明 |
|
|
175
|
+
|------|------|
|
|
176
|
+
| `cosineSimilarity(a, b)` | 余弦相似度,零向量/长度不齐返回 0,数值稳定 |
|
|
177
|
+
| `kMeans(vectors, k, opts)` | K-Means(k-means++ 播种 + 空簇修复),返回每簇索引数组 |
|
|
178
|
+
| `clusterMemories(memories, vectors, k)` | 按向量把记忆分组,返回簇(保留原对象引用) |
|
|
179
|
+
| `findPotentialConflicts(memories, vectors, threshold = 0.85)` | 同类型、高相似(可能矛盾)的记忆对 |
|
|
180
|
+
|
|
181
|
+
autoDream 调度器在触发巩固时:对候选记忆做 `clusterMemories` 分组 → 每组送 LLM 决策(merge/archive/conflict),并用 `findPotentialConflicts` 预先标出疑似矛盾对,减少漏判。
|
|
182
|
+
|
|
183
|
+
## 5. 降级策略
|
|
184
|
+
|
|
185
|
+
```
|
|
186
|
+
首选 回退 1 回退 2 兜底
|
|
187
|
+
LocalEmbedder(ONNX) ──失败──► OllamaEmbedder ──失败──► OpenAIEmbedder ──失败/未配置──► 关键词 LIKE
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
- Embedder 方法**抛错**,由上层 `createEmbedderByProvider` 编排的降级链逐级捕获切换;全链路失败则回退到 v0.1 的 LIKE 子串关键词搜索(本库无 FTS5,关键词召回即 LIKE 子串扫描)
|
|
191
|
+
- **原则**:语义链路任何一环失败都**静默降级、不阻断记忆读写**——写入照常落库,只是当次不补向量
|
|
192
|
+
- **降级是每请求动态的**:同一会话内 ONNX 偶发失败,下一次请求自动重试首选路径,不固化到降级档
|
|
193
|
+
- **索引一致性**:Embedder 的 `modelHash` 变化(换模型/换维度)时,旧向量与新向量不可混算余弦,需触发「重建索引」(复用 v0.1 的 `reindexMissing` 补建缺失向量)
|
|
194
|
+
|
|
195
|
+
## 6. 模型清单
|
|
196
|
+
|
|
197
|
+
| 用途 | 模型 | 后端 | 维度 | 说明 |
|
|
198
|
+
|------|------|------|------|------|
|
|
199
|
+
| Embedding | `Xenova/bge-small-zh-v1.5` | ONNX(默认) | 512 | 中文优化,全离线,体积小 |
|
|
200
|
+
| Embedding | `bge-m3` | Ollama | 1024 | 多语言(中/英/日等),质量高 |
|
|
201
|
+
| Embedding | `nomic-embed-text` | Ollama | 768 | 英文为主,通用 |
|
|
202
|
+
| Embedding | `text-embedding-3-small` | OpenAI 兼容 | 1536 | v0.1 云端默认,升级后仍可用 |
|
|
203
|
+
| Rerank | `Xenova/bge-reranker-base` | ONNX(Phase 2) | — | 交叉编码,首候选精排 |
|
|
204
|
+
| Rerank | `bge-reranker-v2-m3` | 云/本地 | — | 更强精排,多语言 |
|
|
205
|
+
|
|
206
|
+
> 维度随模型而定:换用不同维度的模型后,已存向量与模型指纹不匹配,需重建索引。
|
|
207
|
+
|
|
208
|
+
## 7. 配置新增项
|
|
209
|
+
|
|
210
|
+
见 [CONFIGURATION.md](CONFIGURATION.md) 的检索与运行时键表。核心:`embedProvider`(`local` / `ollama` / `openai`,默认 `openai` 保持 v0.1 行为)、`localEmbedModel`、`localEmbedDimension`、`localEmbedDevice`、`localEmbedBatchSize`、`ollamaBaseUrl`、`ollamaModel`、`embedModelCacheDir`、`rerankEnabled`、`rerankModel`。
|
|
211
|
+
|
|
212
|
+
## 8. 测试与基准
|
|
213
|
+
|
|
214
|
+
- 单元:`src/local-embedder.js` 三后端接口一致性、`clustering.js` 聚类正确性(node:test)
|
|
215
|
+
- 降级链:mock 各后端抛错,断言逐级回退到 LIKE
|
|
216
|
+
- 基准:`scripts/benchmark-embed.js`(嵌入延迟/吞吐)、`scripts/benchmark-rerank.js`(Rerank 延迟 vs 候选规模)
|
|
217
|
+
|
|
218
|
+
## 9. 反思更新(v0.2.1 `update` 决策 + 失败追踪)
|
|
219
|
+
|
|
220
|
+
### 9.1 `update` 决策类型
|
|
221
|
+
|
|
222
|
+
autoDream 整理时,LLM 可输出第 5 种决策 `update`,直接修正单条记忆的过时/错误内容:
|
|
223
|
+
|
|
224
|
+
- **约束**:`ids` 只能含一个 id;必须产生实际字段变化;不能更新 `summary`;新建 < 24h 的记忆不可 update
|
|
225
|
+
- **频率限制**:每次 autoDream 最多 `reflectionUpdateMaxPerRun`(默认 2)个 update,超限整单拒绝
|
|
226
|
+
- **幂等**:字段已与目标一致时跳过(可重放无副作用)
|
|
227
|
+
- **审计**:`dream_runs` 记录 update 的 `_before` 快照,可回溯变更前内容
|
|
228
|
+
- **向量同步**:update 后删除旧向量、按新内容重嵌入,保持索引一致
|
|
229
|
+
|
|
230
|
+
### 9.2 失败追踪(failure_memories)
|
|
231
|
+
|
|
232
|
+
`failure_memories` 表记录记忆纠正/失败事件,为后续反思进化积累数据:
|
|
233
|
+
|
|
234
|
+
```sql
|
|
235
|
+
CREATE TABLE failure_memories (
|
|
236
|
+
id TEXT PRIMARY KEY,
|
|
237
|
+
query TEXT, -- 用户原始查询/意图(可选)
|
|
238
|
+
expected TEXT, -- 期望结果
|
|
239
|
+
actual TEXT, -- 实际结果
|
|
240
|
+
failure_type TEXT, -- outdated / miss / wrong / user_correction
|
|
241
|
+
memory_id TEXT, -- 关联的记忆 id
|
|
242
|
+
created_at TEXT NOT NULL
|
|
243
|
+
);
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
- **触发**:用户调用 `memory_update` 且内容变化时,若 `reflectionFailureTracking` 开启则记录 `user_correction` 行(actual=旧值、expected=新值)
|
|
247
|
+
- **查询**:`store.listFailures`(过滤/分页)、`store.getFailureStats`(按类型计数)
|
|
248
|
+
|
|
249
|
+
### 9.3 配置项
|
|
250
|
+
|
|
251
|
+
```javascript
|
|
252
|
+
reflectionUpdateEnabled: true, // update 决策总开关
|
|
253
|
+
reflectionFailureTracking: true, // 失败追踪总开关
|
|
254
|
+
reflectionUpdateMaxPerRun: 2, // 每次整理最多 update 数
|
|
255
|
+
reflectionUpdateMinAgeHours: 24 // 新建记忆保护期(小时)
|
|
256
|
+
```
|