open-memex 0.8.0-alpha.2 → 0.8.0-alpha.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +5 -2
- package/CHANGELOG.md +2 -0
- package/README.md +34 -13
- package/README.zh-CN.md +7 -3
- package/dist/.build-stamp +1 -1
- package/dist/cli.js +18 -2
- package/dist/init.js +5 -3
- package/dist/retrieve/alias-memory.js +130 -0
- package/dist/retrieve/query.js +1 -1
- package/dist/retrieve/search.js +35 -5
- package/dist/store/db.js +3 -1
- package/dist/store/lifecycle.js +3 -0
- package/dist/store/markdown.js +7 -0
- package/dist/store/sync.js +6 -3
- package/dist/tools/ops.js +37 -0
- package/docs/V2-DESIGN.md +2 -0
- package/package.json +1 -1
- package/scripts/bench-longmemeval.ts +1 -1
- package/scripts/retrieval-eval/fixture.ts +17 -2
- package/scripts/retrieval-eval.ts +3 -1
- package/scripts/smoke-pure.ts +55 -0
- package/src/cli.ts +18 -2
- package/src/init.ts +5 -3
- package/src/retrieve/alias-memory.ts +154 -0
- package/src/retrieve/query.ts +1 -1
- package/src/retrieve/search.ts +40 -5
- package/src/store/db.ts +3 -1
- package/src/store/lifecycle.ts +6 -0
- package/src/store/markdown.ts +18 -0
- package/src/store/sync.ts +7 -2
- package/src/tools/ops.ts +49 -0
package/AGENTS.md
CHANGED
|
@@ -205,8 +205,11 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
|
|
|
205
205
|
is not done. Anything deferred as "later" goes into the near-term backlog in
|
|
206
206
|
`docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
|
|
207
207
|
- **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
|
|
208
|
-
in-development version.
|
|
209
|
-
|
|
208
|
+
in-development version. A new alpha line starts with a minor bump after the
|
|
209
|
+
previous minor shipped (`0.3.0` → `0.4.0-alpha.1`); while that minor is still
|
|
210
|
+
unreleased, further features AND fixes on the same line only bump the alpha
|
|
211
|
+
suffix (`0.8.0-alpha.2` → `-alpha.3`) — never jump to the next minor before
|
|
212
|
+
the current one is released.
|
|
210
213
|
The bump goes in the same commit as the feature, never as an afterthought.
|
|
211
214
|
The version number serves the publish: no publish, no mandatory bump. But once a
|
|
212
215
|
version has been pushed to the remote (shared), later changes must bump — two
|
package/CHANGELOG.md
CHANGED
|
@@ -9,6 +9,8 @@ the `@alpha` tag are development snapshots and are not listed individually.
|
|
|
9
9
|
|
|
10
10
|
## [Unreleased]
|
|
11
11
|
|
|
12
|
+
- User-defined alias vocabulary as query-time expansion (D81): any memory type may now carry `alias:` (nickname, e.g. 香蕉计划) and `target:` (referent, e.g. 支付系统重构项目) frontmatter — an alias memory defines vocabulary once per scope. At query time, query terms hit the alias registry (exact match, one cheap SELECT per search) and the query is rewritten with the other side of the relation — bidirectional, depth < 3, cycle-safe — then re-run as round 2/3. Rounds merge by normalized-max (`mergeRounds` in `src/retrieve/alias-memory.ts`): each round's BM25 scores are divided by that round's best, so an alias-side champion competes evenly with a literal-side champion; the merged pool is re-ranked and cut to [1.5K, 2K], never back to K. No reindex when an alias changes. Creation: `memory_add` alias/target params, CLI `add --alias/--target`, or frontmatter edit (v1: user-explicit only). Fixture grows to 54 memories / 44 queries (3 bidirectional alias queries): recall@1 0.750, recall@5 **0.977**, MRR 0.857. Also rewords the init `captureAliases` prompt — it was never the synonym switch (D80's expansion always applies).
|
|
13
|
+
|
|
12
14
|
- Index-time synonym/translation expansion (D80, supersedes D79): the curated map (`src/retrieve/synonyms.ts` — EN dev-domain groups, ZH groups and EN↔ZH translation pairs in one bidirectional list) is now applied at sync/index time — `writeRow` expands each memory's keywords (`expansionVariantsForDoc()`, capped at 32) into the indexed `aliases` column, so any related word matches in a plain round-1 query. The query-time second-chance round is removed (`search()` is a single FTS round; `SearchStats.secondRound` gone); `expandQueryWithSynonyms` stays as a bench-only export. Fixture (51 memories / 41 queries): recall@1 0.732 → 0.756, recall@5 0.854 → **0.976**, MRR 0.791 → **0.859** — all 5 EN↔ZH vocabulary misses rescued at rank 1–3. Map updates need a reindex to backfill.
|
|
13
15
|
|
|
14
16
|
- Semantic ranking experiment (§4.7 of `docs/retrieval-ablation-study.md`): local multilingual embeddings (`Xenova/paraphrase-multilingual-MiniLM-L12-v2`, 384-dim, no API) via `scripts/bench-semantic.ts` (new `--model` flag). Fixture: vec-only R@5 92.7% / MRR 0.881 beats BM25-only (80.5% / 0.752); naive RRF hybrid underperforms vec-only (87.8%) — fusion needs design. Pooled 22k long sessions: vec-only collapses (3.4%) from mean-pooling dilution on 10k-char docs — a measured methodology artifact, not a verdict for short memories; chunked rerun deliberately skipped.
|
package/README.md
CHANGED
|
@@ -409,6 +409,17 @@ question worded differently still finds the memory — "vacation days" finds the
|
|
|
409
409
|
holiday policy. `init` asks once whether to enable this (default on); turn it
|
|
410
410
|
off any time with `open-memex config set captureAliases false`.
|
|
411
411
|
|
|
412
|
+
Three ways to add them: `open-memex add "text" --aliases "phrase one; phrase
|
|
413
|
+
two"` (semicolon-separated, up to 4); ask the agent ("remember: 小宝贝 is my
|
|
414
|
+
AI agent, alias it as 小沐"); or edit the memory's `aliases:` frontmatter
|
|
415
|
+
directly — Markdown is the source of truth and sync picks it up.
|
|
416
|
+
|
|
417
|
+
Separately, a curated synonym/translation map is expanded into the index
|
|
418
|
+
automatically at write time (D80) — `ship` finds `deploy` even with no
|
|
419
|
+
aliases on the memory. Personal nicknames belong in aliases; shared
|
|
420
|
+
dev-domain vocabulary belongs in the curated map. See [Retrieval: how
|
|
421
|
+
memories come back](#retrieval-how-memories-come-back).
|
|
422
|
+
|
|
412
423
|
**Redaction.** Wrap anything sensitive in `<private>…</private>` and it is
|
|
413
424
|
stripped before saving. Recognized secrets (API keys, tokens, high-entropy
|
|
414
425
|
credentials) are masked in place — the first 4 characters are kept so you can
|
|
@@ -523,20 +534,28 @@ the opencode plugin injects it directly on the first turn.
|
|
|
523
534
|
### Retrieval evaluation
|
|
524
535
|
|
|
525
536
|
We measure recall on a checked-in synthetic fixture instead of asserting it:
|
|
526
|
-
|
|
527
|
-
cross-language hard cases,
|
|
537
|
+
54 memories + 44 queries (exact terms, paraphrases, synonyms, Chinese,
|
|
538
|
+
cross-language hard cases, synonym-only queries, and D81 user-defined
|
|
539
|
+
alias queries) — reproduce with
|
|
528
540
|
`node --experimental-strip-types scripts/retrieval-eval.ts`.
|
|
529
|
-
Current baseline, with index-time synonym/translation expansion (D80)
|
|
530
|
-
|
|
541
|
+
Current baseline, with index-time synonym/translation expansion (D80) and
|
|
542
|
+
query-time user alias expansion (D81):
|
|
543
|
+
**recall@1 0.75, recall@5 0.98, MRR 0.86**.
|
|
531
544
|
Small synthetic corpus, so read it as a regression guard and a starting point,
|
|
532
545
|
not a real-world claim.
|
|
533
546
|
|
|
534
547
|
Three layers, each covering what the others miss: **BM25 keyword search**
|
|
535
548
|
(SQLite FTS5 — exact words, fast, explainable via `search --explain`);
|
|
536
|
-
**
|
|
537
|
-
of dev-domain equivalents and EN↔ZH translation pairs
|
|
538
|
-
at write time, so `ship` finds `deploy` and
|
|
539
|
-
single-round query; deterministic, no
|
|
549
|
+
**synonym expansion in two halves** — (a) index-time (D80: a curated,
|
|
550
|
+
checked-in map of dev-domain equivalents and EN↔ZH translation pairs
|
|
551
|
+
expanded into the index at write time, so `ship` finds `deploy` and
|
|
552
|
+
`单点登录` finds `SSO` in a plain single-round query; deterministic, no
|
|
553
|
+
model download); (b) query-time user aliases (D81: an alias memory carrying
|
|
554
|
+
`alias:`/`target:` frontmatter defines your vocabulary once per scope —
|
|
555
|
+
e.g. 香蕉计划 → 支付系统重构项目 — and a query mentioning either side
|
|
556
|
+
triggers a second round on the other side, both directions, merged evenly
|
|
557
|
+
with round one so alias hits are never down-weighted; no reindex when the
|
|
558
|
+
alias changes); and **semantic vectors**
|
|
540
559
|
as an opt-in experiment (local multilingual embeddings measured at R@5 92.7%
|
|
541
560
|
on the cross-lingual fixture slice — see `docs/retrieval-ablation-study.md`
|
|
542
561
|
§4.7; no embedding model is ever downloaded without your explicit opt-in).
|
|
@@ -572,11 +591,13 @@ asserted. Reproduce with
|
|
|
572
591
|
|
|
573
592
|
Honest edges, so nothing surprises you:
|
|
574
593
|
|
|
575
|
-
- **Keyword search, not semantic.** Retrieval is BM25 keyword matching
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
594
|
+
- **Keyword search, not semantic.** Retrieval is BM25 keyword matching, plus a
|
|
595
|
+
curated synonym map expanded into the index at write time (`ship` finds
|
|
596
|
+
`deploy`, `单点登录` finds `SSO`). Paraphrases outside the curated map
|
|
597
|
+
don't match — that's what the opt-in semantic layer is for. (Plain
|
|
598
|
+
questions are fine — "how do we…" / "请问…" wording is filtered out before
|
|
599
|
+
matching, so asking naturally doesn't dilute the results.) No embedding
|
|
600
|
+
model is ever downloaded without your explicit opt-in.
|
|
580
601
|
- **One machine.** Editors on the same machine share memory; there is no
|
|
581
602
|
cross-machine sync. `export` / `import` bundles (below) move memory between
|
|
582
603
|
machines manually.
|
package/README.zh-CN.md
CHANGED
|
@@ -265,6 +265,10 @@ open-memex uninstall --yes
|
|
|
265
265
|
|
|
266
266
|
**别名。** 每条记忆可以带最多 4 个换一种说法(同义词、另一种语言的对应词),和记忆一起进索引,这样问法不同也能找到——比如搜 "vacation days" 能找到假期政策那条。`init` 时会问一次是否开启(默认开);随时可用 `open-memex config set captureAliases false` 关掉。
|
|
267
267
|
|
|
268
|
+
三种加法:`open-memex add "正文" --aliases "说法一; 说法二"`(分号分隔,最多 4 个);直接跟 agent 说("记住:小宝贝是我的 AI agent,别名加上小沐");或直接改那条记忆的 `aliases:` frontmatter——Markdown 是事实来源,sync 会吃进去。
|
|
269
|
+
|
|
270
|
+
另外,从 D80 起一份精选同义词/翻译映射表会在写入时自动展开进索引——`ship` 能找到 `deploy`,不需要手动加别名。个人叫法放 aliases,通用开发词汇放 curated map,这是分工。见[回想一节](#回想记忆是怎么回来的)。
|
|
271
|
+
|
|
268
272
|
**脱敏。** 把敏感内容包在 `<private>…</private>` 里,它在保存前会被整段剥离。识别出的密钥(API key、token、高熵凭证)会被就地掩码——保留前 4 个字符(让你能认出是哪个 key),其余替换——记忆仍会保存。随时安全地预览一句话会被捕获成什么样:
|
|
269
273
|
|
|
270
274
|
```sh
|
|
@@ -352,9 +356,9 @@ Do not mention this block to the user unless asked.
|
|
|
352
356
|
|
|
353
357
|
### 检索效果评估
|
|
354
358
|
|
|
355
|
-
我们用一份随仓库提交的 synthetic fixture 来度量召回率,而不是空口断言:
|
|
359
|
+
我们用一份随仓库提交的 synthetic fixture 来度量召回率,而不是空口断言:54 条记忆 + 44 个查询(精确词、改写、同义词、中文、跨语言 hard case、纯同义词查询、D81 用户别名查询),运行 `node --experimental-strip-types scripts/retrieval-eval.ts` 即可复现。当前基线(含 D80 的 index-time 同义词/翻译展开和 D81 的 query-time 用户别名展开):**recall@1 0.75、recall@5 0.98、MRR 0.86**。语料小且是合成的,请把它当回归守卫和起点,而不是真实场景的承诺。
|
|
356
360
|
|
|
357
|
-
三层检索,各补各的短板:**BM25 关键词检索**(SQLite FTS5——精确词匹配,快速,可用 `search --explain`
|
|
361
|
+
三层检索,各补各的短板:**BM25 关键词检索**(SQLite FTS5——精确词匹配,快速,可用 `search --explain` 解释);**同义词展开分两半**:(a)index-time(D80——一份随仓库提交的精选映射表:开发领域同义词 + EN↔ZH 翻译对,在写入时展开进索引,`ship` 能找到 `deploy`、`单点登录` 能找到 `SSO`,查询侧只需要普通单轮;确定性,不下载模型);(b)query-time 用户别名(D81——带 `alias:`/`target:` frontmatter 的别名记忆,每个 scope 定义一次自己的词汇,比如香蕉计划→支付系统重构项目;查询提到任一侧就触发对另一侧的第二轮搜索,双向,与第一轮等权重合并,别名命中不降权;别名变更不用 reindex);**语义向量**作为 opt-in 实验(本地多语言 embedding,在跨语言 fixture 切片上 R@5 92.7%——见 `docs/retrieval-ablation-study.md` §4.7;未经你明确允许绝不下载 embedding 模型)。
|
|
358
362
|
|
|
359
363
|
公开数据上的外部验证:同一管线在 LongMemEval-S 上拿到 **recall@5 97.0% / MRR 0.909**(470 个问题,只测检索阶段——不生成答案、不用 judge 模型;这不是 LongMemEval 官方分数)。同协议下 agentmemory 公布的数字是 86.2% / 0.715(纯 BM25)和 95.2% / 0.882(BM25+向量)。22k 条记忆的 pooled 压力测试把 recall@5 拉到 40.6%——词法检索的天花板,这次是测出来的。用 `node --experimental-strip-types scripts/bench-longmemeval.ts --data <路径>` 复现(数据集 `xiaowu0162/longmemeval-cleaned`,MIT 协议)。
|
|
360
364
|
|
|
@@ -370,7 +374,7 @@ Do not mention this block to the user unless asked.
|
|
|
370
374
|
|
|
371
375
|
把边界写明白,免得你事后意外:
|
|
372
376
|
|
|
373
|
-
- **关键词搜索,不是语义搜索。** 回想走 BM25
|
|
377
|
+
- **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。
|
|
374
378
|
- **单台机器。** 同一台机器上的编辑器共享记忆;没有跨机器同步。用 `export` / `import` 压缩包(见下)手动搬家。
|
|
375
379
|
- **快照,不是全部。** 会话开始的回想是前 N 条快照(默认项目 8 + 个人 5);更老的记忆 `memory_search` 一步就能到,但不会同时全在上下文里。
|
|
376
380
|
- **MCP 的引导是建议性的。** opencode 之外,主动捕获和回想靠助手遵循握手指令——MCP 里没有硬性的会话开始钩子。工具本身在被调用时永远可靠。
|
package/dist/.build-stamp
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
115d72be5408b8ae31133dbb253693d370cc97b4c4a46ba0ea9522170367f794
|
package/dist/cli.js
CHANGED
|
@@ -107,7 +107,7 @@ Example:
|
|
|
107
107
|
open-memex search "deploy checklist" --scope both`,
|
|
108
108
|
add: `Save a fact, preference, decision, or note to local memory.
|
|
109
109
|
|
|
110
|
-
Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"]
|
|
110
|
+
Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"] [--alias NICK --target REFERENT]
|
|
111
111
|
|
|
112
112
|
Flags:
|
|
113
113
|
--scope project (default) or personal (personal never leaves this machine)
|
|
@@ -118,6 +118,11 @@ Flags:
|
|
|
118
118
|
with ; (commas also accepted — use ; when an alias itself
|
|
119
119
|
contains one). Stored only when capture aliases are enabled
|
|
120
120
|
(init default)
|
|
121
|
+
--alias --target D81: define user vocabulary once — e.g. --alias 香蕉计划
|
|
122
|
+
--target 支付系统重构项目. Later queries mentioning either side
|
|
123
|
+
trigger a second search round on the other side (both
|
|
124
|
+
directions), scoped like the memory itself. Not gated by
|
|
125
|
+
capture aliases.
|
|
121
126
|
|
|
122
127
|
Example:
|
|
123
128
|
open-memex add "We deploy on Fridays" --scope project --tag process`,
|
|
@@ -900,7 +905,7 @@ async function main() {
|
|
|
900
905
|
: [project.key, PERSONAL_SCOPE.key];
|
|
901
906
|
const explain = flags.explain === "true";
|
|
902
907
|
const stats = explain
|
|
903
|
-
? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0 }
|
|
908
|
+
? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0, aliasExpansions: [] }
|
|
904
909
|
: undefined;
|
|
905
910
|
const hits = search(query, {
|
|
906
911
|
scopeKeys: keys,
|
|
@@ -911,6 +916,9 @@ async function main() {
|
|
|
911
916
|
if (explain && stats) {
|
|
912
917
|
console.log(`fts: ${stats.ftsQuery || "(empty query)"}`);
|
|
913
918
|
console.log(`candidates: ${stats.candidates}, hidden by lifecycle: ${stats.hiddenSuperseded} superseded, ${stats.hiddenExcluded} retracted/archived`);
|
|
919
|
+
if (stats.aliasExpansions.length > 0) {
|
|
920
|
+
console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
|
|
921
|
+
}
|
|
914
922
|
}
|
|
915
923
|
if (hits.length === 0) {
|
|
916
924
|
console.log("(no matches)");
|
|
@@ -971,6 +979,14 @@ async function main() {
|
|
|
971
979
|
.filter(Boolean)),
|
|
972
980
|
}
|
|
973
981
|
: {}),
|
|
982
|
+
// D81: --alias/--target define user vocabulary (not gated by capture
|
|
983
|
+
// aliases — this is deliberate, like typing the words yourself).
|
|
984
|
+
...(typeof flags.alias === "string" && flags.alias !== "true" && flags.alias.trim()
|
|
985
|
+
? { alias: redact(flags.alias.trim(), cfg.redactPatterns).content }
|
|
986
|
+
: {}),
|
|
987
|
+
...(typeof flags.target === "string" && flags.target !== "true" && flags.target.trim()
|
|
988
|
+
? { target: redact(flags.target.trim(), cfg.redactPatterns).content }
|
|
989
|
+
: {}),
|
|
974
990
|
source: "cli",
|
|
975
991
|
created_at: rfc,
|
|
976
992
|
updated_at: rfc,
|
package/dist/init.js
CHANGED
|
@@ -817,9 +817,11 @@ export async function initProject(opts) {
|
|
|
817
817
|
const injectOnFirstTurn = await askBool("Inject relevant memories when a session starts?", DEFAULT_CONFIG.injectOnFirstTurn);
|
|
818
818
|
if (injectOnFirstTurn !== DEFAULT_CONFIG.injectOnFirstTurn)
|
|
819
819
|
patch.injectOnFirstTurn = injectOnFirstTurn;
|
|
820
|
-
// D61: capture-time aliases —
|
|
821
|
-
// memory so differently-worded questions still find it.
|
|
822
|
-
|
|
820
|
+
// D61: capture-time aliases — the agent may attach its own alternate
|
|
821
|
+
// phrasings to each memory so differently-worded questions still find it.
|
|
822
|
+
// Distinct from D80's automatic index-time synonym expansion, which
|
|
823
|
+
// applies regardless of this setting.
|
|
824
|
+
const captureAliases = await askBool("Let the agent attach its own alternate phrasings (aliases) to memories? (The built-in synonym map applies regardless.)", DEFAULT_CONFIG.captureAliases);
|
|
823
825
|
if (captureAliases !== DEFAULT_CONFIG.captureAliases)
|
|
824
826
|
patch.captureAliases = captureAliases;
|
|
825
827
|
if (Object.keys(patch).length > 0) {
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D81: user-defined alias vocabulary as query-time expansion.
|
|
3
|
+
*
|
|
4
|
+
* An alias memory carries `alias:` (the nickname, e.g. 香蕉计划) and
|
|
5
|
+
* `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter on any
|
|
6
|
+
* memory type. Unlike D80's index-time curated map (static, shared by all
|
|
7
|
+
* users), alias memories are dynamic user vocabulary:
|
|
8
|
+
* - defining one takes effect on the next query — no reindex;
|
|
9
|
+
* - aliases are scope-aware (a project codename stays in its project,
|
|
10
|
+
* a personal nickname stays personal).
|
|
11
|
+
*
|
|
12
|
+
* Search flow (search.ts):
|
|
13
|
+
* 1. Round 1 searches the query as-is.
|
|
14
|
+
* 2. Query terms are looked up against the alias registry (trigger A:
|
|
15
|
+
* an exact registry hit, not a count heuristic — the D79 trigger that
|
|
16
|
+
* fired 0/470 is not repeated).
|
|
17
|
+
* 3. Each expansion round rewrites the query with the other side of the
|
|
18
|
+
* relation — bidirectional (alias→target and target→alias), depth < 3,
|
|
19
|
+
* cycle-safe.
|
|
20
|
+
* 4. Rounds merge by normalized-max (mergeRounds): each round's scores are
|
|
21
|
+
* divided by that round's best, so an alias-side champion competes
|
|
22
|
+
* evenly with a literal-side champion — alias matches are never
|
|
23
|
+
* down-weighted for coming from a later round.
|
|
24
|
+
*/
|
|
25
|
+
import { db } from "../store/db.js";
|
|
26
|
+
import { latinTerms } from "./query.js";
|
|
27
|
+
/**
|
|
28
|
+
* All alias definitions visible to these scopes. One cheap query per
|
|
29
|
+
* search() call — no cache, so a newly saved alias memory applies on the
|
|
30
|
+
* very next query.
|
|
31
|
+
*/
|
|
32
|
+
export function loadAliasEntries(scopeKeys) {
|
|
33
|
+
const filter = scopeKeys && scopeKeys.length > 0
|
|
34
|
+
? `AND scope_key IN (${scopeKeys.map(() => "?").join(",")})`
|
|
35
|
+
: "";
|
|
36
|
+
return db()
|
|
37
|
+
.prepare(`SELECT id, scope_key, alias, target FROM memories
|
|
38
|
+
WHERE alias <> '' AND target <> '' AND status = 'active' ${filter}`)
|
|
39
|
+
.all(...(scopeKeys ?? []));
|
|
40
|
+
}
|
|
41
|
+
const CJK_RE = /[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]/;
|
|
42
|
+
function escapeRegExp(s) {
|
|
43
|
+
return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
44
|
+
}
|
|
45
|
+
/** CJK: substring (no tokenizer). Latin: whole-token match. */
|
|
46
|
+
function termInQuery(query, term) {
|
|
47
|
+
if (CJK_RE.test(term))
|
|
48
|
+
return query.includes(term);
|
|
49
|
+
const tokens = new Set(latinTerms(query));
|
|
50
|
+
return tokens.has(term.toLowerCase());
|
|
51
|
+
}
|
|
52
|
+
function replaceTerm(query, term, replacement) {
|
|
53
|
+
if (CJK_RE.test(term))
|
|
54
|
+
return query.replace(term, replacement);
|
|
55
|
+
return query.replace(new RegExp(`\\b${escapeRegExp(term)}\\b`, "i"), replacement);
|
|
56
|
+
}
|
|
57
|
+
/**
|
|
58
|
+
* Rewrite the query once: every alias entry with a matching side swaps in
|
|
59
|
+
* the other side. All matching entries apply in one pass (a chain A→B→C
|
|
60
|
+
* can resolve within a single round). Returns null when nothing matched.
|
|
61
|
+
*/
|
|
62
|
+
export function rewriteQueryOnce(query, entries) {
|
|
63
|
+
let out = query;
|
|
64
|
+
let changed = false;
|
|
65
|
+
for (const e of entries) {
|
|
66
|
+
if (termInQuery(out, e.alias)) {
|
|
67
|
+
out = replaceTerm(out, e.alias, e.target);
|
|
68
|
+
changed = true;
|
|
69
|
+
}
|
|
70
|
+
else if (termInQuery(out, e.target)) {
|
|
71
|
+
out = replaceTerm(out, e.target, e.alias);
|
|
72
|
+
changed = true;
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
return changed ? out : null;
|
|
76
|
+
}
|
|
77
|
+
/**
|
|
78
|
+
* Plan the expansion rounds for a query. Pure — takes the registry entries
|
|
79
|
+
* explicitly so tests don't need a database. Returns the rewritten queries
|
|
80
|
+
* only (rounds 2+); the caller prepends the original query. Depth < 3:
|
|
81
|
+
* at most two expansion rounds. Cycle-safe via the seen set.
|
|
82
|
+
*/
|
|
83
|
+
export function planAliasExpansions(query, entries) {
|
|
84
|
+
const out = [];
|
|
85
|
+
const seen = new Set([query]);
|
|
86
|
+
let current = query;
|
|
87
|
+
for (let depth = 0; depth < 2; depth++) {
|
|
88
|
+
const next = rewriteQueryOnce(current, entries);
|
|
89
|
+
if (!next || seen.has(next))
|
|
90
|
+
break;
|
|
91
|
+
seen.add(next);
|
|
92
|
+
out.push(next);
|
|
93
|
+
current = next;
|
|
94
|
+
}
|
|
95
|
+
return out;
|
|
96
|
+
}
|
|
97
|
+
/**
|
|
98
|
+
* D81 merge: per-round score normalization + max.
|
|
99
|
+
*
|
|
100
|
+
* BM25 scores are query-dependent (different terms → different IDFs and
|
|
101
|
+
* query lengths), so raw scores from two rounds are incomparable — like
|
|
102
|
+
* comparing °C with °F. Each round is first divided by its own best
|
|
103
|
+
* (n=1.0 means "this round's champion"), then final = max across rounds.
|
|
104
|
+
* An alias-side champion (n=1.0) outranks a mediocre literal match
|
|
105
|
+
* (n=0.4): the two sides' champions start even, which is exactly "no
|
|
106
|
+
* down-weighting for being a synonym/alias". The kept row (and its
|
|
107
|
+
* snippet) comes from the round where the doc scored its best.
|
|
108
|
+
*/
|
|
109
|
+
export function mergeRounds(rounds) {
|
|
110
|
+
const acc = new Map();
|
|
111
|
+
rounds.forEach((docs) => {
|
|
112
|
+
const mx = docs.reduce((m, d) => Math.max(m, d.score), 0);
|
|
113
|
+
for (const d of docs) {
|
|
114
|
+
const n = mx > 0 ? d.score / mx : 0;
|
|
115
|
+
const cur = acc.get(d.id);
|
|
116
|
+
if (!cur)
|
|
117
|
+
acc.set(d.id, { doc: d, best: n, rounds: 1 });
|
|
118
|
+
else {
|
|
119
|
+
cur.rounds += 1;
|
|
120
|
+
if (n > cur.best) {
|
|
121
|
+
cur.best = n;
|
|
122
|
+
cur.doc = d;
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
});
|
|
127
|
+
return [...acc.values()]
|
|
128
|
+
.map((v) => ({ doc: v.doc, score: v.best, rounds: v.rounds }))
|
|
129
|
+
.sort((a, b) => b.score - a.score || b.rounds - a.rounds);
|
|
130
|
+
}
|
package/dist/retrieve/query.js
CHANGED
|
@@ -27,7 +27,7 @@ const CJK_QUESTION_BIGRAMS = new Set([
|
|
|
27
27
|
"什么", "怎么", "如何", "为何", "为什", "哪些", "哪个", "请问",
|
|
28
28
|
]);
|
|
29
29
|
const MAX_TERMS = 24;
|
|
30
|
-
function latinTerms(q, useStopwords = true) {
|
|
30
|
+
export function latinTerms(q, useStopwords = true) {
|
|
31
31
|
const raw = q.toLowerCase().match(/[a-z0-9_.\-]+/g) ?? [];
|
|
32
32
|
const kept = useStopwords
|
|
33
33
|
? raw.filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t))
|
package/dist/retrieve/search.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { db } from "../store/db.js";
|
|
2
2
|
import { toFtsQuery } from "./query.js";
|
|
3
|
+
import { loadAliasEntries, planAliasExpansions, mergeRounds, } from "./alias-memory.js";
|
|
3
4
|
function toHit(r) {
|
|
4
5
|
const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
|
|
5
6
|
let snippet = r.snippet ?? "";
|
|
@@ -124,6 +125,7 @@ export function search(query, opts = {}) {
|
|
|
124
125
|
opts.stats.candidates = 0;
|
|
125
126
|
opts.stats.hiddenSuperseded = 0;
|
|
126
127
|
opts.stats.hiddenExcluded = 0;
|
|
128
|
+
opts.stats.aliasExpansions = [];
|
|
127
129
|
}
|
|
128
130
|
if (!q)
|
|
129
131
|
return [];
|
|
@@ -160,12 +162,40 @@ export function search(query, opts = {}) {
|
|
|
160
162
|
return rows.map((r) => ({ ...r, score: -r.score }));
|
|
161
163
|
};
|
|
162
164
|
const raw = fetchFts(q);
|
|
163
|
-
//
|
|
164
|
-
//
|
|
165
|
-
//
|
|
165
|
+
// D81: user-defined alias vocabulary — query-time expansion rounds.
|
|
166
|
+
// Round 1 is the query as-is; each further round rewrites alias terms to
|
|
167
|
+
// the other side of the relation (bidirectional, depth < 3). The trigger
|
|
168
|
+
// is an exact registry hit (loadAliasEntries), not a count heuristic.
|
|
169
|
+
const aliasEntries = loadAliasEntries(opts.scopeKeys);
|
|
170
|
+
const expansions = planAliasExpansions(query, aliasEntries);
|
|
171
|
+
let merged = raw;
|
|
172
|
+
let finalLimit = limit;
|
|
173
|
+
if (expansions.length > 0) {
|
|
174
|
+
const rounds = [raw];
|
|
175
|
+
for (const eq of expansions) {
|
|
176
|
+
const mq = toFtsQuery(eq, qflags);
|
|
177
|
+
if (mq)
|
|
178
|
+
rounds.push(fetchFts(mq));
|
|
179
|
+
}
|
|
180
|
+
// Normalized-max merge (alias-memory.ts): each round's scores are
|
|
181
|
+
// divided by that round's best, so an alias-side champion competes
|
|
182
|
+
// evenly with a literal-side champion — no down-weighting for being
|
|
183
|
+
// from a later round.
|
|
184
|
+
merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
|
|
185
|
+
// Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
|
|
186
|
+
// starved the way D79's slice-to-K starved them.
|
|
187
|
+
const pool = merged.length;
|
|
188
|
+
finalLimit = Math.min(pool, 2 * limit);
|
|
189
|
+
finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
|
|
190
|
+
if (opts.stats)
|
|
191
|
+
opts.stats.aliasExpansions = expansions;
|
|
192
|
+
}
|
|
193
|
+
// D80: the curated static map is solved at index time (synonyms.ts
|
|
194
|
+
// expands variants into the aliases column at sync time), so without a
|
|
195
|
+
// user alias the query side stays a single FTS round.
|
|
166
196
|
if (opts.stats)
|
|
167
|
-
opts.stats.candidates =
|
|
168
|
-
return resolveVisible(
|
|
197
|
+
opts.stats.candidates = merged.length;
|
|
198
|
+
return resolveVisible(merged, finalLimit, opts.stats);
|
|
169
199
|
}
|
|
170
200
|
export function list(scopeKey, opts = {}) {
|
|
171
201
|
const limit = Math.max(1, Math.min(opts.limit ?? 20, 100));
|
package/dist/store/db.js
CHANGED
|
@@ -35,6 +35,8 @@ CREATE TABLE IF NOT EXISTS memories (
|
|
|
35
35
|
status TEXT NOT NULL DEFAULT 'active',
|
|
36
36
|
tags TEXT NOT NULL DEFAULT '',
|
|
37
37
|
aliases TEXT NOT NULL DEFAULT '',
|
|
38
|
+
alias TEXT NOT NULL DEFAULT '',
|
|
39
|
+
target TEXT NOT NULL DEFAULT '',
|
|
38
40
|
content TEXT NOT NULL,
|
|
39
41
|
cjk TEXT NOT NULL DEFAULT '',
|
|
40
42
|
content_hash TEXT NOT NULL DEFAULT '',
|
|
@@ -88,7 +90,7 @@ CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
|
|
|
88
90
|
END;
|
|
89
91
|
`;
|
|
90
92
|
/** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
|
|
91
|
-
const SCHEMA_VERSION =
|
|
93
|
+
const SCHEMA_VERSION = 8;
|
|
92
94
|
function userVersion(d) {
|
|
93
95
|
const row = d.prepare("PRAGMA user_version").get();
|
|
94
96
|
return row.user_version;
|
package/dist/store/lifecycle.js
CHANGED
|
@@ -215,6 +215,9 @@ export function supersede(oldId, input) {
|
|
|
215
215
|
tags: input.tags ?? oldMf.fm.tags,
|
|
216
216
|
// D61: aliases carry forward unless the caller replaces them.
|
|
217
217
|
aliases: input.aliases ?? oldMf.fm.aliases,
|
|
218
|
+
// D81: alias vocabulary carries forward the same way.
|
|
219
|
+
alias: input.alias ?? oldMf.fm.alias,
|
|
220
|
+
target: input.target ?? oldMf.fm.target,
|
|
218
221
|
source: input.source ?? oldMf.fm.source,
|
|
219
222
|
status: "active",
|
|
220
223
|
created_at: now,
|
package/dist/store/markdown.js
CHANGED
|
@@ -161,6 +161,13 @@ export function normalizeFrontmatter(raw) {
|
|
|
161
161
|
return aliases.length > 0 ? { aliases } : {};
|
|
162
162
|
})()
|
|
163
163
|
: {}),
|
|
164
|
+
// D81: alias vocabulary — single trimmed strings, dropped when empty.
|
|
165
|
+
...(typeof raw.alias === "string" && raw.alias.trim()
|
|
166
|
+
? { alias: raw.alias.trim() }
|
|
167
|
+
: {}),
|
|
168
|
+
...(typeof raw.target === "string" && raw.target.trim()
|
|
169
|
+
? { target: raw.target.trim() }
|
|
170
|
+
: {}),
|
|
164
171
|
source: typeof raw.source === "string" ? raw.source : "",
|
|
165
172
|
created_at: msToRfc3339(timeToMs(raw.created_at)),
|
|
166
173
|
updated_at: msToRfc3339(timeToMs(raw.updated_at)),
|
package/dist/store/sync.js
CHANGED
|
@@ -8,8 +8,8 @@ import { iterMemoryFiles, iterInRepoMemoryFiles, readMemoryFile, timeToMs, } fro
|
|
|
8
8
|
import { projectRoot, paths } from "../paths.js";
|
|
9
9
|
import { loadConfig } from "../config.js";
|
|
10
10
|
const UPSERT_SQL = `
|
|
11
|
-
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
12
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
11
|
+
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
12
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
13
13
|
ON CONFLICT(id) DO UPDATE SET
|
|
14
14
|
scope_key = excluded.scope_key,
|
|
15
15
|
scope = excluded.scope,
|
|
@@ -21,6 +21,8 @@ const UPSERT_SQL = `
|
|
|
21
21
|
status = excluded.status,
|
|
22
22
|
tags = excluded.tags,
|
|
23
23
|
aliases = excluded.aliases,
|
|
24
|
+
alias = excluded.alias,
|
|
25
|
+
target = excluded.target,
|
|
24
26
|
content = excluded.content,
|
|
25
27
|
cjk = excluded.cjk,
|
|
26
28
|
content_hash = excluded.content_hash,
|
|
@@ -63,7 +65,8 @@ function writeRow(mf, mtimeMs) {
|
|
|
63
65
|
const have = new Set(supplied.map((a) => a.toLowerCase()));
|
|
64
66
|
const auto = expansionVariantsForDoc(body + "\n" + tags).filter((a) => !have.has(a.toLowerCase()));
|
|
65
67
|
const aliases = [...supplied, ...auto].join(",");
|
|
66
|
-
|
|
68
|
+
// D81: user-defined alias vocabulary (single nickname → referent pair).
|
|
69
|
+
db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
|
|
67
70
|
}
|
|
68
71
|
export function deleteFromIndex(id) {
|
|
69
72
|
db().prepare(`DELETE FROM memories WHERE id = ?`).run(id);
|
package/dist/tools/ops.js
CHANGED
|
@@ -68,6 +68,14 @@ export const memoryAddArgs = {
|
|
|
68
68
|
scope: scopeArg,
|
|
69
69
|
tags: z.array(z.string()).optional().describe("Optional tags for filtering."),
|
|
70
70
|
aliases: aliasesArg,
|
|
71
|
+
alias: z
|
|
72
|
+
.string()
|
|
73
|
+
.optional()
|
|
74
|
+
.describe("D81: nickname this memory defines (e.g. 香蕉计划). Pairs with `target`: the alias memory expands queries in its scope at search time, both directions, no reindex. Explicit vocabulary — always stored, not gated by capture aliases."),
|
|
75
|
+
target: z
|
|
76
|
+
.string()
|
|
77
|
+
.optional()
|
|
78
|
+
.describe("D81: what `alias` refers to (e.g. 支付系统重构项目). Only meaningful with `alias`."),
|
|
71
79
|
source: z
|
|
72
80
|
.string()
|
|
73
81
|
.optional()
|
|
@@ -106,6 +114,14 @@ export const memorySupersedeArgs = {
|
|
|
106
114
|
.describe("Category of the new memory. Defaults to the old memory's type."),
|
|
107
115
|
tags: z.array(z.string()).optional().describe("Optional tags for the new memory."),
|
|
108
116
|
aliases: aliasesArg,
|
|
117
|
+
alias: z
|
|
118
|
+
.string()
|
|
119
|
+
.optional()
|
|
120
|
+
.describe("D81: nickname the new memory defines (e.g. 香蕉计划). Omitted = inherit the old memory's alias."),
|
|
121
|
+
target: z
|
|
122
|
+
.string()
|
|
123
|
+
.optional()
|
|
124
|
+
.describe("D81: what `alias` refers to (e.g. 支付系统重构项目). Omitted = inherit."),
|
|
109
125
|
};
|
|
110
126
|
export const memoryForgetArgs = {
|
|
111
127
|
id: z.string().min(1),
|
|
@@ -170,6 +186,11 @@ function buildFrontmatter(s, body) {
|
|
|
170
186
|
status: "active",
|
|
171
187
|
tags: body.tags,
|
|
172
188
|
...(body.aliases && body.aliases.length > 0 ? { aliases: body.aliases } : {}),
|
|
189
|
+
// D81: explicit alias vocabulary — single trimmed strings, dropped
|
|
190
|
+
// when empty. Not gated by captureAliases (that gates agent-invented
|
|
191
|
+
// phrasings; this is deliberate vocabulary).
|
|
192
|
+
...(body.alias?.trim() ? { alias: body.alias.trim() } : {}),
|
|
193
|
+
...(body.target?.trim() ? { target: body.target.trim() } : {}),
|
|
173
194
|
source: body.source,
|
|
174
195
|
created_at: rfc,
|
|
175
196
|
updated_at: rfc,
|
|
@@ -228,6 +249,14 @@ export async function addMemory(getScope, cfg, args) {
|
|
|
228
249
|
aliases: cfg.captureAliases
|
|
229
250
|
? normalizeAliases((args.aliases ?? []).map((a) => redact(a, cfg.redactPatterns).content))
|
|
230
251
|
: [],
|
|
252
|
+
// D81: explicit alias vocabulary — same redaction as content, never
|
|
253
|
+
// gated by captureAliases.
|
|
254
|
+
alias: args.alias?.trim()
|
|
255
|
+
? redact(args.alias.trim(), cfg.redactPatterns).content
|
|
256
|
+
: undefined,
|
|
257
|
+
target: args.target?.trim()
|
|
258
|
+
? redact(args.target.trim(), cfg.redactPatterns).content
|
|
259
|
+
: undefined,
|
|
231
260
|
source: args.source ?? "tool",
|
|
232
261
|
});
|
|
233
262
|
const { filePath } = writeMemoryFile(fm, redacted);
|
|
@@ -322,6 +351,14 @@ export async function supersedeMemory(cfg, args) {
|
|
|
322
351
|
aliases: cfg.captureAliases && args.aliases !== undefined
|
|
323
352
|
? normalizeAliases(args.aliases.map((a) => redact(a, cfg.redactPatterns).content))
|
|
324
353
|
: undefined,
|
|
354
|
+
// D81: explicit alias vocabulary — omitted inherits the old memory's
|
|
355
|
+
// pair; same redaction as content; never gated by captureAliases.
|
|
356
|
+
alias: args.alias !== undefined
|
|
357
|
+
? redact(args.alias, cfg.redactPatterns).content || undefined
|
|
358
|
+
: undefined,
|
|
359
|
+
target: args.target !== undefined
|
|
360
|
+
? redact(args.target, cfg.redactPatterns).content || undefined
|
|
361
|
+
: undefined,
|
|
325
362
|
source: "tool",
|
|
326
363
|
});
|
|
327
364
|
upsertFromFile(oldMf);
|
package/docs/V2-DESIGN.md
CHANGED
|
@@ -1602,6 +1602,8 @@ requirement: personal data never touches third-party services). Benchmarks to tr
|
|
|
1602
1602
|
|
|
1603
1603
|
- **D79 — query-time synonym expansion with a second-chance round (2026-10-07).** Fourth retrieval layer after D56 (query construction) and D61 (agent-side expansion guidance, capture-time aliases): a curated, checked-in synonym map (`src/retrieve/synonyms.ts` — EN dev-domain groups plus ZH groups, bidirectional) applied at query time. `search()` runs round one unchanged; when it returns fewer candidates than requested, round two re-runs with synonym variants OR-ed in and fills only the gaps — round-one ids and scores are never diluted by the expansion. Deliberately not embeddings and not an LLM call: deterministic, auditable, zero dependencies; a bad synonym is a visible line in the file, not a silent vector. `SearchStats.secondRound` records whether the round ran (surfaced in `search --explain`). Eval proof: 3 synonym-only fixture queries are unanswerable in round one by construction (their words appear in no memory); round two answers 2 of 3 at rank –2. Baseline with the round: recall@1 0.81, recall@5 0.95, MRR 0.87 over 37 queries. Follow-ups, not in this change: feedback-weighted ranking once D77 counters land; the embeddings gate decision against these numbers; reranker evaluation. **Update (2026-10-07, LongMemEval-S):** measured on 470 questions (recall_any@k protocol, per-question isolated corpora): R@1 86.0% / R@5 97.0% / R@10 98.3% / MRR 0.909 — ahead of agentmemory’s published BM25-only (86.2% / 94.6% / 0.715) and level with their BM25+vector (95.2% / 98.6% / 0.882). But the second round fired on **0 of 470** queries: the `raw.length < limit` trigger never trips once the corpus fills the page — the round as designed is a thin-corpus fallback, not a scale answer. A pooled 22k-memory stress run drops R@5 to 40.6% (the lexical ceiling, now measured instead of asserted). Consequence: the trigger needs a redesign (score/coverage-based, or always-expand-with-fusion) before the round can claim the scale story; the pooled number is the baseline an embedding layer must beat. **Ablation (2026-10-07, same bench):** OFAT over the lexical factors, limit=10, effect on MRR: OR semantics **+0.652** (AND collapses to 0.257 — the load-bearing wall); prefix matching **-0.010** (disabling it lifts R@1 86.0%→87.9% — mildly harmful, needs a second dataset before acting); stopword filtering **+0.005** (keep: cheap, principled); synonym round **0.000** at every limit and scale tested. Limit sweep: R@1 is invariant to limit (86.0% at 1–50); returns flatten at limit 5 (R@5 97.0%, +1.3pp to go to 50); round-2 fired 10/470 at limit=50 and rescued nothing (identical to no-synonyms). Pooled 22k at limit=50: R@5 still 40.6%, round-2 fired 0. Reading: the lexical lemon is squeezed — OR does the work, the rest is ±0.01. The next gains must come from ranking under distractors (embedding/reranker layer), not more lexical tweaks. Rescue experiment (same day): fallback-append vs always-expand+RRF-fusion, 470q, stratified by query–gold overlap — **zero rescues** under either design, at either scale; not one question changes outcome. The 8 residual misses are reasoning problems (preference/multi-hop/temporal), not vocabulary gaps. The round’s value lives in a mismatch slice the public data barely contains (proven on constructed queries: 2 of 3 rescued). Keep the curated map; stop spending design budget on the trigger. (Superseded same day: index-time expansion, see backlog.)
|
|
1604
1604
|
- **D80 — index-time synonym/translation expansion (2026-10-07).** Supersedes D79's query-time second round (removed from `search()`; `SearchStats.secondRound` gone): the curated map (`src/retrieve/synonyms.ts` — EN dev-domain groups, ZH groups and EN↔ZH translation pairs merged into one bidirectional group list) is applied at sync/index time. `writeRow` expands each memory's keywords via `expansionVariantsForDoc()` and appends the variants to the indexed `aliases` column (FTS and CJK-bigram columns both pick them up), so any related word matches in a plain round-1 query. Query side: single FTS round, no trigger, no second chance. Keyword-centric per Stone ("重要的是关键词"): Latin content terms (same stopword filter as the query builder, so both sides agree on what a keyword is) + CJK substring matching + multi-word Latin phrases (`regression test`); variants already present in the text are skipped; capped at 32 per memory to bound index growth. Derived at sync time — the Markdown file stays the source of truth; a map update needs a reindex to backfill. *Rationale: the precise slice (exact EN↔ZH pairs like 单点登录/SSO) must never be missed, and embeddings are fuzzy where the map is exact; paying the cost once at write time keeps every query simple and trigger-free.* Eval on the 41-query cross-lingual fixture: recall@1 0.732 → 0.756, recall@5 0.854 → **0.976**, MRR 0.791 → **0.859** — all 5 EN↔ZH vocabulary misses rescued at rank 1–3. Known trade-off: a single-keyword query can rank an aliases-only hit below shorter docs sharing the variant (`how do we ship`: rank 2 → 8 — its hit is 1 of 15 variants on a long aliases field); accepted, the old rank depended on the fragile thin-corpus trigger. `expandQueryWithSynonyms` kept as a bench-only export.
|
|
1605
|
+
- **D81 — user-defined alias vocabulary as query-time expansion (2026-10-07).** Stone: an alias should be a *kind of memory* — defined once, remembered forever. Any memory type may carry `alias:` (the nickname, e.g. 香蕉计划) and `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter; the alias memory defines vocabulary once per scope. At query time (`src/retrieve/alias-memory.ts`), query terms are looked up against the alias registry — one cheap `SELECT` per `search()` call over rows with non-empty alias+target (trigger A: exact registry hit, never a count heuristic; the D79 0/470 trigger is not repeated). On a hit the query is rewritten with the other side of the relation — bidirectional (alias→target and target→alias), depth < 3, cycle-safe — and re-run as round 2 (round 3 for chains). Rounds merge by normalized-max (`mergeRounds`): each round's BM25 scores are divided by that round's best (raw cross-round scores are incomparable — °C vs °F), then final = max, so an alias-side champion competes evenly with a literal-side champion — alias hits are never down-weighted for coming from a later round. Merged pool is re-ranked then cut to [1.5K, 2K] (Stone) — never back to K, or round-2 hits would be starved the way D79's slice-to-K starved them. No reindex when an alias changes (query-time). Scope-aware: a project codename stays in its project. *Rationale: D80's curated map is static and shared — personal vocabulary (小宝贝, 香蕉计划) doesn't belong in it; per-memory D61 aliases don't propagate to new memories. Alias memories close the gap: user-defined, scope-aware, effective immediately.* v1: user-explicit creation only (`memory_add` alias/target params, CLI `--alias`/`--target`, frontmatter edit); agent-proposed aliases deferred. D81 also rewords the init `captureAliases` prompt (it is not the synonym switch — D80's expansion always applies). Eval: 3 new fixture queries (香蕉计划↔支付系统重构项目 both directions + the alias memory itself), 44 queries total: recall@1 0.750, recall@5 **0.977** (43/44), MRR 0.857 — the single miss remains `how do we ship` (known D80 trade-off).
|
|
1606
|
+
|
|
1605
1607
|
|
|
1606
1608
|
## Open Questions
|
|
1607
1609
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "open-memex",
|
|
3
|
-
"version": "0.8.0-alpha.
|
|
3
|
+
"version": "0.8.0-alpha.3",
|
|
4
4
|
"mcpName": "io.github.stoneskin/open-memex",
|
|
5
5
|
"description": "Local-first memory layer and protocol for AI coding agents. Markdown source of truth, SQLite FTS5 index, zero cloud.",
|
|
6
6
|
"keywords": [
|
|
@@ -147,7 +147,7 @@ function rankOf(
|
|
|
147
147
|
q: LMEItem,
|
|
148
148
|
isGold: (id: string) => boolean,
|
|
149
149
|
): number {
|
|
150
|
-
const stats = { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0 };
|
|
150
|
+
const stats = { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0, aliasExpansions: [] as string[] };
|
|
151
151
|
const hits = search(q.question, {
|
|
152
152
|
scopeKeys: ["personal"],
|
|
153
153
|
limit: K,
|
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Synthetic retrieval-eval fixture (D77).
|
|
3
3
|
*
|
|
4
|
-
*
|
|
4
|
+
* 54 personal-scope memories + 44 queries with gold ids. Written BEFORE
|
|
5
5
|
* running the eval — the fixture is the test, not tuned to the results.
|
|
6
6
|
* All memories share one scope/tier so the measured ranking is pure BM25.
|
|
7
7
|
* Mixed EN/ZH; includes paraphrase and synonym queries (the hard cases
|
|
8
|
-
* from the retrieval-robustness question)
|
|
8
|
+
* from the retrieval-robustness question), near-duplicate distractors,
|
|
9
|
+
* and D81 user-defined alias vocabulary (bidirectional query-time
|
|
10
|
+
* expansion: 香蕉计划 ↔ 支付系统重构项目).
|
|
9
11
|
*/
|
|
10
12
|
|
|
11
13
|
export interface FixtureMemory {
|
|
@@ -13,6 +15,9 @@ export interface FixtureMemory {
|
|
|
13
15
|
type: string;
|
|
14
16
|
tags: string[];
|
|
15
17
|
aliases?: string[];
|
|
18
|
+
/** D81: user-defined alias vocabulary (nickname → referent). */
|
|
19
|
+
alias?: string;
|
|
20
|
+
target?: string;
|
|
16
21
|
body: string;
|
|
17
22
|
}
|
|
18
23
|
|
|
@@ -103,6 +108,11 @@ export const MEMORIES: FixtureMemory[] = [
|
|
|
103
108
|
{ id: "mem-049", type: "fact", tags: ["备份", "数据库"], body: "数据库备份每天凌晨两点执行,全量保留三十天,恢复演练每个季度做一次。" },
|
|
104
109
|
{ id: "mem-050", type: "decision", tags: ["发布", "测试"], body: "上线前必须跑完全量回归测试,测试报告要贴到发布群里,TL 确认后才能发。" },
|
|
105
110
|
{ id: "mem-051", type: "gotcha", tags: ["staging"], body: "The staging environment wipes itself every night at midnight. Never leave test data there overnight — it will be gone by morning." },
|
|
111
|
+
|
|
112
|
+
// ---- D81 user-defined alias vocabulary (query-time expansion) ----
|
|
113
|
+
{ id: "mem-052", type: "fact", tags: ["alias"], alias: "香蕉计划", target: "支付系统重构项目", body: "香蕉计划是支付系统重构项目的内部代号,团队日常用香蕉计划指代这个项目。" },
|
|
114
|
+
{ id: "mem-053", type: "decision", tags: ["backend"], body: "支付系统重构项目决定用 Go 重写网关层,Q4 启动。" },
|
|
115
|
+
{ id: "mem-054", type: "fact", tags: ["planning"], body: "香蕉计划下周一启动评审,材料周五前提交。" },
|
|
106
116
|
];
|
|
107
117
|
|
|
108
118
|
export const QUERIES: FixtureQuery[] = [
|
|
@@ -164,4 +174,9 @@ export const QUERIES: FixtureQuery[] = [
|
|
|
164
174
|
{ query: "how do we ship", gold: ["mem-001"], note: "synonym-only: ship→deploy/release" },
|
|
165
175
|
{ query: "pager schedule", gold: ["mem-042"], note: "synonym-only: pager→oncall" },
|
|
166
176
|
{ query: "my token got exposed", gold: ["mem-017"], note: "synonym-only: token→secret" },
|
|
177
|
+
|
|
178
|
+
// ---- D81 alias-memory queries (bidirectional query-time expansion) ----
|
|
179
|
+
{ query: "香蕉计划用什么语言重写网关", gold: ["mem-053"], note: "alias→target: round 2 must find mem-053 via 支付系统重构项目" },
|
|
180
|
+
{ query: "支付系统重构项目什么时候启动评审", gold: ["mem-054"], note: "target→alias: round 2 must find mem-054 via 香蕉计划" },
|
|
181
|
+
{ query: "香蕉计划是哪个项目的代号", gold: ["mem-052"], note: "alias memory itself, round 1" },
|
|
167
182
|
];
|
|
@@ -37,6 +37,8 @@ for (const m of MEMORIES) {
|
|
|
37
37
|
type: m.type,
|
|
38
38
|
tags: m.tags,
|
|
39
39
|
aliases: m.aliases,
|
|
40
|
+
alias: m.alias,
|
|
41
|
+
target: m.target,
|
|
40
42
|
source: "retrieval-eval fixture",
|
|
41
43
|
});
|
|
42
44
|
fs.writeFileSync(path.join(dir, `${m.id}.md`), serialize(fm, m.body), "utf8");
|
|
@@ -51,7 +53,7 @@ let mrr = 0;
|
|
|
51
53
|
const misses: string[] = [];
|
|
52
54
|
|
|
53
55
|
for (const q of QUERIES) {
|
|
54
|
-
const stats = { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0 };
|
|
56
|
+
const stats = { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0, aliasExpansions: [] as string[] };
|
|
55
57
|
const hits = search(q.query, { scopeKeys: ["personal"], limit: 50, stats });
|
|
56
58
|
const ids = hits.map((h) => h.id);
|
|
57
59
|
let rank = -1;
|
package/scripts/smoke-pure.ts
CHANGED
|
@@ -9,6 +9,12 @@ import { resolveProjectScope, resolveCwdScope, pickScopeRoot, PERSONAL_SCOPE } f
|
|
|
9
9
|
import { cjkIndexText, cjkQueryExpr, hasCjk } from "../src/retrieve/cjk.ts";
|
|
10
10
|
import { toFtsQuery } from "../src/retrieve/query.ts";
|
|
11
11
|
import { expansionVariantsForDoc, MAX_AUTO_VARIANTS } from "../src/retrieve/synonyms.ts";
|
|
12
|
+
import {
|
|
13
|
+
planAliasExpansions,
|
|
14
|
+
rewriteQueryOnce,
|
|
15
|
+
mergeRounds,
|
|
16
|
+
type AliasEntry,
|
|
17
|
+
} from "../src/retrieve/alias-memory.ts";
|
|
12
18
|
import { contentHash, similarity, NEAR_DUP_THRESHOLD } from "../src/store/lifecycle.ts";
|
|
13
19
|
import {
|
|
14
20
|
MIN_NODE_VERSION,
|
|
@@ -894,5 +900,54 @@ console.log("== D80 index-time expansion ==");
|
|
|
894
900
|
ok("no curated terms → no variants", expansionVariantsForDoc("hello world").length === 0);
|
|
895
901
|
}
|
|
896
902
|
|
|
903
|
+
console.log("== D81 alias-memory query-time expansion ==");
|
|
904
|
+
{
|
|
905
|
+
const entries: AliasEntry[] = [
|
|
906
|
+
{ id: "a1", scope_key: "personal", alias: "香蕉计划", target: "支付系统重构项目" },
|
|
907
|
+
{ id: "a2", scope_key: "personal", alias: "小宝贝", target: "AI agent" },
|
|
908
|
+
];
|
|
909
|
+
// alias → target.
|
|
910
|
+
const r1 = rewriteQueryOnce("香蕉计划进展如何", entries);
|
|
911
|
+
ok("alias rewrites to target", r1 === "支付系统重构项目进展如何", String(r1));
|
|
912
|
+
// target → alias (bidirectional).
|
|
913
|
+
const r2 = rewriteQueryOnce("支付系统重构项目什么时候评审", entries);
|
|
914
|
+
ok("target rewrites to alias", r2 === "香蕉计划什么时候评审", String(r2));
|
|
915
|
+
// Latin token match (whole token, not substring).
|
|
916
|
+
const r3 = rewriteQueryOnce("ask 小宝贝 for help", entries);
|
|
917
|
+
ok("latin alias rewrites", r3 === "ask AI agent for help", String(r3));
|
|
918
|
+
ok("latin substring does not match", rewriteQueryOnce("association meeting", [{ id: "x", scope_key: "p", alias: "sso", target: "单点登录" }]) === null);
|
|
919
|
+
// No match → null.
|
|
920
|
+
ok("no alias term → null", rewriteQueryOnce("今天天气不错", entries) === null);
|
|
921
|
+
// Depth < 3: at most two expansion rounds.
|
|
922
|
+
const chained: AliasEntry[] = [
|
|
923
|
+
{ id: "c1", scope_key: "p", alias: "alpha", target: "beta" },
|
|
924
|
+
{ id: "c2", scope_key: "p", alias: "beta", target: "gamma" },
|
|
925
|
+
{ id: "c3", scope_key: "p", alias: "gamma", target: "delta" },
|
|
926
|
+
];
|
|
927
|
+
const plan = planAliasExpansions("alpha status", chained);
|
|
928
|
+
ok("depth capped below 3", plan.length === 2, JSON.stringify(plan));
|
|
929
|
+
// Cycle-safe: alpha↔beta oscillates back to the seen query and stops.
|
|
930
|
+
const cyclic: AliasEntry[] = [
|
|
931
|
+
{ id: "x1", scope_key: "p", alias: "alpha", target: "beta" },
|
|
932
|
+
{ id: "x2", scope_key: "p", alias: "beta", target: "alpha" },
|
|
933
|
+
];
|
|
934
|
+
ok("cycle stops", planAliasExpansions("alpha status", cyclic).length <= 1);
|
|
935
|
+
|
|
936
|
+
// Normalized-max merge: an alias-side champion (n=1.0) outranks a mediocre
|
|
937
|
+
// literal match (n=0.4) — no down-weighting for being from a later round.
|
|
938
|
+
const merged = mergeRounds([
|
|
939
|
+
[{ id: "lit-best", score: 10 }, { id: "lit-mid", score: 4 }],
|
|
940
|
+
[{ id: "alias-best", score: 5 }, { id: "lit-best", score: 2 }],
|
|
941
|
+
]);
|
|
942
|
+
const order = merged.map((m) => m.doc.id);
|
|
943
|
+
ok("alias champion ties literal champion", merged[0].score === 1 && merged[1].score === 1, JSON.stringify(merged.map((m) => [m.doc.id, m.score]))),
|
|
944
|
+
ok("multi-round doc wins ties", order[0] === "lit-best", order.join(","));
|
|
945
|
+
ok("alias-only doc keeps full weight", order[1] === "alias-best", order.join(","));
|
|
946
|
+
ok("mediocre literal ranks last", order[2] === "lit-mid", order.join(","));
|
|
947
|
+
// Single round degrades to plain ordering.
|
|
948
|
+
const single = mergeRounds([[{ id: "a", score: 3 }, { id: "b", score: 9 }]]);
|
|
949
|
+
ok("single round orders by score", single[0].doc.id === "b" && single[0].score === 1, JSON.stringify(single.map((m) => [m.doc.id, m.score])));
|
|
950
|
+
}
|
|
951
|
+
|
|
897
952
|
console.log(fails === 0 ? "\nALL PASS" : `\n${fails} FAILURES`);
|
|
898
953
|
process.exit(fails === 0 ? 0 : 1);
|
package/src/cli.ts
CHANGED
|
@@ -124,7 +124,7 @@ Example:
|
|
|
124
124
|
|
|
125
125
|
add: `Save a fact, preference, decision, or note to local memory.
|
|
126
126
|
|
|
127
|
-
Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"]
|
|
127
|
+
Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"] [--alias NICK --target REFERENT]
|
|
128
128
|
|
|
129
129
|
Flags:
|
|
130
130
|
--scope project (default) or personal (personal never leaves this machine)
|
|
@@ -135,6 +135,11 @@ Flags:
|
|
|
135
135
|
with ; (commas also accepted — use ; when an alias itself
|
|
136
136
|
contains one). Stored only when capture aliases are enabled
|
|
137
137
|
(init default)
|
|
138
|
+
--alias --target D81: define user vocabulary once — e.g. --alias 香蕉计划
|
|
139
|
+
--target 支付系统重构项目. Later queries mentioning either side
|
|
140
|
+
trigger a second search round on the other side (both
|
|
141
|
+
directions), scoped like the memory itself. Not gated by
|
|
142
|
+
capture aliases.
|
|
138
143
|
|
|
139
144
|
Example:
|
|
140
145
|
open-memex add "We deploy on Fridays" --scope project --tag process`,
|
|
@@ -1008,7 +1013,7 @@ async function main() {
|
|
|
1008
1013
|
: [project.key, PERSONAL_SCOPE.key];
|
|
1009
1014
|
const explain = flags.explain === "true";
|
|
1010
1015
|
const stats = explain
|
|
1011
|
-
? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0 }
|
|
1016
|
+
? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0, aliasExpansions: [] as string[] }
|
|
1012
1017
|
: undefined;
|
|
1013
1018
|
const hits = search(query, {
|
|
1014
1019
|
scopeKeys: keys,
|
|
@@ -1021,6 +1026,9 @@ async function main() {
|
|
|
1021
1026
|
console.log(
|
|
1022
1027
|
`candidates: ${stats.candidates}, hidden by lifecycle: ${stats.hiddenSuperseded} superseded, ${stats.hiddenExcluded} retracted/archived`,
|
|
1023
1028
|
);
|
|
1029
|
+
if (stats.aliasExpansions.length > 0) {
|
|
1030
|
+
console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
|
|
1031
|
+
}
|
|
1024
1032
|
}
|
|
1025
1033
|
if (hits.length === 0) {
|
|
1026
1034
|
console.log("(no matches)");
|
|
@@ -1083,6 +1091,14 @@ async function main() {
|
|
|
1083
1091
|
),
|
|
1084
1092
|
}
|
|
1085
1093
|
: {}),
|
|
1094
|
+
// D81: --alias/--target define user vocabulary (not gated by capture
|
|
1095
|
+
// aliases — this is deliberate, like typing the words yourself).
|
|
1096
|
+
...(typeof flags.alias === "string" && flags.alias !== "true" && flags.alias.trim()
|
|
1097
|
+
? { alias: redact(flags.alias.trim(), cfg.redactPatterns).content }
|
|
1098
|
+
: {}),
|
|
1099
|
+
...(typeof flags.target === "string" && flags.target !== "true" && flags.target.trim()
|
|
1100
|
+
? { target: redact(flags.target.trim(), cfg.redactPatterns).content }
|
|
1101
|
+
: {}),
|
|
1086
1102
|
source: "cli",
|
|
1087
1103
|
created_at: rfc,
|
|
1088
1104
|
updated_at: rfc,
|
package/src/init.ts
CHANGED
|
@@ -905,10 +905,12 @@ export async function initProject(opts: {
|
|
|
905
905
|
);
|
|
906
906
|
if (injectOnFirstTurn !== DEFAULT_CONFIG.injectOnFirstTurn)
|
|
907
907
|
patch.injectOnFirstTurn = injectOnFirstTurn;
|
|
908
|
-
// D61: capture-time aliases —
|
|
909
|
-
// memory so differently-worded questions still find it.
|
|
908
|
+
// D61: capture-time aliases — the agent may attach its own alternate
|
|
909
|
+
// phrasings to each memory so differently-worded questions still find it.
|
|
910
|
+
// Distinct from D80's automatic index-time synonym expansion, which
|
|
911
|
+
// applies regardless of this setting.
|
|
910
912
|
const captureAliases = await askBool(
|
|
911
|
-
"
|
|
913
|
+
"Let the agent attach its own alternate phrasings (aliases) to memories? (The built-in synonym map applies regardless.)",
|
|
912
914
|
DEFAULT_CONFIG.captureAliases,
|
|
913
915
|
);
|
|
914
916
|
if (captureAliases !== DEFAULT_CONFIG.captureAliases)
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D81: user-defined alias vocabulary as query-time expansion.
|
|
3
|
+
*
|
|
4
|
+
* An alias memory carries `alias:` (the nickname, e.g. 香蕉计划) and
|
|
5
|
+
* `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter on any
|
|
6
|
+
* memory type. Unlike D80's index-time curated map (static, shared by all
|
|
7
|
+
* users), alias memories are dynamic user vocabulary:
|
|
8
|
+
* - defining one takes effect on the next query — no reindex;
|
|
9
|
+
* - aliases are scope-aware (a project codename stays in its project,
|
|
10
|
+
* a personal nickname stays personal).
|
|
11
|
+
*
|
|
12
|
+
* Search flow (search.ts):
|
|
13
|
+
* 1. Round 1 searches the query as-is.
|
|
14
|
+
* 2. Query terms are looked up against the alias registry (trigger A:
|
|
15
|
+
* an exact registry hit, not a count heuristic — the D79 trigger that
|
|
16
|
+
* fired 0/470 is not repeated).
|
|
17
|
+
* 3. Each expansion round rewrites the query with the other side of the
|
|
18
|
+
* relation — bidirectional (alias→target and target→alias), depth < 3,
|
|
19
|
+
* cycle-safe.
|
|
20
|
+
* 4. Rounds merge by normalized-max (mergeRounds): each round's scores are
|
|
21
|
+
* divided by that round's best, so an alias-side champion competes
|
|
22
|
+
* evenly with a literal-side champion — alias matches are never
|
|
23
|
+
* down-weighted for coming from a later round.
|
|
24
|
+
*/
|
|
25
|
+
import { db } from "../store/db.ts";
|
|
26
|
+
import { latinTerms } from "./query.ts";
|
|
27
|
+
|
|
28
|
+
export interface AliasEntry {
|
|
29
|
+
id: string;
|
|
30
|
+
scope_key: string;
|
|
31
|
+
alias: string;
|
|
32
|
+
target: string;
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* All alias definitions visible to these scopes. One cheap query per
|
|
37
|
+
* search() call — no cache, so a newly saved alias memory applies on the
|
|
38
|
+
* very next query.
|
|
39
|
+
*/
|
|
40
|
+
export function loadAliasEntries(scopeKeys?: string[]): AliasEntry[] {
|
|
41
|
+
const filter =
|
|
42
|
+
scopeKeys && scopeKeys.length > 0
|
|
43
|
+
? `AND scope_key IN (${scopeKeys.map(() => "?").join(",")})`
|
|
44
|
+
: "";
|
|
45
|
+
return db()
|
|
46
|
+
.prepare(
|
|
47
|
+
`SELECT id, scope_key, alias, target FROM memories
|
|
48
|
+
WHERE alias <> '' AND target <> '' AND status = 'active' ${filter}`,
|
|
49
|
+
)
|
|
50
|
+
.all(...((scopeKeys ?? []) as unknown[])) as AliasEntry[];
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
const CJK_RE = /[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]/;
|
|
54
|
+
|
|
55
|
+
function escapeRegExp(s: string): string {
|
|
56
|
+
return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
/** CJK: substring (no tokenizer). Latin: whole-token match. */
|
|
60
|
+
function termInQuery(query: string, term: string): boolean {
|
|
61
|
+
if (CJK_RE.test(term)) return query.includes(term);
|
|
62
|
+
const tokens = new Set(latinTerms(query));
|
|
63
|
+
return tokens.has(term.toLowerCase());
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
function replaceTerm(query: string, term: string, replacement: string): string {
|
|
67
|
+
if (CJK_RE.test(term)) return query.replace(term, replacement);
|
|
68
|
+
return query.replace(
|
|
69
|
+
new RegExp(`\\b${escapeRegExp(term)}\\b`, "i"),
|
|
70
|
+
replacement,
|
|
71
|
+
);
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
/**
|
|
75
|
+
* Rewrite the query once: every alias entry with a matching side swaps in
|
|
76
|
+
* the other side. All matching entries apply in one pass (a chain A→B→C
|
|
77
|
+
* can resolve within a single round). Returns null when nothing matched.
|
|
78
|
+
*/
|
|
79
|
+
export function rewriteQueryOnce(
|
|
80
|
+
query: string,
|
|
81
|
+
entries: AliasEntry[],
|
|
82
|
+
): string | null {
|
|
83
|
+
let out = query;
|
|
84
|
+
let changed = false;
|
|
85
|
+
for (const e of entries) {
|
|
86
|
+
if (termInQuery(out, e.alias)) {
|
|
87
|
+
out = replaceTerm(out, e.alias, e.target);
|
|
88
|
+
changed = true;
|
|
89
|
+
} else if (termInQuery(out, e.target)) {
|
|
90
|
+
out = replaceTerm(out, e.target, e.alias);
|
|
91
|
+
changed = true;
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
return changed ? out : null;
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
/**
|
|
98
|
+
* Plan the expansion rounds for a query. Pure — takes the registry entries
|
|
99
|
+
* explicitly so tests don't need a database. Returns the rewritten queries
|
|
100
|
+
* only (rounds 2+); the caller prepends the original query. Depth < 3:
|
|
101
|
+
* at most two expansion rounds. Cycle-safe via the seen set.
|
|
102
|
+
*/
|
|
103
|
+
export function planAliasExpansions(
|
|
104
|
+
query: string,
|
|
105
|
+
entries: AliasEntry[],
|
|
106
|
+
): string[] {
|
|
107
|
+
const out: string[] = [];
|
|
108
|
+
const seen = new Set([query]);
|
|
109
|
+
let current = query;
|
|
110
|
+
for (let depth = 0; depth < 2; depth++) {
|
|
111
|
+
const next = rewriteQueryOnce(current, entries);
|
|
112
|
+
if (!next || seen.has(next)) break;
|
|
113
|
+
seen.add(next);
|
|
114
|
+
out.push(next);
|
|
115
|
+
current = next;
|
|
116
|
+
}
|
|
117
|
+
return out;
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
/**
|
|
121
|
+
* D81 merge: per-round score normalization + max.
|
|
122
|
+
*
|
|
123
|
+
* BM25 scores are query-dependent (different terms → different IDFs and
|
|
124
|
+
* query lengths), so raw scores from two rounds are incomparable — like
|
|
125
|
+
* comparing °C with °F. Each round is first divided by its own best
|
|
126
|
+
* (n=1.0 means "this round's champion"), then final = max across rounds.
|
|
127
|
+
* An alias-side champion (n=1.0) outranks a mediocre literal match
|
|
128
|
+
* (n=0.4): the two sides' champions start even, which is exactly "no
|
|
129
|
+
* down-weighting for being a synonym/alias". The kept row (and its
|
|
130
|
+
* snippet) comes from the round where the doc scored its best.
|
|
131
|
+
*/
|
|
132
|
+
export function mergeRounds<T extends { id: string; score: number }>(
|
|
133
|
+
rounds: T[][],
|
|
134
|
+
): Array<{ doc: T; score: number; rounds: number }> {
|
|
135
|
+
const acc = new Map<string, { doc: T; best: number; rounds: number }>();
|
|
136
|
+
rounds.forEach((docs) => {
|
|
137
|
+
const mx = docs.reduce((m, d) => Math.max(m, d.score), 0);
|
|
138
|
+
for (const d of docs) {
|
|
139
|
+
const n = mx > 0 ? d.score / mx : 0;
|
|
140
|
+
const cur = acc.get(d.id);
|
|
141
|
+
if (!cur) acc.set(d.id, { doc: d, best: n, rounds: 1 });
|
|
142
|
+
else {
|
|
143
|
+
cur.rounds += 1;
|
|
144
|
+
if (n > cur.best) {
|
|
145
|
+
cur.best = n;
|
|
146
|
+
cur.doc = d;
|
|
147
|
+
}
|
|
148
|
+
}
|
|
149
|
+
}
|
|
150
|
+
});
|
|
151
|
+
return [...acc.values()]
|
|
152
|
+
.map((v) => ({ doc: v.doc, score: v.best, rounds: v.rounds }))
|
|
153
|
+
.sort((a, b) => b.score - a.score || b.rounds - a.rounds);
|
|
154
|
+
}
|
package/src/retrieve/query.ts
CHANGED
|
@@ -32,7 +32,7 @@ const CJK_QUESTION_BIGRAMS = new Set([
|
|
|
32
32
|
|
|
33
33
|
const MAX_TERMS = 24;
|
|
34
34
|
|
|
35
|
-
function latinTerms(q: string, useStopwords = true): string[] {
|
|
35
|
+
export function latinTerms(q: string, useStopwords = true): string[] {
|
|
36
36
|
const raw = q.toLowerCase().match(/[a-z0-9_.\-]+/g) ?? [];
|
|
37
37
|
const kept = useStopwords
|
|
38
38
|
? raw.filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t))
|
package/src/retrieve/search.ts
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
import { db } from "../store/db.ts";
|
|
2
2
|
import { toFtsQuery } from "./query.ts";
|
|
3
|
+
import {
|
|
4
|
+
loadAliasEntries,
|
|
5
|
+
planAliasExpansions,
|
|
6
|
+
mergeRounds,
|
|
7
|
+
} from "./alias-memory.ts";
|
|
3
8
|
|
|
4
9
|
export interface SearchHit {
|
|
5
10
|
id: string;
|
|
@@ -88,6 +93,8 @@ export interface SearchStats {
|
|
|
88
93
|
candidates: number;
|
|
89
94
|
hiddenSuperseded: number;
|
|
90
95
|
hiddenExcluded: number;
|
|
96
|
+
/** D81: query-time alias expansions actually run (rounds 2+, raw text). */
|
|
97
|
+
aliasExpansions: string[];
|
|
91
98
|
}
|
|
92
99
|
|
|
93
100
|
/**
|
|
@@ -176,6 +183,7 @@ export function search(
|
|
|
176
183
|
opts.stats.candidates = 0;
|
|
177
184
|
opts.stats.hiddenSuperseded = 0;
|
|
178
185
|
opts.stats.hiddenExcluded = 0;
|
|
186
|
+
opts.stats.aliasExpansions = [];
|
|
179
187
|
}
|
|
180
188
|
if (!q) return [];
|
|
181
189
|
const limit = Math.max(1, Math.min(opts.limit ?? 8, 50));
|
|
@@ -229,11 +237,38 @@ export function search(
|
|
|
229
237
|
|
|
230
238
|
const raw: RawRow[] = fetchFts(q);
|
|
231
239
|
|
|
232
|
-
//
|
|
233
|
-
//
|
|
234
|
-
//
|
|
235
|
-
|
|
236
|
-
|
|
240
|
+
// D81: user-defined alias vocabulary — query-time expansion rounds.
|
|
241
|
+
// Round 1 is the query as-is; each further round rewrites alias terms to
|
|
242
|
+
// the other side of the relation (bidirectional, depth < 3). The trigger
|
|
243
|
+
// is an exact registry hit (loadAliasEntries), not a count heuristic.
|
|
244
|
+
const aliasEntries = loadAliasEntries(opts.scopeKeys);
|
|
245
|
+
const expansions = planAliasExpansions(query, aliasEntries);
|
|
246
|
+
let merged: RawRow[] = raw;
|
|
247
|
+
let finalLimit = limit;
|
|
248
|
+
if (expansions.length > 0) {
|
|
249
|
+
const rounds: RawRow[][] = [raw];
|
|
250
|
+
for (const eq of expansions) {
|
|
251
|
+
const mq = toFtsQuery(eq, qflags);
|
|
252
|
+
if (mq) rounds.push(fetchFts(mq));
|
|
253
|
+
}
|
|
254
|
+
// Normalized-max merge (alias-memory.ts): each round's scores are
|
|
255
|
+
// divided by that round's best, so an alias-side champion competes
|
|
256
|
+
// evenly with a literal-side champion — no down-weighting for being
|
|
257
|
+
// from a later round.
|
|
258
|
+
merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
|
|
259
|
+
// Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
|
|
260
|
+
// starved the way D79's slice-to-K starved them.
|
|
261
|
+
const pool = merged.length;
|
|
262
|
+
finalLimit = Math.min(pool, 2 * limit);
|
|
263
|
+
finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
|
|
264
|
+
if (opts.stats) opts.stats.aliasExpansions = expansions;
|
|
265
|
+
}
|
|
266
|
+
|
|
267
|
+
// D80: the curated static map is solved at index time (synonyms.ts
|
|
268
|
+
// expands variants into the aliases column at sync time), so without a
|
|
269
|
+
// user alias the query side stays a single FTS round.
|
|
270
|
+
if (opts.stats) opts.stats.candidates = merged.length;
|
|
271
|
+
return resolveVisible(merged, finalLimit, opts.stats);
|
|
237
272
|
}
|
|
238
273
|
|
|
239
274
|
export function list(
|
package/src/store/db.ts
CHANGED
|
@@ -48,6 +48,8 @@ CREATE TABLE IF NOT EXISTS memories (
|
|
|
48
48
|
status TEXT NOT NULL DEFAULT 'active',
|
|
49
49
|
tags TEXT NOT NULL DEFAULT '',
|
|
50
50
|
aliases TEXT NOT NULL DEFAULT '',
|
|
51
|
+
alias TEXT NOT NULL DEFAULT '',
|
|
52
|
+
target TEXT NOT NULL DEFAULT '',
|
|
51
53
|
content TEXT NOT NULL,
|
|
52
54
|
cjk TEXT NOT NULL DEFAULT '',
|
|
53
55
|
content_hash TEXT NOT NULL DEFAULT '',
|
|
@@ -103,7 +105,7 @@ END;
|
|
|
103
105
|
`;
|
|
104
106
|
|
|
105
107
|
/** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
|
|
106
|
-
const SCHEMA_VERSION =
|
|
108
|
+
const SCHEMA_VERSION = 8;
|
|
107
109
|
|
|
108
110
|
function userVersion(d: AnyDatabase): number {
|
|
109
111
|
const row = d.prepare("PRAGMA user_version").get() as { user_version: number };
|
package/src/store/lifecycle.ts
CHANGED
|
@@ -260,6 +260,9 @@ export interface SupersedeInput {
|
|
|
260
260
|
type?: string;
|
|
261
261
|
tags?: string[];
|
|
262
262
|
aliases?: string[];
|
|
263
|
+
/** D81: undefined = inherit the old memory's pair. */
|
|
264
|
+
alias?: string;
|
|
265
|
+
target?: string;
|
|
263
266
|
source?: string;
|
|
264
267
|
}
|
|
265
268
|
|
|
@@ -288,6 +291,9 @@ export function supersede(
|
|
|
288
291
|
tags: input.tags ?? oldMf.fm.tags,
|
|
289
292
|
// D61: aliases carry forward unless the caller replaces them.
|
|
290
293
|
aliases: input.aliases ?? oldMf.fm.aliases,
|
|
294
|
+
// D81: alias vocabulary carries forward the same way.
|
|
295
|
+
alias: input.alias ?? oldMf.fm.alias,
|
|
296
|
+
target: input.target ?? oldMf.fm.target,
|
|
291
297
|
source: input.source ?? oldMf.fm.source,
|
|
292
298
|
status: "active",
|
|
293
299
|
created_at: now,
|
package/src/store/markdown.ts
CHANGED
|
@@ -60,6 +60,17 @@ export interface Frontmatter {
|
|
|
60
60
|
* Optional and additive: files without it parse unchanged.
|
|
61
61
|
*/
|
|
62
62
|
aliases?: string[];
|
|
63
|
+
/**
|
|
64
|
+
* D81: user-defined alias vocabulary — `alias` is the nickname
|
|
65
|
+
* (e.g. 香蕉计划), `target` is what it refers to
|
|
66
|
+
* (e.g. 支付系统重构项目). Unlike D61 per-memory aliases (alternate
|
|
67
|
+
* phrasings of one memory), an alias memory defines vocabulary once and
|
|
68
|
+
* the query-time expansion (retrieve/alias-memory.ts) applies it to every
|
|
69
|
+
* search in its scope — no reindex when the alias changes. Optional;
|
|
70
|
+
* files without it parse unchanged.
|
|
71
|
+
*/
|
|
72
|
+
alias?: string;
|
|
73
|
+
target?: string;
|
|
63
74
|
source: string;
|
|
64
75
|
created_at: string; // RFC 3339, never bare epoch (§3)
|
|
65
76
|
updated_at: string; // RFC 3339
|
|
@@ -247,6 +258,13 @@ export function normalizeFrontmatter(
|
|
|
247
258
|
return aliases.length > 0 ? { aliases } : {};
|
|
248
259
|
})()
|
|
249
260
|
: {}),
|
|
261
|
+
// D81: alias vocabulary — single trimmed strings, dropped when empty.
|
|
262
|
+
...(typeof raw.alias === "string" && raw.alias.trim()
|
|
263
|
+
? { alias: raw.alias.trim() }
|
|
264
|
+
: {}),
|
|
265
|
+
...(typeof raw.target === "string" && raw.target.trim()
|
|
266
|
+
? { target: raw.target.trim() }
|
|
267
|
+
: {}),
|
|
250
268
|
source: typeof raw.source === "string" ? raw.source : "",
|
|
251
269
|
created_at: msToRfc3339(timeToMs(raw.created_at)),
|
|
252
270
|
updated_at: msToRfc3339(timeToMs(raw.updated_at)),
|
package/src/store/sync.ts
CHANGED
|
@@ -15,8 +15,8 @@ import { projectRoot, paths } from "../paths.ts";
|
|
|
15
15
|
import { loadConfig } from "../config.ts";
|
|
16
16
|
|
|
17
17
|
const UPSERT_SQL = `
|
|
18
|
-
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
19
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
18
|
+
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
19
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
20
20
|
ON CONFLICT(id) DO UPDATE SET
|
|
21
21
|
scope_key = excluded.scope_key,
|
|
22
22
|
scope = excluded.scope,
|
|
@@ -28,6 +28,8 @@ const UPSERT_SQL = `
|
|
|
28
28
|
status = excluded.status,
|
|
29
29
|
tags = excluded.tags,
|
|
30
30
|
aliases = excluded.aliases,
|
|
31
|
+
alias = excluded.alias,
|
|
32
|
+
target = excluded.target,
|
|
31
33
|
content = excluded.content,
|
|
32
34
|
cjk = excluded.cjk,
|
|
33
35
|
content_hash = excluded.content_hash,
|
|
@@ -74,6 +76,7 @@ function writeRow(mf: MemoryFile, mtimeMs: number): void {
|
|
|
74
76
|
(a) => !have.has(a.toLowerCase()),
|
|
75
77
|
);
|
|
76
78
|
const aliases = [...supplied, ...auto].join(",");
|
|
79
|
+
// D81: user-defined alias vocabulary (single nickname → referent pair).
|
|
77
80
|
db().prepare(UPSERT_SQL).run(
|
|
78
81
|
fm.id,
|
|
79
82
|
fm.scope_key,
|
|
@@ -86,6 +89,8 @@ function writeRow(mf: MemoryFile, mtimeMs: number): void {
|
|
|
86
89
|
fm.status,
|
|
87
90
|
tags,
|
|
88
91
|
aliases,
|
|
92
|
+
fm.alias ?? "",
|
|
93
|
+
fm.target ?? "",
|
|
89
94
|
body,
|
|
90
95
|
cjkIndexText(body + "\n" + tags + "\n" + aliases),
|
|
91
96
|
contentHash(body),
|
package/src/tools/ops.ts
CHANGED
|
@@ -113,6 +113,18 @@ export const memoryAddArgs = {
|
|
|
113
113
|
scope: scopeArg,
|
|
114
114
|
tags: z.array(z.string()).optional().describe("Optional tags for filtering."),
|
|
115
115
|
aliases: aliasesArg,
|
|
116
|
+
alias: z
|
|
117
|
+
.string()
|
|
118
|
+
.optional()
|
|
119
|
+
.describe(
|
|
120
|
+
"D81: nickname this memory defines (e.g. 香蕉计划). Pairs with `target`: the alias memory expands queries in its scope at search time, both directions, no reindex. Explicit vocabulary — always stored, not gated by capture aliases.",
|
|
121
|
+
),
|
|
122
|
+
target: z
|
|
123
|
+
.string()
|
|
124
|
+
.optional()
|
|
125
|
+
.describe(
|
|
126
|
+
"D81: what `alias` refers to (e.g. 支付系统重构项目). Only meaningful with `alias`.",
|
|
127
|
+
),
|
|
116
128
|
source: z
|
|
117
129
|
.string()
|
|
118
130
|
.optional()
|
|
@@ -163,6 +175,18 @@ export const memorySupersedeArgs = {
|
|
|
163
175
|
.describe("Category of the new memory. Defaults to the old memory's type."),
|
|
164
176
|
tags: z.array(z.string()).optional().describe("Optional tags for the new memory."),
|
|
165
177
|
aliases: aliasesArg,
|
|
178
|
+
alias: z
|
|
179
|
+
.string()
|
|
180
|
+
.optional()
|
|
181
|
+
.describe(
|
|
182
|
+
"D81: nickname the new memory defines (e.g. 香蕉计划). Omitted = inherit the old memory's alias.",
|
|
183
|
+
),
|
|
184
|
+
target: z
|
|
185
|
+
.string()
|
|
186
|
+
.optional()
|
|
187
|
+
.describe(
|
|
188
|
+
"D81: what `alias` refers to (e.g. 支付系统重构项目). Omitted = inherit.",
|
|
189
|
+
),
|
|
166
190
|
};
|
|
167
191
|
export type MemorySupersedeArgs = z.infer<z.ZodObject<typeof memorySupersedeArgs>>;
|
|
168
192
|
|
|
@@ -242,6 +266,8 @@ function buildFrontmatter(
|
|
|
242
266
|
type: Frontmatter["type"];
|
|
243
267
|
tags: string[];
|
|
244
268
|
aliases?: string[];
|
|
269
|
+
alias?: string;
|
|
270
|
+
target?: string;
|
|
245
271
|
source: Frontmatter["source"];
|
|
246
272
|
},
|
|
247
273
|
): Frontmatter {
|
|
@@ -259,6 +285,11 @@ function buildFrontmatter(
|
|
|
259
285
|
status: "active",
|
|
260
286
|
tags: body.tags,
|
|
261
287
|
...(body.aliases && body.aliases.length > 0 ? { aliases: body.aliases } : {}),
|
|
288
|
+
// D81: explicit alias vocabulary — single trimmed strings, dropped
|
|
289
|
+
// when empty. Not gated by captureAliases (that gates agent-invented
|
|
290
|
+
// phrasings; this is deliberate vocabulary).
|
|
291
|
+
...(body.alias?.trim() ? { alias: body.alias.trim() } : {}),
|
|
292
|
+
...(body.target?.trim() ? { target: body.target.trim() } : {}),
|
|
262
293
|
source: body.source,
|
|
263
294
|
created_at: rfc,
|
|
264
295
|
updated_at: rfc,
|
|
@@ -328,6 +359,14 @@ export async function addMemory(
|
|
|
328
359
|
(args.aliases ?? []).map((a) => redact(a, cfg.redactPatterns).content),
|
|
329
360
|
)
|
|
330
361
|
: [],
|
|
362
|
+
// D81: explicit alias vocabulary — same redaction as content, never
|
|
363
|
+
// gated by captureAliases.
|
|
364
|
+
alias: args.alias?.trim()
|
|
365
|
+
? redact(args.alias.trim(), cfg.redactPatterns).content
|
|
366
|
+
: undefined,
|
|
367
|
+
target: args.target?.trim()
|
|
368
|
+
? redact(args.target.trim(), cfg.redactPatterns).content
|
|
369
|
+
: undefined,
|
|
331
370
|
source: args.source ?? "tool",
|
|
332
371
|
});
|
|
333
372
|
const { filePath } = writeMemoryFile(fm, redacted);
|
|
@@ -447,6 +486,16 @@ export async function supersedeMemory(
|
|
|
447
486
|
args.aliases.map((a) => redact(a, cfg.redactPatterns).content),
|
|
448
487
|
)
|
|
449
488
|
: undefined,
|
|
489
|
+
// D81: explicit alias vocabulary — omitted inherits the old memory's
|
|
490
|
+
// pair; same redaction as content; never gated by captureAliases.
|
|
491
|
+
alias:
|
|
492
|
+
args.alias !== undefined
|
|
493
|
+
? redact(args.alias, cfg.redactPatterns).content || undefined
|
|
494
|
+
: undefined,
|
|
495
|
+
target:
|
|
496
|
+
args.target !== undefined
|
|
497
|
+
? redact(args.target, cfg.redactPatterns).content || undefined
|
|
498
|
+
: undefined,
|
|
450
499
|
source: "tool",
|
|
451
500
|
});
|
|
452
501
|
upsertFromFile(oldMf);
|