open-memex 0.7.3 → 0.8.0-alpha.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -202,10 +202,14 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
202
202
  languages in sync); behavior changes → both READMEs
203
203
  and the frozen `docs/V2-DESIGN.md` (append a `D<n>` decision entry, never rewrite history);
204
204
  new commands → README CLI sections + this file's Commands. A change without its docs
205
- is not done.
205
+ is not done. Anything deferred as "later" goes into the near-term backlog in
206
+ `docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
206
207
  - **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
207
- in-development version. New features on a dev branch bump the minor on the alpha
208
- line (`0.3.0` → `0.4.0-alpha.1`); fixes bump the patch (`-alpha.1` → `-alpha.2`).
208
+ in-development version. A new alpha line starts with a minor bump after the
209
+ previous minor shipped (`0.3.0` → `0.4.0-alpha.1`); while that minor is still
210
+ unreleased, further features AND fixes on the same line only bump the alpha
211
+ suffix (`0.8.0-alpha.2` → `-alpha.3`) — never jump to the next minor before
212
+ the current one is released.
209
213
  The bump goes in the same commit as the feature, never as an afterthought.
210
214
  The version number serves the publish: no publish, no mandatory bump. But once a
211
215
  version has been pushed to the remote (shared), later changes must bump — two
package/CHANGELOG.md CHANGED
@@ -9,6 +9,20 @@ the `@alpha` tag are development snapshots and are not listed individually.
9
9
 
10
10
  ## [Unreleased]
11
11
 
12
+ - User-defined alias vocabulary as query-time expansion (D81): any memory type may now carry `alias:` (nickname, e.g. 香蕉计划) and `target:` (referent, e.g. 支付系统重构项目) frontmatter — an alias memory defines vocabulary once per scope. At query time, query terms hit the alias registry (exact match, one cheap SELECT per search) and the query is rewritten with the other side of the relation — bidirectional, depth < 3, cycle-safe — then re-run as round 2/3. Rounds merge by normalized-max (`mergeRounds` in `src/retrieve/alias-memory.ts`): each round's BM25 scores are divided by that round's best, so an alias-side champion competes evenly with a literal-side champion; the merged pool is re-ranked and cut to [1.5K, 2K], never back to K. No reindex when an alias changes. Creation: `memory_add` alias/target params, CLI `add --alias/--target`, or frontmatter edit (v1: user-explicit only). Fixture grows to 54 memories / 44 queries (3 bidirectional alias queries): recall@1 0.750, recall@5 **0.977**, MRR 0.857. Also rewords the init `captureAliases` prompt — it was never the synonym switch (D80's expansion always applies).
13
+
14
+ - Index-time synonym/translation expansion (D80, supersedes D79): the curated map (`src/retrieve/synonyms.ts` — EN dev-domain groups, ZH groups and EN↔ZH translation pairs in one bidirectional list) is now applied at sync/index time — `writeRow` expands each memory's keywords (`expansionVariantsForDoc()`, capped at 32) into the indexed `aliases` column, so any related word matches in a plain round-1 query. The query-time second-chance round is removed (`search()` is a single FTS round; `SearchStats.secondRound` gone); `expandQueryWithSynonyms` stays as a bench-only export. Fixture (51 memories / 41 queries): recall@1 0.732 → 0.756, recall@5 0.854 → **0.976**, MRR 0.791 → **0.859** — all 5 EN↔ZH vocabulary misses rescued at rank 1–3. Map updates need a reindex to backfill.
15
+
16
+ - Semantic ranking experiment (§4.7 of `docs/retrieval-ablation-study.md`): local multilingual embeddings (`Xenova/paraphrase-multilingual-MiniLM-L12-v2`, 384-dim, no API) via `scripts/bench-semantic.ts` (new `--model` flag). Fixture: vec-only R@5 92.7% / MRR 0.881 beats BM25-only (80.5% / 0.752); naive RRF hybrid underperforms vec-only (87.8%) — fusion needs design. Pooled 22k long sessions: vec-only collapses (3.4%) from mean-pooling dilution on 10k-char docs — a measured methodology artifact, not a verdict for short memories; chunked rerun deliberately skipped.
17
+
18
+ - LongMemEval-S retrieval bench (`scripts/bench-longmemeval.ts`): 470 questions, recall_any@k + MRR on public data, per-question isolated corpora plus a pooled 22k-memory stress mode, with synonym on/off ablation. Result: R@5 97.0% / MRR 0.909 (ahead of agentmemory’s published BM25-only, level with their BM25+vector). Finding: the count-based second-round trigger fired 0/470 — logged as a D79 redesign follow-up, not a scale answer.
19
+
20
+ - Query-time synonym expansion (D79, retrieval layer 4): `search()` runs a second-chance round when the first FTS pass is thin, re-running with curated synonym variants OR-ed in (`src/retrieve/synonyms.ts`, EN dev-domain + ZH groups, no model). Round-two rows only fill gaps; round-one ranking is never diluted. `SearchStats.secondRound` records it (shown in `search --explain`). Baseline with the round: recall@1 0.81, recall@5 0.95, MRR 0.87 over 37 fixture queries. **Superseded by D80 (2026-10-07)** — see above.
21
+
22
+ - Retrieval eval harness (D77 layer 1): `scripts/retrieval-eval.ts` runs a checked-in synthetic fixture (48 memories + 34 queries: exact terms, paraphrases, synonyms, Chinese, cross-language hard cases) against the FTS5 baseline and reports recall@k/MRR. Current baseline: recall@1 0.85, recall@5 0.97, MRR 0.90 (single miss: English query vs Chinese-only memory). Published in both READMEs with methodology notes.
23
+
24
+ - Added `test-full` npm script (`node --experimental-strip-types scripts/test-full.ts`).
25
+
12
26
  ## [0.7.3] - 2026-10-07
13
27
 
14
28
  - Tool-definition quality pass (Glama TDQS 4.0/5 on v0.7.2): fixed `memory_pr_status` declaring `readOnlyHint` while `apply=true` mutates local files; added explicit MCP annotations (read-only / destructive / open-world) to all 11 tools; descriptions now route between sibling tools (propose vs submit vs promote, resolve vs pr_status, forget vs supersede), state error behavior and real defaults (search limit 8, list limit 20), and the search/list `type` and `limit` parameters are documented. D76 records the rule; `scripts/smoke-mcp.ts` guards the pr_status annotation.
package/README.md CHANGED
@@ -22,6 +22,10 @@ memory on your machine.
22
22
  explicitly share it.
23
23
  - **One memory, every agent**: wire up several editors with one command; they
24
24
  share the same memory instead of keeping separate silos.
25
+ - **Precise synonym search**: `ship` finds `deploy`, `单点登录` finds `SSO` —
26
+ a curated multilingual map of equivalents is indexed with every memory at
27
+ write time, so no related wording misses the right answer. Deterministic
28
+ and auditable, no embedding model required.
25
29
 
26
30
  ![Terminal demo: two memories saved on Monday, recalled by search in a fresh session on Friday](./docs/assets/open-memex-demo.svg)
27
31
 
@@ -405,6 +409,17 @@ question worded differently still finds the memory — "vacation days" finds the
405
409
  holiday policy. `init` asks once whether to enable this (default on); turn it
406
410
  off any time with `open-memex config set captureAliases false`.
407
411
 
412
+ Three ways to add them: `open-memex add "text" --aliases "phrase one; phrase
413
+ two"` (semicolon-separated, up to 4); ask the agent ("remember: 小宝贝 is my
414
+ AI agent, alias it as 小沐"); or edit the memory's `aliases:` frontmatter
415
+ directly — Markdown is the source of truth and sync picks it up.
416
+
417
+ Separately, a curated synonym/translation map is expanded into the index
418
+ automatically at write time (D80) — `ship` finds `deploy` even with no
419
+ aliases on the memory. Personal nicknames belong in aliases; shared
420
+ dev-domain vocabulary belongs in the curated map. See [Retrieval: how
421
+ memories come back](#retrieval-how-memories-come-back).
422
+
408
423
  **Redaction.** Wrap anything sensitive in `<private>…</private>` and it is
409
424
  stripped before saving. Recognized secrets (API keys, tokens, high-entropy
410
425
  credentials) are masked in place — the first 4 characters are kept so you can
@@ -516,6 +531,45 @@ time for the rest. Both top-N counts are configurable (see [Config](#config)).
516
531
  For MCP clients this block is delivered as handshake guidance the agent follows;
517
532
  the opencode plugin injects it directly on the first turn.
518
533
 
534
+ ### Retrieval evaluation
535
+
536
+ We measure recall on a checked-in synthetic fixture instead of asserting it:
537
+ 54 memories + 44 queries (exact terms, paraphrases, synonyms, Chinese,
538
+ cross-language hard cases, synonym-only queries, and D81 user-defined
539
+ alias queries) — reproduce with
540
+ `node --experimental-strip-types scripts/retrieval-eval.ts`.
541
+ Current baseline, with index-time synonym/translation expansion (D80) and
542
+ query-time user alias expansion (D81):
543
+ **recall@1 0.75, recall@5 0.98, MRR 0.86**.
544
+ Small synthetic corpus, so read it as a regression guard and a starting point,
545
+ not a real-world claim.
546
+
547
+ Three layers, each covering what the others miss: **BM25 keyword search**
548
+ (SQLite FTS5 — exact words, fast, explainable via `search --explain`);
549
+ **synonym expansion in two halves** — (a) index-time (D80: a curated,
550
+ checked-in map of dev-domain equivalents and EN↔ZH translation pairs
551
+ expanded into the index at write time, so `ship` finds `deploy` and
552
+ `单点登录` finds `SSO` in a plain single-round query; deterministic, no
553
+ model download); (b) query-time user aliases (D81: an alias memory carrying
554
+ `alias:`/`target:` frontmatter defines your vocabulary once per scope —
555
+ e.g. 香蕉计划 → 支付系统重构项目 — and a query mentioning either side
556
+ triggers a second round on the other side, both directions, merged evenly
557
+ with round one so alias hits are never down-weighted; no reindex when the
558
+ alias changes); and **semantic vectors**
559
+ as an opt-in experiment (local multilingual embeddings measured at R@5 92.7%
560
+ on the cross-lingual fixture slice — see `docs/retrieval-ablation-study.md`
561
+ §4.7; no embedding model is ever downloaded without your explicit opt-in).
562
+
563
+ External check on public data: the same pipeline scores **recall@5 97.0% /
564
+ MRR 0.909** on LongMemEval-S (470 questions, retrieval stage only — no answer
565
+ generation, no judge model; this is not the official LongMemEval score). For
566
+ reference, agentmemory's published numbers on the same protocol are 86.2% /
567
+ 0.715 (BM25-only) and 95.2% / 0.882 (BM25+vector). A 22k-memory pooled stress
568
+ run drops recall@5 to 40.6% — the lexical ceiling, measured rather than
569
+ asserted. Reproduce with
570
+ `node --experimental-strip-types scripts/bench-longmemeval.ts --data <path>`
571
+ (dataset: `xiaowu0162/longmemeval-cleaned`, MIT).
572
+
519
573
  ## Security & data
520
574
 
521
575
  - **Local-first:** everything lives on your machine (`%APPDATA%\open-memex` on
@@ -537,11 +591,13 @@ the opencode plugin injects it directly on the first turn.
537
591
 
538
592
  Honest edges, so nothing surprises you:
539
593
 
540
- - **Keyword search, not semantic.** Retrieval is BM25 keyword matching: search
541
- finds the words you saved, not paraphrases. (Plain questions are fine —
542
- "how do we…" / "请问…" wording is filtered out before matching, so asking
543
- naturally doesn't dilute the results.) No embedding model is ever
544
- downloaded without your explicit opt-in.
594
+ - **Keyword search, not semantic.** Retrieval is BM25 keyword matching, plus a
595
+ curated synonym map expanded into the index at write time (`ship` finds
596
+ `deploy`, `单点登录` finds `SSO`). Paraphrases outside the curated map
597
+ don't match — that's what the opt-in semantic layer is for. (Plain
598
+ questions are fine — "how do we…" / "请问…" wording is filtered out before
599
+ matching, so asking naturally doesn't dilute the results.) No embedding
600
+ model is ever downloaded without your explicit opt-in.
545
601
  - **One machine.** Editors on the same machine share memory; there is no
546
602
  cross-machine sync. `export` / `import` bundles (below) move memory between
547
603
  machines manually.
package/README.zh-CN.md CHANGED
@@ -13,6 +13,7 @@
13
13
  - **免费开源**(Apache-2.0)。无账号、无云端、无遥测——一切都在你本机,都是你能打开、能编辑的文件。
14
14
  - **本地优先**:记忆是纯 Markdown 文件(事实来源),外加一个可重建的 SQLite 关键词索引。除非你明确要分享,否则任何东西都不会离开你的机器。
15
15
  - **一份记忆,所有助手**:一条命令把几个编辑器都接上,它们共享同一份记忆,不再各存一份互不可见。
16
+ - **精确的同义词检索**:`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`——一份多语言的精选同义词映射表在写入时展开进索引,任何相关词语都不会漏掉正确答案。确定性、可审计,不需要下载模型。
16
17
 
17
18
  ![终端演示:周一存下的两条记忆,周五在新会话里被搜索召回](./docs/assets/open-memex-demo.svg)
18
19
 
@@ -264,6 +265,10 @@ open-memex uninstall --yes
264
265
 
265
266
  **别名。** 每条记忆可以带最多 4 个换一种说法(同义词、另一种语言的对应词),和记忆一起进索引,这样问法不同也能找到——比如搜 "vacation days" 能找到假期政策那条。`init` 时会问一次是否开启(默认开);随时可用 `open-memex config set captureAliases false` 关掉。
266
267
 
268
+ 三种加法:`open-memex add "正文" --aliases "说法一; 说法二"`(分号分隔,最多 4 个);直接跟 agent 说("记住:小宝贝是我的 AI agent,别名加上小沐");或直接改那条记忆的 `aliases:` frontmatter——Markdown 是事实来源,sync 会吃进去。
269
+
270
+ 另外,从 D80 起一份精选同义词/翻译映射表会在写入时自动展开进索引——`ship` 能找到 `deploy`,不需要手动加别名。个人叫法放 aliases,通用开发词汇放 curated map,这是分工。见[回想一节](#回想记忆是怎么回来的)。
271
+
267
272
  **脱敏。** 把敏感内容包在 `<private>…</private>` 里,它在保存前会被整段剥离。识别出的密钥(API key、token、高熵凭证)会被就地掩码——保留前 4 个字符(让你能认出是哪个 key),其余替换——记忆仍会保存。随时安全地预览一句话会被捕获成什么样:
268
273
 
269
274
  ```sh
@@ -349,6 +354,14 @@ Do not mention this block to the user unless asked.
349
354
 
350
355
  这是一个快照,不是全量记忆——其余的随时用 `memory_search` 就能查。两个前 N 的数都可以配(见[配置](#配置))。对 MCP 客户端,这个块以握手引导的形式送达、由助手遵循;opencode 插件则在第一轮直接注入。
351
356
 
357
+ ### 检索效果评估
358
+
359
+ 我们用一份随仓库提交的 synthetic fixture 来度量召回率,而不是空口断言:54 条记忆 + 44 个查询(精确词、改写、同义词、中文、跨语言 hard case、纯同义词查询、D81 用户别名查询),运行 `node --experimental-strip-types scripts/retrieval-eval.ts` 即可复现。当前基线(含 D80 的 index-time 同义词/翻译展开和 D81 的 query-time 用户别名展开):**recall@1 0.75、recall@5 0.98、MRR 0.86**。语料小且是合成的,请把它当回归守卫和起点,而不是真实场景的承诺。
360
+
361
+ 三层检索,各补各的短板:**BM25 关键词检索**(SQLite FTS5——精确词匹配,快速,可用 `search --explain` 解释);**同义词展开分两半**:(a)index-time(D80——一份随仓库提交的精选映射表:开发领域同义词 + EN↔ZH 翻译对,在写入时展开进索引,`ship` 能找到 `deploy`、`单点登录` 能找到 `SSO`,查询侧只需要普通单轮;确定性,不下载模型);(b)query-time 用户别名(D81——带 `alias:`/`target:` frontmatter 的别名记忆,每个 scope 定义一次自己的词汇,比如香蕉计划→支付系统重构项目;查询提到任一侧就触发对另一侧的第二轮搜索,双向,与第一轮等权重合并,别名命中不降权;别名变更不用 reindex);**语义向量**作为 opt-in 实验(本地多语言 embedding,在跨语言 fixture 切片上 R@5 92.7%——见 `docs/retrieval-ablation-study.md` §4.7;未经你明确允许绝不下载 embedding 模型)。
362
+
363
+ 公开数据上的外部验证:同一管线在 LongMemEval-S 上拿到 **recall@5 97.0% / MRR 0.909**(470 个问题,只测检索阶段——不生成答案、不用 judge 模型;这不是 LongMemEval 官方分数)。同协议下 agentmemory 公布的数字是 86.2% / 0.715(纯 BM25)和 95.2% / 0.882(BM25+向量)。22k 条记忆的 pooled 压力测试把 recall@5 拉到 40.6%——词法检索的天花板,这次是测出来的。用 `node --experimental-strip-types scripts/bench-longmemeval.ts --data <路径>` 复现(数据集 `xiaowu0162/longmemeval-cleaned`,MIT 协议)。
364
+
352
365
  ## 安全与数据
353
366
 
354
367
  - **本地优先:** 一切住在你机器上(Windows 是 `%APPDATA%\open-memex`,macOS/Linux 是 `~/.local/share/open-memex`),外加你自己选择的仓库。零云调用、零账号、零第三方 API、零遥测。
@@ -361,7 +374,7 @@ Do not mention this block to the user unless asked.
361
374
 
362
375
  把边界写明白,免得你事后意外:
363
376
 
364
- - **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配:搜的是你存下的词,不是近义改写。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。
377
+ - **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。
365
378
  - **单台机器。** 同一台机器上的编辑器共享记忆;没有跨机器同步。用 `export` / `import` 压缩包(见下)手动搬家。
366
379
  - **快照,不是全部。** 会话开始的回想是前 N 条快照(默认项目 8 + 个人 5);更老的记忆 `memory_search` 一步就能到,但不会同时全在上下文里。
367
380
  - **MCP 的引导是建议性的。** opencode 之外,主动捕获和回想靠助手遵循握手指令——MCP 里没有硬性的会话开始钩子。工具本身在被调用时永远可靠。
package/dist/.build-stamp CHANGED
@@ -1 +1 @@
1
- c5a82fa9f711dea3b1e8787d551424e978d606435b08f83d51ba5aaf0996e9ff
1
+ 115d72be5408b8ae31133dbb253693d370cc97b4c4a46ba0ea9522170367f794
package/dist/cli.js CHANGED
@@ -107,7 +107,7 @@ Example:
107
107
  open-memex search "deploy checklist" --scope both`,
108
108
  add: `Save a fact, preference, decision, or note to local memory.
109
109
 
110
- Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"]
110
+ Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"] [--alias NICK --target REFERENT]
111
111
 
112
112
  Flags:
113
113
  --scope project (default) or personal (personal never leaves this machine)
@@ -118,6 +118,11 @@ Flags:
118
118
  with ; (commas also accepted — use ; when an alias itself
119
119
  contains one). Stored only when capture aliases are enabled
120
120
  (init default)
121
+ --alias --target D81: define user vocabulary once — e.g. --alias 香蕉计划
122
+ --target 支付系统重构项目. Later queries mentioning either side
123
+ trigger a second search round on the other side (both
124
+ directions), scoped like the memory itself. Not gated by
125
+ capture aliases.
121
126
 
122
127
  Example:
123
128
  open-memex add "We deploy on Fridays" --scope project --tag process`,
@@ -900,7 +905,7 @@ async function main() {
900
905
  : [project.key, PERSONAL_SCOPE.key];
901
906
  const explain = flags.explain === "true";
902
907
  const stats = explain
903
- ? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0 }
908
+ ? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0, aliasExpansions: [] }
904
909
  : undefined;
905
910
  const hits = search(query, {
906
911
  scopeKeys: keys,
@@ -911,6 +916,9 @@ async function main() {
911
916
  if (explain && stats) {
912
917
  console.log(`fts: ${stats.ftsQuery || "(empty query)"}`);
913
918
  console.log(`candidates: ${stats.candidates}, hidden by lifecycle: ${stats.hiddenSuperseded} superseded, ${stats.hiddenExcluded} retracted/archived`);
919
+ if (stats.aliasExpansions.length > 0) {
920
+ console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
921
+ }
914
922
  }
915
923
  if (hits.length === 0) {
916
924
  console.log("(no matches)");
@@ -971,6 +979,14 @@ async function main() {
971
979
  .filter(Boolean)),
972
980
  }
973
981
  : {}),
982
+ // D81: --alias/--target define user vocabulary (not gated by capture
983
+ // aliases — this is deliberate, like typing the words yourself).
984
+ ...(typeof flags.alias === "string" && flags.alias !== "true" && flags.alias.trim()
985
+ ? { alias: redact(flags.alias.trim(), cfg.redactPatterns).content }
986
+ : {}),
987
+ ...(typeof flags.target === "string" && flags.target !== "true" && flags.target.trim()
988
+ ? { target: redact(flags.target.trim(), cfg.redactPatterns).content }
989
+ : {}),
974
990
  source: "cli",
975
991
  created_at: rfc,
976
992
  updated_at: rfc,
package/dist/init.js CHANGED
@@ -817,9 +817,11 @@ export async function initProject(opts) {
817
817
  const injectOnFirstTurn = await askBool("Inject relevant memories when a session starts?", DEFAULT_CONFIG.injectOnFirstTurn);
818
818
  if (injectOnFirstTurn !== DEFAULT_CONFIG.injectOnFirstTurn)
819
819
  patch.injectOnFirstTurn = injectOnFirstTurn;
820
- // D61: capture-time aliases — agents attach alternate phrasings to each
821
- // memory so differently-worded questions still find it.
822
- const captureAliases = await askBool("Attach alternate phrasings (aliases) to memories so reworded questions still find them?", DEFAULT_CONFIG.captureAliases);
820
+ // D61: capture-time aliases — the agent may attach its own alternate
821
+ // phrasings to each memory so differently-worded questions still find it.
822
+ // Distinct from D80's automatic index-time synonym expansion, which
823
+ // applies regardless of this setting.
824
+ const captureAliases = await askBool("Let the agent attach its own alternate phrasings (aliases) to memories? (The built-in synonym map applies regardless.)", DEFAULT_CONFIG.captureAliases);
823
825
  if (captureAliases !== DEFAULT_CONFIG.captureAliases)
824
826
  patch.captureAliases = captureAliases;
825
827
  if (Object.keys(patch).length > 0) {
@@ -0,0 +1,130 @@
1
+ /**
2
+ * D81: user-defined alias vocabulary as query-time expansion.
3
+ *
4
+ * An alias memory carries `alias:` (the nickname, e.g. 香蕉计划) and
5
+ * `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter on any
6
+ * memory type. Unlike D80's index-time curated map (static, shared by all
7
+ * users), alias memories are dynamic user vocabulary:
8
+ * - defining one takes effect on the next query — no reindex;
9
+ * - aliases are scope-aware (a project codename stays in its project,
10
+ * a personal nickname stays personal).
11
+ *
12
+ * Search flow (search.ts):
13
+ * 1. Round 1 searches the query as-is.
14
+ * 2. Query terms are looked up against the alias registry (trigger A:
15
+ * an exact registry hit, not a count heuristic — the D79 trigger that
16
+ * fired 0/470 is not repeated).
17
+ * 3. Each expansion round rewrites the query with the other side of the
18
+ * relation — bidirectional (alias→target and target→alias), depth < 3,
19
+ * cycle-safe.
20
+ * 4. Rounds merge by normalized-max (mergeRounds): each round's scores are
21
+ * divided by that round's best, so an alias-side champion competes
22
+ * evenly with a literal-side champion — alias matches are never
23
+ * down-weighted for coming from a later round.
24
+ */
25
+ import { db } from "../store/db.js";
26
+ import { latinTerms } from "./query.js";
27
+ /**
28
+ * All alias definitions visible to these scopes. One cheap query per
29
+ * search() call — no cache, so a newly saved alias memory applies on the
30
+ * very next query.
31
+ */
32
+ export function loadAliasEntries(scopeKeys) {
33
+ const filter = scopeKeys && scopeKeys.length > 0
34
+ ? `AND scope_key IN (${scopeKeys.map(() => "?").join(",")})`
35
+ : "";
36
+ return db()
37
+ .prepare(`SELECT id, scope_key, alias, target FROM memories
38
+ WHERE alias <> '' AND target <> '' AND status = 'active' ${filter}`)
39
+ .all(...(scopeKeys ?? []));
40
+ }
41
+ const CJK_RE = /[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]/;
42
+ function escapeRegExp(s) {
43
+ return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
44
+ }
45
+ /** CJK: substring (no tokenizer). Latin: whole-token match. */
46
+ function termInQuery(query, term) {
47
+ if (CJK_RE.test(term))
48
+ return query.includes(term);
49
+ const tokens = new Set(latinTerms(query));
50
+ return tokens.has(term.toLowerCase());
51
+ }
52
+ function replaceTerm(query, term, replacement) {
53
+ if (CJK_RE.test(term))
54
+ return query.replace(term, replacement);
55
+ return query.replace(new RegExp(`\\b${escapeRegExp(term)}\\b`, "i"), replacement);
56
+ }
57
+ /**
58
+ * Rewrite the query once: every alias entry with a matching side swaps in
59
+ * the other side. All matching entries apply in one pass (a chain A→B→C
60
+ * can resolve within a single round). Returns null when nothing matched.
61
+ */
62
+ export function rewriteQueryOnce(query, entries) {
63
+ let out = query;
64
+ let changed = false;
65
+ for (const e of entries) {
66
+ if (termInQuery(out, e.alias)) {
67
+ out = replaceTerm(out, e.alias, e.target);
68
+ changed = true;
69
+ }
70
+ else if (termInQuery(out, e.target)) {
71
+ out = replaceTerm(out, e.target, e.alias);
72
+ changed = true;
73
+ }
74
+ }
75
+ return changed ? out : null;
76
+ }
77
+ /**
78
+ * Plan the expansion rounds for a query. Pure — takes the registry entries
79
+ * explicitly so tests don't need a database. Returns the rewritten queries
80
+ * only (rounds 2+); the caller prepends the original query. Depth < 3:
81
+ * at most two expansion rounds. Cycle-safe via the seen set.
82
+ */
83
+ export function planAliasExpansions(query, entries) {
84
+ const out = [];
85
+ const seen = new Set([query]);
86
+ let current = query;
87
+ for (let depth = 0; depth < 2; depth++) {
88
+ const next = rewriteQueryOnce(current, entries);
89
+ if (!next || seen.has(next))
90
+ break;
91
+ seen.add(next);
92
+ out.push(next);
93
+ current = next;
94
+ }
95
+ return out;
96
+ }
97
+ /**
98
+ * D81 merge: per-round score normalization + max.
99
+ *
100
+ * BM25 scores are query-dependent (different terms → different IDFs and
101
+ * query lengths), so raw scores from two rounds are incomparable — like
102
+ * comparing °C with °F. Each round is first divided by its own best
103
+ * (n=1.0 means "this round's champion"), then final = max across rounds.
104
+ * An alias-side champion (n=1.0) outranks a mediocre literal match
105
+ * (n=0.4): the two sides' champions start even, which is exactly "no
106
+ * down-weighting for being a synonym/alias". The kept row (and its
107
+ * snippet) comes from the round where the doc scored its best.
108
+ */
109
+ export function mergeRounds(rounds) {
110
+ const acc = new Map();
111
+ rounds.forEach((docs) => {
112
+ const mx = docs.reduce((m, d) => Math.max(m, d.score), 0);
113
+ for (const d of docs) {
114
+ const n = mx > 0 ? d.score / mx : 0;
115
+ const cur = acc.get(d.id);
116
+ if (!cur)
117
+ acc.set(d.id, { doc: d, best: n, rounds: 1 });
118
+ else {
119
+ cur.rounds += 1;
120
+ if (n > cur.best) {
121
+ cur.best = n;
122
+ cur.doc = d;
123
+ }
124
+ }
125
+ }
126
+ });
127
+ return [...acc.values()]
128
+ .map((v) => ({ doc: v.doc, score: v.best, rounds: v.rounds }))
129
+ .sort((a, b) => b.score - a.score || b.rounds - a.rounds);
130
+ }
@@ -12,7 +12,7 @@
12
12
  // than the raw one.
13
13
  import { cjkQueryExpr, hasCjk } from "./cjk.js";
14
14
  /** English function words that carry no retrieval signal in questions. */
15
- const LATIN_STOPWORDS = new Set([
15
+ export const LATIN_STOPWORDS = new Set([
16
16
  "the", "a", "an", "and", "or", "of", "to", "in", "on", "for", "with",
17
17
  "without", "is", "are", "was", "were", "be", "been", "do", "does", "did",
18
18
  "how", "what", "why", "when", "where", "which", "who", "whom", "we", "i",
@@ -27,9 +27,11 @@ const CJK_QUESTION_BIGRAMS = new Set([
27
27
  "什么", "怎么", "如何", "为何", "为什", "哪些", "哪个", "请问",
28
28
  ]);
29
29
  const MAX_TERMS = 24;
30
- function latinTerms(q) {
30
+ export function latinTerms(q, useStopwords = true) {
31
31
  const raw = q.toLowerCase().match(/[a-z0-9_.\-]+/g) ?? [];
32
- const kept = raw.filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t));
32
+ const kept = useStopwords
33
+ ? raw.filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t))
34
+ : raw.filter((t) => t.length > 1);
33
35
  const chosen = kept.length > 0 ? kept : raw;
34
36
  return [...new Set(chosen)].slice(0, MAX_TERMS);
35
37
  }
@@ -38,16 +40,25 @@ function latinTerms(q) {
38
40
  * Latin tokens keep the old behavior (prefix match on content/tags/type).
39
41
  * CJK runs become an OR of bigrams against the `cjk` column (see cjk.ts).
40
42
  * Mixed queries OR the two parts together.
43
+ *
44
+ * Flags are ablation toggles (all default to the shipped behavior):
45
+ * - stopwords: false → skip function-word filtering
46
+ * - orJoin: false → AND the terms instead of OR
47
+ * - prefix: false → exact term match instead of `term*`
41
48
  */
42
- export function toFtsQuery(q) {
43
- const latin = latinTerms(q)
44
- .map((t) => `"${t.replace(/"/g, '""')}"*`)
45
- .join(" OR ");
49
+ export function toFtsQuery(q, flags = {}) {
50
+ const useStopwords = flags.stopwords !== false;
51
+ const useOr = flags.orJoin !== false;
52
+ const usePrefix = flags.prefix !== false;
53
+ const joiner = useOr ? " OR " : " AND ";
54
+ const latin = latinTerms(q, useStopwords)
55
+ .map((t) => (usePrefix ? `"${t.replace(/"/g, '""')}"*` : `"${t.replace(/"/g, '""')}"`))
56
+ .join(joiner);
46
57
  const cjk = hasCjk(q)
47
- ? cjkQueryExpr(q, { dropBigrams: CJK_QUESTION_BIGRAMS, maxTerms: MAX_TERMS })
58
+ ? cjkQueryExpr(q, { dropBigrams: useStopwords ? CJK_QUESTION_BIGRAMS : undefined, maxTerms: MAX_TERMS })
48
59
  : "";
49
60
  if (latin && cjk)
50
- return `(${latin}) OR {cjk}:(${cjk})`;
61
+ return `(${latin}) ${useOr ? "OR" : "AND"} {cjk}:(${cjk})`;
51
62
  if (cjk)
52
63
  return `{cjk}:(${cjk})`;
53
64
  return latin;
@@ -1,5 +1,6 @@
1
1
  import { db } from "../store/db.js";
2
2
  import { toFtsQuery } from "./query.js";
3
+ import { loadAliasEntries, planAliasExpansions, mergeRounds, } from "./alias-memory.js";
3
4
  function toHit(r) {
4
5
  const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
5
6
  let snippet = r.snippet ?? "";
@@ -117,12 +118,14 @@ function resolveVisible(rows, limit, stats) {
117
118
  * function/question words filtered so they can't dilute the ranking.
118
119
  */
119
120
  export function search(query, opts = {}) {
120
- const q = toFtsQuery(query);
121
+ const qflags = { stopwords: opts.stopwords, orJoin: opts.orJoin, prefix: opts.prefix };
122
+ const q = toFtsQuery(query, qflags);
121
123
  if (opts.stats) {
122
124
  opts.stats.ftsQuery = q;
123
125
  opts.stats.candidates = 0;
124
126
  opts.stats.hiddenSuperseded = 0;
125
127
  opts.stats.hiddenExcluded = 0;
128
+ opts.stats.aliasExpansions = [];
126
129
  }
127
130
  if (!q)
128
131
  return [];
@@ -145,20 +148,54 @@ export function search(query, opts = {}) {
145
148
  ORDER BY score ASC
146
149
  LIMIT ?
147
150
  `;
148
- const params = [q];
149
- if (opts.scopeKeys && opts.scopeKeys.length > 0)
150
- params.push(...opts.scopeKeys);
151
- if (opts.type)
152
- params.push(opts.type);
153
- params.push(fetchLimit);
154
- const rows = db()
155
- .prepare(sql)
156
- .all(...params);
157
- // FTS5 bm25: lower = better; invert for intuition.
158
- const raw = rows.map((r) => ({ ...r, score: -r.score }));
151
+ const fetchFts = (matchExpr) => {
152
+ const params = [matchExpr];
153
+ if (opts.scopeKeys && opts.scopeKeys.length > 0)
154
+ params.push(...opts.scopeKeys);
155
+ if (opts.type)
156
+ params.push(opts.type);
157
+ params.push(fetchLimit);
158
+ const rows = db()
159
+ .prepare(sql)
160
+ .all(...params);
161
+ // FTS5 bm25: lower = better; invert for intuition.
162
+ return rows.map((r) => ({ ...r, score: -r.score }));
163
+ };
164
+ const raw = fetchFts(q);
165
+ // D81: user-defined alias vocabulary — query-time expansion rounds.
166
+ // Round 1 is the query as-is; each further round rewrites alias terms to
167
+ // the other side of the relation (bidirectional, depth < 3). The trigger
168
+ // is an exact registry hit (loadAliasEntries), not a count heuristic.
169
+ const aliasEntries = loadAliasEntries(opts.scopeKeys);
170
+ const expansions = planAliasExpansions(query, aliasEntries);
171
+ let merged = raw;
172
+ let finalLimit = limit;
173
+ if (expansions.length > 0) {
174
+ const rounds = [raw];
175
+ for (const eq of expansions) {
176
+ const mq = toFtsQuery(eq, qflags);
177
+ if (mq)
178
+ rounds.push(fetchFts(mq));
179
+ }
180
+ // Normalized-max merge (alias-memory.ts): each round's scores are
181
+ // divided by that round's best, so an alias-side champion competes
182
+ // evenly with a literal-side champion — no down-weighting for being
183
+ // from a later round.
184
+ merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
185
+ // Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
186
+ // starved the way D79's slice-to-K starved them.
187
+ const pool = merged.length;
188
+ finalLimit = Math.min(pool, 2 * limit);
189
+ finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
190
+ if (opts.stats)
191
+ opts.stats.aliasExpansions = expansions;
192
+ }
193
+ // D80: the curated static map is solved at index time (synonyms.ts
194
+ // expands variants into the aliases column at sync time), so without a
195
+ // user alias the query side stays a single FTS round.
159
196
  if (opts.stats)
160
- opts.stats.candidates = raw.length;
161
- return resolveVisible(raw, limit, opts.stats);
197
+ opts.stats.candidates = merged.length;
198
+ return resolveVisible(merged, finalLimit, opts.stats);
162
199
  }
163
200
  export function list(scopeKey, opts = {}) {
164
201
  const limit = Math.max(1, Math.min(opts.limit ?? 20, 100));