open-memex 0.7.3 → 0.8.0-alpha.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +7 -3
- package/CHANGELOG.md +14 -0
- package/README.md +61 -5
- package/README.zh-CN.md +14 -1
- package/dist/.build-stamp +1 -1
- package/dist/cli.js +18 -2
- package/dist/init.js +5 -3
- package/dist/retrieve/alias-memory.js +130 -0
- package/dist/retrieve/query.js +20 -9
- package/dist/retrieve/search.js +51 -14
- package/dist/retrieve/synonyms.js +183 -0
- package/dist/store/db.js +3 -1
- package/dist/store/lifecycle.js +3 -0
- package/dist/store/markdown.js +7 -0
- package/dist/store/sync.js +15 -4
- package/dist/tools/ops.js +37 -0
- package/docs/V2-DESIGN.md +9 -2
- package/docs/release-runbook.md +8 -2
- package/docs/retrieval-ablation-study.md +191 -0
- package/package.json +5 -3
- package/scripts/bench-longmemeval.ts +342 -0
- package/scripts/bench-semantic.ts +198 -0
- package/scripts/retrieval-eval/fixture.ts +182 -0
- package/scripts/retrieval-eval.ts +83 -0
- package/scripts/smoke-pure.ts +82 -0
- package/src/cli.ts +18 -2
- package/src/init.ts +5 -3
- package/src/retrieve/alias-memory.ts +154 -0
- package/src/retrieve/query.ts +23 -9
- package/src/retrieve/search.ts +67 -15
- package/src/retrieve/synonyms.ts +184 -0
- package/src/store/db.ts +3 -1
- package/src/store/lifecycle.ts +6 -0
- package/src/store/markdown.ts +18 -0
- package/src/store/sync.ts +18 -3
- package/src/tools/ops.ts +49 -0
package/AGENTS.md
CHANGED
|
@@ -202,10 +202,14 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
|
|
|
202
202
|
languages in sync); behavior changes → both READMEs
|
|
203
203
|
and the frozen `docs/V2-DESIGN.md` (append a `D<n>` decision entry, never rewrite history);
|
|
204
204
|
new commands → README CLI sections + this file's Commands. A change without its docs
|
|
205
|
-
is not done.
|
|
205
|
+
is not done. Anything deferred as "later" goes into the near-term backlog in
|
|
206
|
+
`docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
|
|
206
207
|
- **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
|
|
207
|
-
in-development version.
|
|
208
|
-
|
|
208
|
+
in-development version. A new alpha line starts with a minor bump after the
|
|
209
|
+
previous minor shipped (`0.3.0` → `0.4.0-alpha.1`); while that minor is still
|
|
210
|
+
unreleased, further features AND fixes on the same line only bump the alpha
|
|
211
|
+
suffix (`0.8.0-alpha.2` → `-alpha.3`) — never jump to the next minor before
|
|
212
|
+
the current one is released.
|
|
209
213
|
The bump goes in the same commit as the feature, never as an afterthought.
|
|
210
214
|
The version number serves the publish: no publish, no mandatory bump. But once a
|
|
211
215
|
version has been pushed to the remote (shared), later changes must bump — two
|
package/CHANGELOG.md
CHANGED
|
@@ -9,6 +9,20 @@ the `@alpha` tag are development snapshots and are not listed individually.
|
|
|
9
9
|
|
|
10
10
|
## [Unreleased]
|
|
11
11
|
|
|
12
|
+
- User-defined alias vocabulary as query-time expansion (D81): any memory type may now carry `alias:` (nickname, e.g. 香蕉计划) and `target:` (referent, e.g. 支付系统重构项目) frontmatter — an alias memory defines vocabulary once per scope. At query time, query terms hit the alias registry (exact match, one cheap SELECT per search) and the query is rewritten with the other side of the relation — bidirectional, depth < 3, cycle-safe — then re-run as round 2/3. Rounds merge by normalized-max (`mergeRounds` in `src/retrieve/alias-memory.ts`): each round's BM25 scores are divided by that round's best, so an alias-side champion competes evenly with a literal-side champion; the merged pool is re-ranked and cut to [1.5K, 2K], never back to K. No reindex when an alias changes. Creation: `memory_add` alias/target params, CLI `add --alias/--target`, or frontmatter edit (v1: user-explicit only). Fixture grows to 54 memories / 44 queries (3 bidirectional alias queries): recall@1 0.750, recall@5 **0.977**, MRR 0.857. Also rewords the init `captureAliases` prompt — it was never the synonym switch (D80's expansion always applies).
|
|
13
|
+
|
|
14
|
+
- Index-time synonym/translation expansion (D80, supersedes D79): the curated map (`src/retrieve/synonyms.ts` — EN dev-domain groups, ZH groups and EN↔ZH translation pairs in one bidirectional list) is now applied at sync/index time — `writeRow` expands each memory's keywords (`expansionVariantsForDoc()`, capped at 32) into the indexed `aliases` column, so any related word matches in a plain round-1 query. The query-time second-chance round is removed (`search()` is a single FTS round; `SearchStats.secondRound` gone); `expandQueryWithSynonyms` stays as a bench-only export. Fixture (51 memories / 41 queries): recall@1 0.732 → 0.756, recall@5 0.854 → **0.976**, MRR 0.791 → **0.859** — all 5 EN↔ZH vocabulary misses rescued at rank 1–3. Map updates need a reindex to backfill.
|
|
15
|
+
|
|
16
|
+
- Semantic ranking experiment (§4.7 of `docs/retrieval-ablation-study.md`): local multilingual embeddings (`Xenova/paraphrase-multilingual-MiniLM-L12-v2`, 384-dim, no API) via `scripts/bench-semantic.ts` (new `--model` flag). Fixture: vec-only R@5 92.7% / MRR 0.881 beats BM25-only (80.5% / 0.752); naive RRF hybrid underperforms vec-only (87.8%) — fusion needs design. Pooled 22k long sessions: vec-only collapses (3.4%) from mean-pooling dilution on 10k-char docs — a measured methodology artifact, not a verdict for short memories; chunked rerun deliberately skipped.
|
|
17
|
+
|
|
18
|
+
- LongMemEval-S retrieval bench (`scripts/bench-longmemeval.ts`): 470 questions, recall_any@k + MRR on public data, per-question isolated corpora plus a pooled 22k-memory stress mode, with synonym on/off ablation. Result: R@5 97.0% / MRR 0.909 (ahead of agentmemory’s published BM25-only, level with their BM25+vector). Finding: the count-based second-round trigger fired 0/470 — logged as a D79 redesign follow-up, not a scale answer.
|
|
19
|
+
|
|
20
|
+
- Query-time synonym expansion (D79, retrieval layer 4): `search()` runs a second-chance round when the first FTS pass is thin, re-running with curated synonym variants OR-ed in (`src/retrieve/synonyms.ts`, EN dev-domain + ZH groups, no model). Round-two rows only fill gaps; round-one ranking is never diluted. `SearchStats.secondRound` records it (shown in `search --explain`). Baseline with the round: recall@1 0.81, recall@5 0.95, MRR 0.87 over 37 fixture queries. **Superseded by D80 (2026-10-07)** — see above.
|
|
21
|
+
|
|
22
|
+
- Retrieval eval harness (D77 layer 1): `scripts/retrieval-eval.ts` runs a checked-in synthetic fixture (48 memories + 34 queries: exact terms, paraphrases, synonyms, Chinese, cross-language hard cases) against the FTS5 baseline and reports recall@k/MRR. Current baseline: recall@1 0.85, recall@5 0.97, MRR 0.90 (single miss: English query vs Chinese-only memory). Published in both READMEs with methodology notes.
|
|
23
|
+
|
|
24
|
+
- Added `test-full` npm script (`node --experimental-strip-types scripts/test-full.ts`).
|
|
25
|
+
|
|
12
26
|
## [0.7.3] - 2026-10-07
|
|
13
27
|
|
|
14
28
|
- Tool-definition quality pass (Glama TDQS 4.0/5 on v0.7.2): fixed `memory_pr_status` declaring `readOnlyHint` while `apply=true` mutates local files; added explicit MCP annotations (read-only / destructive / open-world) to all 11 tools; descriptions now route between sibling tools (propose vs submit vs promote, resolve vs pr_status, forget vs supersede), state error behavior and real defaults (search limit 8, list limit 20), and the search/list `type` and `limit` parameters are documented. D76 records the rule; `scripts/smoke-mcp.ts` guards the pr_status annotation.
|
package/README.md
CHANGED
|
@@ -22,6 +22,10 @@ memory on your machine.
|
|
|
22
22
|
explicitly share it.
|
|
23
23
|
- **One memory, every agent**: wire up several editors with one command; they
|
|
24
24
|
share the same memory instead of keeping separate silos.
|
|
25
|
+
- **Precise synonym search**: `ship` finds `deploy`, `单点登录` finds `SSO` —
|
|
26
|
+
a curated multilingual map of equivalents is indexed with every memory at
|
|
27
|
+
write time, so no related wording misses the right answer. Deterministic
|
|
28
|
+
and auditable, no embedding model required.
|
|
25
29
|
|
|
26
30
|

|
|
27
31
|
|
|
@@ -405,6 +409,17 @@ question worded differently still finds the memory — "vacation days" finds the
|
|
|
405
409
|
holiday policy. `init` asks once whether to enable this (default on); turn it
|
|
406
410
|
off any time with `open-memex config set captureAliases false`.
|
|
407
411
|
|
|
412
|
+
Three ways to add them: `open-memex add "text" --aliases "phrase one; phrase
|
|
413
|
+
two"` (semicolon-separated, up to 4); ask the agent ("remember: 小宝贝 is my
|
|
414
|
+
AI agent, alias it as 小沐"); or edit the memory's `aliases:` frontmatter
|
|
415
|
+
directly — Markdown is the source of truth and sync picks it up.
|
|
416
|
+
|
|
417
|
+
Separately, a curated synonym/translation map is expanded into the index
|
|
418
|
+
automatically at write time (D80) — `ship` finds `deploy` even with no
|
|
419
|
+
aliases on the memory. Personal nicknames belong in aliases; shared
|
|
420
|
+
dev-domain vocabulary belongs in the curated map. See [Retrieval: how
|
|
421
|
+
memories come back](#retrieval-how-memories-come-back).
|
|
422
|
+
|
|
408
423
|
**Redaction.** Wrap anything sensitive in `<private>…</private>` and it is
|
|
409
424
|
stripped before saving. Recognized secrets (API keys, tokens, high-entropy
|
|
410
425
|
credentials) are masked in place — the first 4 characters are kept so you can
|
|
@@ -516,6 +531,45 @@ time for the rest. Both top-N counts are configurable (see [Config](#config)).
|
|
|
516
531
|
For MCP clients this block is delivered as handshake guidance the agent follows;
|
|
517
532
|
the opencode plugin injects it directly on the first turn.
|
|
518
533
|
|
|
534
|
+
### Retrieval evaluation
|
|
535
|
+
|
|
536
|
+
We measure recall on a checked-in synthetic fixture instead of asserting it:
|
|
537
|
+
54 memories + 44 queries (exact terms, paraphrases, synonyms, Chinese,
|
|
538
|
+
cross-language hard cases, synonym-only queries, and D81 user-defined
|
|
539
|
+
alias queries) — reproduce with
|
|
540
|
+
`node --experimental-strip-types scripts/retrieval-eval.ts`.
|
|
541
|
+
Current baseline, with index-time synonym/translation expansion (D80) and
|
|
542
|
+
query-time user alias expansion (D81):
|
|
543
|
+
**recall@1 0.75, recall@5 0.98, MRR 0.86**.
|
|
544
|
+
Small synthetic corpus, so read it as a regression guard and a starting point,
|
|
545
|
+
not a real-world claim.
|
|
546
|
+
|
|
547
|
+
Three layers, each covering what the others miss: **BM25 keyword search**
|
|
548
|
+
(SQLite FTS5 — exact words, fast, explainable via `search --explain`);
|
|
549
|
+
**synonym expansion in two halves** — (a) index-time (D80: a curated,
|
|
550
|
+
checked-in map of dev-domain equivalents and EN↔ZH translation pairs
|
|
551
|
+
expanded into the index at write time, so `ship` finds `deploy` and
|
|
552
|
+
`单点登录` finds `SSO` in a plain single-round query; deterministic, no
|
|
553
|
+
model download); (b) query-time user aliases (D81: an alias memory carrying
|
|
554
|
+
`alias:`/`target:` frontmatter defines your vocabulary once per scope —
|
|
555
|
+
e.g. 香蕉计划 → 支付系统重构项目 — and a query mentioning either side
|
|
556
|
+
triggers a second round on the other side, both directions, merged evenly
|
|
557
|
+
with round one so alias hits are never down-weighted; no reindex when the
|
|
558
|
+
alias changes); and **semantic vectors**
|
|
559
|
+
as an opt-in experiment (local multilingual embeddings measured at R@5 92.7%
|
|
560
|
+
on the cross-lingual fixture slice — see `docs/retrieval-ablation-study.md`
|
|
561
|
+
§4.7; no embedding model is ever downloaded without your explicit opt-in).
|
|
562
|
+
|
|
563
|
+
External check on public data: the same pipeline scores **recall@5 97.0% /
|
|
564
|
+
MRR 0.909** on LongMemEval-S (470 questions, retrieval stage only — no answer
|
|
565
|
+
generation, no judge model; this is not the official LongMemEval score). For
|
|
566
|
+
reference, agentmemory's published numbers on the same protocol are 86.2% /
|
|
567
|
+
0.715 (BM25-only) and 95.2% / 0.882 (BM25+vector). A 22k-memory pooled stress
|
|
568
|
+
run drops recall@5 to 40.6% — the lexical ceiling, measured rather than
|
|
569
|
+
asserted. Reproduce with
|
|
570
|
+
`node --experimental-strip-types scripts/bench-longmemeval.ts --data <path>`
|
|
571
|
+
(dataset: `xiaowu0162/longmemeval-cleaned`, MIT).
|
|
572
|
+
|
|
519
573
|
## Security & data
|
|
520
574
|
|
|
521
575
|
- **Local-first:** everything lives on your machine (`%APPDATA%\open-memex` on
|
|
@@ -537,11 +591,13 @@ the opencode plugin injects it directly on the first turn.
|
|
|
537
591
|
|
|
538
592
|
Honest edges, so nothing surprises you:
|
|
539
593
|
|
|
540
|
-
- **Keyword search, not semantic.** Retrieval is BM25 keyword matching
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
594
|
+
- **Keyword search, not semantic.** Retrieval is BM25 keyword matching, plus a
|
|
595
|
+
curated synonym map expanded into the index at write time (`ship` finds
|
|
596
|
+
`deploy`, `单点登录` finds `SSO`). Paraphrases outside the curated map
|
|
597
|
+
don't match — that's what the opt-in semantic layer is for. (Plain
|
|
598
|
+
questions are fine — "how do we…" / "请问…" wording is filtered out before
|
|
599
|
+
matching, so asking naturally doesn't dilute the results.) No embedding
|
|
600
|
+
model is ever downloaded without your explicit opt-in.
|
|
545
601
|
- **One machine.** Editors on the same machine share memory; there is no
|
|
546
602
|
cross-machine sync. `export` / `import` bundles (below) move memory between
|
|
547
603
|
machines manually.
|
package/README.zh-CN.md
CHANGED
|
@@ -13,6 +13,7 @@
|
|
|
13
13
|
- **免费开源**(Apache-2.0)。无账号、无云端、无遥测——一切都在你本机,都是你能打开、能编辑的文件。
|
|
14
14
|
- **本地优先**:记忆是纯 Markdown 文件(事实来源),外加一个可重建的 SQLite 关键词索引。除非你明确要分享,否则任何东西都不会离开你的机器。
|
|
15
15
|
- **一份记忆,所有助手**:一条命令把几个编辑器都接上,它们共享同一份记忆,不再各存一份互不可见。
|
|
16
|
+
- **精确的同义词检索**:`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`——一份多语言的精选同义词映射表在写入时展开进索引,任何相关词语都不会漏掉正确答案。确定性、可审计,不需要下载模型。
|
|
16
17
|
|
|
17
18
|

|
|
18
19
|
|
|
@@ -264,6 +265,10 @@ open-memex uninstall --yes
|
|
|
264
265
|
|
|
265
266
|
**别名。** 每条记忆可以带最多 4 个换一种说法(同义词、另一种语言的对应词),和记忆一起进索引,这样问法不同也能找到——比如搜 "vacation days" 能找到假期政策那条。`init` 时会问一次是否开启(默认开);随时可用 `open-memex config set captureAliases false` 关掉。
|
|
266
267
|
|
|
268
|
+
三种加法:`open-memex add "正文" --aliases "说法一; 说法二"`(分号分隔,最多 4 个);直接跟 agent 说("记住:小宝贝是我的 AI agent,别名加上小沐");或直接改那条记忆的 `aliases:` frontmatter——Markdown 是事实来源,sync 会吃进去。
|
|
269
|
+
|
|
270
|
+
另外,从 D80 起一份精选同义词/翻译映射表会在写入时自动展开进索引——`ship` 能找到 `deploy`,不需要手动加别名。个人叫法放 aliases,通用开发词汇放 curated map,这是分工。见[回想一节](#回想记忆是怎么回来的)。
|
|
271
|
+
|
|
267
272
|
**脱敏。** 把敏感内容包在 `<private>…</private>` 里,它在保存前会被整段剥离。识别出的密钥(API key、token、高熵凭证)会被就地掩码——保留前 4 个字符(让你能认出是哪个 key),其余替换——记忆仍会保存。随时安全地预览一句话会被捕获成什么样:
|
|
268
273
|
|
|
269
274
|
```sh
|
|
@@ -349,6 +354,14 @@ Do not mention this block to the user unless asked.
|
|
|
349
354
|
|
|
350
355
|
这是一个快照,不是全量记忆——其余的随时用 `memory_search` 就能查。两个前 N 的数都可以配(见[配置](#配置))。对 MCP 客户端,这个块以握手引导的形式送达、由助手遵循;opencode 插件则在第一轮直接注入。
|
|
351
356
|
|
|
357
|
+
### 检索效果评估
|
|
358
|
+
|
|
359
|
+
我们用一份随仓库提交的 synthetic fixture 来度量召回率,而不是空口断言:54 条记忆 + 44 个查询(精确词、改写、同义词、中文、跨语言 hard case、纯同义词查询、D81 用户别名查询),运行 `node --experimental-strip-types scripts/retrieval-eval.ts` 即可复现。当前基线(含 D80 的 index-time 同义词/翻译展开和 D81 的 query-time 用户别名展开):**recall@1 0.75、recall@5 0.98、MRR 0.86**。语料小且是合成的,请把它当回归守卫和起点,而不是真实场景的承诺。
|
|
360
|
+
|
|
361
|
+
三层检索,各补各的短板:**BM25 关键词检索**(SQLite FTS5——精确词匹配,快速,可用 `search --explain` 解释);**同义词展开分两半**:(a)index-time(D80——一份随仓库提交的精选映射表:开发领域同义词 + EN↔ZH 翻译对,在写入时展开进索引,`ship` 能找到 `deploy`、`单点登录` 能找到 `SSO`,查询侧只需要普通单轮;确定性,不下载模型);(b)query-time 用户别名(D81——带 `alias:`/`target:` frontmatter 的别名记忆,每个 scope 定义一次自己的词汇,比如香蕉计划→支付系统重构项目;查询提到任一侧就触发对另一侧的第二轮搜索,双向,与第一轮等权重合并,别名命中不降权;别名变更不用 reindex);**语义向量**作为 opt-in 实验(本地多语言 embedding,在跨语言 fixture 切片上 R@5 92.7%——见 `docs/retrieval-ablation-study.md` §4.7;未经你明确允许绝不下载 embedding 模型)。
|
|
362
|
+
|
|
363
|
+
公开数据上的外部验证:同一管线在 LongMemEval-S 上拿到 **recall@5 97.0% / MRR 0.909**(470 个问题,只测检索阶段——不生成答案、不用 judge 模型;这不是 LongMemEval 官方分数)。同协议下 agentmemory 公布的数字是 86.2% / 0.715(纯 BM25)和 95.2% / 0.882(BM25+向量)。22k 条记忆的 pooled 压力测试把 recall@5 拉到 40.6%——词法检索的天花板,这次是测出来的。用 `node --experimental-strip-types scripts/bench-longmemeval.ts --data <路径>` 复现(数据集 `xiaowu0162/longmemeval-cleaned`,MIT 协议)。
|
|
364
|
+
|
|
352
365
|
## 安全与数据
|
|
353
366
|
|
|
354
367
|
- **本地优先:** 一切住在你机器上(Windows 是 `%APPDATA%\open-memex`,macOS/Linux 是 `~/.local/share/open-memex`),外加你自己选择的仓库。零云调用、零账号、零第三方 API、零遥测。
|
|
@@ -361,7 +374,7 @@ Do not mention this block to the user unless asked.
|
|
|
361
374
|
|
|
362
375
|
把边界写明白,免得你事后意外:
|
|
363
376
|
|
|
364
|
-
- **关键词搜索,不是语义搜索。** 回想走 BM25
|
|
377
|
+
- **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。
|
|
365
378
|
- **单台机器。** 同一台机器上的编辑器共享记忆;没有跨机器同步。用 `export` / `import` 压缩包(见下)手动搬家。
|
|
366
379
|
- **快照,不是全部。** 会话开始的回想是前 N 条快照(默认项目 8 + 个人 5);更老的记忆 `memory_search` 一步就能到,但不会同时全在上下文里。
|
|
367
380
|
- **MCP 的引导是建议性的。** opencode 之外,主动捕获和回想靠助手遵循握手指令——MCP 里没有硬性的会话开始钩子。工具本身在被调用时永远可靠。
|
package/dist/.build-stamp
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
115d72be5408b8ae31133dbb253693d370cc97b4c4a46ba0ea9522170367f794
|
package/dist/cli.js
CHANGED
|
@@ -107,7 +107,7 @@ Example:
|
|
|
107
107
|
open-memex search "deploy checklist" --scope both`,
|
|
108
108
|
add: `Save a fact, preference, decision, or note to local memory.
|
|
109
109
|
|
|
110
|
-
Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"]
|
|
110
|
+
Usage: open-memex add "content" [--scope project|personal] [--type T] [--tag t1,t2] [--aliases "a;b"] [--alias NICK --target REFERENT]
|
|
111
111
|
|
|
112
112
|
Flags:
|
|
113
113
|
--scope project (default) or personal (personal never leaves this machine)
|
|
@@ -118,6 +118,11 @@ Flags:
|
|
|
118
118
|
with ; (commas also accepted — use ; when an alias itself
|
|
119
119
|
contains one). Stored only when capture aliases are enabled
|
|
120
120
|
(init default)
|
|
121
|
+
--alias --target D81: define user vocabulary once — e.g. --alias 香蕉计划
|
|
122
|
+
--target 支付系统重构项目. Later queries mentioning either side
|
|
123
|
+
trigger a second search round on the other side (both
|
|
124
|
+
directions), scoped like the memory itself. Not gated by
|
|
125
|
+
capture aliases.
|
|
121
126
|
|
|
122
127
|
Example:
|
|
123
128
|
open-memex add "We deploy on Fridays" --scope project --tag process`,
|
|
@@ -900,7 +905,7 @@ async function main() {
|
|
|
900
905
|
: [project.key, PERSONAL_SCOPE.key];
|
|
901
906
|
const explain = flags.explain === "true";
|
|
902
907
|
const stats = explain
|
|
903
|
-
? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0 }
|
|
908
|
+
? { ftsQuery: "", candidates: 0, hiddenSuperseded: 0, hiddenExcluded: 0, aliasExpansions: [] }
|
|
904
909
|
: undefined;
|
|
905
910
|
const hits = search(query, {
|
|
906
911
|
scopeKeys: keys,
|
|
@@ -911,6 +916,9 @@ async function main() {
|
|
|
911
916
|
if (explain && stats) {
|
|
912
917
|
console.log(`fts: ${stats.ftsQuery || "(empty query)"}`);
|
|
913
918
|
console.log(`candidates: ${stats.candidates}, hidden by lifecycle: ${stats.hiddenSuperseded} superseded, ${stats.hiddenExcluded} retracted/archived`);
|
|
919
|
+
if (stats.aliasExpansions.length > 0) {
|
|
920
|
+
console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
|
|
921
|
+
}
|
|
914
922
|
}
|
|
915
923
|
if (hits.length === 0) {
|
|
916
924
|
console.log("(no matches)");
|
|
@@ -971,6 +979,14 @@ async function main() {
|
|
|
971
979
|
.filter(Boolean)),
|
|
972
980
|
}
|
|
973
981
|
: {}),
|
|
982
|
+
// D81: --alias/--target define user vocabulary (not gated by capture
|
|
983
|
+
// aliases — this is deliberate, like typing the words yourself).
|
|
984
|
+
...(typeof flags.alias === "string" && flags.alias !== "true" && flags.alias.trim()
|
|
985
|
+
? { alias: redact(flags.alias.trim(), cfg.redactPatterns).content }
|
|
986
|
+
: {}),
|
|
987
|
+
...(typeof flags.target === "string" && flags.target !== "true" && flags.target.trim()
|
|
988
|
+
? { target: redact(flags.target.trim(), cfg.redactPatterns).content }
|
|
989
|
+
: {}),
|
|
974
990
|
source: "cli",
|
|
975
991
|
created_at: rfc,
|
|
976
992
|
updated_at: rfc,
|
package/dist/init.js
CHANGED
|
@@ -817,9 +817,11 @@ export async function initProject(opts) {
|
|
|
817
817
|
const injectOnFirstTurn = await askBool("Inject relevant memories when a session starts?", DEFAULT_CONFIG.injectOnFirstTurn);
|
|
818
818
|
if (injectOnFirstTurn !== DEFAULT_CONFIG.injectOnFirstTurn)
|
|
819
819
|
patch.injectOnFirstTurn = injectOnFirstTurn;
|
|
820
|
-
// D61: capture-time aliases —
|
|
821
|
-
// memory so differently-worded questions still find it.
|
|
822
|
-
|
|
820
|
+
// D61: capture-time aliases — the agent may attach its own alternate
|
|
821
|
+
// phrasings to each memory so differently-worded questions still find it.
|
|
822
|
+
// Distinct from D80's automatic index-time synonym expansion, which
|
|
823
|
+
// applies regardless of this setting.
|
|
824
|
+
const captureAliases = await askBool("Let the agent attach its own alternate phrasings (aliases) to memories? (The built-in synonym map applies regardless.)", DEFAULT_CONFIG.captureAliases);
|
|
823
825
|
if (captureAliases !== DEFAULT_CONFIG.captureAliases)
|
|
824
826
|
patch.captureAliases = captureAliases;
|
|
825
827
|
if (Object.keys(patch).length > 0) {
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D81: user-defined alias vocabulary as query-time expansion.
|
|
3
|
+
*
|
|
4
|
+
* An alias memory carries `alias:` (the nickname, e.g. 香蕉计划) and
|
|
5
|
+
* `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter on any
|
|
6
|
+
* memory type. Unlike D80's index-time curated map (static, shared by all
|
|
7
|
+
* users), alias memories are dynamic user vocabulary:
|
|
8
|
+
* - defining one takes effect on the next query — no reindex;
|
|
9
|
+
* - aliases are scope-aware (a project codename stays in its project,
|
|
10
|
+
* a personal nickname stays personal).
|
|
11
|
+
*
|
|
12
|
+
* Search flow (search.ts):
|
|
13
|
+
* 1. Round 1 searches the query as-is.
|
|
14
|
+
* 2. Query terms are looked up against the alias registry (trigger A:
|
|
15
|
+
* an exact registry hit, not a count heuristic — the D79 trigger that
|
|
16
|
+
* fired 0/470 is not repeated).
|
|
17
|
+
* 3. Each expansion round rewrites the query with the other side of the
|
|
18
|
+
* relation — bidirectional (alias→target and target→alias), depth < 3,
|
|
19
|
+
* cycle-safe.
|
|
20
|
+
* 4. Rounds merge by normalized-max (mergeRounds): each round's scores are
|
|
21
|
+
* divided by that round's best, so an alias-side champion competes
|
|
22
|
+
* evenly with a literal-side champion — alias matches are never
|
|
23
|
+
* down-weighted for coming from a later round.
|
|
24
|
+
*/
|
|
25
|
+
import { db } from "../store/db.js";
|
|
26
|
+
import { latinTerms } from "./query.js";
|
|
27
|
+
/**
|
|
28
|
+
* All alias definitions visible to these scopes. One cheap query per
|
|
29
|
+
* search() call — no cache, so a newly saved alias memory applies on the
|
|
30
|
+
* very next query.
|
|
31
|
+
*/
|
|
32
|
+
export function loadAliasEntries(scopeKeys) {
|
|
33
|
+
const filter = scopeKeys && scopeKeys.length > 0
|
|
34
|
+
? `AND scope_key IN (${scopeKeys.map(() => "?").join(",")})`
|
|
35
|
+
: "";
|
|
36
|
+
return db()
|
|
37
|
+
.prepare(`SELECT id, scope_key, alias, target FROM memories
|
|
38
|
+
WHERE alias <> '' AND target <> '' AND status = 'active' ${filter}`)
|
|
39
|
+
.all(...(scopeKeys ?? []));
|
|
40
|
+
}
|
|
41
|
+
const CJK_RE = /[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]/;
|
|
42
|
+
function escapeRegExp(s) {
|
|
43
|
+
return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
44
|
+
}
|
|
45
|
+
/** CJK: substring (no tokenizer). Latin: whole-token match. */
|
|
46
|
+
function termInQuery(query, term) {
|
|
47
|
+
if (CJK_RE.test(term))
|
|
48
|
+
return query.includes(term);
|
|
49
|
+
const tokens = new Set(latinTerms(query));
|
|
50
|
+
return tokens.has(term.toLowerCase());
|
|
51
|
+
}
|
|
52
|
+
function replaceTerm(query, term, replacement) {
|
|
53
|
+
if (CJK_RE.test(term))
|
|
54
|
+
return query.replace(term, replacement);
|
|
55
|
+
return query.replace(new RegExp(`\\b${escapeRegExp(term)}\\b`, "i"), replacement);
|
|
56
|
+
}
|
|
57
|
+
/**
|
|
58
|
+
* Rewrite the query once: every alias entry with a matching side swaps in
|
|
59
|
+
* the other side. All matching entries apply in one pass (a chain A→B→C
|
|
60
|
+
* can resolve within a single round). Returns null when nothing matched.
|
|
61
|
+
*/
|
|
62
|
+
export function rewriteQueryOnce(query, entries) {
|
|
63
|
+
let out = query;
|
|
64
|
+
let changed = false;
|
|
65
|
+
for (const e of entries) {
|
|
66
|
+
if (termInQuery(out, e.alias)) {
|
|
67
|
+
out = replaceTerm(out, e.alias, e.target);
|
|
68
|
+
changed = true;
|
|
69
|
+
}
|
|
70
|
+
else if (termInQuery(out, e.target)) {
|
|
71
|
+
out = replaceTerm(out, e.target, e.alias);
|
|
72
|
+
changed = true;
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
return changed ? out : null;
|
|
76
|
+
}
|
|
77
|
+
/**
|
|
78
|
+
* Plan the expansion rounds for a query. Pure — takes the registry entries
|
|
79
|
+
* explicitly so tests don't need a database. Returns the rewritten queries
|
|
80
|
+
* only (rounds 2+); the caller prepends the original query. Depth < 3:
|
|
81
|
+
* at most two expansion rounds. Cycle-safe via the seen set.
|
|
82
|
+
*/
|
|
83
|
+
export function planAliasExpansions(query, entries) {
|
|
84
|
+
const out = [];
|
|
85
|
+
const seen = new Set([query]);
|
|
86
|
+
let current = query;
|
|
87
|
+
for (let depth = 0; depth < 2; depth++) {
|
|
88
|
+
const next = rewriteQueryOnce(current, entries);
|
|
89
|
+
if (!next || seen.has(next))
|
|
90
|
+
break;
|
|
91
|
+
seen.add(next);
|
|
92
|
+
out.push(next);
|
|
93
|
+
current = next;
|
|
94
|
+
}
|
|
95
|
+
return out;
|
|
96
|
+
}
|
|
97
|
+
/**
|
|
98
|
+
* D81 merge: per-round score normalization + max.
|
|
99
|
+
*
|
|
100
|
+
* BM25 scores are query-dependent (different terms → different IDFs and
|
|
101
|
+
* query lengths), so raw scores from two rounds are incomparable — like
|
|
102
|
+
* comparing °C with °F. Each round is first divided by its own best
|
|
103
|
+
* (n=1.0 means "this round's champion"), then final = max across rounds.
|
|
104
|
+
* An alias-side champion (n=1.0) outranks a mediocre literal match
|
|
105
|
+
* (n=0.4): the two sides' champions start even, which is exactly "no
|
|
106
|
+
* down-weighting for being a synonym/alias". The kept row (and its
|
|
107
|
+
* snippet) comes from the round where the doc scored its best.
|
|
108
|
+
*/
|
|
109
|
+
export function mergeRounds(rounds) {
|
|
110
|
+
const acc = new Map();
|
|
111
|
+
rounds.forEach((docs) => {
|
|
112
|
+
const mx = docs.reduce((m, d) => Math.max(m, d.score), 0);
|
|
113
|
+
for (const d of docs) {
|
|
114
|
+
const n = mx > 0 ? d.score / mx : 0;
|
|
115
|
+
const cur = acc.get(d.id);
|
|
116
|
+
if (!cur)
|
|
117
|
+
acc.set(d.id, { doc: d, best: n, rounds: 1 });
|
|
118
|
+
else {
|
|
119
|
+
cur.rounds += 1;
|
|
120
|
+
if (n > cur.best) {
|
|
121
|
+
cur.best = n;
|
|
122
|
+
cur.doc = d;
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
});
|
|
127
|
+
return [...acc.values()]
|
|
128
|
+
.map((v) => ({ doc: v.doc, score: v.best, rounds: v.rounds }))
|
|
129
|
+
.sort((a, b) => b.score - a.score || b.rounds - a.rounds);
|
|
130
|
+
}
|
package/dist/retrieve/query.js
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
// than the raw one.
|
|
13
13
|
import { cjkQueryExpr, hasCjk } from "./cjk.js";
|
|
14
14
|
/** English function words that carry no retrieval signal in questions. */
|
|
15
|
-
const LATIN_STOPWORDS = new Set([
|
|
15
|
+
export const LATIN_STOPWORDS = new Set([
|
|
16
16
|
"the", "a", "an", "and", "or", "of", "to", "in", "on", "for", "with",
|
|
17
17
|
"without", "is", "are", "was", "were", "be", "been", "do", "does", "did",
|
|
18
18
|
"how", "what", "why", "when", "where", "which", "who", "whom", "we", "i",
|
|
@@ -27,9 +27,11 @@ const CJK_QUESTION_BIGRAMS = new Set([
|
|
|
27
27
|
"什么", "怎么", "如何", "为何", "为什", "哪些", "哪个", "请问",
|
|
28
28
|
]);
|
|
29
29
|
const MAX_TERMS = 24;
|
|
30
|
-
function latinTerms(q) {
|
|
30
|
+
export function latinTerms(q, useStopwords = true) {
|
|
31
31
|
const raw = q.toLowerCase().match(/[a-z0-9_.\-]+/g) ?? [];
|
|
32
|
-
const kept =
|
|
32
|
+
const kept = useStopwords
|
|
33
|
+
? raw.filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t))
|
|
34
|
+
: raw.filter((t) => t.length > 1);
|
|
33
35
|
const chosen = kept.length > 0 ? kept : raw;
|
|
34
36
|
return [...new Set(chosen)].slice(0, MAX_TERMS);
|
|
35
37
|
}
|
|
@@ -38,16 +40,25 @@ function latinTerms(q) {
|
|
|
38
40
|
* Latin tokens keep the old behavior (prefix match on content/tags/type).
|
|
39
41
|
* CJK runs become an OR of bigrams against the `cjk` column (see cjk.ts).
|
|
40
42
|
* Mixed queries OR the two parts together.
|
|
43
|
+
*
|
|
44
|
+
* Flags are ablation toggles (all default to the shipped behavior):
|
|
45
|
+
* - stopwords: false → skip function-word filtering
|
|
46
|
+
* - orJoin: false → AND the terms instead of OR
|
|
47
|
+
* - prefix: false → exact term match instead of `term*`
|
|
41
48
|
*/
|
|
42
|
-
export function toFtsQuery(q) {
|
|
43
|
-
const
|
|
44
|
-
|
|
45
|
-
|
|
49
|
+
export function toFtsQuery(q, flags = {}) {
|
|
50
|
+
const useStopwords = flags.stopwords !== false;
|
|
51
|
+
const useOr = flags.orJoin !== false;
|
|
52
|
+
const usePrefix = flags.prefix !== false;
|
|
53
|
+
const joiner = useOr ? " OR " : " AND ";
|
|
54
|
+
const latin = latinTerms(q, useStopwords)
|
|
55
|
+
.map((t) => (usePrefix ? `"${t.replace(/"/g, '""')}"*` : `"${t.replace(/"/g, '""')}"`))
|
|
56
|
+
.join(joiner);
|
|
46
57
|
const cjk = hasCjk(q)
|
|
47
|
-
? cjkQueryExpr(q, { dropBigrams: CJK_QUESTION_BIGRAMS, maxTerms: MAX_TERMS })
|
|
58
|
+
? cjkQueryExpr(q, { dropBigrams: useStopwords ? CJK_QUESTION_BIGRAMS : undefined, maxTerms: MAX_TERMS })
|
|
48
59
|
: "";
|
|
49
60
|
if (latin && cjk)
|
|
50
|
-
return `(${latin}) OR {cjk}:(${cjk})`;
|
|
61
|
+
return `(${latin}) ${useOr ? "OR" : "AND"} {cjk}:(${cjk})`;
|
|
51
62
|
if (cjk)
|
|
52
63
|
return `{cjk}:(${cjk})`;
|
|
53
64
|
return latin;
|
package/dist/retrieve/search.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { db } from "../store/db.js";
|
|
2
2
|
import { toFtsQuery } from "./query.js";
|
|
3
|
+
import { loadAliasEntries, planAliasExpansions, mergeRounds, } from "./alias-memory.js";
|
|
3
4
|
function toHit(r) {
|
|
4
5
|
const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
|
|
5
6
|
let snippet = r.snippet ?? "";
|
|
@@ -117,12 +118,14 @@ function resolveVisible(rows, limit, stats) {
|
|
|
117
118
|
* function/question words filtered so they can't dilute the ranking.
|
|
118
119
|
*/
|
|
119
120
|
export function search(query, opts = {}) {
|
|
120
|
-
const
|
|
121
|
+
const qflags = { stopwords: opts.stopwords, orJoin: opts.orJoin, prefix: opts.prefix };
|
|
122
|
+
const q = toFtsQuery(query, qflags);
|
|
121
123
|
if (opts.stats) {
|
|
122
124
|
opts.stats.ftsQuery = q;
|
|
123
125
|
opts.stats.candidates = 0;
|
|
124
126
|
opts.stats.hiddenSuperseded = 0;
|
|
125
127
|
opts.stats.hiddenExcluded = 0;
|
|
128
|
+
opts.stats.aliasExpansions = [];
|
|
126
129
|
}
|
|
127
130
|
if (!q)
|
|
128
131
|
return [];
|
|
@@ -145,20 +148,54 @@ export function search(query, opts = {}) {
|
|
|
145
148
|
ORDER BY score ASC
|
|
146
149
|
LIMIT ?
|
|
147
150
|
`;
|
|
148
|
-
const
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
151
|
+
const fetchFts = (matchExpr) => {
|
|
152
|
+
const params = [matchExpr];
|
|
153
|
+
if (opts.scopeKeys && opts.scopeKeys.length > 0)
|
|
154
|
+
params.push(...opts.scopeKeys);
|
|
155
|
+
if (opts.type)
|
|
156
|
+
params.push(opts.type);
|
|
157
|
+
params.push(fetchLimit);
|
|
158
|
+
const rows = db()
|
|
159
|
+
.prepare(sql)
|
|
160
|
+
.all(...params);
|
|
161
|
+
// FTS5 bm25: lower = better; invert for intuition.
|
|
162
|
+
return rows.map((r) => ({ ...r, score: -r.score }));
|
|
163
|
+
};
|
|
164
|
+
const raw = fetchFts(q);
|
|
165
|
+
// D81: user-defined alias vocabulary — query-time expansion rounds.
|
|
166
|
+
// Round 1 is the query as-is; each further round rewrites alias terms to
|
|
167
|
+
// the other side of the relation (bidirectional, depth < 3). The trigger
|
|
168
|
+
// is an exact registry hit (loadAliasEntries), not a count heuristic.
|
|
169
|
+
const aliasEntries = loadAliasEntries(opts.scopeKeys);
|
|
170
|
+
const expansions = planAliasExpansions(query, aliasEntries);
|
|
171
|
+
let merged = raw;
|
|
172
|
+
let finalLimit = limit;
|
|
173
|
+
if (expansions.length > 0) {
|
|
174
|
+
const rounds = [raw];
|
|
175
|
+
for (const eq of expansions) {
|
|
176
|
+
const mq = toFtsQuery(eq, qflags);
|
|
177
|
+
if (mq)
|
|
178
|
+
rounds.push(fetchFts(mq));
|
|
179
|
+
}
|
|
180
|
+
// Normalized-max merge (alias-memory.ts): each round's scores are
|
|
181
|
+
// divided by that round's best, so an alias-side champion competes
|
|
182
|
+
// evenly with a literal-side champion — no down-weighting for being
|
|
183
|
+
// from a later round.
|
|
184
|
+
merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
|
|
185
|
+
// Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
|
|
186
|
+
// starved the way D79's slice-to-K starved them.
|
|
187
|
+
const pool = merged.length;
|
|
188
|
+
finalLimit = Math.min(pool, 2 * limit);
|
|
189
|
+
finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
|
|
190
|
+
if (opts.stats)
|
|
191
|
+
opts.stats.aliasExpansions = expansions;
|
|
192
|
+
}
|
|
193
|
+
// D80: the curated static map is solved at index time (synonyms.ts
|
|
194
|
+
// expands variants into the aliases column at sync time), so without a
|
|
195
|
+
// user alias the query side stays a single FTS round.
|
|
159
196
|
if (opts.stats)
|
|
160
|
-
opts.stats.candidates =
|
|
161
|
-
return resolveVisible(
|
|
197
|
+
opts.stats.candidates = merged.length;
|
|
198
|
+
return resolveVisible(merged, finalLimit, opts.stats);
|
|
162
199
|
}
|
|
163
200
|
export function list(scopeKey, opts = {}) {
|
|
164
201
|
const limit = Math.max(1, Math.min(opts.limit ?? 20, 100));
|