open-memex 0.7.3 → 0.8.0-alpha.2

package/AGENTS.md

@@ -202,7 +202,8 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
   languages in sync); behavior changes → both READMEs
   and the frozen `docs/V2-DESIGN.md` (append a `D<n>` decision entry, never rewrite history);
   new commands → README CLI sections + this file's Commands. A change without its docs
-  is not done.
+  is not done. Anything deferred as "later" goes into the near-term backlog in
+  `docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
 - **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
   in-development version. New features on a dev branch bump the minor on the alpha
   line (`0.3.0` → `0.4.0-alpha.1`); fixes bump the patch (`-alpha.1` → `-alpha.2`).

package/CHANGELOG.md

@@ -9,6 +9,18 @@ the `@alpha` tag are development snapshots and are not listed individually.
 
 ## [Unreleased]
 
+- Index-time synonym/translation expansion (D80, supersedes D79): the curated map (`src/retrieve/synonyms.ts` — EN dev-domain groups, ZH groups and EN↔ZH translation pairs in one bidirectional list) is now applied at sync/index time — `writeRow` expands each memory's keywords (`expansionVariantsForDoc()`, capped at 32) into the indexed `aliases` column, so any related word matches in a plain round-1 query. The query-time second-chance round is removed (`search()` is a single FTS round; `SearchStats.secondRound` gone); `expandQueryWithSynonyms` stays as a bench-only export. Fixture (51 memories / 41 queries): recall@1 0.732 → 0.756, recall@5 0.854 → **0.976**, MRR 0.791 → **0.859** — all 5 EN↔ZH vocabulary misses rescued at rank 1–3. Map updates need a reindex to backfill.
+
+- Semantic ranking experiment (§4.7 of `docs/retrieval-ablation-study.md`): local multilingual embeddings (`Xenova/paraphrase-multilingual-MiniLM-L12-v2`, 384-dim, no API) via `scripts/bench-semantic.ts` (new `--model` flag). Fixture: vec-only R@5 92.7% / MRR 0.881 beats BM25-only (80.5% / 0.752); naive RRF hybrid underperforms vec-only (87.8%) — fusion needs design. Pooled 22k long sessions: vec-only collapses (3.4%) from mean-pooling dilution on 10k-char docs — a measured methodology artifact, not a verdict for short memories; chunked rerun deliberately skipped.
+
+- LongMemEval-S retrieval bench (`scripts/bench-longmemeval.ts`): 470 questions, recall_any@k + MRR on public data, per-question isolated corpora plus a pooled 22k-memory stress mode, with synonym on/off ablation. Result: R@5 97.0% / MRR 0.909 (ahead of agentmemory’s published BM25-only, level with their BM25+vector). Finding: the count-based second-round trigger fired 0/470 — logged as a D79 redesign follow-up, not a scale answer.
+
+- Query-time synonym expansion (D79, retrieval layer 4): `search()` runs a second-chance round when the first FTS pass is thin, re-running with curated synonym variants OR-ed in (`src/retrieve/synonyms.ts`, EN dev-domain + ZH groups, no model). Round-two rows only fill gaps; round-one ranking is never diluted. `SearchStats.secondRound` records it (shown in `search --explain`). Baseline with the round: recall@1 0.81, recall@5 0.95, MRR 0.87 over 37 fixture queries. **Superseded by D80 (2026-10-07)** — see above.
+
+- Retrieval eval harness (D77 layer 1): `scripts/retrieval-eval.ts` runs a checked-in synthetic fixture (48 memories + 34 queries: exact terms, paraphrases, synonyms, Chinese, cross-language hard cases) against the FTS5 baseline and reports recall@k/MRR. Current baseline: recall@1 0.85, recall@5 0.97, MRR 0.90 (single miss: English query vs Chinese-only memory). Published in both READMEs with methodology notes.
+
+- Added `test-full` npm script (`node --experimental-strip-types scripts/test-full.ts`).
+
 ## [0.7.3] - 2026-10-07
 
 - Tool-definition quality pass (Glama TDQS 4.0/5 on v0.7.2): fixed `memory_pr_status` declaring `readOnlyHint` while `apply=true` mutates local files; added explicit MCP annotations (read-only / destructive / open-world) to all 11 tools; descriptions now route between sibling tools (propose vs submit vs promote, resolve vs pr_status, forget vs supersede), state error behavior and real defaults (search limit 8, list limit 20), and the search/list `type` and `limit` parameters are documented. D76 records the rule; `scripts/smoke-mcp.ts` guards the pr_status annotation.

package/README.md

@@ -22,6 +22,10 @@ memory on your machine.
   explicitly share it.
 - **One memory, every agent**: wire up several editors with one command; they
   share the same memory instead of keeping separate silos.
+- **Precise synonym search**: `ship` finds `deploy`, `单点登录` finds `SSO` —
+  a curated multilingual map of equivalents is indexed with every memory at
+  write time, so no related wording misses the right answer. Deterministic
+  and auditable, no embedding model required.
 
 ![Terminal demo: two memories saved on Monday, recalled by search in a fresh session on Friday](./docs/assets/open-memex-demo.svg)
 
@@ -516,6 +520,37 @@ time for the rest. Both top-N counts are configurable (see [Config](#config)).
 For MCP clients this block is delivered as handshake guidance the agent follows;
 the opencode plugin injects it directly on the first turn.
 
+### Retrieval evaluation
+
+We measure recall on a checked-in synthetic fixture instead of asserting it:
+51 memories + 41 queries (exact terms, paraphrases, synonyms, Chinese,
+cross-language hard cases, and synonym-only queries) — reproduce with
+`node --experimental-strip-types scripts/retrieval-eval.ts`.
+Current baseline, with index-time synonym/translation expansion (D80):
+**recall@1 0.76, recall@5 0.98, MRR 0.86**.
+Small synthetic corpus, so read it as a regression guard and a starting point,
+not a real-world claim.
+
+Three layers, each covering what the others miss: **BM25 keyword search**
+(SQLite FTS5 — exact words, fast, explainable via `search --explain`);
+**index-time synonym/translation expansion** (D80 — a curated, checked-in map
+of dev-domain equivalents and EN↔ZH translation pairs expanded into the index
+at write time, so `ship` finds `deploy` and `单点登录` finds `SSO` in a plain
+single-round query; deterministic, no model download); and **semantic vectors**
+as an opt-in experiment (local multilingual embeddings measured at R@5 92.7%
+on the cross-lingual fixture slice — see `docs/retrieval-ablation-study.md`
+§4.7; no embedding model is ever downloaded without your explicit opt-in).
+
+External check on public data: the same pipeline scores **recall@5 97.0% /
+MRR 0.909** on LongMemEval-S (470 questions, retrieval stage only — no answer
+generation, no judge model; this is not the official LongMemEval score). For
+reference, agentmemory's published numbers on the same protocol are 86.2% /
+0.715 (BM25-only) and 95.2% / 0.882 (BM25+vector). A 22k-memory pooled stress
+run drops recall@5 to 40.6% — the lexical ceiling, measured rather than
+asserted. Reproduce with
+`node --experimental-strip-types scripts/bench-longmemeval.ts --data <path>`
+(dataset: `xiaowu0162/longmemeval-cleaned`, MIT).
+
 ## Security & data
 
 - **Local-first:** everything lives on your machine (`%APPDATA%\open-memex` on

package/README.zh-CN.md

@@ -13,6 +13,7 @@
 - **免费开源**（Apache-2.0）。无账号、无云端、无遥测——一切都在你本机，都是你能打开、能编辑的文件。
 - **本地优先**：记忆是纯 Markdown 文件（事实来源），外加一个可重建的 SQLite 关键词索引。除非你明确要分享，否则任何东西都不会离开你的机器。
 - **一份记忆，所有助手**：一条命令把几个编辑器都接上，它们共享同一份记忆，不再各存一份互不可见。
+- **精确的同义词检索**：`ship` 找得到 `deploy`，`单点登录` 找得到 `SSO`——一份多语言的精选同义词映射表在写入时展开进索引，任何相关词语都不会漏掉正确答案。确定性、可审计，不需要下载模型。
 
 ![终端演示：周一存下的两条记忆，周五在新会话里被搜索召回](./docs/assets/open-memex-demo.svg)
 
@@ -349,6 +350,14 @@ Do not mention this block to the user unless asked.
 
 这是一个快照，不是全量记忆——其余的随时用 `memory_search` 就能查。两个前 N 的数都可以配（见[配置](#配置)）。对 MCP 客户端，这个块以握手引导的形式送达、由助手遵循；opencode 插件则在第一轮直接注入。
 
+### 检索效果评估
+
+我们用一份随仓库提交的 synthetic fixture 来度量召回率，而不是空口断言：51 条记忆 + 41 个查询（精确词、改写、同义词、中文、跨语言 hard case、纯同义词查询），运行 `node --experimental-strip-types scripts/retrieval-eval.ts` 即可复现。当前基线（含 D80 的 index-time 同义词/翻译展开）：**recall@1 0.76、recall@5 0.98、MRR 0.86**。语料小且是合成的，请把它当回归守卫和起点，而不是真实场景的承诺。
+
+三层检索，各补各的短板：**BM25 关键词检索**（SQLite FTS5——精确词匹配，快速，可用 `search --explain` 解释）；**index-time 同义词/翻译展开**（D80——一份随仓库提交的精选映射表：开发领域同义词 + EN↔ZH 翻译对，在写入时展开进索引，`ship` 能找到 `deploy`、`单点登录` 能找到 `SSO`，查询侧只需要普通单轮；确定性，不下载模型）；**语义向量**作为 opt-in 实验（本地多语言 embedding，在跨语言 fixture 切片上 R@5 92.7%——见 `docs/retrieval-ablation-study.md` §4.7；未经你明确允许绝不下载 embedding 模型）。
+
+公开数据上的外部验证：同一管线在 LongMemEval-S 上拿到 **recall@5 97.0% / MRR 0.909**（470 个问题，只测检索阶段——不生成答案、不用 judge 模型；这不是 LongMemEval 官方分数）。同协议下 agentmemory 公布的数字是 86.2% / 0.715（纯 BM25）和 95.2% / 0.882（BM25+向量）。22k 条记忆的 pooled 压力测试把 recall@5 拉到 40.6%——词法检索的天花板，这次是测出来的。用 `node --experimental-strip-types scripts/bench-longmemeval.ts --data <路径>` 复现（数据集 `xiaowu0162/longmemeval-cleaned`，MIT 协议）。
+
 ## 安全与数据
 
 - **本地优先：** 一切住在你机器上（Windows 是 `%APPDATA%\open-memex`，macOS/Linux 是 `~/.local/share/open-memex`），外加你自己选择的仓库。零云调用、零账号、零第三方 API、零遥测。

package/dist/.build-stamp

@@ -1 +1 @@
-c5a82fa9f711dea3b1e8787d551424e978d606435b08f83d51ba5aaf0996e9ff
+9f341cad8c580b840308fbdb525de70a108bc40aa372f47c4b248baa67ab14f8

package/dist/retrieve/query.js

@@ -12,7 +12,7 @@
 // than the raw one.
 import { cjkQueryExpr, hasCjk } from "./cjk.js";
 /** English function words that carry no retrieval signal in questions. */
-const LATIN_STOPWORDS = new Set([
+export const LATIN_STOPWORDS = new Set([
     "the", "a", "an", "and", "or", "of", "to", "in", "on", "for", "with",
     "without", "is", "are", "was", "were", "be", "been", "do", "does", "did",
     "how", "what", "why", "when", "where", "which", "who", "whom", "we", "i",
@@ -27,9 +27,11 @@ const CJK_QUESTION_BIGRAMS = new Set([
     "什么", "怎么", "如何", "为何", "为什", "哪些", "哪个", "请问",
 ]);
 const MAX_TERMS = 24;
-function latinTerms(q) {
+function latinTerms(q, useStopwords = true) {
     const raw = q.toLowerCase().match(/[a-z0-9_.\-]+/g) ?? [];
-    const kept = raw.filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t));
+    const kept = useStopwords
+        ? raw.filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t))
+        : raw.filter((t) => t.length > 1);
     const chosen = kept.length > 0 ? kept : raw;
     return [...new Set(chosen)].slice(0, MAX_TERMS);
 }
@@ -38,16 +40,25 @@ function latinTerms(q) {
  * Latin tokens keep the old behavior (prefix match on content/tags/type).
  * CJK runs become an OR of bigrams against the `cjk` column (see cjk.ts).
  * Mixed queries OR the two parts together.
+ *
+ * Flags are ablation toggles (all default to the shipped behavior):
+ * - stopwords: false  → skip function-word filtering
+ * - orJoin: false     → AND the terms instead of OR
+ * - prefix: false     → exact term match instead of `term*`
  */
-export function toFtsQuery(q) {
-    const latin = latinTerms(q)
-        .map((t) => `"${t.replace(/"/g, '""')}"*`)
-        .join(" OR ");
+export function toFtsQuery(q, flags = {}) {
+    const useStopwords = flags.stopwords !== false;
+    const useOr = flags.orJoin !== false;
+    const usePrefix = flags.prefix !== false;
+    const joiner = useOr ? " OR " : " AND ";
+    const latin = latinTerms(q, useStopwords)
+        .map((t) => (usePrefix ? `"${t.replace(/"/g, '""')}"*` : `"${t.replace(/"/g, '""')}"`))
+        .join(joiner);
     const cjk = hasCjk(q)
-        ? cjkQueryExpr(q, { dropBigrams: CJK_QUESTION_BIGRAMS, maxTerms: MAX_TERMS })
+        ? cjkQueryExpr(q, { dropBigrams: useStopwords ? CJK_QUESTION_BIGRAMS : undefined, maxTerms: MAX_TERMS })
         : "";
     if (latin && cjk)
-        return `(${latin}) OR {cjk}:(${cjk})`;
+        return `(${latin}) ${useOr ? "OR" : "AND"} {cjk}:(${cjk})`;
     if (cjk)
         return `{cjk}:(${cjk})`;
     return latin;

package/dist/retrieve/search.js

@@ -117,7 +117,8 @@ function resolveVisible(rows, limit, stats) {
  * function/question words filtered so they can't dilute the ranking.
  */
 export function search(query, opts = {}) {
-    const q = toFtsQuery(query);
+    const qflags = { stopwords: opts.stopwords, orJoin: opts.orJoin, prefix: opts.prefix };
+    const q = toFtsQuery(query, qflags);
     if (opts.stats) {
         opts.stats.ftsQuery = q;
         opts.stats.candidates = 0;
@@ -145,17 +146,23 @@ export function search(query, opts = {}) {
     ORDER BY score ASC
     LIMIT ?
   `;
-    const params = [q];
-    if (opts.scopeKeys && opts.scopeKeys.length > 0)
-        params.push(...opts.scopeKeys);
-    if (opts.type)
-        params.push(opts.type);
-    params.push(fetchLimit);
-    const rows = db()
-        .prepare(sql)
-        .all(...params);
-    // FTS5 bm25: lower = better; invert for intuition.
-    const raw = rows.map((r) => ({ ...r, score: -r.score }));
+    const fetchFts = (matchExpr) => {
+        const params = [matchExpr];
+        if (opts.scopeKeys && opts.scopeKeys.length > 0)
+            params.push(...opts.scopeKeys);
+        if (opts.type)
+            params.push(opts.type);
+        params.push(fetchLimit);
+        const rows = db()
+            .prepare(sql)
+            .all(...params);
+        // FTS5 bm25: lower = better; invert for intuition.
+        return rows.map((r) => ({ ...r, score: -r.score }));
+    };
+    const raw = fetchFts(q);
+    // D80: vocabulary mismatch is solved at index time (synonyms.ts expands
+    // curated variants into the aliases column at sync time), so the query
+    // side is a single FTS round — no trigger, no second chance.
     if (opts.stats)
         opts.stats.candidates = raw.length;
     return resolveVisible(raw, limit, opts.stats);

package/dist/retrieve/synonyms.js

@@ -0,0 +1,183 @@
+/**
+ * Index-time synonym/translation expansion (D80).
+ *
+ * Stone's rule: solve vocabulary mismatch at write time, so any related
+ * word finds the answer with a plain round-1 query. At sync/index time
+ * every memory's keywords are expanded through a curated, checked-in
+ * equivalence map and the variants are written into the indexed `aliases`
+ * column (FTS + CJK bigram columns both pick them up via sync.ts). The
+ * query side stays simple: one FTS round, no trigger, no second chance.
+ *
+ * Deliberately NOT an embedding model or an LLM call: deterministic,
+ * auditable, zero dependencies, and the failure mode (a bad synonym)
+ * is a visible line in this file, not a silent vector.
+ *
+ * Groups are bidirectional — any member expands to the other members.
+ * Keep groups tight: an over-broad synonym costs precision on every
+ * query, and at index time a bad entry pollutes every document instead
+ * of one query. When in doubt, leave it out. A map update only reaches
+ * already-indexed memories after a reindex.
+ *
+ * Design note (D80): the map covers the *precise* slice — EN↔ZH
+ * translation pairs and dev-domain equivalents where embeddings are
+ * fuzzy (ship/deploy are neighbors in vector space but "单点登录"/SSO
+ * is an exact pair no embedding guarantees). Embeddings cover the
+ * semantic long tail; this map covers what must never be missed.
+ */
+import { LATIN_STOPWORDS } from "./query.js";
+const CJK_RE = /[\u4e00-\u9fff]/;
+/**
+ * Curated equivalence groups. EN dev-domain vocabulary, ZH equivalents,
+ * and EN↔ZH translation pairs live in one bidirectional group: any member
+ * expands to all the others, in either direction.
+ */
+const GROUPS = [
+    ["deploy", "ship", "release", "rollout", "publish", "部署", "发布", "上线"],
+    ["test", "testing", "tests", "测试", "单测", "集成测试"],
+    ["regression test", "回归测试"],
+    ["rollback", "revert", "回滚", "撤回", "撤销", "回退"],
+    ["incident", "outage", "postmortem", "故障", "事故", "宕机"],
+    ["auth", "authentication", "login", "sso", "single sign-on", "单点登录", "登录"],
+    ["secret", "secrets", "credential", "credentials", "token", "密钥"],
+    ["database", "db", "数据库"],
+    ["backup", "restore", "备份", "恢复"],
+    ["retention", "保留"],
+    ["production", "生产", "生产环境"],
+    ["staging", "预发布环境", "预发布"],
+    ["wipe", "wipes", "清空", "清除"],
+    ["nightly", "night", "每晚"],
+    ["thursday", "周四", "星期四"],
+    ["mandatory", "强制", "必须"],
+    ["flaky", "flake", "unreliable", "intermittent"],
+    ["bug", "defect", "issue"],
+    ["ci", "pipeline"],
+    ["cache", "caching", "cached"],
+    ["invalidate", "invalidation", "expire", "expiry"],
+    ["oncall", "on-call", "pager"],
+    ["migration", "migrate", "migrating"],
+    ["index", "indexing", "indexed"],
+    ["slow", "latency", "performance", "perf"],
+    ["refactor", "refactoring"],
+    ["merge", "merging"],
+    ["branch", "branches"],
+    ["commit", "commits"],
+    ["review", "code review", "评审", "代码评审"],
+    ["mock", "stub"],
+    ["debug", "debugging", "troubleshoot", "troubleshooting"],
+    ["log", "logs", "logging"],
+    ["monitor", "monitoring", "alert", "alerting"],
+    ["scale", "scaling"],
+    ["timeout", "timeouts"],
+    ["retry", "retries"],
+    ["queue", "job", "worker"],
+    ["api", "endpoint"],
+    ["docker", "container", "image"],
+    ["kubernetes", "k8s"],
+    ["env", "environment"],
+    ["config", "configuration", "setting", "settings"],
+    ["hotfix", "hot fix"],
+    ["estimate", "estimation"],
+    ["乱码", "编码问题"],
+    ["并发", "并行"],
+    ["重构"],
+    ["规范", "约定"],
+];
+const termLookup = new Map();
+const phraseEntries = [];
+const cjkEntries = [];
+for (const g of GROUPS) {
+    const cjkKeys = g.filter((m) => CJK_RE.test(m));
+    if (cjkKeys.length > 0)
+        cjkEntries.push({ keys: cjkKeys, siblings: [...g] });
+    for (const m of g) {
+        if (CJK_RE.test(m))
+            continue;
+        const siblings = g.filter((x) => x !== m);
+        if (m.includes(" "))
+            phraseEntries.push({ phrase: m.toLowerCase(), siblings });
+        else
+            termLookup.set(m.toLowerCase(), siblings);
+    }
+}
+/**
+ * Synonym variants for one lowercased Latin term (excluding the term
+ * itself). Empty array when the term has no curated synonyms.
+ */
+export function synonymVariants(term) {
+    return termLookup.get(term.toLowerCase()) ?? [];
+}
+/** Cap on auto-generated variants per memory — bounds index growth. */
+export const MAX_AUTO_VARIANTS = 32;
+/**
+ * Index-time expansion for one memory's text (body + tags). Keyword-
+ * centric (Stone): extract content terms, expand each through the
+ * curated map, return variants not already present in the text.
+ * Pure logic — safe to unit test, runs once per memory at sync time.
+ */
+export function expansionVariantsForDoc(text) {
+    const lower = text.toLowerCase();
+    // Content terms only: function words carry no signal (same filter the
+    // query builder applies, so both sides agree on what a keyword is).
+    const latinTokens = new Set((lower.match(/[a-z0-9_.\-]+/g) ?? []).filter((t) => t.length > 1 && !LATIN_STOPWORDS.has(t)));
+    const out = [];
+    const seen = new Set();
+    const push = (v) => {
+        const key = v.toLowerCase();
+        if (seen.has(key) || out.length >= MAX_AUTO_VARIANTS)
+            return;
+        seen.add(key);
+        out.push(v);
+    };
+    // A variant is "already present" when the doc has it as a token (Latin),
+    // as a substring (Latin phrase), or as a substring (CJK — no tokenizer).
+    const present = (v) => CJK_RE.test(v)
+        ? text.includes(v)
+        : v.includes(" ")
+            ? lower.includes(v.toLowerCase())
+            : latinTokens.has(v.toLowerCase());
+    for (const t of latinTokens) {
+        for (const v of termLookup.get(t) ?? [])
+            if (!present(v))
+                push(v);
+    }
+    for (const { phrase, siblings } of phraseEntries) {
+        if (!lower.includes(phrase))
+            continue;
+        for (const v of siblings)
+            if (!present(v))
+                push(v);
+    }
+    for (const { keys, siblings } of cjkEntries) {
+        if (!keys.some((k) => text.includes(k)))
+            continue;
+        for (const v of siblings)
+            if (!present(v))
+                push(v);
+    }
+    return out;
+}
+/**
+ * Expand a raw query string with synonym variants (legacy query-time
+ * helper, kept for bench scripts). The product no longer expands at
+ * query time — D80 moved expansion to the index.
+ */
+export function expandQueryWithSynonyms(q) {
+    const extra = new Set();
+    const latinTerms = q.toLowerCase().match(/[a-z0-9_.\-]+/g) ?? [];
+    for (const t of latinTerms) {
+        for (const s of synonymVariants(t))
+            extra.add(s);
+    }
+    for (const { keys, siblings } of cjkEntries) {
+        if (keys.some((k) => q.includes(k))) {
+            for (const s of siblings)
+                extra.add(s);
+        }
+    }
+    // Don't re-add terms already in the query.
+    const have = new Set(latinTerms);
+    const additions = [...extra].filter((s) => !have.has(s.toLowerCase()) && !q.includes(s));
+    if (additions.length === 0)
+        return q;
+    return `${q} ${additions.join(" ")}`;
+}

package/dist/store/sync.js

@@ -2,6 +2,7 @@ import fs from "node:fs";
 import path from "node:path";
 import { db } from "./db.js";
 import { cjkIndexText } from "../retrieve/cjk.js";
+import { expansionVariantsForDoc } from "../retrieve/synonyms.js";
 import { contentHash, repairChain } from "./lifecycle.js";
 import { iterMemoryFiles, iterInRepoMemoryFiles, readMemoryFile, timeToMs, } from "./markdown.js";
 import { projectRoot, paths } from "../paths.js";
@@ -54,7 +55,14 @@ export function upsertFromFile(mf) {
 function writeRow(mf, mtimeMs) {
     const { fm, body, filePath } = mf;
     const tags = (fm.tags ?? []).join(",");
-    const aliases = (fm.aliases ?? []).join(",");
+    // D80 index-time expansion: curated synonym/translation variants for the
+    // memory's keywords are indexed in the aliases column (and therefore the
+    // CJK bigram column below), so any related word matches in round one.
+    // Derived at sync time — the Markdown file stays the source of truth.
+    const supplied = fm.aliases ?? [];
+    const have = new Set(supplied.map((a) => a.toLowerCase()));
+    const auto = expansionVariantsForDoc(body + "\n" + tags).filter((a) => !have.has(a.toLowerCase()));
+    const aliases = [...supplied, ...auto].join(",");
     db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, body, cjkIndexText(body + "\n" + tags + "\n" + aliases), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
 }
 export function deleteFromIndex(id) {

package/docs/V2-DESIGN.md

@@ -1,6 +1,6 @@
 # OpenMemex — Design Document (protocol v0.2)
 
-**Status:** FROZEN — protocol v0.2 (2026-09-26). Decisions D1–D78 are settled; open questions are tracked at the end of this document.
+**Status:** FROZEN — protocol v0.2 (2026-09-26). Decisions D1–D79 are settled; open questions are tracked at the end of this document.
 **Author:** Stone, with 小沐
 **Changelog vs v1:** incorporates round-3 review from Perplexity, Grok, Gemini, ChatGPT, DeepSeek.
 Key changes: Design Principles section; `role` separated from `type`; two iron rules;
@@ -562,6 +562,8 @@ requirement: personal data never touches third-party services). Benchmarks to tr
 - **Phase 4 — Future, signal-gated.** Cloud `RemoteProvider` customization only on: multi-private-repo
   sharing needs, fine-grained ACL, audit/compliance mandates · optional API-backed exporters/providers
   for enterprise knowledge systems.
+- **Near-term backlog — committed direction, not yet built (2026-10-07).** Stone's rule: anything said as "later" is written down here. D77-L2: per-memory `retrieved_count`/`useful_count` (`useful_self` vs `useful_team`) + `memory_useful` tool. D77-L3: `memory_status` reuse report (top reused, zero-reuse, per-scope reuse rate) feeding the Spark Tank pilot metrics. D77: external bench — LongMemEval-S run 2026-10-07 (`scripts/bench-longmemeval.ts`, 470 questions, recall_any@k: R@5 97.0% / MRR 0.909 per-question, R@5 40.6% on a pooled 22k corpus; ahead of agentmemory’s BM25-only, level with their BM25+vector). MemoryAgentBench still open as a second external data point. D79 follow-up SUPERSEDED → built as **D80** (2026-10-07): index-time synonym/translation expansion shipped; 41-query fixture recall@5 0.854 → 0.976, MRR 0.791 → 0.859. Embedding-model configurability (Stone 2026-10-07): the multilingual model is his personal need (EN↔ZH), not everyone’s — the model must be user-configurable. Timing: (1) NOW in the bench only — a `--model` flag to measure the multilingual tax (multilingual vs English-only on English data), which decides the DEFAULT. (2) AT SHIP TIME — `embedding.model` in `MyOMemoryConfig` + `SETTABLE_KEYS`, model-agnostic storage from day one (dim + model id stored with vectors, cache dir), download UX and `doctor` checks. Not before the gate passes: building config UX for a layer we haven’t decided to ship is premature. If multilingual tax ≈ 0, default multilingual (covers everyone); else default English-only with a docs note.D78: curate+submit — pre-submit lint, then confidence-tier auto-curation (auto-apply / batch-confirm / explicit human). D79 follow-ups: feedback-weighted ranking once D77 counters land; embeddings gate decision against the published baseline; reranker evaluation. TDQS nit (separate PR): tighten search/list `type` to `z.enum`. Retrieval eval: grow the fixture (project-scope tier cases) as retrieval logic evolves. Retrieval split — local vs share server (Stone 2026-10-07): the two environments emphasize different mechanisms. Local (personal machine): BM25 + curated index-time expansion as the primary — deterministic, zero model download (D7), fast on CPU, index bloat bounded by the small corpus; embeddings opt-in only. Share server (team/org, enterprise): embeddings + ANN + reranker as the primary — server-grade compute with models resident, corpus too large and multi-author for any curated map; the curated map stays as a precision layer plus org glossary. The server additionally aggregates D77 useful_count across users into feedback-weighted ranking (already in D79 follow-ups) — a signal a single machine can never have. Bottom-up loop: curated pairs that many users converge on locally become org-glossary candidates. Economics flip: N users each downloading a model vs one model amortized on the server.
+
 - **Phase 5 — Candidates recorded 2026-10-04 (not committed).** From the competitor scan (D75): an offline FTS5 retrieval eval with a published recall number (gates any semantic-layer decision) [promoted to next build, D77] · per-client trust levels on the review ladder · cold-start Markdown import into outbox drafts · `doctor` check that each editor's MCP registration points at the intended data root (mis-rooted servers are a real failure mode elsewhere). Sequencing and scope TBD per candidate.
 
 ## 19. Migration v1 → v2
@@ -1598,6 +1600,9 @@ requirement: personal data never touches third-party services). Benchmarks to tr
 
 - **D78 — pre-submit curation: human-on-the-loop, not in-the-loop (2026-10-07).** Fragmented captures must be organized before submit, but the submitter's patience is the scarce resource — every manual gate lowers the submit rate, so curation defaults to automatic and the human handles exceptions. Confidence tiers: **auto-apply** (exact duplicates, trivial fragments, inferred type/tags, title normalization); **batch-confirm** (semantic merges, splits — one summary screen, one tap to accept or undo, never per-item approval); **explicit human decision** (dropping substantive content, and submit itself — the D-hard boundary "project scope never auto-submits" stands: automation covers organizing, sharing stays human). The safety net is the supersede chain: every auto-merge preserves provenance and is undoable, so pre-approval is replaced by reversibility (undo-send, not confirm-dialog). Trigger: no new step to remember — `submit` becomes curate+submit in one command, with the curation summary attached to the PR description; session wrap-up is the alternative hook (batch-silent, per the D61 checkpoint posture). Closed loop: human undos feed back into confidence thresholds, consuming the D77 `useful_count` signals. This revises the earlier review-ladder instinct of per-item approval: the human moves from in-the-loop to on-the-loop.
 
+- **D79 — query-time synonym expansion with a second-chance round (2026-10-07).** Fourth retrieval layer after D56 (query construction) and D61 (agent-side expansion guidance, capture-time aliases): a curated, checked-in synonym map (`src/retrieve/synonyms.ts` — EN dev-domain groups plus ZH groups, bidirectional) applied at query time. `search()` runs round one unchanged; when it returns fewer candidates than requested, round two re-runs with synonym variants OR-ed in and fills only the gaps — round-one ids and scores are never diluted by the expansion. Deliberately not embeddings and not an LLM call: deterministic, auditable, zero dependencies; a bad synonym is a visible line in the file, not a silent vector. `SearchStats.secondRound` records whether the round ran (surfaced in `search --explain`). Eval proof: 3 synonym-only fixture queries are unanswerable in round one by construction (their words appear in no memory); round two answers 2 of 3 at rank –2. Baseline with the round: recall@1 0.81, recall@5 0.95, MRR 0.87 over 37 queries. Follow-ups, not in this change: feedback-weighted ranking once D77 counters land; the embeddings gate decision against these numbers; reranker evaluation. **Update (2026-10-07, LongMemEval-S):** measured on 470 questions (recall_any@k protocol, per-question isolated corpora): R@1 86.0% / R@5 97.0% / R@10 98.3% / MRR 0.909 — ahead of agentmemory’s published BM25-only (86.2% / 94.6% / 0.715) and level with their BM25+vector (95.2% / 98.6% / 0.882). But the second round fired on **0 of 470** queries: the `raw.length < limit` trigger never trips once the corpus fills the page — the round as designed is a thin-corpus fallback, not a scale answer. A pooled 22k-memory stress run drops R@5 to 40.6% (the lexical ceiling, now measured instead of asserted). Consequence: the trigger needs a redesign (score/coverage-based, or always-expand-with-fusion) before the round can claim the scale story; the pooled number is the baseline an embedding layer must beat. **Ablation (2026-10-07, same bench):** OFAT over the lexical factors, limit=10, effect on MRR: OR semantics **+0.652** (AND collapses to 0.257 — the load-bearing wall); prefix matching **-0.010** (disabling it lifts R@1 86.0%→87.9% — mildly harmful, needs a second dataset before acting); stopword filtering **+0.005** (keep: cheap, principled); synonym round **0.000** at every limit and scale tested. Limit sweep: R@1 is invariant to limit (86.0% at 1–50); returns flatten at limit 5 (R@5 97.0%, +1.3pp to go to 50); round-2 fired 10/470 at limit=50 and rescued nothing (identical to no-synonyms). Pooled 22k at limit=50: R@5 still 40.6%, round-2 fired 0. Reading: the lexical lemon is squeezed — OR does the work, the rest is ±0.01. The next gains must come from ranking under distractors (embedding/reranker layer), not more lexical tweaks. Rescue experiment (same day): fallback-append vs always-expand+RRF-fusion, 470q, stratified by query–gold overlap — **zero rescues** under either design, at either scale; not one question changes outcome. The 8 residual misses are reasoning problems (preference/multi-hop/temporal), not vocabulary gaps. The round’s value lives in a mismatch slice the public data barely contains (proven on constructed queries: 2 of 3 rescued). Keep the curated map; stop spending design budget on the trigger. (Superseded same day: index-time expansion, see backlog.)
+- **D80 — index-time synonym/translation expansion (2026-10-07).** Supersedes D79's query-time second round (removed from `search()`; `SearchStats.secondRound` gone): the curated map (`src/retrieve/synonyms.ts` — EN dev-domain groups, ZH groups and EN↔ZH translation pairs merged into one bidirectional group list) is applied at sync/index time. `writeRow` expands each memory's keywords via `expansionVariantsForDoc()` and appends the variants to the indexed `aliases` column (FTS and CJK-bigram columns both pick them up), so any related word matches in a plain round-1 query. Query side: single FTS round, no trigger, no second chance. Keyword-centric per Stone ("重要的是关键词"): Latin content terms (same stopword filter as the query builder, so both sides agree on what a keyword is) + CJK substring matching + multi-word Latin phrases (`regression test`); variants already present in the text are skipped; capped at 32 per memory to bound index growth. Derived at sync time — the Markdown file stays the source of truth; a map update needs a reindex to backfill. *Rationale: the precise slice (exact EN↔ZH pairs like 单点登录/SSO) must never be missed, and embeddings are fuzzy where the map is exact; paying the cost once at write time keeps every query simple and trigger-free.* Eval on the 41-query cross-lingual fixture: recall@1 0.732 → 0.756, recall@5 0.854 → **0.976**, MRR 0.791 → **0.859** — all 5 EN↔ZH vocabulary misses rescued at rank 1–3. Known trade-off: a single-keyword query can rank an aliases-only hit below shorter docs sharing the variant (`how do we ship`: rank 2 → 8 — its hit is 1 of 15 variants on a long aliases field); accepted, the old rank depended on the fragile thin-corpus trigger. `expandQueryWithSynonyms` kept as a bench-only export.
+
 ## Open Questions
 
 _Pruned 2026-10-04 (D75): entries decided in D56/D57/D61/D68 and the shipped layers of the retrieval-robustness question were removed — the decisions are the record. What remains below is genuinely open._
@@ -1608,7 +1613,7 @@ _Pruned 2026-10-04 (D75): entries decided in D56/D57/D61/D68 and the shipped lay
 
 - **Novice experience: the 60-second magic moment and the plain-language layer (open, 2026-10-03; trimmed 2026-10-04):** the privacy model is now stated in plain words at init (D56). Still open: init offering to save one sample memory and immediately having the user's agent answer a question with it — the user *sees* recall work before being asked to trust it — and a plain-language layer over the review ladder ("saved on this computer" / "shared with the team") with outbox/submit/promote vocabulary becoming the advanced view. This is the same vocabulary wall the PM/BA question above hits from the other side.
 
-- **Retrieval evaluation: measure recall before choosing a semantic layer (open, 2026-10-04):** the retrieval-robustness question closed with all three cheap layers shipped (D56 query construction; D61 agent-side expansion + capture-time aliases), and embeddings stayed deferred *by judgment, without numbers*. A comparable project (agent-memory) publishes an offline recall@5 of 0.98 and wears it as a badge. open-memex should build the same kind of small, fixed eval — a query set over a seeded store, recall@k / MRR for the current FTS5 pipeline — and publish the number. The semantic-layer decision then gets a gate: if FTS5 + aliases is good enough, embeddings stay deferred; if a query class measurably fails, that class defines what an optional local-embedding layer must beat. Eval first, architecture second.
+- **Retrieval evaluation: measure recall before choosing a semantic layer (open, 2026-10-04):** the retrieval-robustness question closed with all three cheap layers shipped (D56 query construction; D61 agent-side expansion + capture-time aliases), and embeddings stayed deferred *by judgment, without numbers*. A comparable project (agent-memory) publishes an offline recall@5 of 0.98 and wears it as a badge. open-memex should build the same kind of small, fixed eval — a query set over a seeded store, recall@k / MRR for the current FTS5 pipeline — and publish the number. The semantic-layer decision then gets a gate: if FTS5 + aliases is good enough, embeddings stay deferred; if a query class measurably fails, that class defines what an optional local-embedding layer must beat. Eval first, architecture second. **Update (2026-10-07):** the eval shipped (D77, `scripts/retrieval-eval.ts`, baseline recall@1 0.81 / recall@5 0.95 / MRR 0.87); query-time synonym expansion landed as retrieval layer 4 (D79) — embeddings stay deferred, now with numbers behind the judgment.
 
 - **Per-client trust levels (open, 2026-10-04):** the review ladder is currently global: every host's writes face the same gates. A comparable tool (auxly) sets a trust level per agent — one agent's writes land immediately, another's queue for approval, a third is read-only. That maps naturally onto open-memex, since the ladder already exists; the open questions are the config surface (per host? per MCP client name? how does the server learn the client's identity reliably?), the default for unknown clients, and whether trust should also gate *reads* of the personal scope. Signal to watch: one user running several agents with different autonomy expectations.
 

package/docs/release-runbook.md

@@ -34,7 +34,7 @@ npm run build
 最后跑全量测试。`test-full` 之前必须先 `build`：测试里有 dist 内容指纹守卫，没 build 会误报失败。
 
 ```powershell
-npm run test-full
+node --experimental-strip-types scripts/test-full.ts
 ```
 
 测试全绿再往下走，中间任何一步报错就停。
@@ -91,7 +91,13 @@ npm publish
 ```
 
 3. 把版本号告诉小沐（例如"0.7.2 published"）：小沐核验 npm latest，然后打 git tag `vX.Y.Z` 并建 GitHub Release。
-4. 同步 MCP Registry。**必须等 npm 新版生效后再跑**：registry 会去 npm 抓新版本的实际包验 `mcpName` 和版本。`npm publish` 成功后先等 1–2 分钟，再执行发布（登录态过期时才需要先重跑 `login github`，也是一条一条来）：
+4. 同步 MCP Registry。**必须等 npm 新版生效后再跑**：registry 会去 npm 抓新版本的实际包验 `mcpName` 和版本。`npm publish` 成功后先等 1–2 分钟，再执行发布（一条一条来）。如果登录态过期（`publish` 报认证错误），先重跑登录：
+
+```powershell
+& "$env:USERPROFILE\tools\mcp-publisher\mcp-publisher.exe" login github
+```
+
+登录态没过期就跳过这一步，直接发布：
 
 ```powershell
 & "$env:USERPROFILE\tools\mcp-publisher\mcp-publisher.exe" publish