open-memex 0.8.0 → 0.8.2-alpha.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -205,6 +205,10 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
205
205
  new commands → README CLI sections + this file's Commands. A change without its docs
206
206
  is not done. Anything deferred as "later" goes into the near-term backlog in
207
207
  `docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
208
+ - **Measurement before mechanism.** A retrieval change lands its fixture +
209
+ baseline number in a commit with no retrieval change at all, so the delta is
210
+ unambiguous. `scripts/retrieval-eval.ts` is the gate: no retrieval PR merges
211
+ without before/after recall@k + MRR on the checked-in fixture.
208
212
  - **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
209
213
  in-development version. A new alpha line starts with a minor bump after the
210
214
  previous minor shipped (`0.3.0` → `0.4.0-alpha.1`); while that minor is still
package/CHANGELOG.md CHANGED
@@ -9,6 +9,18 @@ the `@alpha` tag are development snapshots and are not listed individually.
9
9
 
10
10
  ## [Unreleased]
11
11
 
12
+ - fix: the legacy-scope warning on plugin load now suggests the migrate command with an explicit `--to <current-scope>`. Without it, `--to` defaults to the scope of the directory the command runs from, so running the suggested command from the open-memex checkout would move the memories into the wrong scope. No behavior change: the warning condition and the migrate contract are untouched.
13
+ - fix: a pre-D81 index store (SCHEMA_VERSION 7, ≤0.7.x) no longer wedges every open with `no such column: alias` (D87). D85's partial index references the `alias`/`target` columns D81 added, and `CREATE TABLE IF NOT EXISTS` adds no columns to an existing table — so the index creation threw inside `db()` before the version-check rebuild could run. `db()` now rebuilds a stale table *before* creating the schema; the Markdown files are the source of truth, so the rebuild is lossless. Regression test in `scripts/test-full.ts` §9 builds a faithful v7 store and asserts the open succeeds and the schema lands at version 9.
14
+
15
+ ## [0.8.1] - 2026-10-07
16
+
17
+ - D83 follow-up: the two D83 clauses that shipped as design but not as code. (1) **BM25 weight for the `expansions` column** — `bm25(memories_fts, …)` now takes an explicit weight vector (`src/retrieve/weights.ts`, order `[content, tags, aliases, expansions, type, cjk]`) instead of the implicit all-1.0. `expansions` is `0.75`: measured on equal-length docs, a direct body match and an expansion-only match score *identically* at weight 1.0 (`-3.983` vs `-3.983`), leaving the order to tiebreaks — any weight below 1.0 breaks that tie in the right direction. `0.75` is the highest value the fixture cannot distinguish from 1.0 (sweep: 1.0/0.9/0.75 all give R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 costs MRR, below 0.35 R@5 breaks). Exposed as `opts.expansionsWeight` so the sweep stays measurable; `smoke-pure.ts` pins the vector's shape and ordering. (2) **`search --explain` per-column attribution** — reports `matched via: body=N alias=N expansion=N other=N` plus the active weights. Does **not** fix the D80 `how do we ship` rank-8 case: that is per-column length norm, not weighting. Fixture unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857.
18
+
19
+ - fix: `search()` honours the caller's `limit` again (#53). When a D81 alias round fired, the widened candidate pool leaked out through the return value, so `memory_search` / MCP `memory_search` / `cli search` returned up to **2× `limit`** hits with no signal it happened. The pool is still wider than `limit` for *ranking* (round-2 hits are not starved the way D79's slice-to-K starved them); only the cut changed, to `limit`. Drops the dead `1.5K` line, which could never raise the value. No ranking change — `scripts/retrieval-eval.ts` is unchanged at recall@1 0.750 / recall@5 0.977 / MRR 0.857.
20
+ - perf: partial index on the alias registry (#55): `loadAliasEntries` ran `WHERE alias <> '' AND target <> ''` as a full table scan on **every** search. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`, which runs on every db open — no `SCHEMA_VERSION` bump, no rebuild, existing installs pick it up on next open. Measured at 5k memories: 971ms → 39ms per 2000 lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Deliberately not a result cache: a newly saved alias must still apply on the very next query.
21
+ - D83 (implemented, for #56): D80 index-time expansion variants moved out of the `aliases` column into a dedicated `expansions` index column (own FTS column + triggers; `SearchHit.aliases` and `(aka: …)` surface only user-authored aliases; expansion-only matches report `(synonym: …)`; `--explain` shows the same via the snippet). Schema v8 → v9 with rebuild-from-Markdown migration (verified on a scratch DB). `retrieval-eval` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857 — the `how do we ship` rank stayed 8: expansion-column length-norm dilution is inherent, not caused by the mixing.
22
+ - `memory_search` / `memory_list` `type` is now a closed enum (D82, closes #51): a misspelled type is rejected at the schema (MCP -32602) instead of silently returning zero results. The opencode plugin shares the same args objects, so both surfaces tighten together; `scripts/smoke-mcp.ts` guards the rejection.
23
+
12
24
  ## [0.8.0] - 2026-10-07
13
25
 
14
26
  - User-defined alias vocabulary as query-time expansion (D81): any memory type may now carry `alias:` (nickname, e.g. 香蕉计划) and `target:` (referent, e.g. 支付系统重构项目) frontmatter — an alias memory defines vocabulary once per scope. At query time, query terms hit the alias registry (exact match, one cheap SELECT per search) and the query is rewritten with the other side of the relation — bidirectional, depth < 3, cycle-safe — then re-run as round 2/3. Rounds merge by normalized-max (`mergeRounds` in `src/retrieve/alias-memory.ts`): each round's BM25 scores are divided by that round's best, so an alias-side champion competes evenly with a literal-side champion; the merged pool is re-ranked and cut to [1.5K, 2K], never back to K. No reindex when an alias changes. Creation: `memory_add` alias/target params, CLI `add --alias/--target`, or frontmatter edit (v1: user-explicit only). Fixture grows to 54 memories / 44 queries (3 bidirectional alias queries): recall@1 0.750, recall@5 **0.977**, MRR 0.857. Also removes the init `captureAliases` and keyword-capture questions — both are now always on (Stone 2026-10-07).
package/README.md CHANGED
@@ -89,7 +89,7 @@ open-memex doctor
89
89
 
90
90
  ## Core concepts
91
91
 
92
- Three ideas explain almost everything open-memex does.
92
+ Four ideas explain almost everything open-memex does.
93
93
 
94
94
  ![open-memex architecture: your editors share one local memory — Markdown files as the source of truth, an SQLite FTS5 index for search, personal scope that never leaves the machine, and project scope shared through git PRs](docs/assets/open-memex-architecture-en.png)
95
95
 
@@ -118,6 +118,15 @@ rebuilds from the files at any time (`open-memex reindex`). Team sharing, when
118
118
  you want it, goes through the same review flow as code: nothing is shared
119
119
  automatically (see [Team workflow](#team-workflow-sharing-memories-through-git)).
120
120
 
121
+ **4. Knowledge stays current.**
122
+ Facts change: decisions get reversed, tools get replaced, yesterday's gotcha
123
+ gets fixed. open-memex doesn't let the old version quietly linger —
124
+ `memory_supersede` replaces a memory with its newer version and keeps the
125
+ chain, so retrieval serves the current fact while the history stays auditable.
126
+ Merely stale memories can be deprecated, and `open-memex audit` reports
127
+ duplicates, stale entries, and broken chains. Pruning is a feature, not
128
+ housekeeping: shared memory rots without it.
129
+
121
130
  ## Which editors, which features
122
131
 
123
132
  open-memex talks to editors two ways: a native **opencode plugin**, and a
@@ -159,6 +168,7 @@ instruction set only made agents chatty.
159
168
  |---|---|---|---|---|
160
169
  | Where it lives | Your machine + your repos | In the repo | Vendor servers | Gone when the chat ends |
161
170
  | Who maintains it | Captured as you work; you review | You write and update by hand | The service | — |
171
+ | When facts change | Superseded, not silently stale — retrieval serves the current version, chain preserved | You hand-edit; stale copies linger | Varies; often silently stale | — |
162
172
  | Works across AI tools | Yes — any MCP client (same machine) | One file per tool convention | Per-integration | No |
163
173
  | Review before sharing | Yes — outbox + pull request | Yes — it's just files | Varies | No |
164
174
  | Human-readable | Plain Markdown files | Yes | Dashboard / API | No |
@@ -410,8 +420,8 @@ holiday policy. Always on — the agent may attach up to 4 per memory, and you
410
420
  can pass your own via `open-memex add --aliases`.
411
421
 
412
422
  Three ways to add them: `open-memex add "text" --aliases "phrase one; phrase
413
- two"` (semicolon-separated, up to 4); ask the agent ("remember: 小宝贝 is my
414
- AI agent, alias it as 小沐"); or edit the memory's `aliases:` frontmatter
423
+ two"` (semicolon-separated, up to 4); ask the agent ("remember: Fluffy is my
424
+ AI agent, alias it as Momo"); or edit the memory's `aliases:` frontmatter
415
425
  directly — Markdown is the source of truth and sync picks it up.
416
426
 
417
427
  Separately, a curated synonym/translation map is expanded into the index
@@ -549,10 +559,10 @@ Three layers, each covering what the others miss: **BM25 keyword search**
549
559
  **synonym expansion in two halves** — (a) index-time (D80: a curated,
550
560
  checked-in map of dev-domain equivalents and EN↔ZH translation pairs
551
561
  expanded into the index at write time, so `ship` finds `deploy` and
552
- `单点登录` finds `SSO` in a plain single-round query; deterministic, no
562
+ `bug` finds `defect` in a plain single-round query; deterministic, no
553
563
  model download); (b) query-time user aliases (D81: an alias memory carrying
554
564
  `alias:`/`target:` frontmatter defines your vocabulary once per scope —
555
- e.g. 香蕉计划 → 支付系统重构项目 — and a query mentioning either side
565
+ e.g. Banana Plan → payment system refactor project — and a query mentioning either side
556
566
  triggers a second round on the other side, both directions, merged evenly
557
567
  with round one so alias hits are never down-weighted; no reindex when the
558
568
  alias changes); and **semantic vectors**
@@ -593,11 +603,14 @@ Honest edges, so nothing surprises you:
593
603
 
594
604
  - **Keyword search, not semantic.** Retrieval is BM25 keyword matching, plus a
595
605
  curated synonym map expanded into the index at write time (`ship` finds
596
- `deploy`, `单点登录` finds `SSO`). Paraphrases outside the curated map
606
+ `deploy`, `bug` finds `defect`). Paraphrases outside the curated map
597
607
  don't match — that's what the opt-in semantic layer is for. (Plain
598
608
  questions are fine — "how do we…" / "请问…" wording is filtered out before
599
609
  matching, so asking naturally doesn't dilute the results.) No embedding
600
- model is ever downloaded without your explicit opt-in.
610
+ model is ever downloaded without your explicit opt-in. (Synonym expansions
611
+ live in their own index column, separate from your own aliases — but a
612
+ query matching only via an expansion can still rank below direct matches;
613
+ that's inherent BM25 length-norm behaviour, not a bug.)
601
614
  - **One machine.** Editors on the same machine share memory; there is no
602
615
  cross-machine sync. `export` / `import` bundles (below) move memory between
603
616
  machines manually.
@@ -695,7 +708,7 @@ open-memex config # print effective config
695
708
  open-memex config set <key> <value> # change a setting
696
709
  open-memex doctor # environment health check (incl. plugin entry)
697
710
  open-memex audit # memory health check (duplicates, stale, broken chains)
698
- open-memex capture --dry-run "记住我喜欢简洁的回答" # preview keyword capture
711
+ open-memex capture --dry-run "remember that I like concise answers" # preview keyword capture
699
712
  open-memex mcp --print-config vscode|cursor|claude|opencode|visualstudio
700
713
  open-memex --help # this reference
701
714
  open-memex <command> --help # help for one command
package/README.zh-CN.md CHANGED
@@ -71,7 +71,7 @@ open-memex doctor
71
71
 
72
72
  ## 核心概念
73
73
 
74
- 三个概念能解释 open-memex 几乎所有的事。
74
+ 四个概念能解释 open-memex 几乎所有的事。
75
75
 
76
76
  ![open-memex 结构示意:你用的编辑器共享同一份本地记忆——Markdown 文件是事实来源,SQLite FTS5 索引负责检索,个人记忆只留本机,项目记忆经 Git PR 分享](docs/assets/open-memex-architecture-zh.png)
77
77
 
@@ -89,6 +89,12 @@ open-memex doctor
89
89
  **3. 你的文件,你做主。**
90
90
  Markdown 文件是事实来源——打开、编辑、删除、用 grep 搜都行。旁边的 SQLite 索引只是搜索加速器,随时可以从文件重建(`open-memex reindex`)。团队共享(当你需要时)走的是和代码一样的评审流程:没有任何东西会被自动分享(见[团队工作流](#团队工作流通过-git-共享记忆))。
91
91
 
92
+ **4. 知识保鲜。**
93
+ 事实会变:决策会被推翻、工具会被替换、昨天的坑今天可能已经填上。open-memex
94
+ 不让旧版本默默留着误导人——`memory_supersede` 用新版替换旧版并保留链条,
95
+ 检索永远给现行版本、历史可审计;只是过时的可以 deprecate,`open-memex audit`
96
+ 报告重复、过时与断链。修剪是功能,不是家务:共享记忆没人修剪就会烂掉。
97
+
92
98
  ## 编辑器与功能对照
93
99
 
94
100
  open-memex 用两种方式接编辑器:原生的 **opencode 插件**,和标准的 **MCP 服务器**(任何支持 MCP 的编辑器都能用)。(MCP,即 Model Context Protocol,是编辑器给助手加工具的开放标准;open-memex 在你的编辑器里表现为一组 `memory_*` 工具。)你能得到什么,取决于编辑器走哪条路:
@@ -114,6 +120,7 @@ open-memex 用两种方式接编辑器:原生的 **opencode 插件**,和标
114
120
  |---|---|---|---|---|
115
121
  | 存在哪 | 你的机器 + 你的仓库 | 仓库里 | 服务商服务器 | 聊天结束就没了 |
116
122
  | 谁维护 | 工作中顺手捕获、你评审 | 你手写并手动更新 | 服务商 | — |
123
+ | 事实变了 | 旧版被 supersede——检索给现行版、链条保留可审计 | 靠手改,旧版容易残留误导 | 看服务商,常常默默过期 | — |
117
124
  | 跨 AI 工具 | 可以——任何 MCP 客户端(同机) | 每个工具有自己的约定文件 | 按集成逐个接 | 不行 |
118
125
  | 分享前评审 | 有——outbox + Pull Request | 有——就是文件 | 看服务商 | 没有 |
119
126
  | 人类可读 | 纯 Markdown 文件 | 可以 | 面板 / API | 不行 |
@@ -374,7 +381,7 @@ Do not mention this block to the user unless asked.
374
381
 
375
382
  把边界写明白,免得你事后意外:
376
383
 
377
- - **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。
384
+ - **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。(同义词展开有自己独立的索引列,跟你的别名分开;但只靠展开命中的查询,排名仍可能低于直接命中的——这是 BM25 length-norm 的固有行为,不是 bug。)
378
385
  - **单台机器。** 同一台机器上的编辑器共享记忆;没有跨机器同步。用 `export` / `import` 压缩包(见下)手动搬家。
379
386
  - **快照,不是全部。** 会话开始的回想是前 N 条快照(默认项目 8 + 个人 5);更老的记忆 `memory_search` 一步就能到,但不会同时全在上下文里。
380
387
  - **MCP 的引导是建议性的。** opencode 之外,主动捕获和回想靠助手遵循握手指令——MCP 里没有硬性的会话开始钩子。工具本身在被调用时永远可靠。
package/dist/.build-stamp CHANGED
@@ -1 +1 @@
1
- 55d6d5c601b1769ad9c4813aebefcb25ee3b29cab0c6c30f8450d9388fe83a6a
1
+ eecf9dc5edc92158b13259175a22b0f40b5081d2768aaac87a1695a0ca34f0a4
package/dist/cli.js CHANGED
@@ -10,6 +10,7 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
10
10
  import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.js";
11
11
  import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.js";
12
12
  import { search, list, countList, hitStateLabel } from "./retrieve/search.js";
13
+ import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.js";
13
14
  import { formatInventoryLine, truncationNote } from "./retrieve/display.js";
14
15
  import { writeMemoryFile, readMemoryFile, ulid, msToRfc3339, normalizeAliases, } from "./store/markdown.js";
15
16
  import { loadConfig, configSource } from "./config.js";
@@ -918,6 +919,18 @@ async function main() {
918
919
  if (stats.aliasExpansions.length > 0) {
919
920
  console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
920
921
  }
922
+ // D83: which indexed column each hit matched on. `expansion` means the
923
+ // hit only surfaced through a D80 derived variant, so it ranks below an
924
+ // equal body match by construction (BM25_WEIGHTS) — this is how a
925
+ // surprising rank gets traced instead of guessed at.
926
+ if (hits.length > 0) {
927
+ const via = hits.reduce((acc, h) => {
928
+ acc[h.matchedVia] = (acc[h.matchedVia] ?? 0) + 1;
929
+ return acc;
930
+ }, {});
931
+ console.log(`matched via: ${Object.entries(via).map(([k, n]) => `${k}=${n}`).join(", ")}` +
932
+ ` (bm25 weights: content=1 tags=1 aliases=1 expansions=${DEFAULT_EXPANSIONS_WEIGHT} type=1 cjk=1)`);
933
+ }
921
934
  }
922
935
  if (hits.length === 0) {
923
936
  console.log("(no matches)");
@@ -39,7 +39,7 @@ export function bootstrapPlugin(root) {
39
39
  if (cwdScope.key !== scope.key && scopeHasFiles(cwdScope.key)) {
40
40
  console.warn(`[open-memex] found memories under legacy scope key "${cwdScope.key}". ` +
41
41
  `Current project scope is "${scope.key}". ` +
42
- `To move them: npm run cli -- migrate --from ${cwdScope.key}`);
42
+ `To move them: npm run cli -- migrate --from ${cwdScope.key} --to ${scope.key}`);
43
43
  }
44
44
  }
45
45
  catch (err) {
@@ -1,13 +1,41 @@
1
1
  import { db } from "../store/db.js";
2
2
  import { toFtsQuery } from "./query.js";
3
+ import { BM25_WEIGHTS } from "./weights.js";
3
4
  import { loadAliasEntries, planAliasExpansions, mergeRounds, } from "./alias-memory.js";
4
5
  function toHit(r) {
5
6
  const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
6
7
  let snippet = r.snippet ?? "";
7
- // D61: the snippet highlights matches in the body column only. When the
8
- // match came through an alias (body shows no highlight), say so — the
9
- // hit otherwise looks unrelated to the query.
10
- if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
8
+ // D83: attribution is per-column now. The aliases column holds only
9
+ // user/authored phrasings (D61); the expansions column holds D80 derived
10
+ // variants. Say which one matched — the hit otherwise looks unrelated to
11
+ // the query when the body shows no highlight.
12
+ const hlTerms = (s) => {
13
+ if (!s)
14
+ return [];
15
+ const m = s.match(/\[([^\]]+)\]/g);
16
+ return m ? [...new Set(m.map((x) => x.slice(1, -1)))] : [];
17
+ };
18
+ const aliasTerms = hlTerms(r.alias_hl);
19
+ const expTerms = hlTerms(r.exp_hl);
20
+ // D83: attribute the hit to the column that matched, so `--explain` can
21
+ // trace an unexpected rank. Order matters: a body highlight is the most
22
+ // direct evidence, then an authored alias, then a derived expansion.
23
+ const matchedVia = /\[[^\]]+\]/.test(snippet)
24
+ ? "body"
25
+ : aliasTerms.length > 0
26
+ ? "alias"
27
+ : expTerms.length > 0
28
+ ? "expansion"
29
+ : "other";
30
+ if (aliasTerms.length > 0 && aliases.length > 0) {
31
+ snippet += ` (aka: ${aliases.join(", ")})`;
32
+ }
33
+ else if (expTerms.length > 0) {
34
+ snippet += ` (synonym: ${expTerms.join(", ")})`;
35
+ }
36
+ else if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
37
+ // Match came through another column (tags/type/cjk) — the authored
38
+ // aliases are still honest context for the hit.
11
39
  snippet += ` (aka: ${aliases.join(", ")})`;
12
40
  }
13
41
  return {
@@ -17,6 +45,7 @@ function toHit(r) {
17
45
  type: r.type,
18
46
  tags: r.tags ? r.tags.split(",").filter(Boolean) : [],
19
47
  aliases,
48
+ matchedVia,
20
49
  snippet,
21
50
  score: r.score,
22
51
  updated_at: r.updated_at,
@@ -138,18 +167,26 @@ export function search(query, opts = {}) {
138
167
  : "";
139
168
  const typeFilter = opts.type ? ` AND m.type = ?` : "";
140
169
  const sql = `
141
- SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.updated_at,
170
+ SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.expansions, m.updated_at,
142
171
  m.status, m.superseded_by, m.review_state, m.source, m.created_at,
143
172
  snippet(memories_fts, 0, '[', ']', ' ... ', 12) AS snippet,
144
- bm25(memories_fts) AS score
173
+ highlight(memories_fts, 2, '[', ']') AS alias_hl,
174
+ highlight(memories_fts, 3, '[', ']') AS exp_hl,
175
+ bm25(memories_fts, ?, ?, ?, ?, ?, ?) AS score
145
176
  FROM memories_fts
146
177
  JOIN memories m ON m.rowid = memories_fts.rowid
147
178
  WHERE memories_fts MATCH ?${scopeFilter}${typeFilter}
148
179
  ORDER BY score ASC
149
180
  LIMIT ?
150
181
  `;
182
+ // D83: the weight vector is bound per statement, so it leads the parameter
183
+ // list (it appears in the SELECT list, before the WHERE's MATCH).
184
+ const weights = opts.expansionsWeight === undefined
185
+ ? BM25_WEIGHTS
186
+ : BM25_WEIGHTS.map((w, i) => (i === 3 ? opts.expansionsWeight : w));
151
187
  const fetchFts = (matchExpr) => {
152
- const params = [matchExpr];
188
+ const params = [...weights];
189
+ params.push(matchExpr);
153
190
  if (opts.scopeKeys && opts.scopeKeys.length > 0)
154
191
  params.push(...opts.scopeKeys);
155
192
  if (opts.type)
@@ -169,7 +206,6 @@ export function search(query, opts = {}) {
169
206
  const aliasEntries = loadAliasEntries(opts.scopeKeys);
170
207
  const expansions = planAliasExpansions(query, aliasEntries);
171
208
  let merged = raw;
172
- let finalLimit = limit;
173
209
  if (expansions.length > 0) {
174
210
  const rounds = [raw];
175
211
  for (const eq of expansions) {
@@ -181,21 +217,22 @@ export function search(query, opts = {}) {
181
217
  // divided by that round's best, so an alias-side champion competes
182
218
  // evenly with a literal-side champion — no down-weighting for being
183
219
  // from a later round.
220
+ //
221
+ // The merged pool is deliberately wider than `limit` (each round already
222
+ // over-fetches by fetchLimit) so round-2 hits are never starved the way
223
+ // D79's slice-to-K starved them. #53: that headroom is for RANKING only —
224
+ // the return is still cut to the caller's `limit`, since `limit` is a cap
225
+ // on what every caller (tools, MCP, cli) receives, not a hint.
184
226
  merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
185
- // Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
186
- // starved the way D79's slice-to-K starved them.
187
- const pool = merged.length;
188
- finalLimit = Math.min(pool, 2 * limit);
189
- finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
190
227
  if (opts.stats)
191
228
  opts.stats.aliasExpansions = expansions;
192
229
  }
193
230
  // D80: the curated static map is solved at index time (synonyms.ts
194
- // expands variants into the aliases column at sync time), so without a
195
- // user alias the query side stays a single FTS round.
231
+ // expands variants into the indexed aliases column at sync time), so
232
+ // without a user alias the query side stays a single FTS round.
196
233
  if (opts.stats)
197
234
  opts.stats.candidates = merged.length;
198
- return resolveVisible(merged, finalLimit, opts.stats);
235
+ return resolveVisible(merged, limit, opts.stats);
199
236
  }
200
237
  export function list(scopeKey, opts = {}) {
201
238
  const limit = Math.max(1, Math.min(opts.limit ?? 20, 100));
@@ -0,0 +1,35 @@
1
+ /**
2
+ * D83: BM25 column weights for the FTS index.
3
+ *
4
+ * Split out of `search.ts` deliberately: `smoke-pure.ts` must keep running
5
+ * with no SQLite dependency (its header says "pure-logic, no
6
+ * bun:sqlite"), and importing `search.ts` pulls in `store/db.ts` at module
7
+ * load. This module holds the tunable constant and nothing else, so the
8
+ * guard test can import it without opening a database.
9
+ *
10
+ * `expansions` (D80-derived synonym/translation variants) sits strictly below
11
+ * `aliases` so the preference is explicit rather than incidental: a body
12
+ * match outranks an expansion-only match, and a user-authored alias outranks
13
+ * a derived one. This is not cosmetic — measured on equal-length docs, at
14
+ * weight 1.0 a direct body match and an expansion-only match score
15
+ * *identically* (-3.983 vs -3.983), leaving the order to tiebreaks. Any
16
+ * weight < 1.0 breaks that tie in the right direction.
17
+ *
18
+ * The value is the highest one the checked-in fixture cannot distinguish from
19
+ * 1.0 (`scripts/retrieval-eval.ts`: R@1 0.750 / R@5 0.977 / MRR 0.857 hold
20
+ * from 1.0 down to 0.75; 0.5 already costs MRR, and below 0.35 R@5 breaks).
21
+ * So the ordering guarantee is free, and the fixture is the gate that keeps
22
+ * it honest — retune by re-running the sweep, not by feel.
23
+ *
24
+ * Known non-fix: this does NOT repair D80's `how do we ship` rank-8 case.
25
+ * That is per-column length norm (one variant among many in a long
26
+ * `expansions` field), not a weighting problem; see D83's update.
27
+ */
28
+ /** Weight for the D80 `expansions` column. */
29
+ export const DEFAULT_EXPANSIONS_WEIGHT = 0.75;
30
+ /**
31
+ * Full weight vector for `memories_fts`, in declaration order.
32
+ * MUST stay in sync with FTS_SCHEMA column order in `store/db.ts`:
33
+ * `[content, tags, aliases, expansions, type, cjk]`.
34
+ */
35
+ export const BM25_WEIGHTS = [1, 1, 1, DEFAULT_EXPANSIONS_WEIGHT, 1, 1];
package/dist/store/db.js CHANGED
@@ -35,6 +35,7 @@ CREATE TABLE IF NOT EXISTS memories (
35
35
  status TEXT NOT NULL DEFAULT 'active',
36
36
  tags TEXT NOT NULL DEFAULT '',
37
37
  aliases TEXT NOT NULL DEFAULT '',
38
+ expansions TEXT NOT NULL DEFAULT '',
38
39
  alias TEXT NOT NULL DEFAULT '',
39
40
  target TEXT NOT NULL DEFAULT '',
40
41
  content TEXT NOT NULL,
@@ -53,6 +54,24 @@ CREATE INDEX IF NOT EXISTS idx_memories_scope_updated
53
54
  ON memories(scope_key, updated_at DESC);
54
55
  CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
55
56
  CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
57
+ -- D81: the alias registry (retrieve/alias-memory.ts) runs on every search().
58
+ -- The predicate is a full table scan without this — the rows carrying a
59
+ -- defined alias pair are a tiny subset of a corpus that may be tens of
60
+ -- thousands. A partial index keeps the lookup proportional to the number of
61
+ -- aliases defined, not the size of the store.
62
+ --
63
+ -- No SCHEMA_VERSION bump: TABLE_SCHEMA runs on every db open (see db()), so
64
+ -- existing installs pick this up on next open. Deliberately NOT a result
65
+ -- cache — the alias registry must be re-read per query so a newly saved
66
+ -- alias memory applies on the very next query. This index accelerates the
67
+ -- lookup without caching results.
68
+ -- D87 correction to the "no bump" reasoning above: it holds only for stores
69
+ -- whose tables already have the alias/target columns. A pre-D81 store's
70
+ -- table lacks them, CREATE TABLE IF NOT EXISTS adds none, and creating this
71
+ -- index throws "no such column: alias" before the version check runs. db()
72
+ -- now rebuilds a stale table before creating the schema.
73
+ CREATE INDEX IF NOT EXISTS idx_memories_alias
74
+ ON memories(scope_key) WHERE alias <> '' AND target <> '';
56
75
  `;
57
76
  // The `cjk` column holds pre-tokenized CJK unigrams+bigrams (see
58
77
  // src/retrieve/cjk.ts). FTS5's unicode61 treats a CJK run as one token, so
@@ -64,6 +83,7 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
64
83
  content,
65
84
  tags,
66
85
  aliases,
86
+ expansions,
67
87
  type,
68
88
  cjk,
69
89
  scope_key UNINDEXED,
@@ -73,24 +93,24 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
73
93
  );
74
94
 
75
95
  CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
76
- INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
77
- VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
96
+ INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
97
+ VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
78
98
  END;
79
99
 
80
100
  CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
81
- INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
82
- VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
101
+ INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
102
+ VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
83
103
  END;
84
104
 
85
105
  CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
86
- INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
87
- VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
88
- INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
89
- VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
106
+ INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
107
+ VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
108
+ INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
109
+ VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
90
110
  END;
91
111
  `;
92
112
  /** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
93
- const SCHEMA_VERSION = 8;
113
+ const SCHEMA_VERSION = 9;
94
114
  function userVersion(d) {
95
115
  const row = d.prepare("PRAGMA user_version").get();
96
116
  return row.user_version;
@@ -130,6 +150,19 @@ export function db() {
130
150
  d.exec("PRAGMA busy_timeout = 5000;");
131
151
  d.exec("PRAGMA synchronous = NORMAL;");
132
152
  d.exec("PRAGMA foreign_keys = ON;");
153
+ // D87: a stale on-disk table predates columns that TABLE_SCHEMA's partial
154
+ // index references (D85's idx_memories_alias needs alias/target, added by
155
+ // D81). CREATE TABLE IF NOT EXISTS adds no columns to an existing table,
156
+ // so creating that index first throws "no such column: alias" — and the
157
+ // version check below never runs, wedging every open on a pre-D81 store.
158
+ // When the table exists but the version is old, rebuild first: the table
159
+ // exists, so rebuild's DELETE has something to delete. Fresh stores have
160
+ // no table and fall through to creation below, exactly as before.
161
+ const hasTable = d
162
+ .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'memories'")
163
+ .get();
164
+ if (hasTable && userVersion(d) < SCHEMA_VERSION)
165
+ rebuildIndexSchema(d);
133
166
  d.exec(TABLE_SCHEMA);
134
167
  d.exec(FTS_SCHEMA);
135
168
  if (userVersion(d) < SCHEMA_VERSION)
@@ -8,8 +8,8 @@ import { iterMemoryFiles, iterInRepoMemoryFiles, readMemoryFile, timeToMs, } fro
8
8
  import { projectRoot, paths } from "../paths.js";
9
9
  import { loadConfig } from "../config.js";
10
10
  const UPSERT_SQL = `
11
- INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
12
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
11
+ INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, expansions, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
12
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
13
13
  ON CONFLICT(id) DO UPDATE SET
14
14
  scope_key = excluded.scope_key,
15
15
  scope = excluded.scope,
@@ -21,6 +21,7 @@ const UPSERT_SQL = `
21
21
  status = excluded.status,
22
22
  tags = excluded.tags,
23
23
  aliases = excluded.aliases,
24
+ expansions = excluded.expansions,
24
25
  alias = excluded.alias,
25
26
  target = excluded.target,
26
27
  content = excluded.content,
@@ -57,16 +58,20 @@ export function upsertFromFile(mf) {
57
58
  function writeRow(mf, mtimeMs) {
58
59
  const { fm, body, filePath } = mf;
59
60
  const tags = (fm.tags ?? []).join(",");
60
- // D80 index-time expansion: curated synonym/translation variants for the
61
- // memory's keywords are indexed in the aliases column (and therefore the
62
- // CJK bigram column below), so any related word matches in round one.
63
- // Derived at sync time — the Markdown file stays the source of truth.
61
+ // D80 index-time expansion, D83: curated synonym/translation variants for
62
+ // the memory's keywords are indexed in their own `expansions` column —
63
+ // never mixed into `aliases`, which holds only user/authored alternate
64
+ // phrasings (so `(aka: …)` attribution stays honest). The CJK bigram
65
+ // column still covers expansions, so CJK variants stay substring-
66
+ // searchable. Derived at sync time — the Markdown file stays the source
67
+ // of truth.
64
68
  const supplied = fm.aliases ?? [];
65
69
  const have = new Set(supplied.map((a) => a.toLowerCase()));
66
70
  const auto = expansionVariantsForDoc(body + "\n" + tags).filter((a) => !have.has(a.toLowerCase()));
67
- const aliases = [...supplied, ...auto].join(",");
71
+ const aliases = supplied.join(",");
72
+ const expansions = auto.join(",");
68
73
  // D81: user-defined alias vocabulary (single nickname → referent pair).
69
- db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
74
+ db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, expansions, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases + "\n" + expansions), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
70
75
  }
71
76
  export function deleteFromIndex(id) {
72
77
  db().prepare(`DELETE FROM memories WHERE id = ?`).run(id);
package/dist/tools/ops.js CHANGED
@@ -90,7 +90,10 @@ export const memorySearchArgs = {
90
90
  .enum(["project", "personal", "both"])
91
91
  .optional()
92
92
  .describe("Which scope(s) to search. Default: both."),
93
- type: z.string().optional().describe("Restrict to memories of this type (e.g. decision, preference, gotcha)."),
93
+ type: z
94
+ .enum(MEMORY_TYPE_TAXONOMY)
95
+ .optional()
96
+ .describe("Restrict to memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
94
97
  limit: z.number().int().min(1).max(50).optional().describe("Max results to return (default 8)."),
95
98
  };
96
99
  export const memoryListArgs = {
@@ -98,7 +101,10 @@ export const memoryListArgs = {
98
101
  .enum(["project", "personal", "both"])
99
102
  .optional()
100
103
  .describe("Which scope(s) to list. `project` = tied to this repo. `personal` = global across all your projects. `both` lists each scope's newest. Default: project."),
101
- type: z.string().optional().describe("Only list memories of this type (e.g. decision, preference, gotcha)."),
104
+ type: z
105
+ .enum(MEMORY_TYPE_TAXONOMY)
106
+ .optional()
107
+ .describe("Only list memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
102
108
  limit: z.number().int().min(1).max(100).optional().describe("Max entries to list, newest first (default 20)."),
103
109
  include: z
104
110
  .enum(["active", "all"])