open-memex 0.8.0 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -205,6 +205,10 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
205
205
  new commands → README CLI sections + this file's Commands. A change without its docs
206
206
  is not done. Anything deferred as "later" goes into the near-term backlog in
207
207
  `docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
208
+ - **Measurement before mechanism.** A retrieval change lands its fixture +
209
+ baseline number in a commit with no retrieval change at all, so the delta is
210
+ unambiguous. `scripts/retrieval-eval.ts` is the gate: no retrieval PR merges
211
+ without before/after recall@k + MRR on the checked-in fixture.
208
212
  - **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
209
213
  in-development version. A new alpha line starts with a minor bump after the
210
214
  previous minor shipped (`0.3.0` → `0.4.0-alpha.1`); while that minor is still
package/CHANGELOG.md CHANGED
@@ -9,6 +9,15 @@ the `@alpha` tag are development snapshots and are not listed individually.
9
9
 
10
10
  ## [Unreleased]
11
11
 
12
+ ## [0.8.1] - 2026-10-07
13
+
14
+ - D83 follow-up: the two D83 clauses that shipped as design but not as code. (1) **BM25 weight for the `expansions` column** — `bm25(memories_fts, …)` now takes an explicit weight vector (`src/retrieve/weights.ts`, order `[content, tags, aliases, expansions, type, cjk]`) instead of the implicit all-1.0. `expansions` is `0.75`: measured on equal-length docs, a direct body match and an expansion-only match score *identically* at weight 1.0 (`-3.983` vs `-3.983`), leaving the order to tiebreaks — any weight below 1.0 breaks that tie in the right direction. `0.75` is the highest value the fixture cannot distinguish from 1.0 (sweep: 1.0/0.9/0.75 all give R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 costs MRR, below 0.35 R@5 breaks). Exposed as `opts.expansionsWeight` so the sweep stays measurable; `smoke-pure.ts` pins the vector's shape and ordering. (2) **`search --explain` per-column attribution** — reports `matched via: body=N alias=N expansion=N other=N` plus the active weights. Does **not** fix the D80 `how do we ship` rank-8 case: that is per-column length norm, not weighting. Fixture unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857.
15
+
16
+ - fix: `search()` honours the caller's `limit` again (#53). When a D81 alias round fired, the widened candidate pool leaked out through the return value, so `memory_search` / MCP `memory_search` / `cli search` returned up to **2× `limit`** hits with no signal it happened. The pool is still wider than `limit` for *ranking* (round-2 hits are not starved the way D79's slice-to-K starved them); only the cut changed, to `limit`. Drops the dead `1.5K` line, which could never raise the value. No ranking change — `scripts/retrieval-eval.ts` is unchanged at recall@1 0.750 / recall@5 0.977 / MRR 0.857.
17
+ - perf: partial index on the alias registry (#55): `loadAliasEntries` ran `WHERE alias <> '' AND target <> ''` as a full table scan on **every** search. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`, which runs on every db open — no `SCHEMA_VERSION` bump, no rebuild, existing installs pick it up on next open. Measured at 5k memories: 971ms → 39ms per 2000 lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Deliberately not a result cache: a newly saved alias must still apply on the very next query.
18
+ - D83 (implemented, for #56): D80 index-time expansion variants moved out of the `aliases` column into a dedicated `expansions` index column (own FTS column + triggers; `SearchHit.aliases` and `(aka: …)` surface only user-authored aliases; expansion-only matches report `(synonym: …)`; `--explain` shows the same via the snippet). Schema v8 → v9 with rebuild-from-Markdown migration (verified on a scratch DB). `retrieval-eval` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857 — the `how do we ship` rank stayed 8: expansion-column length-norm dilution is inherent, not caused by the mixing.
19
+ - `memory_search` / `memory_list` `type` is now a closed enum (D82, closes #51): a misspelled type is rejected at the schema (MCP -32602) instead of silently returning zero results. The opencode plugin shares the same args objects, so both surfaces tighten together; `scripts/smoke-mcp.ts` guards the rejection.
20
+
12
21
  ## [0.8.0] - 2026-10-07
13
22
 
14
23
  - User-defined alias vocabulary as query-time expansion (D81): any memory type may now carry `alias:` (nickname, e.g. 香蕉计划) and `target:` (referent, e.g. 支付系统重构项目) frontmatter — an alias memory defines vocabulary once per scope. At query time, query terms hit the alias registry (exact match, one cheap SELECT per search) and the query is rewritten with the other side of the relation — bidirectional, depth < 3, cycle-safe — then re-run as round 2/3. Rounds merge by normalized-max (`mergeRounds` in `src/retrieve/alias-memory.ts`): each round's BM25 scores are divided by that round's best, so an alias-side champion competes evenly with a literal-side champion; the merged pool is re-ranked and cut to [1.5K, 2K], never back to K. No reindex when an alias changes. Creation: `memory_add` alias/target params, CLI `add --alias/--target`, or frontmatter edit (v1: user-explicit only). Fixture grows to 54 memories / 44 queries (3 bidirectional alias queries): recall@1 0.750, recall@5 **0.977**, MRR 0.857. Also removes the init `captureAliases` and keyword-capture questions — both are now always on (Stone 2026-10-07).
package/README.md CHANGED
@@ -410,8 +410,8 @@ holiday policy. Always on — the agent may attach up to 4 per memory, and you
410
410
  can pass your own via `open-memex add --aliases`.
411
411
 
412
412
  Three ways to add them: `open-memex add "text" --aliases "phrase one; phrase
413
- two"` (semicolon-separated, up to 4); ask the agent ("remember: 小宝贝 is my
414
- AI agent, alias it as 小沐"); or edit the memory's `aliases:` frontmatter
413
+ two"` (semicolon-separated, up to 4); ask the agent ("remember: Fluffy is my
414
+ AI agent, alias it as Momo"); or edit the memory's `aliases:` frontmatter
415
415
  directly — Markdown is the source of truth and sync picks it up.
416
416
 
417
417
  Separately, a curated synonym/translation map is expanded into the index
@@ -549,10 +549,10 @@ Three layers, each covering what the others miss: **BM25 keyword search**
549
549
  **synonym expansion in two halves** — (a) index-time (D80: a curated,
550
550
  checked-in map of dev-domain equivalents and EN↔ZH translation pairs
551
551
  expanded into the index at write time, so `ship` finds `deploy` and
552
- `单点登录` finds `SSO` in a plain single-round query; deterministic, no
552
+ `bug` finds `defect` in a plain single-round query; deterministic, no
553
553
  model download); (b) query-time user aliases (D81: an alias memory carrying
554
554
  `alias:`/`target:` frontmatter defines your vocabulary once per scope —
555
- e.g. 香蕉计划 → 支付系统重构项目 — and a query mentioning either side
555
+ e.g. Banana Plan → payment system refactor project — and a query mentioning either side
556
556
  triggers a second round on the other side, both directions, merged evenly
557
557
  with round one so alias hits are never down-weighted; no reindex when the
558
558
  alias changes); and **semantic vectors**
@@ -593,11 +593,14 @@ Honest edges, so nothing surprises you:
593
593
 
594
594
  - **Keyword search, not semantic.** Retrieval is BM25 keyword matching, plus a
595
595
  curated synonym map expanded into the index at write time (`ship` finds
596
- `deploy`, `单点登录` finds `SSO`). Paraphrases outside the curated map
596
+ `deploy`, `bug` finds `defect`). Paraphrases outside the curated map
597
597
  don't match — that's what the opt-in semantic layer is for. (Plain
598
598
  questions are fine — "how do we…" / "请问…" wording is filtered out before
599
599
  matching, so asking naturally doesn't dilute the results.) No embedding
600
- model is ever downloaded without your explicit opt-in.
600
+ model is ever downloaded without your explicit opt-in. (Synonym expansions
601
+ live in their own index column, separate from your own aliases — but a
602
+ query matching only via an expansion can still rank below direct matches;
603
+ that's inherent BM25 length-norm behaviour, not a bug.)
601
604
  - **One machine.** Editors on the same machine share memory; there is no
602
605
  cross-machine sync. `export` / `import` bundles (below) move memory between
603
606
  machines manually.
@@ -695,7 +698,7 @@ open-memex config # print effective config
695
698
  open-memex config set <key> <value> # change a setting
696
699
  open-memex doctor # environment health check (incl. plugin entry)
697
700
  open-memex audit # memory health check (duplicates, stale, broken chains)
698
- open-memex capture --dry-run "记住我喜欢简洁的回答" # preview keyword capture
701
+ open-memex capture --dry-run "remember that I like concise answers" # preview keyword capture
699
702
  open-memex mcp --print-config vscode|cursor|claude|opencode|visualstudio
700
703
  open-memex --help # this reference
701
704
  open-memex <command> --help # help for one command
package/README.zh-CN.md CHANGED
@@ -374,7 +374,7 @@ Do not mention this block to the user unless asked.
374
374
 
375
375
  把边界写明白,免得你事后意外:
376
376
 
377
- - **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。
377
+ - **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。(同义词展开有自己独立的索引列,跟你的别名分开;但只靠展开命中的查询,排名仍可能低于直接命中的——这是 BM25 length-norm 的固有行为,不是 bug。)
378
378
  - **单台机器。** 同一台机器上的编辑器共享记忆;没有跨机器同步。用 `export` / `import` 压缩包(见下)手动搬家。
379
379
  - **快照,不是全部。** 会话开始的回想是前 N 条快照(默认项目 8 + 个人 5);更老的记忆 `memory_search` 一步就能到,但不会同时全在上下文里。
380
380
  - **MCP 的引导是建议性的。** opencode 之外,主动捕获和回想靠助手遵循握手指令——MCP 里没有硬性的会话开始钩子。工具本身在被调用时永远可靠。
package/dist/.build-stamp CHANGED
@@ -1 +1 @@
1
- 55d6d5c601b1769ad9c4813aebefcb25ee3b29cab0c6c30f8450d9388fe83a6a
1
+ db188c4af0c070c360ad3e3694172a66d5503a7fdad1ea2321a56b059228e3e6
package/dist/cli.js CHANGED
@@ -10,6 +10,7 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
10
10
  import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.js";
11
11
  import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.js";
12
12
  import { search, list, countList, hitStateLabel } from "./retrieve/search.js";
13
+ import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.js";
13
14
  import { formatInventoryLine, truncationNote } from "./retrieve/display.js";
14
15
  import { writeMemoryFile, readMemoryFile, ulid, msToRfc3339, normalizeAliases, } from "./store/markdown.js";
15
16
  import { loadConfig, configSource } from "./config.js";
@@ -918,6 +919,18 @@ async function main() {
918
919
  if (stats.aliasExpansions.length > 0) {
919
920
  console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
920
921
  }
922
+ // D83: which indexed column each hit matched on. `expansion` means the
923
+ // hit only surfaced through a D80 derived variant, so it ranks below an
924
+ // equal body match by construction (BM25_WEIGHTS) — this is how a
925
+ // surprising rank gets traced instead of guessed at.
926
+ if (hits.length > 0) {
927
+ const via = hits.reduce((acc, h) => {
928
+ acc[h.matchedVia] = (acc[h.matchedVia] ?? 0) + 1;
929
+ return acc;
930
+ }, {});
931
+ console.log(`matched via: ${Object.entries(via).map(([k, n]) => `${k}=${n}`).join(", ")}` +
932
+ ` (bm25 weights: content=1 tags=1 aliases=1 expansions=${DEFAULT_EXPANSIONS_WEIGHT} type=1 cjk=1)`);
933
+ }
921
934
  }
922
935
  if (hits.length === 0) {
923
936
  console.log("(no matches)");
@@ -1,13 +1,41 @@
1
1
  import { db } from "../store/db.js";
2
2
  import { toFtsQuery } from "./query.js";
3
+ import { BM25_WEIGHTS } from "./weights.js";
3
4
  import { loadAliasEntries, planAliasExpansions, mergeRounds, } from "./alias-memory.js";
4
5
  function toHit(r) {
5
6
  const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
6
7
  let snippet = r.snippet ?? "";
7
- // D61: the snippet highlights matches in the body column only. When the
8
- // match came through an alias (body shows no highlight), say so — the
9
- // hit otherwise looks unrelated to the query.
10
- if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
8
+ // D83: attribution is per-column now. The aliases column holds only
9
+ // user/authored phrasings (D61); the expansions column holds D80 derived
10
+ // variants. Say which one matched — the hit otherwise looks unrelated to
11
+ // the query when the body shows no highlight.
12
+ const hlTerms = (s) => {
13
+ if (!s)
14
+ return [];
15
+ const m = s.match(/\[([^\]]+)\]/g);
16
+ return m ? [...new Set(m.map((x) => x.slice(1, -1)))] : [];
17
+ };
18
+ const aliasTerms = hlTerms(r.alias_hl);
19
+ const expTerms = hlTerms(r.exp_hl);
20
+ // D83: attribute the hit to the column that matched, so `--explain` can
21
+ // trace an unexpected rank. Order matters: a body highlight is the most
22
+ // direct evidence, then an authored alias, then a derived expansion.
23
+ const matchedVia = /\[[^\]]+\]/.test(snippet)
24
+ ? "body"
25
+ : aliasTerms.length > 0
26
+ ? "alias"
27
+ : expTerms.length > 0
28
+ ? "expansion"
29
+ : "other";
30
+ if (aliasTerms.length > 0 && aliases.length > 0) {
31
+ snippet += ` (aka: ${aliases.join(", ")})`;
32
+ }
33
+ else if (expTerms.length > 0) {
34
+ snippet += ` (synonym: ${expTerms.join(", ")})`;
35
+ }
36
+ else if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
37
+ // Match came through another column (tags/type/cjk) — the authored
38
+ // aliases are still honest context for the hit.
11
39
  snippet += ` (aka: ${aliases.join(", ")})`;
12
40
  }
13
41
  return {
@@ -17,6 +45,7 @@ function toHit(r) {
17
45
  type: r.type,
18
46
  tags: r.tags ? r.tags.split(",").filter(Boolean) : [],
19
47
  aliases,
48
+ matchedVia,
20
49
  snippet,
21
50
  score: r.score,
22
51
  updated_at: r.updated_at,
@@ -138,18 +167,26 @@ export function search(query, opts = {}) {
138
167
  : "";
139
168
  const typeFilter = opts.type ? ` AND m.type = ?` : "";
140
169
  const sql = `
141
- SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.updated_at,
170
+ SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.expansions, m.updated_at,
142
171
  m.status, m.superseded_by, m.review_state, m.source, m.created_at,
143
172
  snippet(memories_fts, 0, '[', ']', ' ... ', 12) AS snippet,
144
- bm25(memories_fts) AS score
173
+ highlight(memories_fts, 2, '[', ']') AS alias_hl,
174
+ highlight(memories_fts, 3, '[', ']') AS exp_hl,
175
+ bm25(memories_fts, ?, ?, ?, ?, ?, ?) AS score
145
176
  FROM memories_fts
146
177
  JOIN memories m ON m.rowid = memories_fts.rowid
147
178
  WHERE memories_fts MATCH ?${scopeFilter}${typeFilter}
148
179
  ORDER BY score ASC
149
180
  LIMIT ?
150
181
  `;
182
+ // D83: the weight vector is bound per statement, so it leads the parameter
183
+ // list (it appears in the SELECT list, before the WHERE's MATCH).
184
+ const weights = opts.expansionsWeight === undefined
185
+ ? BM25_WEIGHTS
186
+ : BM25_WEIGHTS.map((w, i) => (i === 3 ? opts.expansionsWeight : w));
151
187
  const fetchFts = (matchExpr) => {
152
- const params = [matchExpr];
188
+ const params = [...weights];
189
+ params.push(matchExpr);
153
190
  if (opts.scopeKeys && opts.scopeKeys.length > 0)
154
191
  params.push(...opts.scopeKeys);
155
192
  if (opts.type)
@@ -169,7 +206,6 @@ export function search(query, opts = {}) {
169
206
  const aliasEntries = loadAliasEntries(opts.scopeKeys);
170
207
  const expansions = planAliasExpansions(query, aliasEntries);
171
208
  let merged = raw;
172
- let finalLimit = limit;
173
209
  if (expansions.length > 0) {
174
210
  const rounds = [raw];
175
211
  for (const eq of expansions) {
@@ -181,21 +217,22 @@ export function search(query, opts = {}) {
181
217
  // divided by that round's best, so an alias-side champion competes
182
218
  // evenly with a literal-side champion — no down-weighting for being
183
219
  // from a later round.
220
+ //
221
+ // The merged pool is deliberately wider than `limit` (each round already
222
+ // over-fetches by fetchLimit) so round-2 hits are never starved the way
223
+ // D79's slice-to-K starved them. #53: that headroom is for RANKING only —
224
+ // the return is still cut to the caller's `limit`, since `limit` is a cap
225
+ // on what every caller (tools, MCP, cli) receives, not a hint.
184
226
  merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
185
- // Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
186
- // starved the way D79's slice-to-K starved them.
187
- const pool = merged.length;
188
- finalLimit = Math.min(pool, 2 * limit);
189
- finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
190
227
  if (opts.stats)
191
228
  opts.stats.aliasExpansions = expansions;
192
229
  }
193
230
  // D80: the curated static map is solved at index time (synonyms.ts
194
- // expands variants into the aliases column at sync time), so without a
195
- // user alias the query side stays a single FTS round.
231
+ // expands variants into the indexed aliases column at sync time), so
232
+ // without a user alias the query side stays a single FTS round.
196
233
  if (opts.stats)
197
234
  opts.stats.candidates = merged.length;
198
- return resolveVisible(merged, finalLimit, opts.stats);
235
+ return resolveVisible(merged, limit, opts.stats);
199
236
  }
200
237
  export function list(scopeKey, opts = {}) {
201
238
  const limit = Math.max(1, Math.min(opts.limit ?? 20, 100));
@@ -0,0 +1,35 @@
1
+ /**
2
+ * D83: BM25 column weights for the FTS index.
3
+ *
4
+ * Split out of `search.ts` deliberately: `smoke-pure.ts` must keep running
5
+ * with no SQLite dependency (its header says "pure-logic, no
6
+ * bun:sqlite"), and importing `search.ts` pulls in `store/db.ts` at module
7
+ * load. This module holds the tunable constant and nothing else, so the
8
+ * guard test can import it without opening a database.
9
+ *
10
+ * `expansions` (D80-derived synonym/translation variants) sits strictly below
11
+ * `aliases` so the preference is explicit rather than incidental: a body
12
+ * match outranks an expansion-only match, and a user-authored alias outranks
13
+ * a derived one. This is not cosmetic — measured on equal-length docs, at
14
+ * weight 1.0 a direct body match and an expansion-only match score
15
+ * *identically* (-3.983 vs -3.983), leaving the order to tiebreaks. Any
16
+ * weight < 1.0 breaks that tie in the right direction.
17
+ *
18
+ * The value is the highest one the checked-in fixture cannot distinguish from
19
+ * 1.0 (`scripts/retrieval-eval.ts`: R@1 0.750 / R@5 0.977 / MRR 0.857 hold
20
+ * from 1.0 down to 0.75; 0.5 already costs MRR, and below 0.35 R@5 breaks).
21
+ * So the ordering guarantee is free, and the fixture is the gate that keeps
22
+ * it honest — retune by re-running the sweep, not by feel.
23
+ *
24
+ * Known non-fix: this does NOT repair D80's `how do we ship` rank-8 case.
25
+ * That is per-column length norm (one variant among many in a long
26
+ * `expansions` field), not a weighting problem; see D83's update.
27
+ */
28
+ /** Weight for the D80 `expansions` column. */
29
+ export const DEFAULT_EXPANSIONS_WEIGHT = 0.75;
30
+ /**
31
+ * Full weight vector for `memories_fts`, in declaration order.
32
+ * MUST stay in sync with FTS_SCHEMA column order in `store/db.ts`:
33
+ * `[content, tags, aliases, expansions, type, cjk]`.
34
+ */
35
+ export const BM25_WEIGHTS = [1, 1, 1, DEFAULT_EXPANSIONS_WEIGHT, 1, 1];
package/dist/store/db.js CHANGED
@@ -35,6 +35,7 @@ CREATE TABLE IF NOT EXISTS memories (
35
35
  status TEXT NOT NULL DEFAULT 'active',
36
36
  tags TEXT NOT NULL DEFAULT '',
37
37
  aliases TEXT NOT NULL DEFAULT '',
38
+ expansions TEXT NOT NULL DEFAULT '',
38
39
  alias TEXT NOT NULL DEFAULT '',
39
40
  target TEXT NOT NULL DEFAULT '',
40
41
  content TEXT NOT NULL,
@@ -53,6 +54,19 @@ CREATE INDEX IF NOT EXISTS idx_memories_scope_updated
53
54
  ON memories(scope_key, updated_at DESC);
54
55
  CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
55
56
  CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
57
+ -- D81: the alias registry (retrieve/alias-memory.ts) runs on every search().
58
+ -- The predicate is a full table scan without this — the rows carrying a
59
+ -- defined alias pair are a tiny subset of a corpus that may be tens of
60
+ -- thousands. A partial index keeps the lookup proportional to the number of
61
+ -- aliases defined, not the size of the store.
62
+ --
63
+ -- No SCHEMA_VERSION bump: TABLE_SCHEMA runs on every db open (see db()), so
64
+ -- existing installs pick this up on next open. Deliberately NOT a result
65
+ -- cache — the alias registry must be re-read per query so a newly saved
66
+ -- alias memory applies on the very next query. This index accelerates the
67
+ -- lookup without caching results.
68
+ CREATE INDEX IF NOT EXISTS idx_memories_alias
69
+ ON memories(scope_key) WHERE alias <> '' AND target <> '';
56
70
  `;
57
71
  // The `cjk` column holds pre-tokenized CJK unigrams+bigrams (see
58
72
  // src/retrieve/cjk.ts). FTS5's unicode61 treats a CJK run as one token, so
@@ -64,6 +78,7 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
64
78
  content,
65
79
  tags,
66
80
  aliases,
81
+ expansions,
67
82
  type,
68
83
  cjk,
69
84
  scope_key UNINDEXED,
@@ -73,24 +88,24 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
73
88
  );
74
89
 
75
90
  CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
76
- INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
77
- VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
91
+ INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
92
+ VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
78
93
  END;
79
94
 
80
95
  CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
81
- INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
82
- VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
96
+ INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
97
+ VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
83
98
  END;
84
99
 
85
100
  CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
86
- INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
87
- VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
88
- INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
89
- VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
101
+ INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
102
+ VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
103
+ INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
104
+ VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
90
105
  END;
91
106
  `;
92
107
  /** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
93
- const SCHEMA_VERSION = 8;
108
+ const SCHEMA_VERSION = 9;
94
109
  function userVersion(d) {
95
110
  const row = d.prepare("PRAGMA user_version").get();
96
111
  return row.user_version;
@@ -8,8 +8,8 @@ import { iterMemoryFiles, iterInRepoMemoryFiles, readMemoryFile, timeToMs, } fro
8
8
  import { projectRoot, paths } from "../paths.js";
9
9
  import { loadConfig } from "../config.js";
10
10
  const UPSERT_SQL = `
11
- INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
12
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
11
+ INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, expansions, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
12
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
13
13
  ON CONFLICT(id) DO UPDATE SET
14
14
  scope_key = excluded.scope_key,
15
15
  scope = excluded.scope,
@@ -21,6 +21,7 @@ const UPSERT_SQL = `
21
21
  status = excluded.status,
22
22
  tags = excluded.tags,
23
23
  aliases = excluded.aliases,
24
+ expansions = excluded.expansions,
24
25
  alias = excluded.alias,
25
26
  target = excluded.target,
26
27
  content = excluded.content,
@@ -57,16 +58,20 @@ export function upsertFromFile(mf) {
57
58
  function writeRow(mf, mtimeMs) {
58
59
  const { fm, body, filePath } = mf;
59
60
  const tags = (fm.tags ?? []).join(",");
60
- // D80 index-time expansion: curated synonym/translation variants for the
61
- // memory's keywords are indexed in the aliases column (and therefore the
62
- // CJK bigram column below), so any related word matches in round one.
63
- // Derived at sync time — the Markdown file stays the source of truth.
61
+ // D80 index-time expansion, D83: curated synonym/translation variants for
62
+ // the memory's keywords are indexed in their own `expansions` column —
63
+ // never mixed into `aliases`, which holds only user/authored alternate
64
+ // phrasings (so `(aka: …)` attribution stays honest). The CJK bigram
65
+ // column still covers expansions, so CJK variants stay substring-
66
+ // searchable. Derived at sync time — the Markdown file stays the source
67
+ // of truth.
64
68
  const supplied = fm.aliases ?? [];
65
69
  const have = new Set(supplied.map((a) => a.toLowerCase()));
66
70
  const auto = expansionVariantsForDoc(body + "\n" + tags).filter((a) => !have.has(a.toLowerCase()));
67
- const aliases = [...supplied, ...auto].join(",");
71
+ const aliases = supplied.join(",");
72
+ const expansions = auto.join(",");
68
73
  // D81: user-defined alias vocabulary (single nickname → referent pair).
69
- db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
74
+ db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, expansions, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases + "\n" + expansions), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
70
75
  }
71
76
  export function deleteFromIndex(id) {
72
77
  db().prepare(`DELETE FROM memories WHERE id = ?`).run(id);
package/dist/tools/ops.js CHANGED
@@ -90,7 +90,10 @@ export const memorySearchArgs = {
90
90
  .enum(["project", "personal", "both"])
91
91
  .optional()
92
92
  .describe("Which scope(s) to search. Default: both."),
93
- type: z.string().optional().describe("Restrict to memories of this type (e.g. decision, preference, gotcha)."),
93
+ type: z
94
+ .enum(MEMORY_TYPE_TAXONOMY)
95
+ .optional()
96
+ .describe("Restrict to memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
94
97
  limit: z.number().int().min(1).max(50).optional().describe("Max results to return (default 8)."),
95
98
  };
96
99
  export const memoryListArgs = {
@@ -98,7 +101,10 @@ export const memoryListArgs = {
98
101
  .enum(["project", "personal", "both"])
99
102
  .optional()
100
103
  .describe("Which scope(s) to list. `project` = tied to this repo. `personal` = global across all your projects. `both` lists each scope's newest. Default: project."),
101
- type: z.string().optional().describe("Only list memories of this type (e.g. decision, preference, gotcha)."),
104
+ type: z
105
+ .enum(MEMORY_TYPE_TAXONOMY)
106
+ .optional()
107
+ .describe("Only list memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
102
108
  limit: z.number().int().min(1).max(100).optional().describe("Max entries to list, newest first (default 20)."),
103
109
  include: z
104
110
  .enum(["active", "all"])
package/docs/V2-DESIGN.md CHANGED
@@ -1605,6 +1605,14 @@ requirement: personal data never touches third-party services). Benchmarks to tr
1605
1605
  - **D81 — user-defined alias vocabulary as query-time expansion (2026-10-07).** Stone: an alias should be a *kind of memory* — defined once, remembered forever. Any memory type may carry `alias:` (the nickname, e.g. 香蕉计划) and `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter; the alias memory defines vocabulary once per scope. At query time (`src/retrieve/alias-memory.ts`), query terms are looked up against the alias registry — one cheap `SELECT` per `search()` call over rows with non-empty alias+target (trigger A: exact registry hit, never a count heuristic; the D79 0/470 trigger is not repeated). On a hit the query is rewritten with the other side of the relation — bidirectional (alias→target and target→alias), depth < 3, cycle-safe — and re-run as round 2 (round 3 for chains). Rounds merge by normalized-max (`mergeRounds`): each round's BM25 scores are divided by that round's best (raw cross-round scores are incomparable — °C vs °F), then final = max, so an alias-side champion competes evenly with a literal-side champion — alias hits are never down-weighted for coming from a later round. Merged pool is re-ranked then cut to [1.5K, 2K] (Stone) — never back to K, or round-2 hits would be starved the way D79's slice-to-K starved them. No reindex when an alias changes (query-time). Scope-aware: a project codename stays in its project. *Rationale: D80's curated map is static and shared — personal vocabulary (小宝贝, 香蕉计划) doesn't belong in it; per-memory D61 aliases don't propagate to new memories. Alias memories close the gap: user-defined, scope-aware, effective immediately.* v1: user-explicit creation only (`memory_add` alias/target params, CLI `--alias`/`--target`, frontmatter edit); agent-proposed aliases deferred. D81 also rewords the init `captureAliases` prompt (it is not the synonym switch — D80's expansion always applies). Eval: 3 new fixture queries (香蕉计划↔支付系统重构项目 both directions + the alias memory itself), 44 queries total: recall@1 0.750, recall@5 **0.977** (43/44), MRR 0.857 — the single miss remains `how do we ship` (known D80 trade-off).
1606
1606
 
1607
1607
 
1608
+ - **D82 — closed `type` taxonomy on search/list (2026-10-07).** Follow-up deferred from the TDQS pass (#50), closes issue #51: `memory_search` / `memory_list` `type` is now `z.enum(MEMORY_TYPE_TAXONOMY)` instead of `z.string()` — a misspelled type (`"decisions"`) is rejected at the schema (MCP -32602) instead of returning zero results the agent can't distinguish from "no memories of that type". Hard rejection, not a soft "no matches" message: closed taxonomy + fail-fast. Existing rows with non-taxonomy types are possible (the frontmatter parser only coerces the v1 `instruction` → `knowledge` case) and are unaffected — they still surface in unfiltered search/list, they just can't be addressed by a type filter. The opencode plugin shares the same args objects, so both surfaces tighten together; `scripts/smoke-mcp.ts` guards the rejection. `memory_submit` keeps `destructiveHint: false` — settled in the #50 review (the outbox unlink happens after the repo copy is committed and hash-verified, so it's recoverable from git); recorded here so it isn't re-litigated.
1609
+
1610
+ - **D83 — D80 expansions get their own index column (2026-10-07).** Design follow-up to issue #56 (deferred from the D80 review): the curated index-time variants (`expansionVariantsForDoc`) move out of the `aliases` column into a dedicated `expansions` column — on `memories`, on `memories_fts`, and in the ai/ad/au triggers. `writeRow` stores user/authored aliases in `aliases` and derived variants in `expansions` (still filtered against the supplied set, still never written to Markdown); the `cjk` bigram column keeps covering body + tags + aliases + expansions so CJK variants stay substring-searchable. Retrieval: the FTS MATCH set gains the column with its own bm25 weight (at or below `aliases` — the fixture is the gate, tune don't guess); `SearchHit.aliases` and the `(aka: …)` attribution surface only user-authored aliases; `search --explain` reports which column matched (alias vs expansion). Migration: `SCHEMA_VERSION` 8 → 9; the bump triggers `rebuildIndexSchema`, which repopulates from Markdown — safe because expansions are derived and Markdown is untouched.
1611
+ *Update (2026-10-07, implemented and measured):* the `how do we ship` rank did **not** improve (still 8) — the dilution for expansion-column matches is inherent to per-column BM25 length norm (32 variants in their own column dilute about as much as 36 did in the shared one), not to the mixing. What the split fixed is provenance (`(aka:)` / `SearchHit.aliases` are user-authored again) and the dilution of deliberate user aliases (a ≤4-term column now scores cleanly). Regression gate passed: `retrieval-eval` R@1 0.750 / R@5 0.977 / MRR 0.857, identical to pre-split; v8→v9 migration verified on a scratch DB (merged aliases re-split correctly, `user_version` 9). *Rationale: an agent reading `(aka: …)` must be able to trust it means "alternate phrasings someone attached to this memory", and a 32-variant derived list must not dilute the weight of 4 deliberate ones.*
1612
+ - **D84 — `limit` is a cap on the return, never a hint (2026-10-07).** Correction to D81's cut rule, raised by the review of #52. D81 widened the merged alias pool to `[1.5K, 2K]` and passed that widened value to `resolveVisible` as the slice — so when an alias round fired, `search()` returned up to **2× `limit`** rows to every caller (`memory_search`, the MCP tool, `cli search`), with nothing in `--explain` saying so. The pool stays wider than `limit` — each round already over-fetches by `fetchLimit`, and the starvation D79's slice-to-K caused is real and still prevented — but the *return* is cut to `limit`. (D81's `1.5K` line was also dead: `max(min(pool,2K), min(pool,1.5K))` is always `min(pool,2K)`, so the documented band never existed; it is gone.) No ranking change — the merge sorts before any cut, so top-k is byte-identical; `scripts/retrieval-eval.ts` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857. *Rationale: `limit` is a contract on what a caller receives. Headroom belongs inside the ranking, not in the result set — an agent paying 2× the tool-result tokens on alias-touching queries has no way to see that it happened.*
1613
+ - **D86 — the `expansions` BM25 weight, set by the fixture (2026-10-07).** Closes the two D83 clauses that were written into the decision log but never landed in code (`00c4d2d` shipped `bm25(memories_fts)` with no weight vector, and left `--explain` reporting only `fts:` / `candidates:` / `alias rounds:`). (1) **Weight vector** — `bm25(memories_fts, …)` now takes an explicit `[content, tags, aliases, expansions, type, cjk]`, with `expansions` at **0.75** (`src/retrieve/weights.ts`, kept out of `search.ts` so `smoke-pure.ts` keeps its no-sqlite guarantee). The measurement that sets it: on equal-length docs at weight 1.0, a direct body match and an expansion-only match score **identically** (`-3.983` vs `-3.983`) — the preference was being decided by tiebreak, not by the ranking function. Any weight < 1.0 breaks that tie toward the body. `0.75` is the *highest* weight the checked-in fixture cannot distinguish from 1.0 (sweep: 1.0 / 0.9 / 0.75 all yield R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 already costs MRR; below 0.35 R@5 breaks), so the ordering guarantee is free and the fixture remains the gate — retune by re-running the sweep, never by intuition. `SearchHit.matchedVia` (`body` | `alias` | `expansion` | `other`) is computed from the same per-column highlights the attribution already used. (2) **`--explain` attribution** — prints `matched via: body=N alias=N expansion=N other=N` alongside the active weights, so a surprising rank is traceable to the column that caused it. *Rationale: D83 promised the split would make the two columns distinguishable; on equal-weight columns they were not distinguishable at all, so the split bought provenance without buying ranking. This is the missing half.* **Explicit non-goal:** this does *not* repair D80's `how do we ship` rank-8 case — that is per-column length norm (one variant among many in a long `expansions` field), not a weighting problem. Fixing that means narrowing *what* lands in the column, not re-weighting it. Fixture unchanged: R@1 0.750 / R@5 0.977 / MRR 0.857.
1614
+ - **D85 — the alias registry is indexed, not cached (2026-10-07).** From the same review (#55). D81's `loadAliasEntries` runs on **every** `search()` call over `alias <> '' AND target <> ''` — a full table scan on a corpus that may be tens of thousands. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`. Two deliberate choices: (1) **no `SCHEMA_VERSION` bump** — `TABLE_SCHEMA` executes on every db open, not only on rebuild, so `CREATE INDEX IF NOT EXISTS` is idempotent and existing installs pick it up on next open with no rebuild; this also keeps the bump free for D83's 8 → 9 schema change. (2) **No result cache** — the registry must be re-read per query so a newly saved alias memory applies on the very next query (D81's freshness property). An index accelerates the lookup without caching results; those two concerns are separable and this resolves only the first. Measured at 5k memories: 971ms → 39ms per 2000 scoped lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Note the *unscoped* variant of the query still prefers `idx_memories_status`; `search()` always passes `scopeKeys`, so the partial index is what ships. *Rationale: the alias table is tiny by construction — one row per defined nickname per scope — so its lookup cost must be proportional to aliases defined, not to store size.*
1615
+
1608
1616
  ## Open Questions
1609
1617
 
1610
1618
  _Pruned 2026-10-04 (D75): entries decided in D56/D57/D61/D68 and the shipped layers of the retrieval-robustness question were removed — the decisions are the record. What remains below is genuinely open._
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "open-memex",
3
- "version": "0.8.0",
3
+ "version": "0.8.1",
4
4
  "mcpName": "io.github.stoneskin/open-memex",
5
5
  "description": "Local-first memory layer and protocol for AI coding agents. Markdown source of truth, SQLite FTS5 index, zero cloud.",
6
6
  "keywords": [
@@ -133,6 +133,18 @@ try {
133
133
  const listBoth = await req("tools/call", { name: "memory_list", arguments: { scope: "both" } });
134
134
  const listBothText = listBoth.result?.content?.[0]?.text ?? "";
135
135
  check("memory_list both scopes (D68)", listBothText.includes("About you"));
136
+ // D82: closed type taxonomy — a misspelled type is rejected at the schema,
137
+ // never silently empty.
138
+ const badType = await req("tools/call", {
139
+ name: "memory_search",
140
+ arguments: { query: "mcp e2e probe", type: "decisions" },
141
+ });
142
+ const badTypeText: string = badType.result?.content?.[0]?.text ?? "";
143
+ check(
144
+ "memory_search rejects misspelled type (D82)",
145
+ /-32602/.test(badTypeText) && /Invalid option/.test(badTypeText),
146
+ badTypeText.slice(0, 160),
147
+ );
136
148
  // D70: numbers run across the whole listing, so "delete #3" names one memory.
137
149
  // Needs a personal entry too, or there is only one section to number.
138
150
  const personalAdd = await req("tools/call", {
@@ -9,6 +9,7 @@ import { resolveProjectScope, resolveCwdScope, pickScopeRoot, PERSONAL_SCOPE } f
9
9
  import { cjkIndexText, cjkQueryExpr, hasCjk } from "../src/retrieve/cjk.ts";
10
10
  import { toFtsQuery } from "../src/retrieve/query.ts";
11
11
  import { expansionVariantsForDoc, MAX_AUTO_VARIANTS } from "../src/retrieve/synonyms.ts";
12
+ import { BM25_WEIGHTS, DEFAULT_EXPANSIONS_WEIGHT } from "../src/retrieve/weights.ts";
12
13
  import {
13
14
  planAliasExpansions,
14
15
  rewriteQueryOnce,
@@ -949,5 +950,20 @@ console.log("== D81 alias-memory query-time expansion ==");
949
950
  ok("single round orders by score", single[0].doc.id === "b" && single[0].score === 1, JSON.stringify(single.map((m) => [m.doc.id, m.score])));
950
951
  }
951
952
 
953
+ console.log("== D83 expansions BM25 weight + matchedVia ==");
954
+ {
955
+ // The weight vector must stay in memories_fts declaration order:
956
+ // [content, tags, aliases, expansions, type, cjk]. A mis-ordered vector
957
+ // silently down-weights the wrong column, so pin the shape.
958
+ ok("BM25_WEIGHTS has one entry per fts column", BM25_WEIGHTS.length === 6, JSON.stringify(BM25_WEIGHTS));
959
+ ok("expansions is index 3", BM25_WEIGHTS[3] === DEFAULT_EXPANSIONS_WEIGHT, JSON.stringify(BM25_WEIGHTS));
960
+ // D83: expansions sits strictly below aliases, so a body match outranks an
961
+ // expansion-only match by construction instead of by tiebreak.
962
+ ok("expansions weighted below aliases", BM25_WEIGHTS[3] < BM25_WEIGHTS[2], `${BM25_WEIGHTS[3]} vs ${BM25_WEIGHTS[2]}`);
963
+ ok("expansions keeps non-zero weight (D80 rescue slice must survive)",
964
+ BM25_WEIGHTS[3] > 0, String(BM25_WEIGHTS[3]));
965
+ ok("body stays the strongest column", BM25_WEIGHTS[0] >= BM25_WEIGHTS[3], JSON.stringify(BM25_WEIGHTS));
966
+ }
967
+
952
968
  console.log(fails === 0 ? "\nALL PASS" : `\n${fails} FAILURES`);
953
969
  process.exit(fails === 0 ? 0 : 1);
package/server.json CHANGED
@@ -3,7 +3,7 @@
3
3
  "name": "io.github.stoneskin/open-memex",
4
4
  "title": "open-memex",
5
5
  "description": "Local-first memory for AI coding agents — Markdown + SQLite FTS5, shared across editors via MCP.",
6
- "version": "0.8.0",
6
+ "version": "0.8.1",
7
7
  "repository": {
8
8
  "url": "https://github.com/stoneskin/open-memex",
9
9
  "source": "github"
@@ -13,7 +13,7 @@
13
13
  {
14
14
  "registryType": "npm",
15
15
  "identifier": "open-memex",
16
- "version": "0.8.0",
16
+ "version": "0.8.1",
17
17
  "runtimeHint": "npx",
18
18
  "packageArguments": [
19
19
  {
package/src/cli.ts CHANGED
@@ -10,6 +10,7 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
10
10
  import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.ts";
11
11
  import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.ts";
12
12
  import { search, list, countList, hitStateLabel } from "./retrieve/search.ts";
13
+ import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.ts";
13
14
  import { formatInventoryLine, truncationNote } from "./retrieve/display.ts";
14
15
  import {
15
16
  writeMemoryFile,
@@ -1028,6 +1029,20 @@ async function main() {
1028
1029
  if (stats.aliasExpansions.length > 0) {
1029
1030
  console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
1030
1031
  }
1032
+ // D83: which indexed column each hit matched on. `expansion` means the
1033
+ // hit only surfaced through a D80 derived variant, so it ranks below an
1034
+ // equal body match by construction (BM25_WEIGHTS) — this is how a
1035
+ // surprising rank gets traced instead of guessed at.
1036
+ if (hits.length > 0) {
1037
+ const via = hits.reduce<Record<string, number>>((acc, h) => {
1038
+ acc[h.matchedVia] = (acc[h.matchedVia] ?? 0) + 1;
1039
+ return acc;
1040
+ }, {});
1041
+ console.log(
1042
+ `matched via: ${Object.entries(via).map(([k, n]) => `${k}=${n}`).join(", ")}` +
1043
+ ` (bm25 weights: content=1 tags=1 aliases=1 expansions=${DEFAULT_EXPANSIONS_WEIGHT} type=1 cjk=1)`,
1044
+ );
1045
+ }
1031
1046
  }
1032
1047
  if (hits.length === 0) {
1033
1048
  console.log("(no matches)");
@@ -1,19 +1,28 @@
1
1
  import { db } from "../store/db.ts";
2
2
  import { toFtsQuery } from "./query.ts";
3
+ import { BM25_WEIGHTS } from "./weights.ts";
3
4
  import {
4
5
  loadAliasEntries,
5
6
  planAliasExpansions,
6
7
  mergeRounds,
7
8
  } from "./alias-memory.ts";
8
9
 
10
+ /** D83: which indexed column a hit's match came from. */
11
+ export type MatchedVia = "body" | "alias" | "expansion" | "other";
12
+
9
13
  export interface SearchHit {
10
14
  id: string;
11
15
  scope_key: string;
12
16
  project_name: string;
13
17
  type: string;
14
18
  tags: string[];
15
- /** D61: capture-time aliases stored on this memory (search hits only). */
19
+ /** D61: capture-time aliases stored on this memory (search hits only).
20
+ * D83: user/authored only — D80 derived variants live in `expansions`. */
16
21
  aliases: string[];
22
+ /** D83: which FTS column produced this hit's match — `body`, `alias`,
23
+ * `expansion`, or `other` (tags/type/cjk). Surfaces in `search --explain`
24
+ * so a surprising rank can be traced to the column that caused it. */
25
+ matchedVia: MatchedVia;
17
26
  snippet: string;
18
27
  score: number;
19
28
  updated_at: number;
@@ -31,6 +40,11 @@ interface RawRow {
31
40
  type: string;
32
41
  tags: string;
33
42
  aliases?: string;
43
+ /** D83: D80 index-time synonym/translation variants — derived, never user-authored. */
44
+ expansions?: string;
45
+ /** D83: per-column FTS highlight, so attribution knows which column matched. */
46
+ alias_hl?: string;
47
+ exp_hl?: string;
34
48
  updated_at: number;
35
49
  snippet: string;
36
50
  score: number;
@@ -44,10 +58,34 @@ interface RawRow {
44
58
  function toHit(r: RawRow): SearchHit {
45
59
  const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
46
60
  let snippet = r.snippet ?? "";
47
- // D61: the snippet highlights matches in the body column only. When the
48
- // match came through an alias (body shows no highlight), say so — the
49
- // hit otherwise looks unrelated to the query.
50
- if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
61
+ // D83: attribution is per-column now. The aliases column holds only
62
+ // user/authored phrasings (D61); the expansions column holds D80 derived
63
+ // variants. Say which one matched — the hit otherwise looks unrelated to
64
+ // the query when the body shows no highlight.
65
+ const hlTerms = (s: string | undefined): string[] => {
66
+ if (!s) return [];
67
+ const m = s.match(/\[([^\]]+)\]/g);
68
+ return m ? [...new Set(m.map((x) => x.slice(1, -1)))] : [];
69
+ };
70
+ const aliasTerms = hlTerms(r.alias_hl);
71
+ const expTerms = hlTerms(r.exp_hl);
72
+ // D83: attribute the hit to the column that matched, so `--explain` can
73
+ // trace an unexpected rank. Order matters: a body highlight is the most
74
+ // direct evidence, then an authored alias, then a derived expansion.
75
+ const matchedVia: MatchedVia = /\[[^\]]+\]/.test(snippet)
76
+ ? "body"
77
+ : aliasTerms.length > 0
78
+ ? "alias"
79
+ : expTerms.length > 0
80
+ ? "expansion"
81
+ : "other";
82
+ if (aliasTerms.length > 0 && aliases.length > 0) {
83
+ snippet += ` (aka: ${aliases.join(", ")})`;
84
+ } else if (expTerms.length > 0) {
85
+ snippet += ` (synonym: ${expTerms.join(", ")})`;
86
+ } else if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
87
+ // Match came through another column (tags/type/cjk) — the authored
88
+ // aliases are still honest context for the hit.
51
89
  snippet += ` (aka: ${aliases.join(", ")})`;
52
90
  }
53
91
  return {
@@ -57,6 +95,7 @@ function toHit(r: RawRow): SearchHit {
57
95
  type: r.type,
58
96
  tags: r.tags ? r.tags.split(",").filter(Boolean) : [],
59
97
  aliases,
98
+ matchedVia,
60
99
  snippet,
61
100
  score: r.score,
62
101
  updated_at: r.updated_at,
@@ -87,6 +126,9 @@ export function hitStateLabel(h: Pick<SearchHit, "scope_key" | "review_state">):
87
126
  return ` [${h.review_state || "draft"}]`;
88
127
  }
89
128
 
129
+ // D83: BM25_WEIGHTS / DEFAULT_EXPANSIONS_WEIGHT live in ./weights.ts, not
130
+ // here — smoke-pure.ts guards them and must stay free of any sqlite import.
131
+
90
132
  /** Optional diagnostics out-param for `search --explain` (D57). */
91
133
  export interface SearchStats {
92
134
  ftsQuery: string;
@@ -174,6 +216,12 @@ export function search(
174
216
  stopwords?: boolean;
175
217
  orJoin?: boolean;
176
218
  prefix?: boolean;
219
+ /**
220
+ * D83: override the BM25 weight of the `expansions` column. Omitted =
221
+ * `DEFAULT_EXPANSIONS_WEIGHT`. An ablation knob — it exists so the
222
+ * fixture can pick the value; not a per-call tuning surface.
223
+ */
224
+ expansionsWeight?: number;
177
225
  } = {},
178
226
  ): SearchHit[] {
179
227
  const qflags = { stopwords: opts.stopwords, orJoin: opts.orJoin, prefix: opts.prefix };
@@ -198,10 +246,12 @@ export function search(
198
246
  const typeFilter = opts.type ? ` AND m.type = ?` : "";
199
247
 
200
248
  const sql = `
201
- SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.updated_at,
249
+ SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.expansions, m.updated_at,
202
250
  m.status, m.superseded_by, m.review_state, m.source, m.created_at,
203
251
  snippet(memories_fts, 0, '[', ']', ' ... ', 12) AS snippet,
204
- bm25(memories_fts) AS score
252
+ highlight(memories_fts, 2, '[', ']') AS alias_hl,
253
+ highlight(memories_fts, 3, '[', ']') AS exp_hl,
254
+ bm25(memories_fts, ?, ?, ?, ?, ?, ?) AS score
205
255
  FROM memories_fts
206
256
  JOIN memories m ON m.rowid = memories_fts.rowid
207
257
  WHERE memories_fts MATCH ?${scopeFilter}${typeFilter}
@@ -214,6 +264,10 @@ export function search(
214
264
  project_name: string;
215
265
  type: string;
216
266
  tags: string;
267
+ aliases?: string;
268
+ expansions?: string;
269
+ alias_hl?: string;
270
+ exp_hl?: string;
217
271
  updated_at: number;
218
272
  status: string;
219
273
  superseded_by: string | null;
@@ -223,8 +277,15 @@ export function search(
223
277
  snippet: string;
224
278
  score: number;
225
279
  };
280
+ // D83: the weight vector is bound per statement, so it leads the parameter
281
+ // list (it appears in the SELECT list, before the WHERE's MATCH).
282
+ const weights =
283
+ opts.expansionsWeight === undefined
284
+ ? BM25_WEIGHTS
285
+ : BM25_WEIGHTS.map((w, i) => (i === 3 ? opts.expansionsWeight! : w));
226
286
  const fetchFts = (matchExpr: string): RawRow[] => {
227
- const params: unknown[] = [matchExpr];
287
+ const params: unknown[] = [...weights];
288
+ params.push(matchExpr);
228
289
  if (opts.scopeKeys && opts.scopeKeys.length > 0) params.push(...opts.scopeKeys);
229
290
  if (opts.type) params.push(opts.type);
230
291
  params.push(fetchLimit);
@@ -244,7 +305,6 @@ export function search(
244
305
  const aliasEntries = loadAliasEntries(opts.scopeKeys);
245
306
  const expansions = planAliasExpansions(query, aliasEntries);
246
307
  let merged: RawRow[] = raw;
247
- let finalLimit = limit;
248
308
  if (expansions.length > 0) {
249
309
  const rounds: RawRow[][] = [raw];
250
310
  for (const eq of expansions) {
@@ -255,20 +315,21 @@ export function search(
255
315
  // divided by that round's best, so an alias-side champion competes
256
316
  // evenly with a literal-side champion — no down-weighting for being
257
317
  // from a later round.
318
+ //
319
+ // The merged pool is deliberately wider than `limit` (each round already
320
+ // over-fetches by fetchLimit) so round-2 hits are never starved the way
321
+ // D79's slice-to-K starved them. #53: that headroom is for RANKING only —
322
+ // the return is still cut to the caller's `limit`, since `limit` is a cap
323
+ // on what every caller (tools, MCP, cli) receives, not a hint.
258
324
  merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
259
- // Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
260
- // starved the way D79's slice-to-K starved them.
261
- const pool = merged.length;
262
- finalLimit = Math.min(pool, 2 * limit);
263
- finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
264
325
  if (opts.stats) opts.stats.aliasExpansions = expansions;
265
326
  }
266
327
 
267
328
  // D80: the curated static map is solved at index time (synonyms.ts
268
- // expands variants into the aliases column at sync time), so without a
269
- // user alias the query side stays a single FTS round.
329
+ // expands variants into the indexed aliases column at sync time), so
330
+ // without a user alias the query side stays a single FTS round.
270
331
  if (opts.stats) opts.stats.candidates = merged.length;
271
- return resolveVisible(merged, finalLimit, opts.stats);
332
+ return resolveVisible(merged, limit, opts.stats);
272
333
  }
273
334
 
274
335
  export function list(
@@ -0,0 +1,37 @@
1
+ /**
2
+ * D83: BM25 column weights for the FTS index.
3
+ *
4
+ * Split out of `search.ts` deliberately: `smoke-pure.ts` must keep running
5
+ * with no SQLite dependency (its header says "pure-logic, no
6
+ * bun:sqlite"), and importing `search.ts` pulls in `store/db.ts` at module
7
+ * load. This module holds the tunable constant and nothing else, so the
8
+ * guard test can import it without opening a database.
9
+ *
10
+ * `expansions` (D80-derived synonym/translation variants) sits strictly below
11
+ * `aliases` so the preference is explicit rather than incidental: a body
12
+ * match outranks an expansion-only match, and a user-authored alias outranks
13
+ * a derived one. This is not cosmetic — measured on equal-length docs, at
14
+ * weight 1.0 a direct body match and an expansion-only match score
15
+ * *identically* (-3.983 vs -3.983), leaving the order to tiebreaks. Any
16
+ * weight < 1.0 breaks that tie in the right direction.
17
+ *
18
+ * The value is the highest one the checked-in fixture cannot distinguish from
19
+ * 1.0 (`scripts/retrieval-eval.ts`: R@1 0.750 / R@5 0.977 / MRR 0.857 hold
20
+ * from 1.0 down to 0.75; 0.5 already costs MRR, and below 0.35 R@5 breaks).
21
+ * So the ordering guarantee is free, and the fixture is the gate that keeps
22
+ * it honest — retune by re-running the sweep, not by feel.
23
+ *
24
+ * Known non-fix: this does NOT repair D80's `how do we ship` rank-8 case.
25
+ * That is per-column length norm (one variant among many in a long
26
+ * `expansions` field), not a weighting problem; see D83's update.
27
+ */
28
+
29
+ /** Weight for the D80 `expansions` column. */
30
+ export const DEFAULT_EXPANSIONS_WEIGHT = 0.75;
31
+
32
+ /**
33
+ * Full weight vector for `memories_fts`, in declaration order.
34
+ * MUST stay in sync with FTS_SCHEMA column order in `store/db.ts`:
35
+ * `[content, tags, aliases, expansions, type, cjk]`.
36
+ */
37
+ export const BM25_WEIGHTS: readonly number[] = [1, 1, 1, DEFAULT_EXPANSIONS_WEIGHT, 1, 1];
package/src/store/db.ts CHANGED
@@ -48,6 +48,7 @@ CREATE TABLE IF NOT EXISTS memories (
48
48
  status TEXT NOT NULL DEFAULT 'active',
49
49
  tags TEXT NOT NULL DEFAULT '',
50
50
  aliases TEXT NOT NULL DEFAULT '',
51
+ expansions TEXT NOT NULL DEFAULT '',
51
52
  alias TEXT NOT NULL DEFAULT '',
52
53
  target TEXT NOT NULL DEFAULT '',
53
54
  content TEXT NOT NULL,
@@ -66,6 +67,19 @@ CREATE INDEX IF NOT EXISTS idx_memories_scope_updated
66
67
  ON memories(scope_key, updated_at DESC);
67
68
  CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
68
69
  CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
70
+ -- D81: the alias registry (retrieve/alias-memory.ts) runs on every search().
71
+ -- The predicate is a full table scan without this — the rows carrying a
72
+ -- defined alias pair are a tiny subset of a corpus that may be tens of
73
+ -- thousands. A partial index keeps the lookup proportional to the number of
74
+ -- aliases defined, not the size of the store.
75
+ --
76
+ -- No SCHEMA_VERSION bump: TABLE_SCHEMA runs on every db open (see db()), so
77
+ -- existing installs pick this up on next open. Deliberately NOT a result
78
+ -- cache — the alias registry must be re-read per query so a newly saved
79
+ -- alias memory applies on the very next query. This index accelerates the
80
+ -- lookup without caching results.
81
+ CREATE INDEX IF NOT EXISTS idx_memories_alias
82
+ ON memories(scope_key) WHERE alias <> '' AND target <> '';
69
83
  `;
70
84
 
71
85
  // The `cjk` column holds pre-tokenized CJK unigrams+bigrams (see
@@ -78,6 +92,7 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
78
92
  content,
79
93
  tags,
80
94
  aliases,
95
+ expansions,
81
96
  type,
82
97
  cjk,
83
98
  scope_key UNINDEXED,
@@ -87,25 +102,25 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
87
102
  );
88
103
 
89
104
  CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
90
- INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
91
- VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
105
+ INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
106
+ VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
92
107
  END;
93
108
 
94
109
  CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
95
- INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
96
- VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
110
+ INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
111
+ VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
97
112
  END;
98
113
 
99
114
  CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
100
- INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
101
- VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
102
- INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
103
- VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
115
+ INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
116
+ VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
117
+ INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
118
+ VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
104
119
  END;
105
120
  `;
106
121
 
107
122
  /** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
108
- const SCHEMA_VERSION = 8;
123
+ const SCHEMA_VERSION = 9;
109
124
 
110
125
  function userVersion(d: AnyDatabase): number {
111
126
  const row = d.prepare("PRAGMA user_version").get() as { user_version: number };
package/src/store/sync.ts CHANGED
@@ -15,8 +15,8 @@ import { projectRoot, paths } from "../paths.ts";
15
15
  import { loadConfig } from "../config.ts";
16
16
 
17
17
  const UPSERT_SQL = `
18
- INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
19
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
18
+ INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, expansions, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
19
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
20
20
  ON CONFLICT(id) DO UPDATE SET
21
21
  scope_key = excluded.scope_key,
22
22
  scope = excluded.scope,
@@ -28,6 +28,7 @@ const UPSERT_SQL = `
28
28
  status = excluded.status,
29
29
  tags = excluded.tags,
30
30
  aliases = excluded.aliases,
31
+ expansions = excluded.expansions,
31
32
  alias = excluded.alias,
32
33
  target = excluded.target,
33
34
  content = excluded.content,
@@ -66,16 +67,20 @@ export function upsertFromFile(mf: MemoryFile): void {
66
67
  function writeRow(mf: MemoryFile, mtimeMs: number): void {
67
68
  const { fm, body, filePath } = mf;
68
69
  const tags = (fm.tags ?? []).join(",");
69
- // D80 index-time expansion: curated synonym/translation variants for the
70
- // memory's keywords are indexed in the aliases column (and therefore the
71
- // CJK bigram column below), so any related word matches in round one.
72
- // Derived at sync time — the Markdown file stays the source of truth.
70
+ // D80 index-time expansion, D83: curated synonym/translation variants for
71
+ // the memory's keywords are indexed in their own `expansions` column —
72
+ // never mixed into `aliases`, which holds only user/authored alternate
73
+ // phrasings (so `(aka: …)` attribution stays honest). The CJK bigram
74
+ // column still covers expansions, so CJK variants stay substring-
75
+ // searchable. Derived at sync time — the Markdown file stays the source
76
+ // of truth.
73
77
  const supplied = fm.aliases ?? [];
74
78
  const have = new Set(supplied.map((a) => a.toLowerCase()));
75
79
  const auto = expansionVariantsForDoc(body + "\n" + tags).filter(
76
80
  (a) => !have.has(a.toLowerCase()),
77
81
  );
78
- const aliases = [...supplied, ...auto].join(",");
82
+ const aliases = supplied.join(",");
83
+ const expansions = auto.join(",");
79
84
  // D81: user-defined alias vocabulary (single nickname → referent pair).
80
85
  db().prepare(UPSERT_SQL).run(
81
86
  fm.id,
@@ -89,10 +94,11 @@ function writeRow(mf: MemoryFile, mtimeMs: number): void {
89
94
  fm.status,
90
95
  tags,
91
96
  aliases,
97
+ expansions,
92
98
  fm.alias ?? "",
93
99
  fm.target ?? "",
94
100
  body,
95
- cjkIndexText(body + "\n" + tags + "\n" + aliases),
101
+ cjkIndexText(body + "\n" + tags + "\n" + aliases + "\n" + expansions),
96
102
  contentHash(body),
97
103
  fm.superseded_by ?? null,
98
104
  fm.source ?? "",
package/src/tools/ops.ts CHANGED
@@ -143,7 +143,12 @@ export const memorySearchArgs = {
143
143
  .enum(["project", "personal", "both"])
144
144
  .optional()
145
145
  .describe("Which scope(s) to search. Default: both."),
146
- type: z.string().optional().describe("Restrict to memories of this type (e.g. decision, preference, gotcha)."),
146
+ type: z
147
+ .enum(MEMORY_TYPE_TAXONOMY)
148
+ .optional()
149
+ .describe(
150
+ "Restrict to memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty.",
151
+ ),
147
152
  limit: z.number().int().min(1).max(50).optional().describe("Max results to return (default 8)."),
148
153
  };
149
154
  export type MemorySearchArgs = z.infer<z.ZodObject<typeof memorySearchArgs>>;
@@ -155,7 +160,12 @@ export const memoryListArgs = {
155
160
  .describe(
156
161
  "Which scope(s) to list. `project` = tied to this repo. `personal` = global across all your projects. `both` lists each scope's newest. Default: project.",
157
162
  ),
158
- type: z.string().optional().describe("Only list memories of this type (e.g. decision, preference, gotcha)."),
163
+ type: z
164
+ .enum(MEMORY_TYPE_TAXONOMY)
165
+ .optional()
166
+ .describe(
167
+ "Only list memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty.",
168
+ ),
159
169
  limit: z.number().int().min(1).max(100).optional().describe("Max entries to list, newest first (default 20)."),
160
170
  include: z
161
171
  .enum(["active", "all"])