open-memex 0.8.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +4 -0
- package/CHANGELOG.md +9 -0
- package/README.md +10 -7
- package/README.zh-CN.md +1 -1
- package/dist/.build-stamp +1 -1
- package/dist/cli.js +13 -0
- package/dist/retrieve/search.js +53 -16
- package/dist/retrieve/weights.js +35 -0
- package/dist/store/db.js +24 -9
- package/dist/store/sync.js +13 -8
- package/dist/tools/ops.js +8 -2
- package/docs/V2-DESIGN.md +8 -0
- package/package.json +1 -1
- package/scripts/smoke-mcp.ts +12 -0
- package/scripts/smoke-pure.ts +16 -0
- package/server.json +2 -2
- package/src/cli.ts +15 -0
- package/src/retrieve/search.ts +78 -17
- package/src/retrieve/weights.ts +37 -0
- package/src/store/db.ts +24 -9
- package/src/store/sync.ts +14 -8
- package/src/tools/ops.ts +12 -2
package/AGENTS.md
CHANGED
|
@@ -205,6 +205,10 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
|
|
|
205
205
|
new commands → README CLI sections + this file's Commands. A change without its docs
|
|
206
206
|
is not done. Anything deferred as "later" goes into the near-term backlog in
|
|
207
207
|
`docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
|
|
208
|
+
- **Measurement before mechanism.** A retrieval change lands its fixture +
|
|
209
|
+
baseline number in a commit with no retrieval change at all, so the delta is
|
|
210
|
+
unambiguous. `scripts/retrieval-eval.ts` is the gate: no retrieval PR merges
|
|
211
|
+
without before/after recall@k + MRR on the checked-in fixture.
|
|
208
212
|
- **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
|
|
209
213
|
in-development version. A new alpha line starts with a minor bump after the
|
|
210
214
|
previous minor shipped (`0.3.0` → `0.4.0-alpha.1`); while that minor is still
|
package/CHANGELOG.md
CHANGED
|
@@ -9,6 +9,15 @@ the `@alpha` tag are development snapshots and are not listed individually.
|
|
|
9
9
|
|
|
10
10
|
## [Unreleased]
|
|
11
11
|
|
|
12
|
+
## [0.8.1] - 2026-10-07
|
|
13
|
+
|
|
14
|
+
- D83 follow-up: the two D83 clauses that shipped as design but not as code. (1) **BM25 weight for the `expansions` column** — `bm25(memories_fts, …)` now takes an explicit weight vector (`src/retrieve/weights.ts`, order `[content, tags, aliases, expansions, type, cjk]`) instead of the implicit all-1.0. `expansions` is `0.75`: measured on equal-length docs, a direct body match and an expansion-only match score *identically* at weight 1.0 (`-3.983` vs `-3.983`), leaving the order to tiebreaks — any weight below 1.0 breaks that tie in the right direction. `0.75` is the highest value the fixture cannot distinguish from 1.0 (sweep: 1.0/0.9/0.75 all give R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 costs MRR, below 0.35 R@5 breaks). Exposed as `opts.expansionsWeight` so the sweep stays measurable; `smoke-pure.ts` pins the vector's shape and ordering. (2) **`search --explain` per-column attribution** — reports `matched via: body=N alias=N expansion=N other=N` plus the active weights. Does **not** fix the D80 `how do we ship` rank-8 case: that is per-column length norm, not weighting. Fixture unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857.
|
|
15
|
+
|
|
16
|
+
- fix: `search()` honours the caller's `limit` again (#53). When a D81 alias round fired, the widened candidate pool leaked out through the return value, so `memory_search` / MCP `memory_search` / `cli search` returned up to **2× `limit`** hits with no signal it happened. The pool is still wider than `limit` for *ranking* (round-2 hits are not starved the way D79's slice-to-K starved them); only the cut changed, to `limit`. Drops the dead `1.5K` line, which could never raise the value. No ranking change — `scripts/retrieval-eval.ts` is unchanged at recall@1 0.750 / recall@5 0.977 / MRR 0.857.
|
|
17
|
+
- perf: partial index on the alias registry (#55): `loadAliasEntries` ran `WHERE alias <> '' AND target <> ''` as a full table scan on **every** search. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`, which runs on every db open — no `SCHEMA_VERSION` bump, no rebuild, existing installs pick it up on next open. Measured at 5k memories: 971ms → 39ms per 2000 lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Deliberately not a result cache: a newly saved alias must still apply on the very next query.
|
|
18
|
+
- D83 (implemented, for #56): D80 index-time expansion variants moved out of the `aliases` column into a dedicated `expansions` index column (own FTS column + triggers; `SearchHit.aliases` and `(aka: …)` surface only user-authored aliases; expansion-only matches report `(synonym: …)`; `--explain` shows the same via the snippet). Schema v8 → v9 with rebuild-from-Markdown migration (verified on a scratch DB). `retrieval-eval` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857 — the `how do we ship` rank stayed 8: expansion-column length-norm dilution is inherent, not caused by the mixing.
|
|
19
|
+
- `memory_search` / `memory_list` `type` is now a closed enum (D82, closes #51): a misspelled type is rejected at the schema (MCP -32602) instead of silently returning zero results. The opencode plugin shares the same args objects, so both surfaces tighten together; `scripts/smoke-mcp.ts` guards the rejection.
|
|
20
|
+
|
|
12
21
|
## [0.8.0] - 2026-10-07
|
|
13
22
|
|
|
14
23
|
- User-defined alias vocabulary as query-time expansion (D81): any memory type may now carry `alias:` (nickname, e.g. 香蕉计划) and `target:` (referent, e.g. 支付系统重构项目) frontmatter — an alias memory defines vocabulary once per scope. At query time, query terms hit the alias registry (exact match, one cheap SELECT per search) and the query is rewritten with the other side of the relation — bidirectional, depth < 3, cycle-safe — then re-run as round 2/3. Rounds merge by normalized-max (`mergeRounds` in `src/retrieve/alias-memory.ts`): each round's BM25 scores are divided by that round's best, so an alias-side champion competes evenly with a literal-side champion; the merged pool is re-ranked and cut to [1.5K, 2K], never back to K. No reindex when an alias changes. Creation: `memory_add` alias/target params, CLI `add --alias/--target`, or frontmatter edit (v1: user-explicit only). Fixture grows to 54 memories / 44 queries (3 bidirectional alias queries): recall@1 0.750, recall@5 **0.977**, MRR 0.857. Also removes the init `captureAliases` and keyword-capture questions — both are now always on (Stone 2026-10-07).
|
package/README.md
CHANGED
|
@@ -410,8 +410,8 @@ holiday policy. Always on — the agent may attach up to 4 per memory, and you
|
|
|
410
410
|
can pass your own via `open-memex add --aliases`.
|
|
411
411
|
|
|
412
412
|
Three ways to add them: `open-memex add "text" --aliases "phrase one; phrase
|
|
413
|
-
two"` (semicolon-separated, up to 4); ask the agent ("remember:
|
|
414
|
-
AI agent, alias it as
|
|
413
|
+
two"` (semicolon-separated, up to 4); ask the agent ("remember: Fluffy is my
|
|
414
|
+
AI agent, alias it as Momo"); or edit the memory's `aliases:` frontmatter
|
|
415
415
|
directly — Markdown is the source of truth and sync picks it up.
|
|
416
416
|
|
|
417
417
|
Separately, a curated synonym/translation map is expanded into the index
|
|
@@ -549,10 +549,10 @@ Three layers, each covering what the others miss: **BM25 keyword search**
|
|
|
549
549
|
**synonym expansion in two halves** — (a) index-time (D80: a curated,
|
|
550
550
|
checked-in map of dev-domain equivalents and EN↔ZH translation pairs
|
|
551
551
|
expanded into the index at write time, so `ship` finds `deploy` and
|
|
552
|
-
|
|
552
|
+
`bug` finds `defect` in a plain single-round query; deterministic, no
|
|
553
553
|
model download); (b) query-time user aliases (D81: an alias memory carrying
|
|
554
554
|
`alias:`/`target:` frontmatter defines your vocabulary once per scope —
|
|
555
|
-
e.g.
|
|
555
|
+
e.g. Banana Plan → payment system refactor project — and a query mentioning either side
|
|
556
556
|
triggers a second round on the other side, both directions, merged evenly
|
|
557
557
|
with round one so alias hits are never down-weighted; no reindex when the
|
|
558
558
|
alias changes); and **semantic vectors**
|
|
@@ -593,11 +593,14 @@ Honest edges, so nothing surprises you:
|
|
|
593
593
|
|
|
594
594
|
- **Keyword search, not semantic.** Retrieval is BM25 keyword matching, plus a
|
|
595
595
|
curated synonym map expanded into the index at write time (`ship` finds
|
|
596
|
-
`deploy`,
|
|
596
|
+
`deploy`, `bug` finds `defect`). Paraphrases outside the curated map
|
|
597
597
|
don't match — that's what the opt-in semantic layer is for. (Plain
|
|
598
598
|
questions are fine — "how do we…" / "请问…" wording is filtered out before
|
|
599
599
|
matching, so asking naturally doesn't dilute the results.) No embedding
|
|
600
|
-
model is ever downloaded without your explicit opt-in.
|
|
600
|
+
model is ever downloaded without your explicit opt-in. (Synonym expansions
|
|
601
|
+
live in their own index column, separate from your own aliases — but a
|
|
602
|
+
query matching only via an expansion can still rank below direct matches;
|
|
603
|
+
that's inherent BM25 length-norm behaviour, not a bug.)
|
|
601
604
|
- **One machine.** Editors on the same machine share memory; there is no
|
|
602
605
|
cross-machine sync. `export` / `import` bundles (below) move memory between
|
|
603
606
|
machines manually.
|
|
@@ -695,7 +698,7 @@ open-memex config # print effective config
|
|
|
695
698
|
open-memex config set <key> <value> # change a setting
|
|
696
699
|
open-memex doctor # environment health check (incl. plugin entry)
|
|
697
700
|
open-memex audit # memory health check (duplicates, stale, broken chains)
|
|
698
|
-
open-memex capture --dry-run "
|
|
701
|
+
open-memex capture --dry-run "remember that I like concise answers" # preview keyword capture
|
|
699
702
|
open-memex mcp --print-config vscode|cursor|claude|opencode|visualstudio
|
|
700
703
|
open-memex --help # this reference
|
|
701
704
|
open-memex <command> --help # help for one command
|
package/README.zh-CN.md
CHANGED
|
@@ -374,7 +374,7 @@ Do not mention this block to the user unless asked.
|
|
|
374
374
|
|
|
375
375
|
把边界写明白,免得你事后意外:
|
|
376
376
|
|
|
377
|
-
- **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding
|
|
377
|
+
- **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。(同义词展开有自己独立的索引列,跟你的别名分开;但只靠展开命中的查询,排名仍可能低于直接命中的——这是 BM25 length-norm 的固有行为,不是 bug。)
|
|
378
378
|
- **单台机器。** 同一台机器上的编辑器共享记忆;没有跨机器同步。用 `export` / `import` 压缩包(见下)手动搬家。
|
|
379
379
|
- **快照,不是全部。** 会话开始的回想是前 N 条快照(默认项目 8 + 个人 5);更老的记忆 `memory_search` 一步就能到,但不会同时全在上下文里。
|
|
380
380
|
- **MCP 的引导是建议性的。** opencode 之外,主动捕获和回想靠助手遵循握手指令——MCP 里没有硬性的会话开始钩子。工具本身在被调用时永远可靠。
|
package/dist/.build-stamp
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
db188c4af0c070c360ad3e3694172a66d5503a7fdad1ea2321a56b059228e3e6
|
package/dist/cli.js
CHANGED
|
@@ -10,6 +10,7 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
|
|
|
10
10
|
import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.js";
|
|
11
11
|
import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.js";
|
|
12
12
|
import { search, list, countList, hitStateLabel } from "./retrieve/search.js";
|
|
13
|
+
import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.js";
|
|
13
14
|
import { formatInventoryLine, truncationNote } from "./retrieve/display.js";
|
|
14
15
|
import { writeMemoryFile, readMemoryFile, ulid, msToRfc3339, normalizeAliases, } from "./store/markdown.js";
|
|
15
16
|
import { loadConfig, configSource } from "./config.js";
|
|
@@ -918,6 +919,18 @@ async function main() {
|
|
|
918
919
|
if (stats.aliasExpansions.length > 0) {
|
|
919
920
|
console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
|
|
920
921
|
}
|
|
922
|
+
// D83: which indexed column each hit matched on. `expansion` means the
|
|
923
|
+
// hit only surfaced through a D80 derived variant, so it ranks below an
|
|
924
|
+
// equal body match by construction (BM25_WEIGHTS) — this is how a
|
|
925
|
+
// surprising rank gets traced instead of guessed at.
|
|
926
|
+
if (hits.length > 0) {
|
|
927
|
+
const via = hits.reduce((acc, h) => {
|
|
928
|
+
acc[h.matchedVia] = (acc[h.matchedVia] ?? 0) + 1;
|
|
929
|
+
return acc;
|
|
930
|
+
}, {});
|
|
931
|
+
console.log(`matched via: ${Object.entries(via).map(([k, n]) => `${k}=${n}`).join(", ")}` +
|
|
932
|
+
` (bm25 weights: content=1 tags=1 aliases=1 expansions=${DEFAULT_EXPANSIONS_WEIGHT} type=1 cjk=1)`);
|
|
933
|
+
}
|
|
921
934
|
}
|
|
922
935
|
if (hits.length === 0) {
|
|
923
936
|
console.log("(no matches)");
|
package/dist/retrieve/search.js
CHANGED
|
@@ -1,13 +1,41 @@
|
|
|
1
1
|
import { db } from "../store/db.js";
|
|
2
2
|
import { toFtsQuery } from "./query.js";
|
|
3
|
+
import { BM25_WEIGHTS } from "./weights.js";
|
|
3
4
|
import { loadAliasEntries, planAliasExpansions, mergeRounds, } from "./alias-memory.js";
|
|
4
5
|
function toHit(r) {
|
|
5
6
|
const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
|
|
6
7
|
let snippet = r.snippet ?? "";
|
|
7
|
-
//
|
|
8
|
-
//
|
|
9
|
-
// hit otherwise looks unrelated to
|
|
10
|
-
|
|
8
|
+
// D83: attribution is per-column now. The aliases column holds only
|
|
9
|
+
// user/authored phrasings (D61); the expansions column holds D80 derived
|
|
10
|
+
// variants. Say which one matched — the hit otherwise looks unrelated to
|
|
11
|
+
// the query when the body shows no highlight.
|
|
12
|
+
const hlTerms = (s) => {
|
|
13
|
+
if (!s)
|
|
14
|
+
return [];
|
|
15
|
+
const m = s.match(/\[([^\]]+)\]/g);
|
|
16
|
+
return m ? [...new Set(m.map((x) => x.slice(1, -1)))] : [];
|
|
17
|
+
};
|
|
18
|
+
const aliasTerms = hlTerms(r.alias_hl);
|
|
19
|
+
const expTerms = hlTerms(r.exp_hl);
|
|
20
|
+
// D83: attribute the hit to the column that matched, so `--explain` can
|
|
21
|
+
// trace an unexpected rank. Order matters: a body highlight is the most
|
|
22
|
+
// direct evidence, then an authored alias, then a derived expansion.
|
|
23
|
+
const matchedVia = /\[[^\]]+\]/.test(snippet)
|
|
24
|
+
? "body"
|
|
25
|
+
: aliasTerms.length > 0
|
|
26
|
+
? "alias"
|
|
27
|
+
: expTerms.length > 0
|
|
28
|
+
? "expansion"
|
|
29
|
+
: "other";
|
|
30
|
+
if (aliasTerms.length > 0 && aliases.length > 0) {
|
|
31
|
+
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
32
|
+
}
|
|
33
|
+
else if (expTerms.length > 0) {
|
|
34
|
+
snippet += ` (synonym: ${expTerms.join(", ")})`;
|
|
35
|
+
}
|
|
36
|
+
else if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
|
|
37
|
+
// Match came through another column (tags/type/cjk) — the authored
|
|
38
|
+
// aliases are still honest context for the hit.
|
|
11
39
|
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
12
40
|
}
|
|
13
41
|
return {
|
|
@@ -17,6 +45,7 @@ function toHit(r) {
|
|
|
17
45
|
type: r.type,
|
|
18
46
|
tags: r.tags ? r.tags.split(",").filter(Boolean) : [],
|
|
19
47
|
aliases,
|
|
48
|
+
matchedVia,
|
|
20
49
|
snippet,
|
|
21
50
|
score: r.score,
|
|
22
51
|
updated_at: r.updated_at,
|
|
@@ -138,18 +167,26 @@ export function search(query, opts = {}) {
|
|
|
138
167
|
: "";
|
|
139
168
|
const typeFilter = opts.type ? ` AND m.type = ?` : "";
|
|
140
169
|
const sql = `
|
|
141
|
-
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.updated_at,
|
|
170
|
+
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.expansions, m.updated_at,
|
|
142
171
|
m.status, m.superseded_by, m.review_state, m.source, m.created_at,
|
|
143
172
|
snippet(memories_fts, 0, '[', ']', ' ... ', 12) AS snippet,
|
|
144
|
-
|
|
173
|
+
highlight(memories_fts, 2, '[', ']') AS alias_hl,
|
|
174
|
+
highlight(memories_fts, 3, '[', ']') AS exp_hl,
|
|
175
|
+
bm25(memories_fts, ?, ?, ?, ?, ?, ?) AS score
|
|
145
176
|
FROM memories_fts
|
|
146
177
|
JOIN memories m ON m.rowid = memories_fts.rowid
|
|
147
178
|
WHERE memories_fts MATCH ?${scopeFilter}${typeFilter}
|
|
148
179
|
ORDER BY score ASC
|
|
149
180
|
LIMIT ?
|
|
150
181
|
`;
|
|
182
|
+
// D83: the weight vector is bound per statement, so it leads the parameter
|
|
183
|
+
// list (it appears in the SELECT list, before the WHERE's MATCH).
|
|
184
|
+
const weights = opts.expansionsWeight === undefined
|
|
185
|
+
? BM25_WEIGHTS
|
|
186
|
+
: BM25_WEIGHTS.map((w, i) => (i === 3 ? opts.expansionsWeight : w));
|
|
151
187
|
const fetchFts = (matchExpr) => {
|
|
152
|
-
const params = [
|
|
188
|
+
const params = [...weights];
|
|
189
|
+
params.push(matchExpr);
|
|
153
190
|
if (opts.scopeKeys && opts.scopeKeys.length > 0)
|
|
154
191
|
params.push(...opts.scopeKeys);
|
|
155
192
|
if (opts.type)
|
|
@@ -169,7 +206,6 @@ export function search(query, opts = {}) {
|
|
|
169
206
|
const aliasEntries = loadAliasEntries(opts.scopeKeys);
|
|
170
207
|
const expansions = planAliasExpansions(query, aliasEntries);
|
|
171
208
|
let merged = raw;
|
|
172
|
-
let finalLimit = limit;
|
|
173
209
|
if (expansions.length > 0) {
|
|
174
210
|
const rounds = [raw];
|
|
175
211
|
for (const eq of expansions) {
|
|
@@ -181,21 +217,22 @@ export function search(query, opts = {}) {
|
|
|
181
217
|
// divided by that round's best, so an alias-side champion competes
|
|
182
218
|
// evenly with a literal-side champion — no down-weighting for being
|
|
183
219
|
// from a later round.
|
|
220
|
+
//
|
|
221
|
+
// The merged pool is deliberately wider than `limit` (each round already
|
|
222
|
+
// over-fetches by fetchLimit) so round-2 hits are never starved the way
|
|
223
|
+
// D79's slice-to-K starved them. #53: that headroom is for RANKING only —
|
|
224
|
+
// the return is still cut to the caller's `limit`, since `limit` is a cap
|
|
225
|
+
// on what every caller (tools, MCP, cli) receives, not a hint.
|
|
184
226
|
merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
|
|
185
|
-
// Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
|
|
186
|
-
// starved the way D79's slice-to-K starved them.
|
|
187
|
-
const pool = merged.length;
|
|
188
|
-
finalLimit = Math.min(pool, 2 * limit);
|
|
189
|
-
finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
|
|
190
227
|
if (opts.stats)
|
|
191
228
|
opts.stats.aliasExpansions = expansions;
|
|
192
229
|
}
|
|
193
230
|
// D80: the curated static map is solved at index time (synonyms.ts
|
|
194
|
-
// expands variants into the aliases column at sync time), so
|
|
195
|
-
// user alias the query side stays a single FTS round.
|
|
231
|
+
// expands variants into the indexed aliases column at sync time), so
|
|
232
|
+
// without a user alias the query side stays a single FTS round.
|
|
196
233
|
if (opts.stats)
|
|
197
234
|
opts.stats.candidates = merged.length;
|
|
198
|
-
return resolveVisible(merged,
|
|
235
|
+
return resolveVisible(merged, limit, opts.stats);
|
|
199
236
|
}
|
|
200
237
|
export function list(scopeKey, opts = {}) {
|
|
201
238
|
const limit = Math.max(1, Math.min(opts.limit ?? 20, 100));
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D83: BM25 column weights for the FTS index.
|
|
3
|
+
*
|
|
4
|
+
* Split out of `search.ts` deliberately: `smoke-pure.ts` must keep running
|
|
5
|
+
* with no SQLite dependency (its header says "pure-logic, no
|
|
6
|
+
* bun:sqlite"), and importing `search.ts` pulls in `store/db.ts` at module
|
|
7
|
+
* load. This module holds the tunable constant and nothing else, so the
|
|
8
|
+
* guard test can import it without opening a database.
|
|
9
|
+
*
|
|
10
|
+
* `expansions` (D80-derived synonym/translation variants) sits strictly below
|
|
11
|
+
* `aliases` so the preference is explicit rather than incidental: a body
|
|
12
|
+
* match outranks an expansion-only match, and a user-authored alias outranks
|
|
13
|
+
* a derived one. This is not cosmetic — measured on equal-length docs, at
|
|
14
|
+
* weight 1.0 a direct body match and an expansion-only match score
|
|
15
|
+
* *identically* (-3.983 vs -3.983), leaving the order to tiebreaks. Any
|
|
16
|
+
* weight < 1.0 breaks that tie in the right direction.
|
|
17
|
+
*
|
|
18
|
+
* The value is the highest one the checked-in fixture cannot distinguish from
|
|
19
|
+
* 1.0 (`scripts/retrieval-eval.ts`: R@1 0.750 / R@5 0.977 / MRR 0.857 hold
|
|
20
|
+
* from 1.0 down to 0.75; 0.5 already costs MRR, and below 0.35 R@5 breaks).
|
|
21
|
+
* So the ordering guarantee is free, and the fixture is the gate that keeps
|
|
22
|
+
* it honest — retune by re-running the sweep, not by feel.
|
|
23
|
+
*
|
|
24
|
+
* Known non-fix: this does NOT repair D80's `how do we ship` rank-8 case.
|
|
25
|
+
* That is per-column length norm (one variant among many in a long
|
|
26
|
+
* `expansions` field), not a weighting problem; see D83's update.
|
|
27
|
+
*/
|
|
28
|
+
/** Weight for the D80 `expansions` column. */
|
|
29
|
+
export const DEFAULT_EXPANSIONS_WEIGHT = 0.75;
|
|
30
|
+
/**
|
|
31
|
+
* Full weight vector for `memories_fts`, in declaration order.
|
|
32
|
+
* MUST stay in sync with FTS_SCHEMA column order in `store/db.ts`:
|
|
33
|
+
* `[content, tags, aliases, expansions, type, cjk]`.
|
|
34
|
+
*/
|
|
35
|
+
export const BM25_WEIGHTS = [1, 1, 1, DEFAULT_EXPANSIONS_WEIGHT, 1, 1];
|
package/dist/store/db.js
CHANGED
|
@@ -35,6 +35,7 @@ CREATE TABLE IF NOT EXISTS memories (
|
|
|
35
35
|
status TEXT NOT NULL DEFAULT 'active',
|
|
36
36
|
tags TEXT NOT NULL DEFAULT '',
|
|
37
37
|
aliases TEXT NOT NULL DEFAULT '',
|
|
38
|
+
expansions TEXT NOT NULL DEFAULT '',
|
|
38
39
|
alias TEXT NOT NULL DEFAULT '',
|
|
39
40
|
target TEXT NOT NULL DEFAULT '',
|
|
40
41
|
content TEXT NOT NULL,
|
|
@@ -53,6 +54,19 @@ CREATE INDEX IF NOT EXISTS idx_memories_scope_updated
|
|
|
53
54
|
ON memories(scope_key, updated_at DESC);
|
|
54
55
|
CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
|
|
55
56
|
CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
|
|
57
|
+
-- D81: the alias registry (retrieve/alias-memory.ts) runs on every search().
|
|
58
|
+
-- The predicate is a full table scan without this — the rows carrying a
|
|
59
|
+
-- defined alias pair are a tiny subset of a corpus that may be tens of
|
|
60
|
+
-- thousands. A partial index keeps the lookup proportional to the number of
|
|
61
|
+
-- aliases defined, not the size of the store.
|
|
62
|
+
--
|
|
63
|
+
-- No SCHEMA_VERSION bump: TABLE_SCHEMA runs on every db open (see db()), so
|
|
64
|
+
-- existing installs pick this up on next open. Deliberately NOT a result
|
|
65
|
+
-- cache — the alias registry must be re-read per query so a newly saved
|
|
66
|
+
-- alias memory applies on the very next query. This index accelerates the
|
|
67
|
+
-- lookup without caching results.
|
|
68
|
+
CREATE INDEX IF NOT EXISTS idx_memories_alias
|
|
69
|
+
ON memories(scope_key) WHERE alias <> '' AND target <> '';
|
|
56
70
|
`;
|
|
57
71
|
// The `cjk` column holds pre-tokenized CJK unigrams+bigrams (see
|
|
58
72
|
// src/retrieve/cjk.ts). FTS5's unicode61 treats a CJK run as one token, so
|
|
@@ -64,6 +78,7 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
|
|
|
64
78
|
content,
|
|
65
79
|
tags,
|
|
66
80
|
aliases,
|
|
81
|
+
expansions,
|
|
67
82
|
type,
|
|
68
83
|
cjk,
|
|
69
84
|
scope_key UNINDEXED,
|
|
@@ -73,24 +88,24 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
|
|
|
73
88
|
);
|
|
74
89
|
|
|
75
90
|
CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
|
|
76
|
-
INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
|
|
77
|
-
VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
|
|
91
|
+
INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
92
|
+
VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
|
|
78
93
|
END;
|
|
79
94
|
|
|
80
95
|
CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
|
|
81
|
-
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
|
|
82
|
-
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
|
|
96
|
+
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
97
|
+
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
|
|
83
98
|
END;
|
|
84
99
|
|
|
85
100
|
CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
|
|
86
|
-
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
|
|
87
|
-
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
|
|
88
|
-
INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
|
|
89
|
-
VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
|
|
101
|
+
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
102
|
+
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
|
|
103
|
+
INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
104
|
+
VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
|
|
90
105
|
END;
|
|
91
106
|
`;
|
|
92
107
|
/** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
|
|
93
|
-
const SCHEMA_VERSION =
|
|
108
|
+
const SCHEMA_VERSION = 9;
|
|
94
109
|
function userVersion(d) {
|
|
95
110
|
const row = d.prepare("PRAGMA user_version").get();
|
|
96
111
|
return row.user_version;
|
package/dist/store/sync.js
CHANGED
|
@@ -8,8 +8,8 @@ import { iterMemoryFiles, iterInRepoMemoryFiles, readMemoryFile, timeToMs, } fro
|
|
|
8
8
|
import { projectRoot, paths } from "../paths.js";
|
|
9
9
|
import { loadConfig } from "../config.js";
|
|
10
10
|
const UPSERT_SQL = `
|
|
11
|
-
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
12
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
11
|
+
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, expansions, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
12
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
13
13
|
ON CONFLICT(id) DO UPDATE SET
|
|
14
14
|
scope_key = excluded.scope_key,
|
|
15
15
|
scope = excluded.scope,
|
|
@@ -21,6 +21,7 @@ const UPSERT_SQL = `
|
|
|
21
21
|
status = excluded.status,
|
|
22
22
|
tags = excluded.tags,
|
|
23
23
|
aliases = excluded.aliases,
|
|
24
|
+
expansions = excluded.expansions,
|
|
24
25
|
alias = excluded.alias,
|
|
25
26
|
target = excluded.target,
|
|
26
27
|
content = excluded.content,
|
|
@@ -57,16 +58,20 @@ export function upsertFromFile(mf) {
|
|
|
57
58
|
function writeRow(mf, mtimeMs) {
|
|
58
59
|
const { fm, body, filePath } = mf;
|
|
59
60
|
const tags = (fm.tags ?? []).join(",");
|
|
60
|
-
// D80 index-time expansion: curated synonym/translation variants for
|
|
61
|
-
// memory's keywords are indexed in
|
|
62
|
-
//
|
|
63
|
-
//
|
|
61
|
+
// D80 index-time expansion, D83: curated synonym/translation variants for
|
|
62
|
+
// the memory's keywords are indexed in their own `expansions` column —
|
|
63
|
+
// never mixed into `aliases`, which holds only user/authored alternate
|
|
64
|
+
// phrasings (so `(aka: …)` attribution stays honest). The CJK bigram
|
|
65
|
+
// column still covers expansions, so CJK variants stay substring-
|
|
66
|
+
// searchable. Derived at sync time — the Markdown file stays the source
|
|
67
|
+
// of truth.
|
|
64
68
|
const supplied = fm.aliases ?? [];
|
|
65
69
|
const have = new Set(supplied.map((a) => a.toLowerCase()));
|
|
66
70
|
const auto = expansionVariantsForDoc(body + "\n" + tags).filter((a) => !have.has(a.toLowerCase()));
|
|
67
|
-
const aliases =
|
|
71
|
+
const aliases = supplied.join(",");
|
|
72
|
+
const expansions = auto.join(",");
|
|
68
73
|
// D81: user-defined alias vocabulary (single nickname → referent pair).
|
|
69
|
-
db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
|
|
74
|
+
db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, expansions, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases + "\n" + expansions), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
|
|
70
75
|
}
|
|
71
76
|
export function deleteFromIndex(id) {
|
|
72
77
|
db().prepare(`DELETE FROM memories WHERE id = ?`).run(id);
|
package/dist/tools/ops.js
CHANGED
|
@@ -90,7 +90,10 @@ export const memorySearchArgs = {
|
|
|
90
90
|
.enum(["project", "personal", "both"])
|
|
91
91
|
.optional()
|
|
92
92
|
.describe("Which scope(s) to search. Default: both."),
|
|
93
|
-
type: z
|
|
93
|
+
type: z
|
|
94
|
+
.enum(MEMORY_TYPE_TAXONOMY)
|
|
95
|
+
.optional()
|
|
96
|
+
.describe("Restrict to memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
|
|
94
97
|
limit: z.number().int().min(1).max(50).optional().describe("Max results to return (default 8)."),
|
|
95
98
|
};
|
|
96
99
|
export const memoryListArgs = {
|
|
@@ -98,7 +101,10 @@ export const memoryListArgs = {
|
|
|
98
101
|
.enum(["project", "personal", "both"])
|
|
99
102
|
.optional()
|
|
100
103
|
.describe("Which scope(s) to list. `project` = tied to this repo. `personal` = global across all your projects. `both` lists each scope's newest. Default: project."),
|
|
101
|
-
type: z
|
|
104
|
+
type: z
|
|
105
|
+
.enum(MEMORY_TYPE_TAXONOMY)
|
|
106
|
+
.optional()
|
|
107
|
+
.describe("Only list memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
|
|
102
108
|
limit: z.number().int().min(1).max(100).optional().describe("Max entries to list, newest first (default 20)."),
|
|
103
109
|
include: z
|
|
104
110
|
.enum(["active", "all"])
|
package/docs/V2-DESIGN.md
CHANGED
|
@@ -1605,6 +1605,14 @@ requirement: personal data never touches third-party services). Benchmarks to tr
|
|
|
1605
1605
|
- **D81 — user-defined alias vocabulary as query-time expansion (2026-10-07).** Stone: an alias should be a *kind of memory* — defined once, remembered forever. Any memory type may carry `alias:` (the nickname, e.g. 香蕉计划) and `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter; the alias memory defines vocabulary once per scope. At query time (`src/retrieve/alias-memory.ts`), query terms are looked up against the alias registry — one cheap `SELECT` per `search()` call over rows with non-empty alias+target (trigger A: exact registry hit, never a count heuristic; the D79 0/470 trigger is not repeated). On a hit the query is rewritten with the other side of the relation — bidirectional (alias→target and target→alias), depth < 3, cycle-safe — and re-run as round 2 (round 3 for chains). Rounds merge by normalized-max (`mergeRounds`): each round's BM25 scores are divided by that round's best (raw cross-round scores are incomparable — °C vs °F), then final = max, so an alias-side champion competes evenly with a literal-side champion — alias hits are never down-weighted for coming from a later round. Merged pool is re-ranked then cut to [1.5K, 2K] (Stone) — never back to K, or round-2 hits would be starved the way D79's slice-to-K starved them. No reindex when an alias changes (query-time). Scope-aware: a project codename stays in its project. *Rationale: D80's curated map is static and shared — personal vocabulary (小宝贝, 香蕉计划) doesn't belong in it; per-memory D61 aliases don't propagate to new memories. Alias memories close the gap: user-defined, scope-aware, effective immediately.* v1: user-explicit creation only (`memory_add` alias/target params, CLI `--alias`/`--target`, frontmatter edit); agent-proposed aliases deferred. D81 also rewords the init `captureAliases` prompt (it is not the synonym switch — D80's expansion always applies). Eval: 3 new fixture queries (香蕉计划↔支付系统重构项目 both directions + the alias memory itself), 44 queries total: recall@1 0.750, recall@5 **0.977** (43/44), MRR 0.857 — the single miss remains `how do we ship` (known D80 trade-off).
|
|
1606
1606
|
|
|
1607
1607
|
|
|
1608
|
+
- **D82 — closed `type` taxonomy on search/list (2026-10-07).** Follow-up deferred from the TDQS pass (#50), closes issue #51: `memory_search` / `memory_list` `type` is now `z.enum(MEMORY_TYPE_TAXONOMY)` instead of `z.string()` — a misspelled type (`"decisions"`) is rejected at the schema (MCP -32602) instead of returning zero results the agent can't distinguish from "no memories of that type". Hard rejection, not a soft "no matches" message: closed taxonomy + fail-fast. Existing rows with non-taxonomy types are possible (the frontmatter parser only coerces the v1 `instruction` → `knowledge` case) and are unaffected — they still surface in unfiltered search/list, they just can't be addressed by a type filter. The opencode plugin shares the same args objects, so both surfaces tighten together; `scripts/smoke-mcp.ts` guards the rejection. `memory_submit` keeps `destructiveHint: false` — settled in the #50 review (the outbox unlink happens after the repo copy is committed and hash-verified, so it's recoverable from git); recorded here so it isn't re-litigated.
|
|
1609
|
+
|
|
1610
|
+
- **D83 — D80 expansions get their own index column (2026-10-07).** Design follow-up to issue #56 (deferred from the D80 review): the curated index-time variants (`expansionVariantsForDoc`) move out of the `aliases` column into a dedicated `expansions` column — on `memories`, on `memories_fts`, and in the ai/ad/au triggers. `writeRow` stores user/authored aliases in `aliases` and derived variants in `expansions` (still filtered against the supplied set, still never written to Markdown); the `cjk` bigram column keeps covering body + tags + aliases + expansions so CJK variants stay substring-searchable. Retrieval: the FTS MATCH set gains the column with its own bm25 weight (at or below `aliases` — the fixture is the gate, tune don't guess); `SearchHit.aliases` and the `(aka: …)` attribution surface only user-authored aliases; `search --explain` reports which column matched (alias vs expansion). Migration: `SCHEMA_VERSION` 8 → 9; the bump triggers `rebuildIndexSchema`, which repopulates from Markdown — safe because expansions are derived and Markdown is untouched.
|
|
1611
|
+
*Update (2026-10-07, implemented and measured):* the `how do we ship` rank did **not** improve (still 8) — the dilution for expansion-column matches is inherent to per-column BM25 length norm (32 variants in their own column dilute about as much as 36 did in the shared one), not to the mixing. What the split fixed is provenance (`(aka:)` / `SearchHit.aliases` are user-authored again) and the dilution of deliberate user aliases (a ≤4-term column now scores cleanly). Regression gate passed: `retrieval-eval` R@1 0.750 / R@5 0.977 / MRR 0.857, identical to pre-split; v8→v9 migration verified on a scratch DB (merged aliases re-split correctly, `user_version` 9). *Rationale: an agent reading `(aka: …)` must be able to trust it means "alternate phrasings someone attached to this memory", and a 32-variant derived list must not dilute the weight of 4 deliberate ones.*
|
|
1612
|
+
- **D84 — `limit` is a cap on the return, never a hint (2026-10-07).** Correction to D81's cut rule, raised by the review of #52. D81 widened the merged alias pool to `[1.5K, 2K]` and passed that widened value to `resolveVisible` as the slice — so when an alias round fired, `search()` returned up to **2× `limit`** rows to every caller (`memory_search`, the MCP tool, `cli search`), with nothing in `--explain` saying so. The pool stays wider than `limit` — each round already over-fetches by `fetchLimit`, and the starvation D79's slice-to-K caused is real and still prevented — but the *return* is cut to `limit`. (D81's `1.5K` line was also dead: `max(min(pool,2K), min(pool,1.5K))` is always `min(pool,2K)`, so the documented band never existed; it is gone.) No ranking change — the merge sorts before any cut, so top-k is byte-identical; `scripts/retrieval-eval.ts` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857. *Rationale: `limit` is a contract on what a caller receives. Headroom belongs inside the ranking, not in the result set — an agent paying 2× the tool-result tokens on alias-touching queries has no way to see that it happened.*
|
|
1613
|
+
- **D86 — the `expansions` BM25 weight, set by the fixture (2026-10-07).** Closes the two D83 clauses that were written into the decision log but never landed in code (`00c4d2d` shipped `bm25(memories_fts)` with no weight vector, and left `--explain` reporting only `fts:` / `candidates:` / `alias rounds:`). (1) **Weight vector** — `bm25(memories_fts, …)` now takes an explicit `[content, tags, aliases, expansions, type, cjk]`, with `expansions` at **0.75** (`src/retrieve/weights.ts`, kept out of `search.ts` so `smoke-pure.ts` keeps its no-sqlite guarantee). The measurement that sets it: on equal-length docs at weight 1.0, a direct body match and an expansion-only match score **identically** (`-3.983` vs `-3.983`) — the preference was being decided by tiebreak, not by the ranking function. Any weight < 1.0 breaks that tie toward the body. `0.75` is the *highest* weight the checked-in fixture cannot distinguish from 1.0 (sweep: 1.0 / 0.9 / 0.75 all yield R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 already costs MRR; below 0.35 R@5 breaks), so the ordering guarantee is free and the fixture remains the gate — retune by re-running the sweep, never by intuition. `SearchHit.matchedVia` (`body` | `alias` | `expansion` | `other`) is computed from the same per-column highlights the attribution already used. (2) **`--explain` attribution** — prints `matched via: body=N alias=N expansion=N other=N` alongside the active weights, so a surprising rank is traceable to the column that caused it. *Rationale: D83 promised the split would make the two columns distinguishable; on equal-weight columns they were not distinguishable at all, so the split bought provenance without buying ranking. This is the missing half.* **Explicit non-goal:** this does *not* repair D80's `how do we ship` rank-8 case — that is per-column length norm (one variant among many in a long `expansions` field), not a weighting problem. Fixing that means narrowing *what* lands in the column, not re-weighting it. Fixture unchanged: R@1 0.750 / R@5 0.977 / MRR 0.857.
|
|
1614
|
+
- **D85 — the alias registry is indexed, not cached (2026-10-07).** From the same review (#55). D81's `loadAliasEntries` runs on **every** `search()` call over `alias <> '' AND target <> ''` — a full table scan on a corpus that may be tens of thousands. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`. Two deliberate choices: (1) **no `SCHEMA_VERSION` bump** — `TABLE_SCHEMA` executes on every db open, not only on rebuild, so `CREATE INDEX IF NOT EXISTS` is idempotent and existing installs pick it up on next open with no rebuild; this also keeps the bump free for D83's 8 → 9 schema change. (2) **No result cache** — the registry must be re-read per query so a newly saved alias memory applies on the very next query (D81's freshness property). An index accelerates the lookup without caching results; those two concerns are separable and this resolves only the first. Measured at 5k memories: 971ms → 39ms per 2000 scoped lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Note the *unscoped* variant of the query still prefers `idx_memories_status`; `search()` always passes `scopeKeys`, so the partial index is what ships. *Rationale: the alias table is tiny by construction — one row per defined nickname per scope — so its lookup cost must be proportional to aliases defined, not to store size.*
|
|
1615
|
+
|
|
1608
1616
|
## Open Questions
|
|
1609
1617
|
|
|
1610
1618
|
_Pruned 2026-10-04 (D75): entries decided in D56/D57/D61/D68 and the shipped layers of the retrieval-robustness question were removed — the decisions are the record. What remains below is genuinely open._
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "open-memex",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"mcpName": "io.github.stoneskin/open-memex",
|
|
5
5
|
"description": "Local-first memory layer and protocol for AI coding agents. Markdown source of truth, SQLite FTS5 index, zero cloud.",
|
|
6
6
|
"keywords": [
|
package/scripts/smoke-mcp.ts
CHANGED
|
@@ -133,6 +133,18 @@ try {
|
|
|
133
133
|
const listBoth = await req("tools/call", { name: "memory_list", arguments: { scope: "both" } });
|
|
134
134
|
const listBothText = listBoth.result?.content?.[0]?.text ?? "";
|
|
135
135
|
check("memory_list both scopes (D68)", listBothText.includes("About you"));
|
|
136
|
+
// D82: closed type taxonomy — a misspelled type is rejected at the schema,
|
|
137
|
+
// never silently empty.
|
|
138
|
+
const badType = await req("tools/call", {
|
|
139
|
+
name: "memory_search",
|
|
140
|
+
arguments: { query: "mcp e2e probe", type: "decisions" },
|
|
141
|
+
});
|
|
142
|
+
const badTypeText: string = badType.result?.content?.[0]?.text ?? "";
|
|
143
|
+
check(
|
|
144
|
+
"memory_search rejects misspelled type (D82)",
|
|
145
|
+
/-32602/.test(badTypeText) && /Invalid option/.test(badTypeText),
|
|
146
|
+
badTypeText.slice(0, 160),
|
|
147
|
+
);
|
|
136
148
|
// D70: numbers run across the whole listing, so "delete #3" names one memory.
|
|
137
149
|
// Needs a personal entry too, or there is only one section to number.
|
|
138
150
|
const personalAdd = await req("tools/call", {
|
package/scripts/smoke-pure.ts
CHANGED
|
@@ -9,6 +9,7 @@ import { resolveProjectScope, resolveCwdScope, pickScopeRoot, PERSONAL_SCOPE } f
|
|
|
9
9
|
import { cjkIndexText, cjkQueryExpr, hasCjk } from "../src/retrieve/cjk.ts";
|
|
10
10
|
import { toFtsQuery } from "../src/retrieve/query.ts";
|
|
11
11
|
import { expansionVariantsForDoc, MAX_AUTO_VARIANTS } from "../src/retrieve/synonyms.ts";
|
|
12
|
+
import { BM25_WEIGHTS, DEFAULT_EXPANSIONS_WEIGHT } from "../src/retrieve/weights.ts";
|
|
12
13
|
import {
|
|
13
14
|
planAliasExpansions,
|
|
14
15
|
rewriteQueryOnce,
|
|
@@ -949,5 +950,20 @@ console.log("== D81 alias-memory query-time expansion ==");
|
|
|
949
950
|
ok("single round orders by score", single[0].doc.id === "b" && single[0].score === 1, JSON.stringify(single.map((m) => [m.doc.id, m.score])));
|
|
950
951
|
}
|
|
951
952
|
|
|
953
|
+
console.log("== D83 expansions BM25 weight + matchedVia ==");
|
|
954
|
+
{
|
|
955
|
+
// The weight vector must stay in memories_fts declaration order:
|
|
956
|
+
// [content, tags, aliases, expansions, type, cjk]. A mis-ordered vector
|
|
957
|
+
// silently down-weights the wrong column, so pin the shape.
|
|
958
|
+
ok("BM25_WEIGHTS has one entry per fts column", BM25_WEIGHTS.length === 6, JSON.stringify(BM25_WEIGHTS));
|
|
959
|
+
ok("expansions is index 3", BM25_WEIGHTS[3] === DEFAULT_EXPANSIONS_WEIGHT, JSON.stringify(BM25_WEIGHTS));
|
|
960
|
+
// D83: expansions sits strictly below aliases, so a body match outranks an
|
|
961
|
+
// expansion-only match by construction instead of by tiebreak.
|
|
962
|
+
ok("expansions weighted below aliases", BM25_WEIGHTS[3] < BM25_WEIGHTS[2], `${BM25_WEIGHTS[3]} vs ${BM25_WEIGHTS[2]}`);
|
|
963
|
+
ok("expansions keeps non-zero weight (D80 rescue slice must survive)",
|
|
964
|
+
BM25_WEIGHTS[3] > 0, String(BM25_WEIGHTS[3]));
|
|
965
|
+
ok("body stays the strongest column", BM25_WEIGHTS[0] >= BM25_WEIGHTS[3], JSON.stringify(BM25_WEIGHTS));
|
|
966
|
+
}
|
|
967
|
+
|
|
952
968
|
console.log(fails === 0 ? "\nALL PASS" : `\n${fails} FAILURES`);
|
|
953
969
|
process.exit(fails === 0 ? 0 : 1);
|
package/server.json
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
"name": "io.github.stoneskin/open-memex",
|
|
4
4
|
"title": "open-memex",
|
|
5
5
|
"description": "Local-first memory for AI coding agents — Markdown + SQLite FTS5, shared across editors via MCP.",
|
|
6
|
-
"version": "0.8.
|
|
6
|
+
"version": "0.8.1",
|
|
7
7
|
"repository": {
|
|
8
8
|
"url": "https://github.com/stoneskin/open-memex",
|
|
9
9
|
"source": "github"
|
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
{
|
|
14
14
|
"registryType": "npm",
|
|
15
15
|
"identifier": "open-memex",
|
|
16
|
-
"version": "0.8.
|
|
16
|
+
"version": "0.8.1",
|
|
17
17
|
"runtimeHint": "npx",
|
|
18
18
|
"packageArguments": [
|
|
19
19
|
{
|
package/src/cli.ts
CHANGED
|
@@ -10,6 +10,7 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
|
|
|
10
10
|
import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.ts";
|
|
11
11
|
import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.ts";
|
|
12
12
|
import { search, list, countList, hitStateLabel } from "./retrieve/search.ts";
|
|
13
|
+
import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.ts";
|
|
13
14
|
import { formatInventoryLine, truncationNote } from "./retrieve/display.ts";
|
|
14
15
|
import {
|
|
15
16
|
writeMemoryFile,
|
|
@@ -1028,6 +1029,20 @@ async function main() {
|
|
|
1028
1029
|
if (stats.aliasExpansions.length > 0) {
|
|
1029
1030
|
console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
|
|
1030
1031
|
}
|
|
1032
|
+
// D83: which indexed column each hit matched on. `expansion` means the
|
|
1033
|
+
// hit only surfaced through a D80 derived variant, so it ranks below an
|
|
1034
|
+
// equal body match by construction (BM25_WEIGHTS) — this is how a
|
|
1035
|
+
// surprising rank gets traced instead of guessed at.
|
|
1036
|
+
if (hits.length > 0) {
|
|
1037
|
+
const via = hits.reduce<Record<string, number>>((acc, h) => {
|
|
1038
|
+
acc[h.matchedVia] = (acc[h.matchedVia] ?? 0) + 1;
|
|
1039
|
+
return acc;
|
|
1040
|
+
}, {});
|
|
1041
|
+
console.log(
|
|
1042
|
+
`matched via: ${Object.entries(via).map(([k, n]) => `${k}=${n}`).join(", ")}` +
|
|
1043
|
+
` (bm25 weights: content=1 tags=1 aliases=1 expansions=${DEFAULT_EXPANSIONS_WEIGHT} type=1 cjk=1)`,
|
|
1044
|
+
);
|
|
1045
|
+
}
|
|
1031
1046
|
}
|
|
1032
1047
|
if (hits.length === 0) {
|
|
1033
1048
|
console.log("(no matches)");
|
package/src/retrieve/search.ts
CHANGED
|
@@ -1,19 +1,28 @@
|
|
|
1
1
|
import { db } from "../store/db.ts";
|
|
2
2
|
import { toFtsQuery } from "./query.ts";
|
|
3
|
+
import { BM25_WEIGHTS } from "./weights.ts";
|
|
3
4
|
import {
|
|
4
5
|
loadAliasEntries,
|
|
5
6
|
planAliasExpansions,
|
|
6
7
|
mergeRounds,
|
|
7
8
|
} from "./alias-memory.ts";
|
|
8
9
|
|
|
10
|
+
/** D83: which indexed column a hit's match came from. */
|
|
11
|
+
export type MatchedVia = "body" | "alias" | "expansion" | "other";
|
|
12
|
+
|
|
9
13
|
export interface SearchHit {
|
|
10
14
|
id: string;
|
|
11
15
|
scope_key: string;
|
|
12
16
|
project_name: string;
|
|
13
17
|
type: string;
|
|
14
18
|
tags: string[];
|
|
15
|
-
/** D61: capture-time aliases stored on this memory (search hits only).
|
|
19
|
+
/** D61: capture-time aliases stored on this memory (search hits only).
|
|
20
|
+
* D83: user/authored only — D80 derived variants live in `expansions`. */
|
|
16
21
|
aliases: string[];
|
|
22
|
+
/** D83: which FTS column produced this hit's match — `body`, `alias`,
|
|
23
|
+
* `expansion`, or `other` (tags/type/cjk). Surfaces in `search --explain`
|
|
24
|
+
* so a surprising rank can be traced to the column that caused it. */
|
|
25
|
+
matchedVia: MatchedVia;
|
|
17
26
|
snippet: string;
|
|
18
27
|
score: number;
|
|
19
28
|
updated_at: number;
|
|
@@ -31,6 +40,11 @@ interface RawRow {
|
|
|
31
40
|
type: string;
|
|
32
41
|
tags: string;
|
|
33
42
|
aliases?: string;
|
|
43
|
+
/** D83: D80 index-time synonym/translation variants — derived, never user-authored. */
|
|
44
|
+
expansions?: string;
|
|
45
|
+
/** D83: per-column FTS highlight, so attribution knows which column matched. */
|
|
46
|
+
alias_hl?: string;
|
|
47
|
+
exp_hl?: string;
|
|
34
48
|
updated_at: number;
|
|
35
49
|
snippet: string;
|
|
36
50
|
score: number;
|
|
@@ -44,10 +58,34 @@ interface RawRow {
|
|
|
44
58
|
function toHit(r: RawRow): SearchHit {
|
|
45
59
|
const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
|
|
46
60
|
let snippet = r.snippet ?? "";
|
|
47
|
-
//
|
|
48
|
-
//
|
|
49
|
-
// hit otherwise looks unrelated to
|
|
50
|
-
|
|
61
|
+
// D83: attribution is per-column now. The aliases column holds only
|
|
62
|
+
// user/authored phrasings (D61); the expansions column holds D80 derived
|
|
63
|
+
// variants. Say which one matched — the hit otherwise looks unrelated to
|
|
64
|
+
// the query when the body shows no highlight.
|
|
65
|
+
const hlTerms = (s: string | undefined): string[] => {
|
|
66
|
+
if (!s) return [];
|
|
67
|
+
const m = s.match(/\[([^\]]+)\]/g);
|
|
68
|
+
return m ? [...new Set(m.map((x) => x.slice(1, -1)))] : [];
|
|
69
|
+
};
|
|
70
|
+
const aliasTerms = hlTerms(r.alias_hl);
|
|
71
|
+
const expTerms = hlTerms(r.exp_hl);
|
|
72
|
+
// D83: attribute the hit to the column that matched, so `--explain` can
|
|
73
|
+
// trace an unexpected rank. Order matters: a body highlight is the most
|
|
74
|
+
// direct evidence, then an authored alias, then a derived expansion.
|
|
75
|
+
const matchedVia: MatchedVia = /\[[^\]]+\]/.test(snippet)
|
|
76
|
+
? "body"
|
|
77
|
+
: aliasTerms.length > 0
|
|
78
|
+
? "alias"
|
|
79
|
+
: expTerms.length > 0
|
|
80
|
+
? "expansion"
|
|
81
|
+
: "other";
|
|
82
|
+
if (aliasTerms.length > 0 && aliases.length > 0) {
|
|
83
|
+
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
84
|
+
} else if (expTerms.length > 0) {
|
|
85
|
+
snippet += ` (synonym: ${expTerms.join(", ")})`;
|
|
86
|
+
} else if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
|
|
87
|
+
// Match came through another column (tags/type/cjk) — the authored
|
|
88
|
+
// aliases are still honest context for the hit.
|
|
51
89
|
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
52
90
|
}
|
|
53
91
|
return {
|
|
@@ -57,6 +95,7 @@ function toHit(r: RawRow): SearchHit {
|
|
|
57
95
|
type: r.type,
|
|
58
96
|
tags: r.tags ? r.tags.split(",").filter(Boolean) : [],
|
|
59
97
|
aliases,
|
|
98
|
+
matchedVia,
|
|
60
99
|
snippet,
|
|
61
100
|
score: r.score,
|
|
62
101
|
updated_at: r.updated_at,
|
|
@@ -87,6 +126,9 @@ export function hitStateLabel(h: Pick<SearchHit, "scope_key" | "review_state">):
|
|
|
87
126
|
return ` [${h.review_state || "draft"}]`;
|
|
88
127
|
}
|
|
89
128
|
|
|
129
|
+
// D83: BM25_WEIGHTS / DEFAULT_EXPANSIONS_WEIGHT live in ./weights.ts, not
|
|
130
|
+
// here — smoke-pure.ts guards them and must stay free of any sqlite import.
|
|
131
|
+
|
|
90
132
|
/** Optional diagnostics out-param for `search --explain` (D57). */
|
|
91
133
|
export interface SearchStats {
|
|
92
134
|
ftsQuery: string;
|
|
@@ -174,6 +216,12 @@ export function search(
|
|
|
174
216
|
stopwords?: boolean;
|
|
175
217
|
orJoin?: boolean;
|
|
176
218
|
prefix?: boolean;
|
|
219
|
+
/**
|
|
220
|
+
* D83: override the BM25 weight of the `expansions` column. Omitted =
|
|
221
|
+
* `DEFAULT_EXPANSIONS_WEIGHT`. An ablation knob — it exists so the
|
|
222
|
+
* fixture can pick the value; not a per-call tuning surface.
|
|
223
|
+
*/
|
|
224
|
+
expansionsWeight?: number;
|
|
177
225
|
} = {},
|
|
178
226
|
): SearchHit[] {
|
|
179
227
|
const qflags = { stopwords: opts.stopwords, orJoin: opts.orJoin, prefix: opts.prefix };
|
|
@@ -198,10 +246,12 @@ export function search(
|
|
|
198
246
|
const typeFilter = opts.type ? ` AND m.type = ?` : "";
|
|
199
247
|
|
|
200
248
|
const sql = `
|
|
201
|
-
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.updated_at,
|
|
249
|
+
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.expansions, m.updated_at,
|
|
202
250
|
m.status, m.superseded_by, m.review_state, m.source, m.created_at,
|
|
203
251
|
snippet(memories_fts, 0, '[', ']', ' ... ', 12) AS snippet,
|
|
204
|
-
|
|
252
|
+
highlight(memories_fts, 2, '[', ']') AS alias_hl,
|
|
253
|
+
highlight(memories_fts, 3, '[', ']') AS exp_hl,
|
|
254
|
+
bm25(memories_fts, ?, ?, ?, ?, ?, ?) AS score
|
|
205
255
|
FROM memories_fts
|
|
206
256
|
JOIN memories m ON m.rowid = memories_fts.rowid
|
|
207
257
|
WHERE memories_fts MATCH ?${scopeFilter}${typeFilter}
|
|
@@ -214,6 +264,10 @@ export function search(
|
|
|
214
264
|
project_name: string;
|
|
215
265
|
type: string;
|
|
216
266
|
tags: string;
|
|
267
|
+
aliases?: string;
|
|
268
|
+
expansions?: string;
|
|
269
|
+
alias_hl?: string;
|
|
270
|
+
exp_hl?: string;
|
|
217
271
|
updated_at: number;
|
|
218
272
|
status: string;
|
|
219
273
|
superseded_by: string | null;
|
|
@@ -223,8 +277,15 @@ export function search(
|
|
|
223
277
|
snippet: string;
|
|
224
278
|
score: number;
|
|
225
279
|
};
|
|
280
|
+
// D83: the weight vector is bound per statement, so it leads the parameter
|
|
281
|
+
// list (it appears in the SELECT list, before the WHERE's MATCH).
|
|
282
|
+
const weights =
|
|
283
|
+
opts.expansionsWeight === undefined
|
|
284
|
+
? BM25_WEIGHTS
|
|
285
|
+
: BM25_WEIGHTS.map((w, i) => (i === 3 ? opts.expansionsWeight! : w));
|
|
226
286
|
const fetchFts = (matchExpr: string): RawRow[] => {
|
|
227
|
-
const params: unknown[] = [
|
|
287
|
+
const params: unknown[] = [...weights];
|
|
288
|
+
params.push(matchExpr);
|
|
228
289
|
if (opts.scopeKeys && opts.scopeKeys.length > 0) params.push(...opts.scopeKeys);
|
|
229
290
|
if (opts.type) params.push(opts.type);
|
|
230
291
|
params.push(fetchLimit);
|
|
@@ -244,7 +305,6 @@ export function search(
|
|
|
244
305
|
const aliasEntries = loadAliasEntries(opts.scopeKeys);
|
|
245
306
|
const expansions = planAliasExpansions(query, aliasEntries);
|
|
246
307
|
let merged: RawRow[] = raw;
|
|
247
|
-
let finalLimit = limit;
|
|
248
308
|
if (expansions.length > 0) {
|
|
249
309
|
const rounds: RawRow[][] = [raw];
|
|
250
310
|
for (const eq of expansions) {
|
|
@@ -255,20 +315,21 @@ export function search(
|
|
|
255
315
|
// divided by that round's best, so an alias-side champion competes
|
|
256
316
|
// evenly with a literal-side champion — no down-weighting for being
|
|
257
317
|
// from a later round.
|
|
318
|
+
//
|
|
319
|
+
// The merged pool is deliberately wider than `limit` (each round already
|
|
320
|
+
// over-fetches by fetchLimit) so round-2 hits are never starved the way
|
|
321
|
+
// D79's slice-to-K starved them. #53: that headroom is for RANKING only —
|
|
322
|
+
// the return is still cut to the caller's `limit`, since `limit` is a cap
|
|
323
|
+
// on what every caller (tools, MCP, cli) receives, not a hint.
|
|
258
324
|
merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
|
|
259
|
-
// Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
|
|
260
|
-
// starved the way D79's slice-to-K starved them.
|
|
261
|
-
const pool = merged.length;
|
|
262
|
-
finalLimit = Math.min(pool, 2 * limit);
|
|
263
|
-
finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
|
|
264
325
|
if (opts.stats) opts.stats.aliasExpansions = expansions;
|
|
265
326
|
}
|
|
266
327
|
|
|
267
328
|
// D80: the curated static map is solved at index time (synonyms.ts
|
|
268
|
-
// expands variants into the aliases column at sync time), so
|
|
269
|
-
// user alias the query side stays a single FTS round.
|
|
329
|
+
// expands variants into the indexed aliases column at sync time), so
|
|
330
|
+
// without a user alias the query side stays a single FTS round.
|
|
270
331
|
if (opts.stats) opts.stats.candidates = merged.length;
|
|
271
|
-
return resolveVisible(merged,
|
|
332
|
+
return resolveVisible(merged, limit, opts.stats);
|
|
272
333
|
}
|
|
273
334
|
|
|
274
335
|
export function list(
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D83: BM25 column weights for the FTS index.
|
|
3
|
+
*
|
|
4
|
+
* Split out of `search.ts` deliberately: `smoke-pure.ts` must keep running
|
|
5
|
+
* with no SQLite dependency (its header says "pure-logic, no
|
|
6
|
+
* bun:sqlite"), and importing `search.ts` pulls in `store/db.ts` at module
|
|
7
|
+
* load. This module holds the tunable constant and nothing else, so the
|
|
8
|
+
* guard test can import it without opening a database.
|
|
9
|
+
*
|
|
10
|
+
* `expansions` (D80-derived synonym/translation variants) sits strictly below
|
|
11
|
+
* `aliases` so the preference is explicit rather than incidental: a body
|
|
12
|
+
* match outranks an expansion-only match, and a user-authored alias outranks
|
|
13
|
+
* a derived one. This is not cosmetic — measured on equal-length docs, at
|
|
14
|
+
* weight 1.0 a direct body match and an expansion-only match score
|
|
15
|
+
* *identically* (-3.983 vs -3.983), leaving the order to tiebreaks. Any
|
|
16
|
+
* weight < 1.0 breaks that tie in the right direction.
|
|
17
|
+
*
|
|
18
|
+
* The value is the highest one the checked-in fixture cannot distinguish from
|
|
19
|
+
* 1.0 (`scripts/retrieval-eval.ts`: R@1 0.750 / R@5 0.977 / MRR 0.857 hold
|
|
20
|
+
* from 1.0 down to 0.75; 0.5 already costs MRR, and below 0.35 R@5 breaks).
|
|
21
|
+
* So the ordering guarantee is free, and the fixture is the gate that keeps
|
|
22
|
+
* it honest — retune by re-running the sweep, not by feel.
|
|
23
|
+
*
|
|
24
|
+
* Known non-fix: this does NOT repair D80's `how do we ship` rank-8 case.
|
|
25
|
+
* That is per-column length norm (one variant among many in a long
|
|
26
|
+
* `expansions` field), not a weighting problem; see D83's update.
|
|
27
|
+
*/
|
|
28
|
+
|
|
29
|
+
/** Weight for the D80 `expansions` column. */
|
|
30
|
+
export const DEFAULT_EXPANSIONS_WEIGHT = 0.75;
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* Full weight vector for `memories_fts`, in declaration order.
|
|
34
|
+
* MUST stay in sync with FTS_SCHEMA column order in `store/db.ts`:
|
|
35
|
+
* `[content, tags, aliases, expansions, type, cjk]`.
|
|
36
|
+
*/
|
|
37
|
+
export const BM25_WEIGHTS: readonly number[] = [1, 1, 1, DEFAULT_EXPANSIONS_WEIGHT, 1, 1];
|
package/src/store/db.ts
CHANGED
|
@@ -48,6 +48,7 @@ CREATE TABLE IF NOT EXISTS memories (
|
|
|
48
48
|
status TEXT NOT NULL DEFAULT 'active',
|
|
49
49
|
tags TEXT NOT NULL DEFAULT '',
|
|
50
50
|
aliases TEXT NOT NULL DEFAULT '',
|
|
51
|
+
expansions TEXT NOT NULL DEFAULT '',
|
|
51
52
|
alias TEXT NOT NULL DEFAULT '',
|
|
52
53
|
target TEXT NOT NULL DEFAULT '',
|
|
53
54
|
content TEXT NOT NULL,
|
|
@@ -66,6 +67,19 @@ CREATE INDEX IF NOT EXISTS idx_memories_scope_updated
|
|
|
66
67
|
ON memories(scope_key, updated_at DESC);
|
|
67
68
|
CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
|
|
68
69
|
CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
|
|
70
|
+
-- D81: the alias registry (retrieve/alias-memory.ts) runs on every search().
|
|
71
|
+
-- The predicate is a full table scan without this — the rows carrying a
|
|
72
|
+
-- defined alias pair are a tiny subset of a corpus that may be tens of
|
|
73
|
+
-- thousands. A partial index keeps the lookup proportional to the number of
|
|
74
|
+
-- aliases defined, not the size of the store.
|
|
75
|
+
--
|
|
76
|
+
-- No SCHEMA_VERSION bump: TABLE_SCHEMA runs on every db open (see db()), so
|
|
77
|
+
-- existing installs pick this up on next open. Deliberately NOT a result
|
|
78
|
+
-- cache — the alias registry must be re-read per query so a newly saved
|
|
79
|
+
-- alias memory applies on the very next query. This index accelerates the
|
|
80
|
+
-- lookup without caching results.
|
|
81
|
+
CREATE INDEX IF NOT EXISTS idx_memories_alias
|
|
82
|
+
ON memories(scope_key) WHERE alias <> '' AND target <> '';
|
|
69
83
|
`;
|
|
70
84
|
|
|
71
85
|
// The `cjk` column holds pre-tokenized CJK unigrams+bigrams (see
|
|
@@ -78,6 +92,7 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
|
|
|
78
92
|
content,
|
|
79
93
|
tags,
|
|
80
94
|
aliases,
|
|
95
|
+
expansions,
|
|
81
96
|
type,
|
|
82
97
|
cjk,
|
|
83
98
|
scope_key UNINDEXED,
|
|
@@ -87,25 +102,25 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
|
|
|
87
102
|
);
|
|
88
103
|
|
|
89
104
|
CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
|
|
90
|
-
INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
|
|
91
|
-
VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
|
|
105
|
+
INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
106
|
+
VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
|
|
92
107
|
END;
|
|
93
108
|
|
|
94
109
|
CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
|
|
95
|
-
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
|
|
96
|
-
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
|
|
110
|
+
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
111
|
+
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
|
|
97
112
|
END;
|
|
98
113
|
|
|
99
114
|
CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
|
|
100
|
-
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
|
|
101
|
-
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
|
|
102
|
-
INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
|
|
103
|
-
VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
|
|
115
|
+
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
116
|
+
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
|
|
117
|
+
INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
118
|
+
VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
|
|
104
119
|
END;
|
|
105
120
|
`;
|
|
106
121
|
|
|
107
122
|
/** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
|
|
108
|
-
const SCHEMA_VERSION =
|
|
123
|
+
const SCHEMA_VERSION = 9;
|
|
109
124
|
|
|
110
125
|
function userVersion(d: AnyDatabase): number {
|
|
111
126
|
const row = d.prepare("PRAGMA user_version").get() as { user_version: number };
|
package/src/store/sync.ts
CHANGED
|
@@ -15,8 +15,8 @@ import { projectRoot, paths } from "../paths.ts";
|
|
|
15
15
|
import { loadConfig } from "../config.ts";
|
|
16
16
|
|
|
17
17
|
const UPSERT_SQL = `
|
|
18
|
-
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
19
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
18
|
+
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, expansions, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
19
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
20
20
|
ON CONFLICT(id) DO UPDATE SET
|
|
21
21
|
scope_key = excluded.scope_key,
|
|
22
22
|
scope = excluded.scope,
|
|
@@ -28,6 +28,7 @@ const UPSERT_SQL = `
|
|
|
28
28
|
status = excluded.status,
|
|
29
29
|
tags = excluded.tags,
|
|
30
30
|
aliases = excluded.aliases,
|
|
31
|
+
expansions = excluded.expansions,
|
|
31
32
|
alias = excluded.alias,
|
|
32
33
|
target = excluded.target,
|
|
33
34
|
content = excluded.content,
|
|
@@ -66,16 +67,20 @@ export function upsertFromFile(mf: MemoryFile): void {
|
|
|
66
67
|
function writeRow(mf: MemoryFile, mtimeMs: number): void {
|
|
67
68
|
const { fm, body, filePath } = mf;
|
|
68
69
|
const tags = (fm.tags ?? []).join(",");
|
|
69
|
-
// D80 index-time expansion: curated synonym/translation variants for
|
|
70
|
-
// memory's keywords are indexed in
|
|
71
|
-
//
|
|
72
|
-
//
|
|
70
|
+
// D80 index-time expansion, D83: curated synonym/translation variants for
|
|
71
|
+
// the memory's keywords are indexed in their own `expansions` column —
|
|
72
|
+
// never mixed into `aliases`, which holds only user/authored alternate
|
|
73
|
+
// phrasings (so `(aka: …)` attribution stays honest). The CJK bigram
|
|
74
|
+
// column still covers expansions, so CJK variants stay substring-
|
|
75
|
+
// searchable. Derived at sync time — the Markdown file stays the source
|
|
76
|
+
// of truth.
|
|
73
77
|
const supplied = fm.aliases ?? [];
|
|
74
78
|
const have = new Set(supplied.map((a) => a.toLowerCase()));
|
|
75
79
|
const auto = expansionVariantsForDoc(body + "\n" + tags).filter(
|
|
76
80
|
(a) => !have.has(a.toLowerCase()),
|
|
77
81
|
);
|
|
78
|
-
const aliases =
|
|
82
|
+
const aliases = supplied.join(",");
|
|
83
|
+
const expansions = auto.join(",");
|
|
79
84
|
// D81: user-defined alias vocabulary (single nickname → referent pair).
|
|
80
85
|
db().prepare(UPSERT_SQL).run(
|
|
81
86
|
fm.id,
|
|
@@ -89,10 +94,11 @@ function writeRow(mf: MemoryFile, mtimeMs: number): void {
|
|
|
89
94
|
fm.status,
|
|
90
95
|
tags,
|
|
91
96
|
aliases,
|
|
97
|
+
expansions,
|
|
92
98
|
fm.alias ?? "",
|
|
93
99
|
fm.target ?? "",
|
|
94
100
|
body,
|
|
95
|
-
cjkIndexText(body + "\n" + tags + "\n" + aliases),
|
|
101
|
+
cjkIndexText(body + "\n" + tags + "\n" + aliases + "\n" + expansions),
|
|
96
102
|
contentHash(body),
|
|
97
103
|
fm.superseded_by ?? null,
|
|
98
104
|
fm.source ?? "",
|
package/src/tools/ops.ts
CHANGED
|
@@ -143,7 +143,12 @@ export const memorySearchArgs = {
|
|
|
143
143
|
.enum(["project", "personal", "both"])
|
|
144
144
|
.optional()
|
|
145
145
|
.describe("Which scope(s) to search. Default: both."),
|
|
146
|
-
type: z
|
|
146
|
+
type: z
|
|
147
|
+
.enum(MEMORY_TYPE_TAXONOMY)
|
|
148
|
+
.optional()
|
|
149
|
+
.describe(
|
|
150
|
+
"Restrict to memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty.",
|
|
151
|
+
),
|
|
147
152
|
limit: z.number().int().min(1).max(50).optional().describe("Max results to return (default 8)."),
|
|
148
153
|
};
|
|
149
154
|
export type MemorySearchArgs = z.infer<z.ZodObject<typeof memorySearchArgs>>;
|
|
@@ -155,7 +160,12 @@ export const memoryListArgs = {
|
|
|
155
160
|
.describe(
|
|
156
161
|
"Which scope(s) to list. `project` = tied to this repo. `personal` = global across all your projects. `both` lists each scope's newest. Default: project.",
|
|
157
162
|
),
|
|
158
|
-
type: z
|
|
163
|
+
type: z
|
|
164
|
+
.enum(MEMORY_TYPE_TAXONOMY)
|
|
165
|
+
.optional()
|
|
166
|
+
.describe(
|
|
167
|
+
"Only list memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty.",
|
|
168
|
+
),
|
|
159
169
|
limit: z.number().int().min(1).max(100).optional().describe("Max entries to list, newest first (default 20)."),
|
|
160
170
|
include: z
|
|
161
171
|
.enum(["active", "all"])
|