open-memex 0.8.0 → 0.8.2-alpha.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +4 -0
- package/CHANGELOG.md +12 -0
- package/README.md +21 -8
- package/README.zh-CN.md +9 -2
- package/dist/.build-stamp +1 -1
- package/dist/cli.js +13 -0
- package/dist/plugin-core.js +1 -1
- package/dist/retrieve/search.js +53 -16
- package/dist/retrieve/weights.js +35 -0
- package/dist/store/db.js +42 -9
- package/dist/store/sync.js +13 -8
- package/dist/tools/ops.js +8 -2
- package/docs/V2-DESIGN.md +15 -1
- package/docs/writing-for-memory.md +94 -0
- package/package.json +1 -1
- package/scripts/smoke-mcp.ts +12 -0
- package/scripts/smoke-pure.ts +16 -0
- package/scripts/test-full.ts +49 -0
- package/server.json +2 -2
- package/src/cli.ts +15 -0
- package/src/plugin-core.ts +1 -1
- package/src/retrieve/search.ts +78 -17
- package/src/retrieve/weights.ts +37 -0
- package/src/store/db.ts +41 -9
- package/src/store/sync.ts +14 -8
- package/src/tools/ops.ts +12 -2
package/AGENTS.md
CHANGED
|
@@ -205,6 +205,10 @@ breaking v1→v2 transition) shipped with 0.3.0. Full rules: `CONTRIBUTING.md`.
|
|
|
205
205
|
new commands → README CLI sections + this file's Commands. A change without its docs
|
|
206
206
|
is not done. Anything deferred as "later" goes into the near-term backlog in
|
|
207
207
|
`docs/V2-DESIGN.md` §18 the same day — unwritten later means never.
|
|
208
|
+
- **Measurement before mechanism.** A retrieval change lands its fixture +
|
|
209
|
+
baseline number in a commit with no retrieval change at all, so the delta is
|
|
210
|
+
unambiguous. `scripts/retrieval-eval.ts` is the gate: no retrieval PR merges
|
|
211
|
+
without before/after recall@k + MRR on the checked-in fixture.
|
|
208
212
|
- **Version bumps ship with features.** `package.json` + `package-lock.json` carry the
|
|
209
213
|
in-development version. A new alpha line starts with a minor bump after the
|
|
210
214
|
previous minor shipped (`0.3.0` → `0.4.0-alpha.1`); while that minor is still
|
package/CHANGELOG.md
CHANGED
|
@@ -9,6 +9,18 @@ the `@alpha` tag are development snapshots and are not listed individually.
|
|
|
9
9
|
|
|
10
10
|
## [Unreleased]
|
|
11
11
|
|
|
12
|
+
- fix: the legacy-scope warning on plugin load now suggests the migrate command with an explicit `--to <current-scope>`. Without it, `--to` defaults to the scope of the directory the command runs from, so running the suggested command from the open-memex checkout would move the memories into the wrong scope. No behavior change: the warning condition and the migrate contract are untouched.
|
|
13
|
+
- fix: a pre-D81 index store (SCHEMA_VERSION 7, ≤0.7.x) no longer wedges every open with `no such column: alias` (D87). D85's partial index references the `alias`/`target` columns D81 added, and `CREATE TABLE IF NOT EXISTS` adds no columns to an existing table — so the index creation threw inside `db()` before the version-check rebuild could run. `db()` now rebuilds a stale table *before* creating the schema; the Markdown files are the source of truth, so the rebuild is lossless. Regression test in `scripts/test-full.ts` §9 builds a faithful v7 store and asserts the open succeeds and the schema lands at version 9.
|
|
14
|
+
|
|
15
|
+
## [0.8.1] - 2026-10-07
|
|
16
|
+
|
|
17
|
+
- D83 follow-up: the two D83 clauses that shipped as design but not as code. (1) **BM25 weight for the `expansions` column** — `bm25(memories_fts, …)` now takes an explicit weight vector (`src/retrieve/weights.ts`, order `[content, tags, aliases, expansions, type, cjk]`) instead of the implicit all-1.0. `expansions` is `0.75`: measured on equal-length docs, a direct body match and an expansion-only match score *identically* at weight 1.0 (`-3.983` vs `-3.983`), leaving the order to tiebreaks — any weight below 1.0 breaks that tie in the right direction. `0.75` is the highest value the fixture cannot distinguish from 1.0 (sweep: 1.0/0.9/0.75 all give R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 costs MRR, below 0.35 R@5 breaks). Exposed as `opts.expansionsWeight` so the sweep stays measurable; `smoke-pure.ts` pins the vector's shape and ordering. (2) **`search --explain` per-column attribution** — reports `matched via: body=N alias=N expansion=N other=N` plus the active weights. Does **not** fix the D80 `how do we ship` rank-8 case: that is per-column length norm, not weighting. Fixture unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857.
|
|
18
|
+
|
|
19
|
+
- fix: `search()` honours the caller's `limit` again (#53). When a D81 alias round fired, the widened candidate pool leaked out through the return value, so `memory_search` / MCP `memory_search` / `cli search` returned up to **2× `limit`** hits with no signal it happened. The pool is still wider than `limit` for *ranking* (round-2 hits are not starved the way D79's slice-to-K starved them); only the cut changed, to `limit`. Drops the dead `1.5K` line, which could never raise the value. No ranking change — `scripts/retrieval-eval.ts` is unchanged at recall@1 0.750 / recall@5 0.977 / MRR 0.857.
|
|
20
|
+
- perf: partial index on the alias registry (#55): `loadAliasEntries` ran `WHERE alias <> '' AND target <> ''` as a full table scan on **every** search. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`, which runs on every db open — no `SCHEMA_VERSION` bump, no rebuild, existing installs pick it up on next open. Measured at 5k memories: 971ms → 39ms per 2000 lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Deliberately not a result cache: a newly saved alias must still apply on the very next query.
|
|
21
|
+
- D83 (implemented, for #56): D80 index-time expansion variants moved out of the `aliases` column into a dedicated `expansions` index column (own FTS column + triggers; `SearchHit.aliases` and `(aka: …)` surface only user-authored aliases; expansion-only matches report `(synonym: …)`; `--explain` shows the same via the snippet). Schema v8 → v9 with rebuild-from-Markdown migration (verified on a scratch DB). `retrieval-eval` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857 — the `how do we ship` rank stayed 8: expansion-column length-norm dilution is inherent, not caused by the mixing.
|
|
22
|
+
- `memory_search` / `memory_list` `type` is now a closed enum (D82, closes #51): a misspelled type is rejected at the schema (MCP -32602) instead of silently returning zero results. The opencode plugin shares the same args objects, so both surfaces tighten together; `scripts/smoke-mcp.ts` guards the rejection.
|
|
23
|
+
|
|
12
24
|
## [0.8.0] - 2026-10-07
|
|
13
25
|
|
|
14
26
|
- User-defined alias vocabulary as query-time expansion (D81): any memory type may now carry `alias:` (nickname, e.g. 香蕉计划) and `target:` (referent, e.g. 支付系统重构项目) frontmatter — an alias memory defines vocabulary once per scope. At query time, query terms hit the alias registry (exact match, one cheap SELECT per search) and the query is rewritten with the other side of the relation — bidirectional, depth < 3, cycle-safe — then re-run as round 2/3. Rounds merge by normalized-max (`mergeRounds` in `src/retrieve/alias-memory.ts`): each round's BM25 scores are divided by that round's best, so an alias-side champion competes evenly with a literal-side champion; the merged pool is re-ranked and cut to [1.5K, 2K], never back to K. No reindex when an alias changes. Creation: `memory_add` alias/target params, CLI `add --alias/--target`, or frontmatter edit (v1: user-explicit only). Fixture grows to 54 memories / 44 queries (3 bidirectional alias queries): recall@1 0.750, recall@5 **0.977**, MRR 0.857. Also removes the init `captureAliases` and keyword-capture questions — both are now always on (Stone 2026-10-07).
|
package/README.md
CHANGED
|
@@ -89,7 +89,7 @@ open-memex doctor
|
|
|
89
89
|
|
|
90
90
|
## Core concepts
|
|
91
91
|
|
|
92
|
-
|
|
92
|
+
Four ideas explain almost everything open-memex does.
|
|
93
93
|
|
|
94
94
|

|
|
95
95
|
|
|
@@ -118,6 +118,15 @@ rebuilds from the files at any time (`open-memex reindex`). Team sharing, when
|
|
|
118
118
|
you want it, goes through the same review flow as code: nothing is shared
|
|
119
119
|
automatically (see [Team workflow](#team-workflow-sharing-memories-through-git)).
|
|
120
120
|
|
|
121
|
+
**4. Knowledge stays current.**
|
|
122
|
+
Facts change: decisions get reversed, tools get replaced, yesterday's gotcha
|
|
123
|
+
gets fixed. open-memex doesn't let the old version quietly linger —
|
|
124
|
+
`memory_supersede` replaces a memory with its newer version and keeps the
|
|
125
|
+
chain, so retrieval serves the current fact while the history stays auditable.
|
|
126
|
+
Merely stale memories can be deprecated, and `open-memex audit` reports
|
|
127
|
+
duplicates, stale entries, and broken chains. Pruning is a feature, not
|
|
128
|
+
housekeeping: shared memory rots without it.
|
|
129
|
+
|
|
121
130
|
## Which editors, which features
|
|
122
131
|
|
|
123
132
|
open-memex talks to editors two ways: a native **opencode plugin**, and a
|
|
@@ -159,6 +168,7 @@ instruction set only made agents chatty.
|
|
|
159
168
|
|---|---|---|---|---|
|
|
160
169
|
| Where it lives | Your machine + your repos | In the repo | Vendor servers | Gone when the chat ends |
|
|
161
170
|
| Who maintains it | Captured as you work; you review | You write and update by hand | The service | — |
|
|
171
|
+
| When facts change | Superseded, not silently stale — retrieval serves the current version, chain preserved | You hand-edit; stale copies linger | Varies; often silently stale | — |
|
|
162
172
|
| Works across AI tools | Yes — any MCP client (same machine) | One file per tool convention | Per-integration | No |
|
|
163
173
|
| Review before sharing | Yes — outbox + pull request | Yes — it's just files | Varies | No |
|
|
164
174
|
| Human-readable | Plain Markdown files | Yes | Dashboard / API | No |
|
|
@@ -410,8 +420,8 @@ holiday policy. Always on — the agent may attach up to 4 per memory, and you
|
|
|
410
420
|
can pass your own via `open-memex add --aliases`.
|
|
411
421
|
|
|
412
422
|
Three ways to add them: `open-memex add "text" --aliases "phrase one; phrase
|
|
413
|
-
two"` (semicolon-separated, up to 4); ask the agent ("remember:
|
|
414
|
-
AI agent, alias it as
|
|
423
|
+
two"` (semicolon-separated, up to 4); ask the agent ("remember: Fluffy is my
|
|
424
|
+
AI agent, alias it as Momo"); or edit the memory's `aliases:` frontmatter
|
|
415
425
|
directly — Markdown is the source of truth and sync picks it up.
|
|
416
426
|
|
|
417
427
|
Separately, a curated synonym/translation map is expanded into the index
|
|
@@ -549,10 +559,10 @@ Three layers, each covering what the others miss: **BM25 keyword search**
|
|
|
549
559
|
**synonym expansion in two halves** — (a) index-time (D80: a curated,
|
|
550
560
|
checked-in map of dev-domain equivalents and EN↔ZH translation pairs
|
|
551
561
|
expanded into the index at write time, so `ship` finds `deploy` and
|
|
552
|
-
|
|
562
|
+
`bug` finds `defect` in a plain single-round query; deterministic, no
|
|
553
563
|
model download); (b) query-time user aliases (D81: an alias memory carrying
|
|
554
564
|
`alias:`/`target:` frontmatter defines your vocabulary once per scope —
|
|
555
|
-
e.g.
|
|
565
|
+
e.g. Banana Plan → payment system refactor project — and a query mentioning either side
|
|
556
566
|
triggers a second round on the other side, both directions, merged evenly
|
|
557
567
|
with round one so alias hits are never down-weighted; no reindex when the
|
|
558
568
|
alias changes); and **semantic vectors**
|
|
@@ -593,11 +603,14 @@ Honest edges, so nothing surprises you:
|
|
|
593
603
|
|
|
594
604
|
- **Keyword search, not semantic.** Retrieval is BM25 keyword matching, plus a
|
|
595
605
|
curated synonym map expanded into the index at write time (`ship` finds
|
|
596
|
-
`deploy`,
|
|
606
|
+
`deploy`, `bug` finds `defect`). Paraphrases outside the curated map
|
|
597
607
|
don't match — that's what the opt-in semantic layer is for. (Plain
|
|
598
608
|
questions are fine — "how do we…" / "请问…" wording is filtered out before
|
|
599
609
|
matching, so asking naturally doesn't dilute the results.) No embedding
|
|
600
|
-
model is ever downloaded without your explicit opt-in.
|
|
610
|
+
model is ever downloaded without your explicit opt-in. (Synonym expansions
|
|
611
|
+
live in their own index column, separate from your own aliases — but a
|
|
612
|
+
query matching only via an expansion can still rank below direct matches;
|
|
613
|
+
that's inherent BM25 length-norm behaviour, not a bug.)
|
|
601
614
|
- **One machine.** Editors on the same machine share memory; there is no
|
|
602
615
|
cross-machine sync. `export` / `import` bundles (below) move memory between
|
|
603
616
|
machines manually.
|
|
@@ -695,7 +708,7 @@ open-memex config # print effective config
|
|
|
695
708
|
open-memex config set <key> <value> # change a setting
|
|
696
709
|
open-memex doctor # environment health check (incl. plugin entry)
|
|
697
710
|
open-memex audit # memory health check (duplicates, stale, broken chains)
|
|
698
|
-
open-memex capture --dry-run "
|
|
711
|
+
open-memex capture --dry-run "remember that I like concise answers" # preview keyword capture
|
|
699
712
|
open-memex mcp --print-config vscode|cursor|claude|opencode|visualstudio
|
|
700
713
|
open-memex --help # this reference
|
|
701
714
|
open-memex <command> --help # help for one command
|
package/README.zh-CN.md
CHANGED
|
@@ -71,7 +71,7 @@ open-memex doctor
|
|
|
71
71
|
|
|
72
72
|
## 核心概念
|
|
73
73
|
|
|
74
|
-
|
|
74
|
+
四个概念能解释 open-memex 几乎所有的事。
|
|
75
75
|
|
|
76
76
|

|
|
77
77
|
|
|
@@ -89,6 +89,12 @@ open-memex doctor
|
|
|
89
89
|
**3. 你的文件,你做主。**
|
|
90
90
|
Markdown 文件是事实来源——打开、编辑、删除、用 grep 搜都行。旁边的 SQLite 索引只是搜索加速器,随时可以从文件重建(`open-memex reindex`)。团队共享(当你需要时)走的是和代码一样的评审流程:没有任何东西会被自动分享(见[团队工作流](#团队工作流通过-git-共享记忆))。
|
|
91
91
|
|
|
92
|
+
**4. 知识保鲜。**
|
|
93
|
+
事实会变:决策会被推翻、工具会被替换、昨天的坑今天可能已经填上。open-memex
|
|
94
|
+
不让旧版本默默留着误导人——`memory_supersede` 用新版替换旧版并保留链条,
|
|
95
|
+
检索永远给现行版本、历史可审计;只是过时的可以 deprecate,`open-memex audit`
|
|
96
|
+
报告重复、过时与断链。修剪是功能,不是家务:共享记忆没人修剪就会烂掉。
|
|
97
|
+
|
|
92
98
|
## 编辑器与功能对照
|
|
93
99
|
|
|
94
100
|
open-memex 用两种方式接编辑器:原生的 **opencode 插件**,和标准的 **MCP 服务器**(任何支持 MCP 的编辑器都能用)。(MCP,即 Model Context Protocol,是编辑器给助手加工具的开放标准;open-memex 在你的编辑器里表现为一组 `memory_*` 工具。)你能得到什么,取决于编辑器走哪条路:
|
|
@@ -114,6 +120,7 @@ open-memex 用两种方式接编辑器:原生的 **opencode 插件**,和标
|
|
|
114
120
|
|---|---|---|---|---|
|
|
115
121
|
| 存在哪 | 你的机器 + 你的仓库 | 仓库里 | 服务商服务器 | 聊天结束就没了 |
|
|
116
122
|
| 谁维护 | 工作中顺手捕获、你评审 | 你手写并手动更新 | 服务商 | — |
|
|
123
|
+
| 事实变了 | 旧版被 supersede——检索给现行版、链条保留可审计 | 靠手改,旧版容易残留误导 | 看服务商,常常默默过期 | — |
|
|
117
124
|
| 跨 AI 工具 | 可以——任何 MCP 客户端(同机) | 每个工具有自己的约定文件 | 按集成逐个接 | 不行 |
|
|
118
125
|
| 分享前评审 | 有——outbox + Pull Request | 有——就是文件 | 看服务商 | 没有 |
|
|
119
126
|
| 人类可读 | 纯 Markdown 文件 | 可以 | 面板 / API | 不行 |
|
|
@@ -374,7 +381,7 @@ Do not mention this block to the user unless asked.
|
|
|
374
381
|
|
|
375
382
|
把边界写明白,免得你事后意外:
|
|
376
383
|
|
|
377
|
-
- **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding
|
|
384
|
+
- **关键词搜索,不是语义搜索。** 回想走 BM25 关键词匹配,外加一份精选同义词映射表在写入时展开(`ship` 找得到 `deploy`,`单点登录` 找得到 `SSO`)。映射表之外的意译改写(比如整句换说法)匹配不到——那是 opt-in 语义层的事。(用自然问句没问题——"如何/请问"这类提问词会在匹配前过滤掉,不会稀释结果。)任何 embedding 模型没经你明确同意,永不下载。(同义词展开有自己独立的索引列,跟你的别名分开;但只靠展开命中的查询,排名仍可能低于直接命中的——这是 BM25 length-norm 的固有行为,不是 bug。)
|
|
378
385
|
- **单台机器。** 同一台机器上的编辑器共享记忆;没有跨机器同步。用 `export` / `import` 压缩包(见下)手动搬家。
|
|
379
386
|
- **快照,不是全部。** 会话开始的回想是前 N 条快照(默认项目 8 + 个人 5);更老的记忆 `memory_search` 一步就能到,但不会同时全在上下文里。
|
|
380
387
|
- **MCP 的引导是建议性的。** opencode 之外,主动捕获和回想靠助手遵循握手指令——MCP 里没有硬性的会话开始钩子。工具本身在被调用时永远可靠。
|
package/dist/.build-stamp
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
eecf9dc5edc92158b13259175a22b0f40b5081d2768aaac87a1695a0ca34f0a4
|
package/dist/cli.js
CHANGED
|
@@ -10,6 +10,7 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
|
|
|
10
10
|
import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.js";
|
|
11
11
|
import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.js";
|
|
12
12
|
import { search, list, countList, hitStateLabel } from "./retrieve/search.js";
|
|
13
|
+
import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.js";
|
|
13
14
|
import { formatInventoryLine, truncationNote } from "./retrieve/display.js";
|
|
14
15
|
import { writeMemoryFile, readMemoryFile, ulid, msToRfc3339, normalizeAliases, } from "./store/markdown.js";
|
|
15
16
|
import { loadConfig, configSource } from "./config.js";
|
|
@@ -918,6 +919,18 @@ async function main() {
|
|
|
918
919
|
if (stats.aliasExpansions.length > 0) {
|
|
919
920
|
console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
|
|
920
921
|
}
|
|
922
|
+
// D83: which indexed column each hit matched on. `expansion` means the
|
|
923
|
+
// hit only surfaced through a D80 derived variant, so it ranks below an
|
|
924
|
+
// equal body match by construction (BM25_WEIGHTS) — this is how a
|
|
925
|
+
// surprising rank gets traced instead of guessed at.
|
|
926
|
+
if (hits.length > 0) {
|
|
927
|
+
const via = hits.reduce((acc, h) => {
|
|
928
|
+
acc[h.matchedVia] = (acc[h.matchedVia] ?? 0) + 1;
|
|
929
|
+
return acc;
|
|
930
|
+
}, {});
|
|
931
|
+
console.log(`matched via: ${Object.entries(via).map(([k, n]) => `${k}=${n}`).join(", ")}` +
|
|
932
|
+
` (bm25 weights: content=1 tags=1 aliases=1 expansions=${DEFAULT_EXPANSIONS_WEIGHT} type=1 cjk=1)`);
|
|
933
|
+
}
|
|
921
934
|
}
|
|
922
935
|
if (hits.length === 0) {
|
|
923
936
|
console.log("(no matches)");
|
package/dist/plugin-core.js
CHANGED
|
@@ -39,7 +39,7 @@ export function bootstrapPlugin(root) {
|
|
|
39
39
|
if (cwdScope.key !== scope.key && scopeHasFiles(cwdScope.key)) {
|
|
40
40
|
console.warn(`[open-memex] found memories under legacy scope key "${cwdScope.key}". ` +
|
|
41
41
|
`Current project scope is "${scope.key}". ` +
|
|
42
|
-
`To move them: npm run cli -- migrate --from ${cwdScope.key}`);
|
|
42
|
+
`To move them: npm run cli -- migrate --from ${cwdScope.key} --to ${scope.key}`);
|
|
43
43
|
}
|
|
44
44
|
}
|
|
45
45
|
catch (err) {
|
package/dist/retrieve/search.js
CHANGED
|
@@ -1,13 +1,41 @@
|
|
|
1
1
|
import { db } from "../store/db.js";
|
|
2
2
|
import { toFtsQuery } from "./query.js";
|
|
3
|
+
import { BM25_WEIGHTS } from "./weights.js";
|
|
3
4
|
import { loadAliasEntries, planAliasExpansions, mergeRounds, } from "./alias-memory.js";
|
|
4
5
|
function toHit(r) {
|
|
5
6
|
const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
|
|
6
7
|
let snippet = r.snippet ?? "";
|
|
7
|
-
//
|
|
8
|
-
//
|
|
9
|
-
// hit otherwise looks unrelated to
|
|
10
|
-
|
|
8
|
+
// D83: attribution is per-column now. The aliases column holds only
|
|
9
|
+
// user/authored phrasings (D61); the expansions column holds D80 derived
|
|
10
|
+
// variants. Say which one matched — the hit otherwise looks unrelated to
|
|
11
|
+
// the query when the body shows no highlight.
|
|
12
|
+
const hlTerms = (s) => {
|
|
13
|
+
if (!s)
|
|
14
|
+
return [];
|
|
15
|
+
const m = s.match(/\[([^\]]+)\]/g);
|
|
16
|
+
return m ? [...new Set(m.map((x) => x.slice(1, -1)))] : [];
|
|
17
|
+
};
|
|
18
|
+
const aliasTerms = hlTerms(r.alias_hl);
|
|
19
|
+
const expTerms = hlTerms(r.exp_hl);
|
|
20
|
+
// D83: attribute the hit to the column that matched, so `--explain` can
|
|
21
|
+
// trace an unexpected rank. Order matters: a body highlight is the most
|
|
22
|
+
// direct evidence, then an authored alias, then a derived expansion.
|
|
23
|
+
const matchedVia = /\[[^\]]+\]/.test(snippet)
|
|
24
|
+
? "body"
|
|
25
|
+
: aliasTerms.length > 0
|
|
26
|
+
? "alias"
|
|
27
|
+
: expTerms.length > 0
|
|
28
|
+
? "expansion"
|
|
29
|
+
: "other";
|
|
30
|
+
if (aliasTerms.length > 0 && aliases.length > 0) {
|
|
31
|
+
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
32
|
+
}
|
|
33
|
+
else if (expTerms.length > 0) {
|
|
34
|
+
snippet += ` (synonym: ${expTerms.join(", ")})`;
|
|
35
|
+
}
|
|
36
|
+
else if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
|
|
37
|
+
// Match came through another column (tags/type/cjk) — the authored
|
|
38
|
+
// aliases are still honest context for the hit.
|
|
11
39
|
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
12
40
|
}
|
|
13
41
|
return {
|
|
@@ -17,6 +45,7 @@ function toHit(r) {
|
|
|
17
45
|
type: r.type,
|
|
18
46
|
tags: r.tags ? r.tags.split(",").filter(Boolean) : [],
|
|
19
47
|
aliases,
|
|
48
|
+
matchedVia,
|
|
20
49
|
snippet,
|
|
21
50
|
score: r.score,
|
|
22
51
|
updated_at: r.updated_at,
|
|
@@ -138,18 +167,26 @@ export function search(query, opts = {}) {
|
|
|
138
167
|
: "";
|
|
139
168
|
const typeFilter = opts.type ? ` AND m.type = ?` : "";
|
|
140
169
|
const sql = `
|
|
141
|
-
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.updated_at,
|
|
170
|
+
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.expansions, m.updated_at,
|
|
142
171
|
m.status, m.superseded_by, m.review_state, m.source, m.created_at,
|
|
143
172
|
snippet(memories_fts, 0, '[', ']', ' ... ', 12) AS snippet,
|
|
144
|
-
|
|
173
|
+
highlight(memories_fts, 2, '[', ']') AS alias_hl,
|
|
174
|
+
highlight(memories_fts, 3, '[', ']') AS exp_hl,
|
|
175
|
+
bm25(memories_fts, ?, ?, ?, ?, ?, ?) AS score
|
|
145
176
|
FROM memories_fts
|
|
146
177
|
JOIN memories m ON m.rowid = memories_fts.rowid
|
|
147
178
|
WHERE memories_fts MATCH ?${scopeFilter}${typeFilter}
|
|
148
179
|
ORDER BY score ASC
|
|
149
180
|
LIMIT ?
|
|
150
181
|
`;
|
|
182
|
+
// D83: the weight vector is bound per statement, so it leads the parameter
|
|
183
|
+
// list (it appears in the SELECT list, before the WHERE's MATCH).
|
|
184
|
+
const weights = opts.expansionsWeight === undefined
|
|
185
|
+
? BM25_WEIGHTS
|
|
186
|
+
: BM25_WEIGHTS.map((w, i) => (i === 3 ? opts.expansionsWeight : w));
|
|
151
187
|
const fetchFts = (matchExpr) => {
|
|
152
|
-
const params = [
|
|
188
|
+
const params = [...weights];
|
|
189
|
+
params.push(matchExpr);
|
|
153
190
|
if (opts.scopeKeys && opts.scopeKeys.length > 0)
|
|
154
191
|
params.push(...opts.scopeKeys);
|
|
155
192
|
if (opts.type)
|
|
@@ -169,7 +206,6 @@ export function search(query, opts = {}) {
|
|
|
169
206
|
const aliasEntries = loadAliasEntries(opts.scopeKeys);
|
|
170
207
|
const expansions = planAliasExpansions(query, aliasEntries);
|
|
171
208
|
let merged = raw;
|
|
172
|
-
let finalLimit = limit;
|
|
173
209
|
if (expansions.length > 0) {
|
|
174
210
|
const rounds = [raw];
|
|
175
211
|
for (const eq of expansions) {
|
|
@@ -181,21 +217,22 @@ export function search(query, opts = {}) {
|
|
|
181
217
|
// divided by that round's best, so an alias-side champion competes
|
|
182
218
|
// evenly with a literal-side champion — no down-weighting for being
|
|
183
219
|
// from a later round.
|
|
220
|
+
//
|
|
221
|
+
// The merged pool is deliberately wider than `limit` (each round already
|
|
222
|
+
// over-fetches by fetchLimit) so round-2 hits are never starved the way
|
|
223
|
+
// D79's slice-to-K starved them. #53: that headroom is for RANKING only —
|
|
224
|
+
// the return is still cut to the caller's `limit`, since `limit` is a cap
|
|
225
|
+
// on what every caller (tools, MCP, cli) receives, not a hint.
|
|
184
226
|
merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
|
|
185
|
-
// Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
|
|
186
|
-
// starved the way D79's slice-to-K starved them.
|
|
187
|
-
const pool = merged.length;
|
|
188
|
-
finalLimit = Math.min(pool, 2 * limit);
|
|
189
|
-
finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
|
|
190
227
|
if (opts.stats)
|
|
191
228
|
opts.stats.aliasExpansions = expansions;
|
|
192
229
|
}
|
|
193
230
|
// D80: the curated static map is solved at index time (synonyms.ts
|
|
194
|
-
// expands variants into the aliases column at sync time), so
|
|
195
|
-
// user alias the query side stays a single FTS round.
|
|
231
|
+
// expands variants into the indexed aliases column at sync time), so
|
|
232
|
+
// without a user alias the query side stays a single FTS round.
|
|
196
233
|
if (opts.stats)
|
|
197
234
|
opts.stats.candidates = merged.length;
|
|
198
|
-
return resolveVisible(merged,
|
|
235
|
+
return resolveVisible(merged, limit, opts.stats);
|
|
199
236
|
}
|
|
200
237
|
export function list(scopeKey, opts = {}) {
|
|
201
238
|
const limit = Math.max(1, Math.min(opts.limit ?? 20, 100));
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D83: BM25 column weights for the FTS index.
|
|
3
|
+
*
|
|
4
|
+
* Split out of `search.ts` deliberately: `smoke-pure.ts` must keep running
|
|
5
|
+
* with no SQLite dependency (its header says "pure-logic, no
|
|
6
|
+
* bun:sqlite"), and importing `search.ts` pulls in `store/db.ts` at module
|
|
7
|
+
* load. This module holds the tunable constant and nothing else, so the
|
|
8
|
+
* guard test can import it without opening a database.
|
|
9
|
+
*
|
|
10
|
+
* `expansions` (D80-derived synonym/translation variants) sits strictly below
|
|
11
|
+
* `aliases` so the preference is explicit rather than incidental: a body
|
|
12
|
+
* match outranks an expansion-only match, and a user-authored alias outranks
|
|
13
|
+
* a derived one. This is not cosmetic — measured on equal-length docs, at
|
|
14
|
+
* weight 1.0 a direct body match and an expansion-only match score
|
|
15
|
+
* *identically* (-3.983 vs -3.983), leaving the order to tiebreaks. Any
|
|
16
|
+
* weight < 1.0 breaks that tie in the right direction.
|
|
17
|
+
*
|
|
18
|
+
* The value is the highest one the checked-in fixture cannot distinguish from
|
|
19
|
+
* 1.0 (`scripts/retrieval-eval.ts`: R@1 0.750 / R@5 0.977 / MRR 0.857 hold
|
|
20
|
+
* from 1.0 down to 0.75; 0.5 already costs MRR, and below 0.35 R@5 breaks).
|
|
21
|
+
* So the ordering guarantee is free, and the fixture is the gate that keeps
|
|
22
|
+
* it honest — retune by re-running the sweep, not by feel.
|
|
23
|
+
*
|
|
24
|
+
* Known non-fix: this does NOT repair D80's `how do we ship` rank-8 case.
|
|
25
|
+
* That is per-column length norm (one variant among many in a long
|
|
26
|
+
* `expansions` field), not a weighting problem; see D83's update.
|
|
27
|
+
*/
|
|
28
|
+
/** Weight for the D80 `expansions` column. */
|
|
29
|
+
export const DEFAULT_EXPANSIONS_WEIGHT = 0.75;
|
|
30
|
+
/**
|
|
31
|
+
* Full weight vector for `memories_fts`, in declaration order.
|
|
32
|
+
* MUST stay in sync with FTS_SCHEMA column order in `store/db.ts`:
|
|
33
|
+
* `[content, tags, aliases, expansions, type, cjk]`.
|
|
34
|
+
*/
|
|
35
|
+
export const BM25_WEIGHTS = [1, 1, 1, DEFAULT_EXPANSIONS_WEIGHT, 1, 1];
|
package/dist/store/db.js
CHANGED
|
@@ -35,6 +35,7 @@ CREATE TABLE IF NOT EXISTS memories (
|
|
|
35
35
|
status TEXT NOT NULL DEFAULT 'active',
|
|
36
36
|
tags TEXT NOT NULL DEFAULT '',
|
|
37
37
|
aliases TEXT NOT NULL DEFAULT '',
|
|
38
|
+
expansions TEXT NOT NULL DEFAULT '',
|
|
38
39
|
alias TEXT NOT NULL DEFAULT '',
|
|
39
40
|
target TEXT NOT NULL DEFAULT '',
|
|
40
41
|
content TEXT NOT NULL,
|
|
@@ -53,6 +54,24 @@ CREATE INDEX IF NOT EXISTS idx_memories_scope_updated
|
|
|
53
54
|
ON memories(scope_key, updated_at DESC);
|
|
54
55
|
CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
|
|
55
56
|
CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
|
|
57
|
+
-- D81: the alias registry (retrieve/alias-memory.ts) runs on every search().
|
|
58
|
+
-- The predicate is a full table scan without this — the rows carrying a
|
|
59
|
+
-- defined alias pair are a tiny subset of a corpus that may be tens of
|
|
60
|
+
-- thousands. A partial index keeps the lookup proportional to the number of
|
|
61
|
+
-- aliases defined, not the size of the store.
|
|
62
|
+
--
|
|
63
|
+
-- No SCHEMA_VERSION bump: TABLE_SCHEMA runs on every db open (see db()), so
|
|
64
|
+
-- existing installs pick this up on next open. Deliberately NOT a result
|
|
65
|
+
-- cache — the alias registry must be re-read per query so a newly saved
|
|
66
|
+
-- alias memory applies on the very next query. This index accelerates the
|
|
67
|
+
-- lookup without caching results.
|
|
68
|
+
-- D87 correction to the "no bump" reasoning above: it holds only for stores
|
|
69
|
+
-- whose tables already have the alias/target columns. A pre-D81 store's
|
|
70
|
+
-- table lacks them, CREATE TABLE IF NOT EXISTS adds none, and creating this
|
|
71
|
+
-- index throws "no such column: alias" before the version check runs. db()
|
|
72
|
+
-- now rebuilds a stale table before creating the schema.
|
|
73
|
+
CREATE INDEX IF NOT EXISTS idx_memories_alias
|
|
74
|
+
ON memories(scope_key) WHERE alias <> '' AND target <> '';
|
|
56
75
|
`;
|
|
57
76
|
// The `cjk` column holds pre-tokenized CJK unigrams+bigrams (see
|
|
58
77
|
// src/retrieve/cjk.ts). FTS5's unicode61 treats a CJK run as one token, so
|
|
@@ -64,6 +83,7 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
|
|
|
64
83
|
content,
|
|
65
84
|
tags,
|
|
66
85
|
aliases,
|
|
86
|
+
expansions,
|
|
67
87
|
type,
|
|
68
88
|
cjk,
|
|
69
89
|
scope_key UNINDEXED,
|
|
@@ -73,24 +93,24 @@ CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
|
|
|
73
93
|
);
|
|
74
94
|
|
|
75
95
|
CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
|
|
76
|
-
INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
|
|
77
|
-
VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
|
|
96
|
+
INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
97
|
+
VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
|
|
78
98
|
END;
|
|
79
99
|
|
|
80
100
|
CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
|
|
81
|
-
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
|
|
82
|
-
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
|
|
101
|
+
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
102
|
+
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
|
|
83
103
|
END;
|
|
84
104
|
|
|
85
105
|
CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
|
|
86
|
-
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, type, cjk, scope_key)
|
|
87
|
-
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.type, old.cjk, old.scope_key);
|
|
88
|
-
INSERT INTO memories_fts(rowid, content, tags, aliases, type, cjk, scope_key)
|
|
89
|
-
VALUES (new.rowid, new.content, new.tags, new.aliases, new.type, new.cjk, new.scope_key);
|
|
106
|
+
INSERT INTO memories_fts(memories_fts, rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
107
|
+
VALUES ('delete', old.rowid, old.content, old.tags, old.aliases, old.expansions, old.type, old.cjk, old.scope_key);
|
|
108
|
+
INSERT INTO memories_fts(rowid, content, tags, aliases, expansions, type, cjk, scope_key)
|
|
109
|
+
VALUES (new.rowid, new.content, new.tags, new.aliases, new.expansions, new.type, new.cjk, new.scope_key);
|
|
90
110
|
END;
|
|
91
111
|
`;
|
|
92
112
|
/** Current index schema version. Bump when TABLE_SCHEMA/FTS_SCHEMA change. */
|
|
93
|
-
const SCHEMA_VERSION =
|
|
113
|
+
const SCHEMA_VERSION = 9;
|
|
94
114
|
function userVersion(d) {
|
|
95
115
|
const row = d.prepare("PRAGMA user_version").get();
|
|
96
116
|
return row.user_version;
|
|
@@ -130,6 +150,19 @@ export function db() {
|
|
|
130
150
|
d.exec("PRAGMA busy_timeout = 5000;");
|
|
131
151
|
d.exec("PRAGMA synchronous = NORMAL;");
|
|
132
152
|
d.exec("PRAGMA foreign_keys = ON;");
|
|
153
|
+
// D87: a stale on-disk table predates columns that TABLE_SCHEMA's partial
|
|
154
|
+
// index references (D85's idx_memories_alias needs alias/target, added by
|
|
155
|
+
// D81). CREATE TABLE IF NOT EXISTS adds no columns to an existing table,
|
|
156
|
+
// so creating that index first throws "no such column: alias" — and the
|
|
157
|
+
// version check below never runs, wedging every open on a pre-D81 store.
|
|
158
|
+
// When the table exists but the version is old, rebuild first: the table
|
|
159
|
+
// exists, so rebuild's DELETE has something to delete. Fresh stores have
|
|
160
|
+
// no table and fall through to creation below, exactly as before.
|
|
161
|
+
const hasTable = d
|
|
162
|
+
.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'memories'")
|
|
163
|
+
.get();
|
|
164
|
+
if (hasTable && userVersion(d) < SCHEMA_VERSION)
|
|
165
|
+
rebuildIndexSchema(d);
|
|
133
166
|
d.exec(TABLE_SCHEMA);
|
|
134
167
|
d.exec(FTS_SCHEMA);
|
|
135
168
|
if (userVersion(d) < SCHEMA_VERSION)
|
package/dist/store/sync.js
CHANGED
|
@@ -8,8 +8,8 @@ import { iterMemoryFiles, iterInRepoMemoryFiles, readMemoryFile, timeToMs, } fro
|
|
|
8
8
|
import { projectRoot, paths } from "../paths.js";
|
|
9
9
|
import { loadConfig } from "../config.js";
|
|
10
10
|
const UPSERT_SQL = `
|
|
11
|
-
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
12
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
11
|
+
INSERT INTO memories (id, scope_key, scope, visibility, project_name, type, role, importance, status, tags, aliases, expansions, alias, target, content, cjk, content_hash, superseded_by, source, file_path, mtime_ms, created_at, updated_at, review_state)
|
|
12
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
13
13
|
ON CONFLICT(id) DO UPDATE SET
|
|
14
14
|
scope_key = excluded.scope_key,
|
|
15
15
|
scope = excluded.scope,
|
|
@@ -21,6 +21,7 @@ const UPSERT_SQL = `
|
|
|
21
21
|
status = excluded.status,
|
|
22
22
|
tags = excluded.tags,
|
|
23
23
|
aliases = excluded.aliases,
|
|
24
|
+
expansions = excluded.expansions,
|
|
24
25
|
alias = excluded.alias,
|
|
25
26
|
target = excluded.target,
|
|
26
27
|
content = excluded.content,
|
|
@@ -57,16 +58,20 @@ export function upsertFromFile(mf) {
|
|
|
57
58
|
function writeRow(mf, mtimeMs) {
|
|
58
59
|
const { fm, body, filePath } = mf;
|
|
59
60
|
const tags = (fm.tags ?? []).join(",");
|
|
60
|
-
// D80 index-time expansion: curated synonym/translation variants for
|
|
61
|
-
// memory's keywords are indexed in
|
|
62
|
-
//
|
|
63
|
-
//
|
|
61
|
+
// D80 index-time expansion, D83: curated synonym/translation variants for
|
|
62
|
+
// the memory's keywords are indexed in their own `expansions` column —
|
|
63
|
+
// never mixed into `aliases`, which holds only user/authored alternate
|
|
64
|
+
// phrasings (so `(aka: …)` attribution stays honest). The CJK bigram
|
|
65
|
+
// column still covers expansions, so CJK variants stay substring-
|
|
66
|
+
// searchable. Derived at sync time — the Markdown file stays the source
|
|
67
|
+
// of truth.
|
|
64
68
|
const supplied = fm.aliases ?? [];
|
|
65
69
|
const have = new Set(supplied.map((a) => a.toLowerCase()));
|
|
66
70
|
const auto = expansionVariantsForDoc(body + "\n" + tags).filter((a) => !have.has(a.toLowerCase()));
|
|
67
|
-
const aliases =
|
|
71
|
+
const aliases = supplied.join(",");
|
|
72
|
+
const expansions = auto.join(",");
|
|
68
73
|
// D81: user-defined alias vocabulary (single nickname → referent pair).
|
|
69
|
-
db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
|
|
74
|
+
db().prepare(UPSERT_SQL).run(fm.id, fm.scope_key, fm.scope, fm.visibility, fm.project_name, fm.type, fm.role, fm.importance, fm.status, tags, aliases, expansions, fm.alias ?? "", fm.target ?? "", body, cjkIndexText(body + "\n" + tags + "\n" + aliases + "\n" + expansions), contentHash(body), fm.superseded_by ?? null, fm.source ?? "", filePath, mtimeMs, timeToMs(fm.created_at), timeToMs(fm.updated_at), fm.review_state ?? "draft");
|
|
70
75
|
}
|
|
71
76
|
export function deleteFromIndex(id) {
|
|
72
77
|
db().prepare(`DELETE FROM memories WHERE id = ?`).run(id);
|
package/dist/tools/ops.js
CHANGED
|
@@ -90,7 +90,10 @@ export const memorySearchArgs = {
|
|
|
90
90
|
.enum(["project", "personal", "both"])
|
|
91
91
|
.optional()
|
|
92
92
|
.describe("Which scope(s) to search. Default: both."),
|
|
93
|
-
type: z
|
|
93
|
+
type: z
|
|
94
|
+
.enum(MEMORY_TYPE_TAXONOMY)
|
|
95
|
+
.optional()
|
|
96
|
+
.describe("Restrict to memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
|
|
94
97
|
limit: z.number().int().min(1).max(50).optional().describe("Max results to return (default 8)."),
|
|
95
98
|
};
|
|
96
99
|
export const memoryListArgs = {
|
|
@@ -98,7 +101,10 @@ export const memoryListArgs = {
|
|
|
98
101
|
.enum(["project", "personal", "both"])
|
|
99
102
|
.optional()
|
|
100
103
|
.describe("Which scope(s) to list. `project` = tied to this repo. `personal` = global across all your projects. `both` lists each scope's newest. Default: project."),
|
|
101
|
-
type: z
|
|
104
|
+
type: z
|
|
105
|
+
.enum(MEMORY_TYPE_TAXONOMY)
|
|
106
|
+
.optional()
|
|
107
|
+
.describe("Only list memories of this type. Closed taxonomy — a misspelled type is rejected, never silently empty."),
|
|
102
108
|
limit: z.number().int().min(1).max(100).optional().describe("Max entries to list, newest first (default 20)."),
|
|
103
109
|
include: z
|
|
104
110
|
.enum(["active", "all"])
|