open-memex 0.8.1 → 0.8.2-alpha.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -9,6 +9,9 @@ the `@alpha` tag are development snapshots and are not listed individually.
9
9
 
10
10
  ## [Unreleased]
11
11
 
12
+ - fix: the legacy-scope warning on plugin load now suggests the migrate command with an explicit `--to <current-scope>`. Without it, `--to` defaults to the scope of the directory the command runs from, so running the suggested command from the open-memex checkout would move the memories into the wrong scope. No behavior change: the warning condition and the migrate contract are untouched.
13
+ - fix: a pre-D81 index store (SCHEMA_VERSION 7, ≤0.7.x) no longer wedges every open with `no such column: alias` (D87). D85's partial index references the `alias`/`target` columns D81 added, and `CREATE TABLE IF NOT EXISTS` adds no columns to an existing table — so the index creation threw inside `db()` before the version-check rebuild could run. `db()` now rebuilds a stale table *before* creating the schema; the Markdown files are the source of truth, so the rebuild is lossless. Regression test in `scripts/test-full.ts` §9 builds a faithful v7 store and asserts the open succeeds and the schema lands at version 9.
14
+
12
15
  ## [0.8.1] - 2026-10-07
13
16
 
14
17
  - D83 follow-up: the two D83 clauses that shipped as design but not as code. (1) **BM25 weight for the `expansions` column** — `bm25(memories_fts, …)` now takes an explicit weight vector (`src/retrieve/weights.ts`, order `[content, tags, aliases, expansions, type, cjk]`) instead of the implicit all-1.0. `expansions` is `0.75`: measured on equal-length docs, a direct body match and an expansion-only match score *identically* at weight 1.0 (`-3.983` vs `-3.983`), leaving the order to tiebreaks — any weight below 1.0 breaks that tie in the right direction. `0.75` is the highest value the fixture cannot distinguish from 1.0 (sweep: 1.0/0.9/0.75 all give R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 costs MRR, below 0.35 R@5 breaks). Exposed as `opts.expansionsWeight` so the sweep stays measurable; `smoke-pure.ts` pins the vector's shape and ordering. (2) **`search --explain` per-column attribution** — reports `matched via: body=N alias=N expansion=N other=N` plus the active weights. Does **not** fix the D80 `how do we ship` rank-8 case: that is per-column length norm, not weighting. Fixture unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857.
package/README.md CHANGED
@@ -89,7 +89,7 @@ open-memex doctor
89
89
 
90
90
  ## Core concepts
91
91
 
92
- Three ideas explain almost everything open-memex does.
92
+ Four ideas explain almost everything open-memex does.
93
93
 
94
94
  ![open-memex architecture: your editors share one local memory — Markdown files as the source of truth, an SQLite FTS5 index for search, personal scope that never leaves the machine, and project scope shared through git PRs](docs/assets/open-memex-architecture-en.png)
95
95
 
@@ -118,6 +118,15 @@ rebuilds from the files at any time (`open-memex reindex`). Team sharing, when
118
118
  you want it, goes through the same review flow as code: nothing is shared
119
119
  automatically (see [Team workflow](#team-workflow-sharing-memories-through-git)).
120
120
 
121
+ **4. Knowledge stays current.**
122
+ Facts change: decisions get reversed, tools get replaced, yesterday's gotcha
123
+ gets fixed. open-memex doesn't let the old version quietly linger —
124
+ `memory_supersede` replaces a memory with its newer version and keeps the
125
+ chain, so retrieval serves the current fact while the history stays auditable.
126
+ Merely stale memories can be deprecated, and `open-memex audit` reports
127
+ duplicates, stale entries, and broken chains. Pruning is a feature, not
128
+ housekeeping: shared memory rots without it.
129
+
121
130
  ## Which editors, which features
122
131
 
123
132
  open-memex talks to editors two ways: a native **opencode plugin**, and a
@@ -159,6 +168,7 @@ instruction set only made agents chatty.
159
168
  |---|---|---|---|---|
160
169
  | Where it lives | Your machine + your repos | In the repo | Vendor servers | Gone when the chat ends |
161
170
  | Who maintains it | Captured as you work; you review | You write and update by hand | The service | — |
171
+ | When facts change | Superseded, not silently stale — retrieval serves the current version, chain preserved | You hand-edit; stale copies linger | Varies; often silently stale | — |
162
172
  | Works across AI tools | Yes — any MCP client (same machine) | One file per tool convention | Per-integration | No |
163
173
  | Review before sharing | Yes — outbox + pull request | Yes — it's just files | Varies | No |
164
174
  | Human-readable | Plain Markdown files | Yes | Dashboard / API | No |
package/README.zh-CN.md CHANGED
@@ -71,7 +71,7 @@ open-memex doctor
71
71
 
72
72
  ## 核心概念
73
73
 
74
- 三个概念能解释 open-memex 几乎所有的事。
74
+ 四个概念能解释 open-memex 几乎所有的事。
75
75
 
76
76
  ![open-memex 结构示意:你用的编辑器共享同一份本地记忆——Markdown 文件是事实来源,SQLite FTS5 索引负责检索,个人记忆只留本机,项目记忆经 Git PR 分享](docs/assets/open-memex-architecture-zh.png)
77
77
 
@@ -89,6 +89,12 @@ open-memex doctor
89
89
  **3. 你的文件,你做主。**
90
90
  Markdown 文件是事实来源——打开、编辑、删除、用 grep 搜都行。旁边的 SQLite 索引只是搜索加速器,随时可以从文件重建(`open-memex reindex`)。团队共享(当你需要时)走的是和代码一样的评审流程:没有任何东西会被自动分享(见[团队工作流](#团队工作流通过-git-共享记忆))。
91
91
 
92
+ **4. 知识保鲜。**
93
+ 事实会变:决策会被推翻、工具会被替换、昨天的坑今天可能已经填上。open-memex
94
+ 不让旧版本默默留着误导人——`memory_supersede` 用新版替换旧版并保留链条,
95
+ 检索永远给现行版本、历史可审计;只是过时的可以 deprecate,`open-memex audit`
96
+ 报告重复、过时与断链。修剪是功能,不是家务:共享记忆没人修剪就会烂掉。
97
+
92
98
  ## 编辑器与功能对照
93
99
 
94
100
  open-memex 用两种方式接编辑器:原生的 **opencode 插件**,和标准的 **MCP 服务器**(任何支持 MCP 的编辑器都能用)。(MCP,即 Model Context Protocol,是编辑器给助手加工具的开放标准;open-memex 在你的编辑器里表现为一组 `memory_*` 工具。)你能得到什么,取决于编辑器走哪条路:
@@ -114,6 +120,7 @@ open-memex 用两种方式接编辑器:原生的 **opencode 插件**,和标
114
120
  |---|---|---|---|---|
115
121
  | 存在哪 | 你的机器 + 你的仓库 | 仓库里 | 服务商服务器 | 聊天结束就没了 |
116
122
  | 谁维护 | 工作中顺手捕获、你评审 | 你手写并手动更新 | 服务商 | — |
123
+ | 事实变了 | 旧版被 supersede——检索给现行版、链条保留可审计 | 靠手改,旧版容易残留误导 | 看服务商,常常默默过期 | — |
117
124
  | 跨 AI 工具 | 可以——任何 MCP 客户端(同机) | 每个工具有自己的约定文件 | 按集成逐个接 | 不行 |
118
125
  | 分享前评审 | 有——outbox + Pull Request | 有——就是文件 | 看服务商 | 没有 |
119
126
  | 人类可读 | 纯 Markdown 文件 | 可以 | 面板 / API | 不行 |
package/dist/.build-stamp CHANGED
@@ -1 +1 @@
1
- db188c4af0c070c360ad3e3694172a66d5503a7fdad1ea2321a56b059228e3e6
1
+ eecf9dc5edc92158b13259175a22b0f40b5081d2768aaac87a1695a0ca34f0a4
@@ -39,7 +39,7 @@ export function bootstrapPlugin(root) {
39
39
  if (cwdScope.key !== scope.key && scopeHasFiles(cwdScope.key)) {
40
40
  console.warn(`[open-memex] found memories under legacy scope key "${cwdScope.key}". ` +
41
41
  `Current project scope is "${scope.key}". ` +
42
- `To move them: npm run cli -- migrate --from ${cwdScope.key}`);
42
+ `To move them: npm run cli -- migrate --from ${cwdScope.key} --to ${scope.key}`);
43
43
  }
44
44
  }
45
45
  catch (err) {
package/dist/store/db.js CHANGED
@@ -65,6 +65,11 @@ CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
65
65
  -- cache — the alias registry must be re-read per query so a newly saved
66
66
  -- alias memory applies on the very next query. This index accelerates the
67
67
  -- lookup without caching results.
68
+ -- D87 correction to the "no bump" reasoning above: it holds only for stores
69
+ -- whose tables already have the alias/target columns. A pre-D81 store's
70
+ -- table lacks them, CREATE TABLE IF NOT EXISTS adds none, and creating this
71
+ -- index throws "no such column: alias" before the version check runs. db()
72
+ -- now rebuilds a stale table before creating the schema.
68
73
  CREATE INDEX IF NOT EXISTS idx_memories_alias
69
74
  ON memories(scope_key) WHERE alias <> '' AND target <> '';
70
75
  `;
@@ -145,6 +150,19 @@ export function db() {
145
150
  d.exec("PRAGMA busy_timeout = 5000;");
146
151
  d.exec("PRAGMA synchronous = NORMAL;");
147
152
  d.exec("PRAGMA foreign_keys = ON;");
153
+ // D87: a stale on-disk table predates columns that TABLE_SCHEMA's partial
154
+ // index references (D85's idx_memories_alias needs alias/target, added by
155
+ // D81). CREATE TABLE IF NOT EXISTS adds no columns to an existing table,
156
+ // so creating that index first throws "no such column: alias" — and the
157
+ // version check below never runs, wedging every open on a pre-D81 store.
158
+ // When the table exists but the version is old, rebuild first: the table
159
+ // exists, so rebuild's DELETE has something to delete. Fresh stores have
160
+ // no table and fall through to creation below, exactly as before.
161
+ const hasTable = d
162
+ .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'memories'")
163
+ .get();
164
+ if (hasTable && userVersion(d) < SCHEMA_VERSION)
165
+ rebuildIndexSchema(d);
148
166
  d.exec(TABLE_SCHEMA);
149
167
  d.exec(FTS_SCHEMA);
150
168
  if (userVersion(d) < SCHEMA_VERSION)
package/docs/V2-DESIGN.md CHANGED
@@ -562,9 +562,13 @@ requirement: personal data never touches third-party services). Benchmarks to tr
562
562
  - **Phase 4 — Future, signal-gated.** Cloud `RemoteProvider` customization only on: multi-private-repo
563
563
  sharing needs, fine-grained ACL, audit/compliance mandates · optional API-backed exporters/providers
564
564
  for enterprise knowledge systems.
565
- - **Near-term backlog — committed direction, not yet built (2026-10-07).** Stone's rule: anything said as "later" is written down here. D77-L2: per-memory `retrieved_count`/`useful_count` (`useful_self` vs `useful_team`) + `memory_useful` tool. D77-L3: `memory_status` reuse report (top reused, zero-reuse, per-scope reuse rate) feeding the Spark Tank pilot metrics. D77: external bench — LongMemEval-S run 2026-10-07 (`scripts/bench-longmemeval.ts`, 470 questions, recall_any@k: R@5 97.0% / MRR 0.909 per-question, R@5 40.6% on a pooled 22k corpus; ahead of agentmemory’s BM25-only, level with their BM25+vector). MemoryAgentBench still open as a second external data point. D79 follow-up SUPERSEDED → built as **D80** (2026-10-07): index-time synonym/translation expansion shipped; 41-query fixture recall@5 0.854 → 0.976, MRR 0.791 → 0.859. Embedding-model configurability (Stone 2026-10-07): the multilingual model is his personal need (EN↔ZH), not everyone’s — the model must be user-configurable. Timing: (1) NOW in the bench only — a `--model` flag to measure the multilingual tax (multilingual vs English-only on English data), which decides the DEFAULT. (2) AT SHIP TIME — `embedding.model` in `MyOMemoryConfig` + `SETTABLE_KEYS`, model-agnostic storage from day one (dim + model id stored with vectors, cache dir), download UX and `doctor` checks. Not before the gate passes: building config UX for a layer we haven’t decided to ship is premature. If multilingual tax ≈ 0, default multilingual (covers everyone); else default English-only with a docs note.D78: curate+submit — pre-submit lint, then confidence-tier auto-curation (auto-apply / batch-confirm / explicit human). D79 follow-ups: feedback-weighted ranking once D77 counters land; embeddings gate decision against the published baseline; reranker evaluation. TDQS nit (separate PR): tighten search/list `type` to `z.enum`. Retrieval eval: grow the fixture (project-scope tier cases) as retrieval logic evolves. Retrieval split — local vs share server (Stone 2026-10-07): the two environments emphasize different mechanisms. Local (personal machine): BM25 + curated index-time expansion as the primary — deterministic, zero model download (D7), fast on CPU, index bloat bounded by the small corpus; embeddings opt-in only. Share server (team/org, enterprise): embeddings + ANN + reranker as the primary — server-grade compute with models resident, corpus too large and multi-author for any curated map; the curated map stays as a precision layer plus org glossary. The server additionally aggregates D77 useful_count across users into feedback-weighted ranking (already in D79 follow-ups) — a signal a single machine can never have. Bottom-up loop: curated pairs that many users converge on locally become org-glossary candidates. Economics flip: N users each downloading a model vs one model amortized on the server.
565
+ - **Near-term backlog — committed direction, not yet built (2026-10-07).** Stone's rule: anything said as "later" is written down here. D77-L2: per-memory `retrieved_count`/`useful_count` (`useful_self` vs `useful_team`) + `memory_useful` tool. D77-L3: `memory_status` reuse report (top reused, zero-reuse, per-scope reuse rate) feeding the Spark Tank pilot metrics. D77: external bench — LongMemEval-S run 2026-10-07 (`scripts/bench-longmemeval.ts`, 470 questions, recall_any@k: R@5 97.0% / MRR 0.909 per-question, R@5 40.6% on a pooled 22k corpus; ahead of agentmemory’s BM25-only, level with their BM25+vector). MemoryAgentBench still open as a second external data point. D79 follow-up SUPERSEDED → built as **D80** (2026-10-07): index-time synonym/translation expansion shipped; 41-query fixture recall@5 0.854 → 0.976, MRR 0.791 → 0.859. Embedding-model configurability (Stone 2026-10-07): the multilingual model is his personal need (EN↔ZH), not everyone’s — the model must be user-configurable. Timing: (1) NOW in the bench only — a `--model` flag to measure the multilingual tax (multilingual vs English-only on English data), which decides the DEFAULT. (2) AT SHIP TIME — `embedding.model` in `MyOMemoryConfig` + `SETTABLE_KEYS`, model-agnostic storage from day one (dim + model id stored with vectors, cache dir), download UX and `doctor` checks. Not before the gate passes: building config UX for a layer we haven’t decided to ship is premature. If multilingual tax ≈ 0, default multilingual (covers everyone); else default English-only with a docs note.D78: curate+submit — pre-submit lint, then confidence-tier auto-curation (auto-apply / batch-confirm / explicit human). D79 follow-ups: feedback-weighted ranking once D77 counters land; embeddings gate decision against the published baseline; reranker evaluation. Cross-lingual embedding rerank (Stone 2026-10-16, from obsidian-mind comparison): ablation showed multilingual vectors beat BM25 by +12pp R@5 on the cross-lingual slice — the one gap lexical can't cover. Design as opt-in rerank for EN↔ZH queries only, not a default path; local stays BM25-first per the retrieval split above. TDQS nit (separate PR): tighten search/list `type` to `z.enum`. Retrieval eval: grow the fixture (project-scope tier cases) as retrieval logic evolves. Retrieval split — local vs share server (Stone 2026-10-07): the two environments emphasize different mechanisms. Local (personal machine): BM25 + curated index-time expansion as the primary — deterministic, zero model download (D7), fast on CPU, index bloat bounded by the small corpus; embeddings opt-in only. Share server (team/org, enterprise): embeddings + ANN + reranker as the primary — server-grade compute with models resident, corpus too large and multi-author for any curated map; the curated map stays as a precision layer plus org glossary. The server additionally aggregates D77 useful_count across users into feedback-weighted ranking (already in D79 follow-ups) — a signal a single machine can never have. Bottom-up loop: curated pairs that many users converge on locally become org-glossary candidates. Economics flip: N users each downloading a model vs one model amortized on the server. OptMem lessons (Stone 2026-10-16, from kafkaup's OptMem post): three friction-reduction ideas worth stealing from the minimalist end of the spectrum. (1) **Token cost as spec** — measure and publish open-memex's prompt/token overhead in the README, the way OptMem advertises its "426-token prompt"; acquisition copy, not just engineering. (2) **`wake` convention** — document a session-start ritual (one `memory_recall` call at the beginning of every session), mirroring OptMem's `memo wake`; the MCP tools exist but the habit isn't specified. (3) **Capture length nudge** — a soft length cap or conciseness hint at capture time, inspired by OptMem's 280-byte hard cap; forces conciseness at write time, cheap to implement, aligned with the distill-to-AGENTS.md direction (D40). Three-tier memory zones (Stone 2026-10-18, from Westenberg "I Deleted My Second Brain" + his own reflection): the second brain should have zones like cache vs database, not "keep everything" vs "delete everything". **Hot** (active zone, cache-like: small, curated, always in context — D81 aliases, WHAT-level) · **Warm** (compost/incubation: not active but not cold; periodic review, rotted items deleted — this is "forget on purpose" as engineering) · **Cold** (full archive, searchable, not resident). The hard part is the **promotion/demotion policy** — what moves between zones, when, by what rule (the cache-replacement problem for knowledge; "useful" ≠ "recently used"). Capture restraint is a separate gate: not everything deserves to enter even cold storage. Spec the policy before building the zones. Auto-capture design brief (Stone 2026-10-19, from ai-memory comparison): open-memex wants auto-capture but under the restraint philosophy (not ai-memory's capture-everything). Five design questions: (1) **End-session trigger** — what hooks does opencode actually expose? (check before designing; ai-memory built finalize-session fallback for harnesses without SessionEnd); candidates: native hook / idle timeout / explicit command / context-near-full. (2) **Distillation** — rules first (what happened: git diff, tool calls, decision sentences; zero LLM), interpretation layer (what it means) left to human/LLM. (3) **Save-value triage** — three tiers: clearly valuable → save; clearly noise → drop; ambiguous → ask human. Boundary principle: **rather over-ask than over-save** (polluted retrieval costs more than a missed memory). (4) **Routing** — four exits: project memory / personal memory / AGENTS.md proposal (distill-agents exists) / discard; keep v1 routing rules simple. (5) **Human interaction** — one batch confirmation at session end ("save these 3? [y/n/edit]"), never per-item popups. Relations: D73 keyword capture is turn-level (no conflict); `wake` convention is session-start (this is session-end — the other bookend); three-tier zones govern where auto-captured items land. Cross-agent study (Stone 2026-10-19): open-memex has cross-agent on the roadmap — study how ai-memory does it (handoff claim-once semantics, lifecycle hooks per harness, VSCode support via MCP-only) before designing ours; borrow the semantics, not the push-injection behavior (discoverability > injection). Injection cost discipline (Stone 2026-10-19, from Harness² paper arXiv:2610.14224): "ALWAYS-ON PROSE IS THE EXPENSIVE OPTION" — every always-on word competes with the agent's output budget. Current session-start injection (8 project + 5 personal) is the expensive path; direction: move "important but not needed every turn" memories to on-demand (tool/skill-triggered loading), keep only truly every-turn items resident. Tolerable now, better not to tolerate. Component cost ladder for reference: SKILL (default, zero-cost until invoked) > SCRIPT > SUBAGENT > PLUGIN ("prose can be skipped, a hook cannot") > ALWAYS-ON TEXT (last resort). Read-less compression (Stone 2026-10-19, from caveman https://github.com/JuliusBrussee/caveman): the execution layer of injection cost discipline — compress what the agent READS before it reaches context, not just what we inject. Caveman's measured numbers: 33.2% fewer input tokens per session (CSV 98.9%, logs 98.5%, YAML 99.1%, JSON 98.5% per-file; HTML has no compressor yet). Direction for open-memex: a pre-retrieval compression pass on tool outputs and retrieved passages (structured data → compact form), originals stay on disk and pullable on demand. Two design principles borrowed: (1) never compress the user's own prompt/input (Adobe CAVEWOMAN finding: cavemanning user prompts makes answers worse); (2) leave code/tool calls/payloads verbatim — compress the wrapper, not the payload. Caveats: caveman's own rules cost ~1,160 input tokens fixed; per-request billing (Copilot-style) nullifies the saving — compression pays off on per-token billing only. Knowledge upkeep as a first-class direction (Stone 2026-10-23; external validation: Hedy Zhang's FDE field retrospective on enterprise AI knowledge bases, X article 2026-10-22 — enterprise knowledge bases die on unowned upkeep, not on retrieval: stale answers served silently, conflicts with no owner, updates nobody owns after delivery). The mechanisms already exist (supersede chains, lifecycle states, `audit` health check, curator convention); the direction is driving the human cost of upkeep toward zero: (1) staleness signals — age, contradiction with newer captures, D77 zero-reuse counters — feeding a standing review queue instead of waiting for someone to remember to run `audit`; (2) a conflict inbox — semantic conflicts and broken chains as a persistent todo; (3) the endgame is D78 confidence-tier auto-curation (auto-apply the safe tier, batch-confirm the rest). Success metric: the base stays trustworthy on minutes-per-week of human attention, not a curator shift. README pitches the shipped half of this as core concept 4 (Knowledge stays current) plus a "When facts change" comparison row.
566
566
 
567
567
  - **Phase 5 — Candidates recorded 2026-10-04 (not committed).** From the competitor scan (D75): an offline FTS5 retrieval eval with a published recall number (gates any semantic-layer decision) [promoted to next build, D77] · per-client trust levels on the review ladder · cold-start Markdown import into outbox drafts · `doctor` check that each editor's MCP registration points at the intended data root (mis-rooted servers are a real failure mode elsewhere). Sequencing and scope TBD per candidate.
568
+ - **Phase 5 — Candidates recorded 2026-10-11 (not committed).** From the XERJ review (xerj-org/xerj, Apache-2.0; founder Ivan N requested open-memex D81 for the XERJ Corpus Hub — agreed 2026-10-11, attribution + repo link): schema-first querying (`xerj_map` — agent fetches the field map before writing query DSL; candidate: a schema/describe helper ahead of `memory_search` to cut dead queries) · token-budgeted MCP results (their `token_budget.rs` — truncate to a token budget, not a fixed row limit) · hybrid retrieval as the scale answer (BM25 + vector kNN with fused ranking, e.g. RRF — the reference pattern for the eventual embedding layer; our lexical ceiling is already measured at R@5 40.6% pooled) · NL-to-validated-query planning (`xerj_plan` — natural language in, validated query out; candidate: a query lint/validate step before search executes) · evidence-carrying memory links (typed links with evidence + confidence, soft-retire never hard-delete — same spirit as our review-state ladder; prior art for a future memory-graph direction) · honest isolation docs (state what a namespace guarantees and does NOT — query-level vs trust boundary; same spirit as our Limitations section) · cost-accounted evals (report tokens + dollars per approach, as in their case studies — the format for proving "memory makes agents cheaper"). Sequencing and scope TBD per candidate.
569
+ - **Phase 5 — Candidates recorded 2026-10-12 (not committed).** From the mattpocock/skills review (writing-for-agents: context pointers, two loads, information hierarchy, completion criteria): a "writing-for-memory" authoring checklist (`docs/writing-for-memory.md`) — trigger branches per entry, tldr+body layered storage, capture criteria (self-contained, dated, D2 role discipline), curation bar (real-session + D77 reuse counters), glossary as first-class memory. Sequencing and scope TBD; implement after the China trip.
570
+ - **Phase 5 — Candidates recorded 2026-10-24 (not committed).** From the RAW-RAG paper (arXiv:2610.30755, 22 Oct 2026; code: Litmeb/RAW-RAG): naive dense retrieval + cross-encoder reranking over raw history beats 12 agentic memory systems on 6 of 7 benchmark–model settings across LoCoMo / LongMemEval / MEME / MemoryAgentBench; ablations make reranking the hardest single lever (removing it −3.92, CI excludes zero) and show added complexity (atomic facts, summaries, graph expansion) hurting on MemoryAgentBench. Assessment: baseline claim strong, "structure is useless" claim medium, generalization to interactive agents weak (paper's own scope limit). (1) **Naive-RAG baseline in the retrieval gate** — every retrieval change must beat a well-specified RAW-RAG baseline on the same fixture before its complexity ships (measurement before mechanism, applied to baselines); MemoryAgentBench is the open second external data point from D77 — reuse its 14 sources / 2,071 questions as the second bench. (2) **Reranker gate, not default** — run BM25 + local small cross-encoder rerank on the D77 fixture and let the gate decide; local CPU + configurable model are preconditions (D7 zero-model-download for local). (3) **Raw-evidence anchors for distilled memories** — recall returns conclusion + source excerpt via canonical_ref (§3.5), so the reader can expand to original text on demand; no change to the no-raw-transcript default. (4) **Retrieve wide, then narrow** — widen the recall pool (e.g. top-50) and let lane fusion + rerank narrow to the injected 3–10, targeting the pooled-corpus gap (D77: per-question R@5 97.0% vs pooled 40.6%). (5) **Path-carrying recall for supersede chains** — when a hit is superseded, return the new conclusion plus the one-hop change reason, not the conclusion alone. Explicitly NOT adopted: storing full raw conversation history (conflicts with §3.5 no-transcript default and capture restraint); LLM-driven extraction / knowledge graphs as default write paths (require a design-doc update first, per AGENTS.md); generalizing benchmark claims to interactive agents. Sequencing and scope TBD per candidate.
571
+ - **Phase 5 — Candidates recorded 2026-10-25 (not committed).** From Tantivy (quickwit-oss/tantivy, Rust Lucene-style full-text library; via @realchendahuang post 2026-10-24; research pass over README, ARCHITECTURE.md, docs.rs): the comparison is Tantivy vs the current SQLite FTS5 + D80 index-time expansion + bigram cjk column, not Tantivy vs hand-rolled BM25. (1) **Chinese search-mode segmentation experiment** — index compound words plus their components (jieba cut_for_search style) against the current unigram + overlapping-bigram cjk column; bigrams are over-splitting (high recall, low precision, bloated column), word + components move toward precision while keeping partial-query recall; gate on the retrieval-eval fixture (R@5/MRR) before any default change; constraint check first: a jieba-style dependency breaks the deliberate zero-native-tokenizer design (cjk.ts runs on both sqlite runtimes). (2) **Field norms as a first-class diagnostic** — the D83 expansions-column length-norm dilution is a field-norm problem, not a weighting problem; Tantivy's per-field 1-byte log-scale fieldnorm confirms the diagnosis; no encoding change (FTS5 already normalizes per column). (3) **Columnar aggregation for the reuse report** — D77-L3 counts by scope/status/age bucket should come from a column scan, not a re-search (Tantivy's fast-field pattern). (4) **Tombstone + background reclaim as confirmed discipline** — Tantivy never mutates published segments (tombstone bitset now, merge reclaims later, manifest swap for visibility); this validates the existing soft-delete + supersede chain, and explains the wipe-path FTS5 'delete' corruption workaround as the cost of in-place mutation — no change, discipline confirmed. (5) **Fuzzy (Levenshtein 1) typo tolerance, low priority** — prefix match + synonym expansion already cover most morphological variation; agent queries rarely need typo tolerance. Already covered, not counted as borrowings: separate stores (inverted match / column filter / fetch content only for displayed hits), per-column snippet + matchedVia attribution (D83), immutable + soft-delete lifecycle. Explicitly NOT adopted: migrating to Tantivy itself (stack + dependency cost, no fixture win shown). Unverified in this pass: which modes tantivy-jieba exposes and jieba-rs dictionary size (algorithm verified from sibling jieba implementations only); Tantivy's BM25 k1/b constants (only "same as Lucene" confirmed). Sequencing and scope TBD per candidate.
568
572
 
569
573
  ## 19. Migration v1 → v2
570
574
 
@@ -1612,6 +1616,7 @@ requirement: personal data never touches third-party services). Benchmarks to tr
1612
1616
  - **D84 — `limit` is a cap on the return, never a hint (2026-10-07).** Correction to D81's cut rule, raised by the review of #52. D81 widened the merged alias pool to `[1.5K, 2K]` and passed that widened value to `resolveVisible` as the slice — so when an alias round fired, `search()` returned up to **2× `limit`** rows to every caller (`memory_search`, the MCP tool, `cli search`), with nothing in `--explain` saying so. The pool stays wider than `limit` — each round already over-fetches by `fetchLimit`, and the starvation D79's slice-to-K caused is real and still prevented — but the *return* is cut to `limit`. (D81's `1.5K` line was also dead: `max(min(pool,2K), min(pool,1.5K))` is always `min(pool,2K)`, so the documented band never existed; it is gone.) No ranking change — the merge sorts before any cut, so top-k is byte-identical; `scripts/retrieval-eval.ts` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857. *Rationale: `limit` is a contract on what a caller receives. Headroom belongs inside the ranking, not in the result set — an agent paying 2× the tool-result tokens on alias-touching queries has no way to see that it happened.*
1613
1617
  - **D86 — the `expansions` BM25 weight, set by the fixture (2026-10-07).** Closes the two D83 clauses that were written into the decision log but never landed in code (`00c4d2d` shipped `bm25(memories_fts)` with no weight vector, and left `--explain` reporting only `fts:` / `candidates:` / `alias rounds:`). (1) **Weight vector** — `bm25(memories_fts, …)` now takes an explicit `[content, tags, aliases, expansions, type, cjk]`, with `expansions` at **0.75** (`src/retrieve/weights.ts`, kept out of `search.ts` so `smoke-pure.ts` keeps its no-sqlite guarantee). The measurement that sets it: on equal-length docs at weight 1.0, a direct body match and an expansion-only match score **identically** (`-3.983` vs `-3.983`) — the preference was being decided by tiebreak, not by the ranking function. Any weight < 1.0 breaks that tie toward the body. `0.75` is the *highest* weight the checked-in fixture cannot distinguish from 1.0 (sweep: 1.0 / 0.9 / 0.75 all yield R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 already costs MRR; below 0.35 R@5 breaks), so the ordering guarantee is free and the fixture remains the gate — retune by re-running the sweep, never by intuition. `SearchHit.matchedVia` (`body` | `alias` | `expansion` | `other`) is computed from the same per-column highlights the attribution already used. (2) **`--explain` attribution** — prints `matched via: body=N alias=N expansion=N other=N` alongside the active weights, so a surprising rank is traceable to the column that caused it. *Rationale: D83 promised the split would make the two columns distinguishable; on equal-weight columns they were not distinguishable at all, so the split bought provenance without buying ranking. This is the missing half.* **Explicit non-goal:** this does *not* repair D80's `how do we ship` rank-8 case — that is per-column length norm (one variant among many in a long `expansions` field), not a weighting problem. Fixing that means narrowing *what* lands in the column, not re-weighting it. Fixture unchanged: R@1 0.750 / R@5 0.977 / MRR 0.857.
1614
1618
  - **D85 — the alias registry is indexed, not cached (2026-10-07).** From the same review (#55). D81's `loadAliasEntries` runs on **every** `search()` call over `alias <> '' AND target <> ''` — a full table scan on a corpus that may be tens of thousands. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`. Two deliberate choices: (1) **no `SCHEMA_VERSION` bump** — `TABLE_SCHEMA` executes on every db open, not only on rebuild, so `CREATE INDEX IF NOT EXISTS` is idempotent and existing installs pick it up on next open with no rebuild; this also keeps the bump free for D83's 8 → 9 schema change. (2) **No result cache** — the registry must be re-read per query so a newly saved alias memory applies on the very next query (D81's freshness property). An index accelerates the lookup without caching results; those two concerns are separable and this resolves only the first. Measured at 5k memories: 971ms → 39ms per 2000 scoped lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Note the *unscoped* variant of the query still prefers `idx_memories_status`; `search()` always passes `scopeKeys`, so the partial index is what ships. *Rationale: the alias table is tiny by construction — one row per defined nickname per scope — so its lookup cost must be proportional to aliases defined, not to store size.*
1619
+ - **D87 — stale stores migrate before the schema is created (2026-10-28).** D85's "no `SCHEMA_VERSION` bump" reasoning was wrong for one population: stores created before D81 (`SCHEMA_VERSION` 7, ≤0.7.x) have no `alias`/`target` columns, and `CREATE TABLE IF NOT EXISTS` adds no columns to an existing table — so `CREATE INDEX IF NOT EXISTS idx_memories_alias ... WHERE alias <> ''` throws `no such column: alias` inside `db()` before the `userVersion < SCHEMA_VERSION` rebuild check ever runs, wedging every open (CLI, MCP, plugin) on that store. Reproduced with a faithful v7 fixture: current code throws exactly that error. Fix, ordering only: in `db()`, when the `memories` table exists but the version is old, `rebuildIndexSchema()` runs *before* `TABLE_SCHEMA`/`FTS_SCHEMA` are executed (the table exists, so rebuild's `DELETE` is valid); fresh stores still fall through to creation and the existing post-create version check is unchanged. No version bump — the guard is ordered before creation, so version 9 still covers the migration, and no new mechanism was needed. Regression test in `test-full.ts` §9 builds a v7 store and asserts the open succeeds, `user_version = 9`, and the three columns are present. The D85 comment in `db.ts` carries a correction note. *Rationale: the wipe-and-repopulate migration only works if it runs before the schema that assumes its result; the bug was ordering, not a missing mechanism.*
1615
1620
 
1616
1621
  ## Open Questions
1617
1622
 
@@ -1638,6 +1643,7 @@ _Pruned 2026-10-04 (D75): entries decided in D56/D57/D61/D68 and the shipped lay
1638
1643
 
1639
1644
  - **Enterprise / pro edition: team server + remote MCP (open, 2026-10-05):** the M365 question above reframes the enterprise surface as a *team-tier* endpoint: personal memory stays local-first, team/project memory is served company-hosted and called by Copilot Studio agents as a remote MCP server. The packaging question is whether that team tier ships as an open-memex "enterprise/pro" edition (hosted service or deployable server + remote MCP) or stays a documented deployment pattern companies run themselves. Adoption insight from large-company anecdotes: internally-built programmer tools get pulled in by BAs/QAs/POs on their own — the wedge is developers, and the non-programmer entry point may not need to be built up front; it arrives via the team surface (M365/Teams) once the team server exists. Open: packaging and monetization model (Stone's current goal for open-memex is personal influence, not user count or revenue — an enterprise edition is a support burden, so timing and shape need a deliberate decision); hosting and auth model for the team server; how the personal/team split is enforced at the protocol level.
1640
1645
  - *Refinement (2026-10-05): the entry point is not built, it is placed.* Employees already live in team chat — asking questions, getting work done. A team-chat AI agent (e.g. a Copilot Studio agent in Teams backed by the team memory server) becomes the entry point by standing where the questions already get asked. Adoption is pull, not push: the agent answers and does work in the chat, and usage follows. This dissolves the "non-programmer entry point" problem rather than solving it — there is no separate door to build.
1646
+ - *Interface constraint (2026-10-16, from tool/skill/MCP boundary audit):* design the MCP interface for dual use from the start — internal agents and external agents (Copilot Studio, team chat) consume the same tools. Reserve org scope alongside project/personal; the review ladder's single-reviewer assumption will need multi-reviewer semantics. No code changes now; the constraint is recorded so the team tier doesn't require an interface rework later.
1641
1647
 
1642
1648
  ---
1643
1649
 
@@ -0,0 +1,94 @@
1
+ # Writing for Memory — authoring checklist
2
+
3
+ > Status: Phase 5 candidate (not committed), recorded 2026-10-12.
4
+ > Adapted from mattpocock/skills `writing-for-agents`
5
+ > (https://github.com/mattpocock/skills).
6
+ > That document is a theory of writing for the agent-as-reader; this is the same
7
+ > theory applied to memory entries. Retrieval (FTS5/BM25, D81 alias expansion)
8
+ > decides *which* entries match — this checklist decides whether a matched entry
9
+ > *deserves* the context it costs.
10
+
11
+ ## 1. Trigger branches — when this memory surfaces
12
+
13
+ A memory entry is a context pointer: its title/summary wording decides when it gets
14
+ retrieved, not just its body. A must-keep fact behind a weakly worded title is a
15
+ variance bug: sharpen the wording first.
16
+
17
+ - [ ] State what the entry is AND the branches that should trigger it (a branch is
18
+ a distinct situation where this memory applies — different runs take
19
+ different branches).
20
+ - [ ] Front-load the leading words: retrieval and reranking weight the front.
21
+ - [ ] One trigger per branch. Synonyms renaming a single branch are one branch
22
+ written twice — collapse them.
23
+ - [ ] Say when it does **not** apply. A memory with no negative scope retrieves
24
+ into situations it doesn't belong in.
25
+
26
+ ## 2. Two budgets — what a retrieved memory costs
27
+
28
+ Every retrieved entry spends context budget on the agent's window, whether or not
29
+ it turns out to be useful that turn.
30
+
31
+ - [ ] Entries injected every session (high importance, always-relevant) earn the
32
+ hardest pruning: each line must justify permanent context spend.
33
+ - [ ] Entries retrieved on demand cost only the retrieval plus their own lines —
34
+ this is where detail belongs, not in the always-injected set.
35
+ - [ ] The human's curation attention is the other budget: capture is cheap,
36
+ curation is the price of a trustworthy store. Spend it where judgement
37
+ matters (D78: human-ON-the-loop).
38
+
39
+ ## 3. Layered entries — tldr + body
40
+
41
+ Progressive disclosure applied to storage: split what every retrieval needs from
42
+ what only some retrievals need.
43
+
44
+ - [ ] Every entry carries a one-line tldr: the whole point, self-contained.
45
+ Retrieval would return the tldr layer first; the body would load on
46
+ demand. (Proposed storage design — Phase 5 candidate, not built yet.)
47
+ - [ ] Inline in the tldr what every branch needs; push branch-specific detail
48
+ into the body.
49
+ - [ ] Never bury the action: if the entry exists to change behavior, the tldr
50
+ says what to do, the body says why.
51
+
52
+ ## 4. Capture criteria — is the entry self-contained?
53
+
54
+ A memory that needs its original session to be understood is a pointer to
55
+ nothing. Each entry must answer, on its own:
56
+
57
+ - [ ] What happened / what was decided (the fact, not the story).
58
+ - [ ] When it applies (trigger branches, §1).
59
+ - [ ] When it does NOT apply (negative scope, §1).
60
+ - [ ] Source: which session/decision it came from, dated. Undated lore rots;
61
+ the reader can't tell stale from live.
62
+ - [ ] Role discipline (D2): records are records. An entry that reads as an
63
+ instruction must carry `role: instruction` explicitly — never let a history
64
+ entry impersonate a current rule.
65
+
66
+ ## 5. Co-location — one entry, one meaning
67
+
68
+ - [ ] A memory's context, decision, and caveats live under one entry, not
69
+ scattered across three. Reading one part should bring its neighbours.
70
+ - [ ] Distinct from duplication: repeating one meaning in two entries is a merge
71
+ candidate; fragmenting one meaning across entries is the failure mode.
72
+ - [ ] Prefer editing the existing entry over appending a contradicting one.
73
+ Contradictions without resolution are retrieval poison — resolve,
74
+ supersede, or deprecate (`status: deprecated`); never hard-delete.
75
+
76
+ ## 6. Curation bar — what earns a place in long-term memory
77
+
78
+ Capture-first means the inbox is generous; the long-term store is not. An entry
79
+ stays only if it clears both:
80
+
81
+ 1. **Recorded from a real session.** What happened, what was decided, what went
82
+ wrong — with a date and source. A hypothetical ("it would be nice to
83
+ remember…") does not clear this bar.
84
+ 2. **Retrieval earns its keep.** D77 counters (`retrieved_count`/`useful_count`,
85
+ not yet built — near-term backlog) are the instrument: entries with sustained
86
+ zero-reuse are prune/merge candidates. Attention, not sentiment, decides.
87
+
88
+ ## 7. Glossary discipline — shared vocabulary as first-class memory
89
+
90
+ - [ ] Project glossaries (D81 alias vocabulary) are curated memory, not just
91
+ query-time expansion machinery: terms, what they mean here, and what they
92
+ are NOT.
93
+ - [ ] New entries should use the glossary's terms. A new term coined in an entry
94
+ without a glossary home is an alias-expansion miss waiting to happen.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "open-memex",
3
- "version": "0.8.1",
3
+ "version": "0.8.2-alpha.2",
4
4
  "mcpName": "io.github.stoneskin/open-memex",
5
5
  "description": "Local-first memory layer and protocol for AI coding agents. Markdown source of truth, SQLite FTS5 index, zero cloud.",
6
6
  "keywords": [
@@ -9,6 +9,7 @@
9
9
  // parsed, and crash-prone commands retry up to 5 times.
10
10
  import { execFileSync, spawn } from "node:child_process";
11
11
  import fs from "node:fs";
12
+ import { createRequire } from "node:module";
12
13
  import os from "node:os";
13
14
  import path from "node:path";
14
15
  import { fileURLToPath, pathToFileURL } from "node:url";
@@ -414,6 +415,54 @@ ok("migrate personal→project runs", r.code === 0, (r.err || r.out).slice(0, 15
414
415
  walk(memRoot);
415
416
  ok("migrated file is now scope: project", /^scope: project$/m.test(moved), moved.slice(0, 200));
416
417
  }
418
+ // D87: a pre-D81 index.db (SCHEMA_VERSION 7 — no alias/target/expansions
419
+ // columns) must open and migrate instead of wedging every command with
420
+ // "no such column: alias": D85's partial index throws before the version
421
+ // check runs. Build a faithful v7 store, then any command that opens the
422
+ // DB must succeed and leave the schema at version 9.
423
+ {
424
+ const require = createRequire(import.meta.url);
425
+ const Database = require("better-sqlite3");
426
+ fs.rmSync(dbPath, { force: true });
427
+ fs.rmSync(dbPath + "-wal", { force: true });
428
+ fs.rmSync(dbPath + "-shm", { force: true });
429
+ const old = new Database(dbPath);
430
+ old.exec(`CREATE TABLE memories (
431
+ id TEXT PRIMARY KEY,
432
+ scope_key TEXT NOT NULL,
433
+ scope TEXT NOT NULL,
434
+ visibility TEXT NOT NULL DEFAULT 'private',
435
+ project_name TEXT NOT NULL DEFAULT '',
436
+ type TEXT NOT NULL DEFAULT 'fact',
437
+ role TEXT NOT NULL DEFAULT 'knowledge',
438
+ importance TEXT NOT NULL DEFAULT 'normal',
439
+ status TEXT NOT NULL DEFAULT 'active',
440
+ tags TEXT NOT NULL DEFAULT '',
441
+ aliases TEXT NOT NULL DEFAULT '',
442
+ content TEXT NOT NULL,
443
+ cjk TEXT NOT NULL DEFAULT '',
444
+ content_hash TEXT NOT NULL DEFAULT '',
445
+ superseded_by TEXT,
446
+ source TEXT NOT NULL DEFAULT '',
447
+ file_path TEXT NOT NULL,
448
+ mtime_ms REAL NOT NULL,
449
+ created_at INTEGER NOT NULL,
450
+ updated_at INTEGER NOT NULL,
451
+ review_state TEXT NOT NULL DEFAULT 'draft'
452
+ );`);
453
+ old.exec("PRAGMA user_version = 7");
454
+ old.prepare("INSERT INTO memories (id, scope_key, scope, file_path, mtime_ms, created_at, updated_at, content) VALUES (?,?,?,?,?,?,?,?)")
455
+ .run("01D87OLD", "personal", "personal", path.join(T, "gone.md"), 1, 1, 1, "stale v7 row");
456
+ old.close();
457
+ r = cliRetry(["list", "--scope", "personal"], PROJ);
458
+ ok("D87: pre-D81 index.db opens instead of wedging", r.code === 0 && !/no such column/.test(r.out + r.err), (r.err || r.out).slice(0, 200));
459
+ const after = new Database(dbPath, { readonly: true });
460
+ const ver = (after.prepare("PRAGMA user_version").get() as { user_version: number }).user_version;
461
+ const cols = (after.prepare("PRAGMA table_info(memories)").all() as { name: string }[]).map((c) => c.name);
462
+ after.close();
463
+ ok("D87: stale store migrates to schema v9", ver === 9, `user_version=${ver}`);
464
+ ok("D87: migrated table has alias/target/expansions", ["alias", "target", "expansions"].every((c) => cols.includes(c)), cols.join(","));
465
+ }
417
466
 
418
467
  // ---------- 10. capture / doctor / mcp --print-config / init ----------
419
468
  console.log("== capture / doctor / mcp / init ==");
@@ -66,7 +66,7 @@ export function bootstrapPlugin(root: string): PluginState {
66
66
  console.warn(
67
67
  `[open-memex] found memories under legacy scope key "${cwdScope.key}". ` +
68
68
  `Current project scope is "${scope.key}". ` +
69
- `To move them: npm run cli -- migrate --from ${cwdScope.key}`,
69
+ `To move them: npm run cli -- migrate --from ${cwdScope.key} --to ${scope.key}`,
70
70
  );
71
71
  }
72
72
  } catch (err) {
package/src/store/db.ts CHANGED
@@ -78,6 +78,11 @@ CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
78
78
  -- cache — the alias registry must be re-read per query so a newly saved
79
79
  -- alias memory applies on the very next query. This index accelerates the
80
80
  -- lookup without caching results.
81
+ -- D87 correction to the "no bump" reasoning above: it holds only for stores
82
+ -- whose tables already have the alias/target columns. A pre-D81 store's
83
+ -- table lacks them, CREATE TABLE IF NOT EXISTS adds none, and creating this
84
+ -- index throws "no such column: alias" before the version check runs. db()
85
+ -- now rebuilds a stale table before creating the schema.
81
86
  CREATE INDEX IF NOT EXISTS idx_memories_alias
82
87
  ON memories(scope_key) WHERE alias <> '' AND target <> '';
83
88
  `;
@@ -162,6 +167,18 @@ export function db(): AnyDatabase {
162
167
  d.exec("PRAGMA busy_timeout = 5000;");
163
168
  d.exec("PRAGMA synchronous = NORMAL;");
164
169
  d.exec("PRAGMA foreign_keys = ON;");
170
+ // D87: a stale on-disk table predates columns that TABLE_SCHEMA's partial
171
+ // index references (D85's idx_memories_alias needs alias/target, added by
172
+ // D81). CREATE TABLE IF NOT EXISTS adds no columns to an existing table,
173
+ // so creating that index first throws "no such column: alias" — and the
174
+ // version check below never runs, wedging every open on a pre-D81 store.
175
+ // When the table exists but the version is old, rebuild first: the table
176
+ // exists, so rebuild's DELETE has something to delete. Fresh stores have
177
+ // no table and fall through to creation below, exactly as before.
178
+ const hasTable = d
179
+ .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'memories'")
180
+ .get() as { name: string } | undefined;
181
+ if (hasTable && userVersion(d) < SCHEMA_VERSION) rebuildIndexSchema(d);
165
182
  d.exec(TABLE_SCHEMA);
166
183
  d.exec(FTS_SCHEMA);
167
184
  if (userVersion(d) < SCHEMA_VERSION) rebuildIndexSchema(d);